NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
PyPI · #610 most downloaded on PyPI
Datadog APM client library
Last release today
30 Sep 2026
Ships on a steady schedule
a new release about every 2 weeks
Nearly every release is documented
notes for 60 of the last 60 stable releases
5 versions withdrawn
withdrawn after publishing
10 years old
852 releases · first in 2017
azure_functions: Resolves an issue where spans generated from timer triggers did not include the trigger type in the resource name.
azure_functions: Resolves an issue where spans generated from timer triggers did not include the trigger type in the resource name.opentracing: Support for OpenTracing has been deprecated and will be removed in v4.0.0. Please use the OpenTelemetry API instead.
opentracing: Support for OpenTracing has been deprecated and will be removed in v4.0.0. Please use the OpenTelemetry API instead.DD_RUNTIME_METRICS_INTERVAL is deprecated and will be removed in the 4.0.0 release.LLM Observability
converse_stream calls to the Bedrock Converse API. converse_stream calls are traced as LLM spans with max_tokens, temperature inference parameters and input/output messages.botocore: Introduces capturing prompts, token usage, and inference parameters for converse_stream calls to the Bedrock API.crewai: Introduces tracing support for the CrewAI SDK, specifically tracing crew, task, agent, and tool executions. See the docs for more information.litellm: Introduces tracing support for the LiteLLM Python SDK's synchronous and asynchronous completion and text_completion methods. See the docs for more information.openai_agents: Introduces tracing support for OpenAI Agents SDK. Agent, guardrail, tool, handoff, response, and custom operations are traced. See the docs for more information.Tracing
Support for Database Monitoring (DBM) propagation in pymongo. Two propagation modes are supported:
This feature can be controlled using the DD_DBM_PROPAGATION_MODE environment variable with values full, service, or disabled. Example of enabling full or service propagation mode: DD_DBM_PROPAGATION_MODE=full|service
Note that this feature requires PyMongo 3.9 or higher.
A subset of the tracer configuration is now stored on disk throughout the tracer's lifetime. This enables the monitoring of which processes are instrumented.
azure_functions: Introduces tracing support for timer triggers.
langgraph: Introduces tracing support for LangGraph, specifically tracing graph and node executions. See the docs for more information.
Code Security
CI Visibility
CODEOWNERS file of the project being tested was looked up in the current working directory rather than the repository root.I/O operation on closed file error was raised while printing ddtrace logs during a pytest session, due to a conflict with pytest's log capture.Profiling
_self_acquired_at attribute.Tracing
Django, Flask, Cherrypy, wsgi, asgi, pyramid, molten, falcon, tornado, aiohttp, bottle, rq, and sanic integrations were unnecessarily running code to activate distributed tracing headers multiple times per request. This is fixed by only activating distributed tracing headers once per request.azure_functions: Resolves an issue where the function name set by the function_name decorator was ignored.azure_functions: Resolves an issue where a function would not start if the trigger_arg_nameparameter was set to anything other than req.azure_functions: Resolves an issue where timer triggers had a span kind of server instead of internalgraphql: Resolves AttributeError raised when used with older versions of graphql-core that do not have formatted property for SourceLocation.opentelemetry: Updates trace flags in opentelemetry.trace.Span.get_span_context() to match OpenTelemetry specification.subprocess: Removes unnecessary debug log variables.One column per quarter.
opentracing: Support for OpenTracing has been deprecated and will be removed in v4.0.0. Please use the OpenTelemetry API instead.
opentracing: Support for OpenTracing has been deprecated and will be removed in v4.0.0. Please use the OpenTelemetry API instead.DD_RUNTIME_METRICS_INTERVAL is deprecated and will be removed in the 4.0.0 release.LLM Observability
converse_stream calls to the Bedrock Converse API. converse_stream calls are traced as LLM spans with max_tokens, temperature inference parameters and input/output messages.botocore: Introduces capturing prompts, token usage, and inference parameters for converse_stream calls to the Bedrock API.crewai: Introduces tracing support for the CrewAI SDK, specifically tracing crew, task, agent, and tool executions. See the docs for more information.litellm: Introduces tracing support for the LiteLLM Python SDK's synchronous and asynchronous completion and text_completion methods. See the docs for more information.openai_agents: Introduces tracing support for OpenAI Agents SDK. Agent, guardrail, tool, handoff, response, and custom operations are traced. See the docs for more information.Tracing
Support for Database Monitoring (DBM) propagation in pymongo. Two propagation modes are supported:
This feature can be controlled using the DD_DBM_PROPAGATION_MODE environment variable with values full, service, or disabled. Example of enabling full or service propagation mode: DD_DBM_PROPAGATION_MODE=full|service
Note that this feature requires PyMongo 3.9 or higher.
A subset of the tracer configuration is now stored on disk throughout the tracer's lifetime. This enables the monitoring of which processes are instrumented.
azure_functions: Introduces tracing support for timer triggers.
langgraph: Introduces tracing support for LangGraph, specifically tracing graph and node executions. See the docs for more information.
Code Security
CI Visibility
CODEOWNERS file of the project being tested was looked up in the current working directory rather than the repository root.I/O operation on closed file error was raised while printing ddtrace logs during a pytest session, due to a conflict with pytest's log capture.Profiling
_self_acquired_at attribute.Tracing
Django, Flask, Cherrypy, wsgi, asgi, pyramid, molten, falcon, tornado, aiohttp, bottle, rq, and sanic integrations were unnecessarily running code to activate distributed tracing headers multiple times per request. This is fixed by only activating distributed tracing headers once per request.azure_functions: Resolves an issue where the function name set by the function_name decorator was ignored.azure_functions: Resolves an issue where a function would not start if the trigger_arg_nameparameter was set to anything other than req.azure_functions: Resolves an issue where timer triggers had a span kind of server instead of internalgraphql: Resolves AttributeError raised when used with older versions of graphql-core that do not have formatted property for SourceLocation.opentelemetry: Updates trace flags in opentelemetry.trace.Span.get_span_context() to match OpenTelemetry specification.subprocess: Removes unnecessary debug log variables.Fixes an infinite recursion from calling logging module when lock profiler fails to delete _self_acquired_at attribute.
Profiling
_self_acquired_at attribute.Tracing
Fixes an issue with PosixPath handling in path operations that could cause errors during taint tracking. This fix improves stability and slightly redu
patch_all is deprecated. As an alternative to patch_all, you can use import ddtrace.auto along with DD_PATCH_MODULES if specific module patching is ne…
echion which includes performance improvements.patch_all is deprecated. As an alternative to patch_all, you can use import ddtrace.auto along with DD_PATCH_MODULES if specific module patching is necessary.Dynamic Instrumentation
LLM Observability
integration tag to LLM Observability spans that are generated by an integration.agentless_enabled and DD_LLMOBS_AGENTLESS_ENABLED now defaults to None, and will be determined automatically based on the presence of a compatible Datadog agent. agentless_enabled and DD_LLMOBS_AGENTLESS_ENABLED can still be set explicitly to True or False</span>, but is now optional for agentless users.Profiling
Tracing
DD_TRACE_NATIVE_SPAN_EVENTS (disabled by default). This requires agent version 7.63.0 or later.http.referrer_hostname tag.Fixes an issue with gevent support and the typing module on CPython >= 3.12.
ASM
NotImplementedError that occurred when trying to deepcopy wrapped builtin functions (like open) while ASM or IAST were enabled. The error was caused by the wrapper not implementing the __deepcopy__ method.CI Visibility
Lib-Injection
LLM Observability
agentless_enabled=True or DD_LLMOBS_AGENTLESS_ENABLED=true.openai: Avoids creating spans for streamed Open AI chat and completion requests that use with_raw_response since the stream cannot be traced in these cases, leading to unfinished spans.Profiling
AttributeError: '_ProfiledThreadingLock' object has no attribute '_self_acquired_at'.Single-Step Instrumentation
Tracing
graphql: Fixes an issue in GraphQL patching that threw an error when locations (an Optional field) is None. This checks for locations before setting attributes.Nothing published for this version
patch_all is deprecated. As an alternative to patch_all, you can use import ddtrace.auto along with DD_PATCH_MODULES if specific module patching is ne…
echion which includes performance improvements.patch_all is deprecated. As an alternative to patch_all, you can use import ddtrace.auto along with DD_PATCH_MODULES if specific module patching is necessary.Dynamic Instrumentation
LLM Observability
integration tag to LLM Observability spans that are generated by an integration.agentless_enabled and DD_LLMOBS_AGENTLESS_ENABLED now defaults to None, and will be determined automatically based on the presence of a compatible Datadog agent. agentless_enabled and DD_LLMOBS_AGENTLESS_ENABLED can still be set explicitly to True or False</span>, but is now optional for agentless users.Profiling
Tracing
DD_TRACE_NATIVE_SPAN_EVENTS (disabled by default). This requires agent version 7.63.0 or later.http.referrer_hostname tag.Fixes an issue with gevent support and the typing module on CPython >= 3.12.
ASM
NotImplementedError that occurred when trying to deepcopy wrapped builtin functions (like open) while ASM or IAST were enabled. The error was caused by the wrapper not implementing the __deepcopy__ method.CI Visibility
Lib-Injection
LLM Observability
agentless_enabled=True or DD_LLMOBS_AGENTLESS_ENABLED=true.openai: Avoids creating spans for streamed Open AI chat and completion requests that use with_raw_response since the stream cannot be traced in these cases, leading to unfinished spans.Profiling
AttributeError: '_ProfiledThreadingLock' object has no attribute '_self_acquired_at'.Single-Step Instrumentation
Tracing
graphql: Fixes an issue in GraphQL patching that threw an error when locations (an Optional field) is None. This checks for locations before setting attributes.Fixes an issue with PosixPath handling in path operations that could cause errors during taint tracking. This fix improves stability and slightly redu
Avoids zombie process from telemetry sender on startup.
Lib-Injection
LLM Observability
agentless_enabled=True or DD_LLMOBS_AGENTLESS_ENABLED=true.Fixes a NotImplementedError that occurred when trying to deepcopy wrapped builtin functions (like open) while ASM or IAST were enabled. The error was
ASM
NotImplementedError that occurred when trying to deepcopy wrapped builtin functions (like open) while ASM or IAST were enabled. The error was caused by the wrapper not implementing the __deepcopy__ method.CI Visibility
Lib-Injection
LLM Observability
Includes dynamodb in the default list of cloud services for payload tagging, i.e. DD_TRACE_CLOUD_PAYLOAD_TAGGING_SERVICES. Note that cloud services pa
Includes dynamodb in the default list of cloud services for payload tagging, i.e. DD_TRACE_CLOUD_PAYLOAD_TAGGING_SERVICES. Note that cloud services payload tagging feature is still gated by DD_TRACE_CLOUD_REQUEST_PAYLOAD_TAGGING and DD_TRACE_CLOUD_RESPONSE_PAYLOAD_TAGGING.
LLM Observability: This introduces tracing for converse calls to the Bedrock Converse API. converse calls are traced as
LLM spans with max_tokens, temperature inference parameters and input/output messages.
botocore: This introduces capturing prompts, token usage, and inference parameters for converse calls to the Bedrock API.
SCA: This add support for extended heartbeat every 24h with dependencies payload.
kafka: Adds messaging.destination.name tag to kafka produce and consume spans.
LLM Observability: Avoids submitting spans to LLM Observability when a non-default base URL is detected in request for the Anthropic, Bedrock, LangChain, Open AI, and Azure Open AI integrations.
ASM: This introduces support for automatic instrumentation of session monitoring and blocking for Django.
ASM: This introduces new metrics span tags to keep track of truncations of WAF payloads.
runtime_metrics: Adds support for sending runtime metrics as gauge metrics (instead of distributions). To enable this feature set DD_TRACE_EXPERIMENTAL_FEATURES_ENABLED=DD_RUNTIME_METRICS_ENABLED.
runtime_metrics: Adds support for tagging runtime metrics with the current runtime ID. To enable tagging, set DD_TRACE_EXPERIMENTAL_RUNTIME_ID_ENABLED=True.
SCA: This fix resolves an issue where some dependencies where reported with an inaccurate name.
profiling: This fix resolves an issue where the Lock profiler would throw an AttributeError: '_ProfiledThreadingLock' object has no attribute '_self_acquired_at'.
tags: Updates DD_TAGS parsing to match the Datadog Agent and other SDKs. Key-value pairs can now be separated by both commas and spaces (e.g., DD_TAGS="key1:value1, key2:value2, key3:value3"). This reverts a change introduced in v2.11.0, which only allowed one separator type.
dynamic sampling: Ensures that dynamic sampling rates are always consistent with the rates received from the agent via remote configuration. Previously, setting a global sample rate and then sampling rules via remote config would result in both configurations being applied. With this fix only the most recent sampling configuration will be applied.
LLM Observability: This fix resolves an issue where enabling LLM Observability caused all botocore submodules to be patched instead
of only bedrock-runtime.
LLM Observability: Resolves issue where Pydantic model objects were encoded as unhelpful default placeholder text.
LLM Observability: Improves encoding of non-JSON serializable I/O objects by attempting to convert to string before resorting to default placeholder text.
tracing: Resolves an issue where trace information, such as span links, baggage, and trace-level tags (ex: sampling decision maker), could be lost when a new process was created while a trace was active.
internal: Fixes an issue where trimming a traceback to attach it to the span could result in the loss of the most recent frames.
pylibmc: fixes an issue where using Client(server=[url]) would throw the error __init__() got multiple values for argument 'servers'
tracing: Ensures remote sampling rules no longer overwrite existing samplers, preserving service-based sample rates set by the Agent and ensuring tracer rate limits configured by ddtrace products (e.g., ASM) are always respected.
DD_TRACE_AGENT_URLDD_DOGSTATSD_URLDD_TRACE_AGENT_HOSTNAMEDD_AGENT_HOSTDD_TRACE_AGENT_PORTDD_DOGSTATSD_PORTDD_AGENT_PORTDD_TRACE_AGENT_TIMEOUT_SECONDSOf the above impacted configurations, these had a precedence change:
DD_TRACE_AGENT_HOSTNAME over DD_AGENT_HOSTDD_TRACE_AGENT_PORT over DD_AGENT_PORTDD_DOGSTATSD_HOST over DD_AGENT_HOSTDD_DOGSTATSD_PORT over DD_AGENT_PORTSCA: Resolves an issue where some dependencies where reported with an inaccurate name.
ASM
Profiling
AttributeError: '_ProfiledThreadingLock' object has no attribute '_self_acquired_at'.Fixes an issue with PosixPath handling in path operations that could cause errors during taint tracking. This fix improves stability and slightly redu
Code Security
Lib-injection
LLM Observability
agentless_enabled=True or DD_LLMOBS_AGENTLESS_ENABLED=true.Tracing
Fixes a NotImplementedError that occurred when trying to deepcopy wrapped builtin functions (like open) while ASM or IAST were enabled. The error was
ASM
NotImplementedError that occurred when trying to deepcopy wrapped builtin functions (like open) while ASM or IAST were enabled. The error was caused by the wrapper not implementing the __deepcopy__ method.CI Visibility
LLM Observability
Profiling
AttributeError: '_ProfiledThreadingLock' object has no attribute '_self_acquired_at'.Tracing
Client(server=[url]) would throw the error __init__() got multiple values for argument 'servers'Fix for release script causing lib injection OCI images to not get published.
Nothing published for this version
Fix for release script causing lib injection OCI images to not get published.
This release is missing the OCI images due to issues with the release pipeline. If you need an OCI image with the features introduced in the 3.2 relea
This release is missing the OCI images due to issues with the release pipeline. If you need an OCI image with the features introduced in the 3.2 release line, please use v3.2.1 instead.
DD_EXPERIMENTAL_APPSEC_STANDALONE_ENABLED with DD_APM_TRACING_ENABLED. Feature set remains the same.Bumps libdatadog dependency to v16.0.3.
Bumps libdatadog dependency to v16.0.3.
ASM
DD_EXPERIMENTAL_APPSEC_STANDALONE_ENABLED with DD_APM_TRACING_ENABLED. Feature set remains the same.CI Visibility
sys.monitoring tools such as pytest-cov from being used.Profiling
Tracing
celery: When multiple broker URLs are provided as a list, use the first broker URL from a list to avoid parsing errors.SCA: Resolves an issue where some dependencies where reported with an inaccurate name.
ASM
CI Visibility
pytest-cov from being used.LLM Observability
Profiling
AttributeError: '_ProfiledThreadingLock' object has no attribute '_self_acquired_at'.Tracing
record_exception call as it was not following OTEL semantics.In some applications, unique cookie names are used per request, leading to a high number of unique cookie vulnerabilities. This fix improves detection…
Dynamic Instrumentation
LLM Observability
Deepseek for the model provider and span name for LLM Observability spans for calls to Deepseek models using the OpenAI SDK.LLMObs.export_span(), LLMObs.inject_distributed_headers(), LLMObs.annotate() to default to the current active LLM Observability-specific span if span is not provided.concurrent.futures.thread module.metadata argument to LLMObs.submit_evaluation_for, which enables users to attach arbitrary structured metadata to custom evaluations.Tracing
api gateway: Adds preview support to create inferred AWS API Gateway spans when certain AWS Gateway headers are passed into the request. Requires DD_TRACE_INFERRED_PROXY_SERVICES_ENABLED=true to enable. For enablement instructions, contact support.graphql: Add span events on GraphQL error spans, allowing multiple errors to be surfaced on the span.graphql: Adds user-provided extensions to graphql error span events through the DD_TRACE_GRAPHQL_ERROR_EXTENSIONS environment variable. This is disabled by default; when set it allows users to capture their extensions.record_exception method that adds an exception to a Span as a span event. Refer to Span.record_exception for more details.ASM
Code Security (IAST)
set_cookies function and adding more context. Warning: This change updates the vulnerability hash, meaning that if it was previously silenced in the web dashboard, it may reappear as a new vulnerability.LLM Observability
botocore: Moves bedrock token usage metrics on APM spans from the meta field under bedrock.usage.{prompt/completion}_tokens to the metrics field under bedrock.response.usage.{prompt/completion}_tokens.Profiling
Single-Step Instrumentation
boto3 from the minimum versions list to avoid blocking SSI on boto3 version.Tracing
asyncio: Fixes an issue where the name of a coroutine was being overridden by a ddtrace function.ot-baggage headers in wsgi apps were not properly recognized due to the http_ header prefix. This ensures consistent baggage propagation in wsgi compatible web frameworks.DD_TRACE_CLOUD_PAYLOAD_TAGGING_SERVICES env variable was not parsed correctlyddtrace.config.<undefined_attribute> would create an IntegrationConfig object instead of raising an AttributeError.bytecode>=0.15.1 for Python 3.12 to avoid known bug.ddtrace internal logger initialization mutating an unlocked shared resource.ddtrace.patch(...) is called with an invalid argument.pymongo: Changes the mongodb.query span tag to json so it can be parsed by the agent.__str__.ModuleNotFoundError errors raised when a OpenTracing tracer is initialized with a AsyncioScopeManager.API Gateway: Add preview support to create inferred AWS API Gateway spans when certain AWS Gateway headers are passed into the request. Requires DD_TRACE_INFERRED_PROXY_SERVICES_ENABLED=true to enable. For enablement instructions, contact support.
LLM Observability: Properly labels Deepseek for the model provider and span name for LLM Observability spans for calls to Deepseek models using the OpenAI SDK.
tracing: Introduces a record_exception method that adds an exception to a Span as a span event. Refer to [Span.record_exception](https://ddtrace.readthedocs.io/en/stable/api.html#ddtrace.trace.Span.record_exception) for more details.
LLM Observability: Introduces an improved automated context management system for LLM Observability-specific spans. Also modifies LLMObs.export_span(), LLMObs.inject_distributed_headers(), LLMObs.annotate() to default to the current active LLM Observability-specific span if span is not provided.
LLM Observability: Introduces automatic distributed tracing support for LLM Observability traces involving the concurrent.futures.thread module.
Symbol Database is now enabled by default.
graphql: Add span events on GraphQL error spans, allowing multiple errors to be surfaced on the span.
graphql: Adds user-provided extensions to graphql error span events through the <span class="title-ref">DD_TRACE_GRAPHQL_ERROR_EXTENSIONS</span> environment variable. This is disabled by default; when set it allows users to capture their extensions.
LLM Observability: This introduces <span class="title-ref">metadata</span> argument to <span class="title-ref">LLMObs.submit_evaluation_for</span>, which enables users to
attach arbitrary structured metadata to custom evaluations.
bytecode>=0.15.1 for Python 3.12 to avoid known bug.boto3 from the minimum versions list to avoid blocking SSI on boto3 version.ddtrace.config.<undefined_attribute> would create an IntegrationConfig object instead of raising an AttributeError.bedrock.usage.{prompt/completion}_tokens to the metrics field under bedrock.response.usage.{prompt/completion}_tokens.ddtrace internal logger initialization mutating an unlocked shared resource.ddtrace.patch(...) is called with an invalid argument.In some applications, unique cookie names are used per request, leading to a high number of unique cookie vulnerabilities. This fix improves detection…
Dynamic Instrumentation
LLM Observability
Deepseek for the model provider and span name for LLM Observability spans for calls to Deepseek models using the OpenAI SDK.LLMObs.export_span(), LLMObs.inject_distributed_headers(), LLMObs.annotate() to default to the current active LLM Observability-specific span if span is not provided.concurrent.futures.thread module.metadata argument to LLMObs.submit_evaluation_for, which enables users to attach arbitrary structured metadata to custom evaluations.Tracing
api gateway: Adds preview support to create inferred AWS API Gateway spans when certain AWS Gateway headers are passed into the request. Requires DD_TRACE_INFERRED_PROXY_SERVICES_ENABLED=true to enable. For enablement instructions, contact support.graphql: Add span events on GraphQL error spans, allowing multiple errors to be surfaced on the span.graphql: Adds user-provided extensions to graphql error span events through the DD_TRACE_GRAPHQL_ERROR_EXTENSIONS environment variable. This is disabled by default; when set it allows users to capture their extensions.record_exception method that adds an exception to a Span as a span event. Refer to Span.record_exception for more details.ASM
Code Security (IAST)
set_cookies function and adding more context. Warning: This change updates the vulnerability hash, meaning that if it was previously silenced in the web dashboard, it may reappear as a new vulnerability.LLM Observability
botocore: Moves bedrock token usage metrics on APM spans from the meta field under bedrock.usage.{prompt/completion}_tokens to the metrics field under bedrock.response.usage.{prompt/completion}_tokens.Profiling
Single-Step Instrumentation
boto3 from the minimum versions list to avoid blocking SSI on boto3 version.Tracing
asyncio: Fixes an issue where the name of a coroutine was being overridden by a ddtrace function.ot-baggage headers in wsgi apps were not properly recognized due to the http_ header prefix. This ensures consistent baggage propagation in wsgi compatible web frameworks.DD_TRACE_CLOUD_PAYLOAD_TAGGING_SERVICES env variable was not parsed correctlyddtrace.config.<undefined_attribute> would create an IntegrationConfig object instead of raising an AttributeError.bytecode>=0.15.1 for Python 3.12 to avoid known bug.ddtrace internal logger initialization mutating an unlocked shared resource.ddtrace.patch(...) is called with an invalid argument.pymongo: Changes the mongodb.query span tag to json so it can be parsed by the agent.__str__.ModuleNotFoundError errors raised when a OpenTracing tracer is initialized with a AsyncioScopeManager.XSS detection for Django applications and Jinja2 (Flask and FastAPI applications), which will be displayed on your DataDog Vulnerability Explorer dash…
We are happy to announce the release of v3.0.0 of ddtrace. This release drops support for Python 3.7 and many previously-deprecated parts of the library interface.
The 2.x release line enters Maintenance Mode with the release of 3.0.0, meaning that it will only receive bugfix changes on its last few minor releases, and 2.21 will be the last 2.x minor version. See the versioning policy for support level definitions.
item.nodeid.TestClass::test_method) for class-based tests.DD_PYTEST_USE_NEW_PLUGIN_BETA environment variable is no longer used.DD_LLMOBS_APP_NAME.parameters argument in LLMObs.annotate() has been removed. Use metadata instead.DD_TRACE_PROPAGATION_STYLE=b3 single header is no longer supported. Use b3 instead.ddtrace.tracer must be used.Tracer.configure(...).Before upgrading to v3, check for deprecation warnings in your application using the latest v2 release (2.21.0):
Enable warnings as errors in pytest:
pytest -W "error::ddtrace.DDTraceDeprecationWarning" tests.py
Set the following environment variable to enable all deprecation warnings:
PYTHONWARNINGS=all python app.py
The following environment variables have been removed or replaced in v3. Use the table below to update your configuration:
| v2 | Removed in v3 | Replaced/Changed in v3 |
|---|---|---|
| LLM Observability | ||
DD_LLMOBS_APP_NAME |
Use DD_LLMOBS_ML_APP instead |
|
_DD_LLMOBS_EVALUATOR_SAMPLING_RULES |
Use DD_LLMOBS_EVALUATOR_SAMPLING_RULES instead |
|
_DD_LLMOBS_EVALUATORS |
Use DD_LLMOBS_EVALUATORS instead |
|
| Tracing | ||
DD_ANALYTICS_ENABLED |
No replacement, this setting is now a no-op. Datadog Analytics is no longer supported. See the ingestion controls documentation for more detail. | |
DD_HTTP_CLIENT_TAG_QUERY_STRING |
Use DD_TRACE_HTTP_CLIENT_TAG_QUERY_STRING. |
|
DD_TRACE_ANALYTICS_ENABLED |
No replacement, this setting is now a no-op. Datadog Analytics is no longer supported. See the ingestion controls documentation for more detail. | |
DD_TRACE_METHODS (using [] notation) |
Must use : notation instead, For example: mod.submod:method1,method2;mod.submod:Class.method1. (docs) |
|
DD_TRACE_PROPAGATION_STYLE=”b3 single header” |
Use DD_TRACE_PROPAGATION_STYLE=b3 for identical behavior |
|
DD_TRACE_SAMPLE_RATE |
Use DD_TRACE_SAMPLING_RULES (docs). |
|
DD_TRACE_SPAN_AGGREGATOR_RLOCK |
No replacement (feature removed). Config variable is now a no-op. | |
| CI Visibility | ||
DD_PYTEST_USE_NEW_PLUGIN_BETA |
No replacement (new pytest plugin is not in beta anymore). |
Use the following patterns to identify the deprecated environment variables in a code base:
git grep -P -e "DD_LLMOBS_APP_NAME" \
-e "_DD_LLMOBS_EVALUATOR_SAMPLING_RULES" \
-e "_DD_LLMOBS_EVALUATORS" \
-e "DD_TRACE_PROPAGATION_STYLE=.*b3 single header" \
-e "DD_TRACE_SAMPLE_RATE" \
-e "DD_TRACE_API_VERSION=v0.3" \
-e "DD_ANALYTICS_ENABLED" \
-e "DD_TRACE_ANALYTICS_ENABLED" \
-e "DD_HTTP_CLIENT_TAG_QUERY_STRING" \
-e "DD_TRACE_SPAN_AGGREGATOR_RLOCK" \
-e "DD_TRACE_METHODS=.*\[\]"
Replace deprecated settings with the recommended alternatives where applicable.
NOTE: The changes to environment variables apply only to the configuration of the dd-trace-py library and not the Datadog Agent.
The following methods and module attributes have been removed or changed. Unless noted, removal means that the functionality pointed to by the removed interface has been disabled.
| v2 | v3 |
|---|---|
| Tracing | |
Tracer.configure(enabled: Optional[bool] = None, hostname: Optional[str] = None, port: Optional[int] = None, uds_path: Optional[str] = None, https: Optional[bool] = None, sampler: Optional[BaseSampler] = None, context_provider: Optional[DefaultContextProvider] = None, wrap_executor: Optional[Callable] = None, priority_sampling: Optional[bool] = None, settings: Optional[Dict[str, Any]] = None, dogstatsd_url: Optional[str] = None, writer: Optional[TraceWriter] = None, partial_flush_enabled: Optional[bool] = None, partial_flush_min_spans: Optional[int] = None, api_version: Optional[str] = None, compute_stats_enabled: Optional[bool] = appsec_enabled: Optional[bool] = None, iast_enabled: Optional[bool] = None, appsec_standalone_enabled: Optional[bool] = None, trace_processors: Optional[List[TraceProcessor]] = None ) |
Tracer.configure(context_provider: Optional[BaseContextProvider] = None, compute_stats_enabled: Optional[bool] = None, appsec_enabled: Optional[bool] = None, iast_enabled: Optional[bool] = None, apm_tracing_disabled: Optional[bool] = None, trace_processors: Optional[List[TraceProcessor]] = None) |
ddtrace.opentracer.tracer.Tracer()._dd_tracer |
ddtrace.tracer |
ddtrace.tracer.sampler.rules[].choose_matcher(lambda: True) |
ddtrace.tracer.sampler.rules[].choose_matcher(“foo”) Callables and regex patterns are no longer allowed as arguments to choose_matcher |
Span.sampled |
Span.context.sampling_priority > 0 |
| LLM Observability | |
LLMObs.annotate(parameters=foo) |
LLMObs.annotate(metadata=foo) |
Much of the library’s functionality has been compatible with Python 3.13 since version 2.20. Some pieces of functionality do not yet work with Python 3.13. Here’s what does not work under Python 3.13 as of dd-trace-py 3.0. 'Not tested' indicates that while the feature may be compatible with Python 3.13, its functionality has not been verified, as it is not included in our automated test coverage.
dd-trace-py doesn’t work on Windows with Python 3.13anthropicconsulfreezegungeventgoogle_generativeaigunicornlangchainmysqlclientopentracingpsycopgpsycopg2pymysqlpytestpytest-bddpytest-benchmarksanicseleniumsqlite3starlettetornadovertexaiBy following these guidelines, you can transition to dd-trace-py v3 gradually, ensuring minimal disruption to your services while taking advantage of the latest features and improvements. Happy migrating!
Removes code and tests related to Python 3.7, breaking the library's compatibility with this Python version.
ASM
Code Security (IAST)
LLM Observability
valkey: Adds automatic instrumentation of the Valkey package. Thank you AhmadMasry!
openai: Introduces tracing support to the OpenAI integration for Python versions 3.12 and 3.13.
Introduces an integration with the RAGAS evaluation framework to continuously monitor the performance of context-augmented LLM generations in production. The integration supports evaluating LLM inferences with the following RAGAS metrics:
To learn more, see the LLM Observability evaluations guide.
CI Visibility
Official release of the new version of the pytest plugin, introducing the following features:
NOTE: this new version of the plugin introduces breaking changes:
module, suite, and test names are now parsed from the item.nodeid attributeA beta version of the plugin had been available since v2.18.0, and could be enabled via the DD_PYTEST_USE_NEW_PLUGIN_BETA environment variable. The new version is now the default, and the environment variable is not used anymore.
LLM Observability
langchain: Removes the langchain.tokens.total_cost span metric for OpenAI calls. For continued cost estimation of OpenAI calls, enable LLM Observability.
langchain: Removes prompt-completion log sampling from the LangChain integration. To continue logging prompt completions, enable LLM Observability.
langchain: Removes integration metrics from the LangChain integration. To continue tracking operational metrics from the OpenAI integration, enable LLM Observability or use trace metrics instead.
langchain: removed tracing support for langchain.chains.base.Chains. To maintain support, migrate to using Langchain's LCEL instead.
- langchain: Drops tracing and LLM Observability support for Langchain v0.0.x versions.
openai: Removes prompt-completion log sampling from the OpenAI integration. To continue logging prompt completions, enable LLM Observability.
openai: Removes integration metrics from the OpenAI integration. To continue tracking operational metrics from the OpenAI integration, enable LLM Observability or use trace metrics instead.
openai: Drops tracing and LLM Observability support for OpenAI 0.x versions. To maintain support, upgrade to openai>=1.0.
Support for the deprecated environment variable DD_LLMOBS_APP_NAME has been removed.
Removes the deprecated parameters argument from LLMObs annotate(). Use metadata instead.
Profiling
Tracing
DD_TRACE_SAMPLE_RATE, use DD_TRACE_SAMPLING_RULES instead.DD_TRACE_API_VERSION=v0.3, the default v0.5 version is used instead.DD_ANALYTICS_ENABLED, Datadog Analytics is no longer supported.DD_TRACE_ANALYTICS_ENABLED, Datadog Analytics is no longer supported.DD_HTTP_CLIENT_TAG_QUERY_STRING, DD_TRACE_HTTP_CLIENT_TAG_QUERY_STRING should be used instead.DD_TRACE_SPAN_AGGREGATOR_RLOCK, disabling the span aggregator rlock is no longer supported.aws_lambda: Drops support for aws_lambda<=6.105.0.DD_TRACE_METHODS with the '[]' notation. Ensure DD_TRACE_METHODS use the ':' notation instead".ddtracer parameter from ddtrace.opentracer.tracer.Tracer().DD_TRACE_PROPAGATION_STYLE to b3 single header is no longer supported. Use b3 instead.ddtrace.tracer.sampler.rules[].choose_matcher(...) method and removes the timestamp_ns parameter from ddtrace.internal.rate_limiter.RateLimiter.is_allowed().Span.sampled propertyddtrace.trace.tracer to access the global tracer instance.ddtrace.trace.PinTracer.configure(...) method and removes the Tracer.sampler attribute.ddtrace.trace.Tracer can not be reinitialized.ASM
LLM Observability
_DD_LLMOBS_EVALUATORS environment variable is deprecated and will be removed in ddtrace 3.0.0. As an alternative to _DD_LLMOBS_EVALUATORS, you can use DD_LLMOBS_EVALUATORS instead. To migrate, replace _DD_LLMOBS_EVALUATORS with DD_LLMOBS_EVALUATORS._DD_LLMOBS_EVALUATOR_SAMPLING_RULES environment variable is deprecated and will be removed in ddtrace 3.0.0. As an alternative to _DD_LLMOBS_EVALUATOR_SAMPLING_RULES, you can use DD_LLMOBS_EVALUATOR_SAMPLING_RULES instead. To migrate, replace _DD_LLMOBS_EVALUATOR_SAMPLING_RULES with DD_LLMOBS_EVALUATOR_SAMPLING_RULES.Tracing
ddtrace.provider.BaseContextProvider to ddtrace.trace.BaseContextProvider. The ddtrace.provider module is deprecated and will be removed in v3.0.0.ddtrace.constants module:
ANALYTICS_SAMPLE_RATE_KEYSAMPLE_RATE_METRIC_KEYSAMPLING_PRIORITY_KEYSAMPLING_AGENT_DECISIONSAMPLING_RULE_DECISIONSAMPLING_LIMIT_DECISIONORIGIN_KEYUSER_ID_KEYHOSTNAME_KEYRUNTIME_FAMILYBASE_SERVICE_KEYSPAN_MEASURED_KEYKEEP_SPANS_RATE_KEYMULTIPLE_IP_HEADERSCONFIG_ENDPOINT_ENVCONFIG_ENDPOINT_RETRIES_ENVCONFIG_ENDPOINT_TIMEOUT_ENVddtrace.settings.config module and deprecates the following ddtrace.config attributes:
http, use DD_TRACE_HEADER_TAGS environment variable instead.http_server, use DD_TRACE_HTTP_SERVER_ERROR_STATUSES environment variable instead.trace_headers, this attribute is internal to the tracer.header_is_traced, this attribute is internal to the tracer.convert_rc_trace_sampling_rules, this attribute is internal to the tracer.enable_remote_configuration, use DD_REMOTE_CONFIGURATION_ENABLED environment variable instead.get_from, use ddtrace.trace.Pin to set instance level configurations.ddtrace.config.ASM
CI Visibility
Code security (IAST)
callonce decorator could trigger an import loopLLM Observability
openai: Fixes a patching issue where asynchronous moderation endpoint calls resulted in coroutine scheduling errors.vertexai: Resolves an issue with chat.send_message() where the content keyword argument was not parsed correctly.LLMObs.annotate() on the same span caused overwriting of metrics, metadata, and prompt dictionaries. Now, calling LLMObs.annotate() will merge new metrics/metadata/prompt values with any existing ones.LLMObs.enable() to configure LLM Observability without environment variables would not automatically propagate distributed tracing headers.Profiling
DD_PROFILING_TIMELINE_ENABLED and DD_PROFILING_STACK_V2_ENABLED.Tracing
aiohttp: Adds the environment variable DD_AIOHTTP_CLIENT_DISABLE_STREAM_TIMING_FOR_MEM_LEAK to address a potential memory leak in the aiohttp integration. When set to true, this flag may cause streamed response span timing to be inaccurate. The flag defaults to false.cassandra, cherrypy, flask_cache, starlette: Ensures a deprecation warning is not raised when patching these integrations via ddtrace-run and import ddtrace.auto.ddtrace package.Ensures that no module from ASM are loaded when ASM is disabled or unavailable.
ASM
Code Security
Tracing
Removes deprecated parameters from Tracer.configure(...) method and removes the Tracer.sampler attribute.
LLM Observability
langchain: Removes the langchain.tokens.total_cost span metric for OpenAI calls. For continued cost estimation of OpenAI calls, enable LLM Observability.langchain: Removes prompt-completion log sampling from the LangChain integration. To continue logging prompt completions, enable LLM Observability.langchain: Removes integration metrics from the LangChain integration. To continue tracking operational metrics from the OpenAI integration, enable LLM Observability or use trace metrics instead.openai: Removes prompt-completion log sampling from the OpenAI integration. To continue logging prompt completions, enable LLM Observability.openai: Removes integration metrics from the OpenAI integration. To continue tracking operational metrics from the OpenAI integration, enable LLM Observability or use trace metrics instead.Tracing
ddtrace.trace.tracer to access the global tracer instance.ddtrace.trace.Tracer can not be reinitialized.Tracer.configure(...) method and removes the Tracer.sampler attribute.DD_TRACE_SAMPLE_RATE, use DD_TRACE_SAMPLING_RULES instead.DD_TRACE_API_VERSION=v0.3, the default v0.5 version is used instead.DD_ANALYTICS_ENABLED, Datadog Analytics is no longer supported.DD_TRACE_ANALYTICS_ENABLED, Datadog Analytics is no longer supported.DD_HTTP_CLIENT_TAG_QUERY_STRING, DD_TRACE_HTTP_CLIENT_TAG_QUERY_STRING should be used instead.DD_TRACE_SPAN_AGGREGATOR_RLOCK, disabling the span aggregator rlock is no longer supported.DD_TRACE_METHODS with the [] notation. Ensure DD_TRACE_METHODS use the : notation instead.ddtracer parameter from ddtrace.opentracer.tracer.Tracer().ddtrace.tracer.sampler.rules[].choose_matcher(...) method and removes the timestamp_ns parameter from ddtrace.internal.rate_limiter.RateLimiter.is_allowed().Span.sampled property.valkey: Adds automatic instrumentation of the Valkey package. Thank you AhmadMasry!aiohttp: Adds the environment variable DD_AIOHTTP_CLIENT_DISABLE_STREAM_TIMING_FOR_MEM_LEAK to address a potential memory leak in the aiohttp integration. When set to true, this flag may cause streamed response span timing to be inaccurate. The flag defaults to false.XSS detection for Django applications, which will be displayed on your DataDog Vulnerability Explorer dashboard. See the Application Vulnerability Man…
CI Visibility
Official release of the new version of the pytest plugin, introducing the following features:
NOTE: this new version of the plugin introduces breaking changes:
module, suite, and test names are now parsed from the item.nodeid attributeA beta version of the plugin had been available since v2.18.0, and could be enabled via the DD_PYTEST_USE_NEW_PLUGIN_BETA environment variable. The new version is now the default, and the environment variable is not used anymore.
LLM Observability
DD_LLMOBS_APP_NAME has been removed.parameters argument from LLMObs annotate(). Use metadata instead.openai: Drops tracing and LLM Observability support for OpenAI 0.x versions. To maintain support, upgrade to openai>=1.0.langchain.chains.base.Chains. To maintain support, migrate to using Langchain's LCEL instead.Profiling
Tracing
DD_TRACE_PROPAGATION_STYLE to b3 single header is no longer supported. Use b3 instead.aws_lambda: Drops support for aws_lambda<=6.105.0.ASM
DD_APPSEC_AUTOMATED_USER_EVENTS_TRACKING is now removed. Please use the new env var DD_APPSEC_AUTO_USER_INSTRUMENTATION_MODE instead.LLM Observability
\_DD_LLMOBS_EVALUATORS environment variable is deprecated and will be removed in ddtrace 3.0.0.\_DD_LLMOBS_EVALUATORS, you can use <span class="title-ref">DD_LLMOBS_EVALUATORS</span> instead. To migrate, replace \_DD_LLMOBS_EVALUATORS with DD_LLMOBS_EVALUATORS.\_DD_LLMOBS_EVALUATOR_SAMPLING_RULES environment variable is deprecated and will be removed in ddtrace 3.0.0.\_DD_LLMOBS_EVALUATOR_SAMPLING_RULES, you can use DD_LLMOBS_EVALUATOR_SAMPLING_RULES instead. To migrate, replace \_DD_LLMOBS_EVALUATOR_SAMPLING_RULES with DD_LLMOBS_EVALUATOR_SAMPLING_RULES.Tracing
ddtrace.provider.BaseContextProvider to ddtrace.trace.BaseContextProvider. The ddtrace.provider module is deprecated and will be removed in v3.0.0.ddtrace.constants module:
ANALYTICS_SAMPLE_RATE_KEYSAMPLE_RATE_METRIC_KEYSAMPLING_PRIORITY_KEYSAMPLING_AGENT_DECISIONSAMPLING_RULE_DECISIONSAMPLING_LIMIT_DECISIONORIGIN_KEYUSER_ID_KEYHOSTNAME_KEYRUNTIME_FAMILYBASE_SERVICE_KEYSPAN_MEASURED_KEYKEEP_SPANS_RATE_KEYMULTIPLE_IP_HEADERSCONFIG_ENDPOINT_ENVCONFIG_ENDPOINT_RETRIES_ENVCONFIG_ENDPOINT_TIMEOUT_ENVddtrace.settings.config module and deprecates the following ddtrace.config attributes:
http, use DD_TRACE_HEADER_TAGS environment variable instead.http_server, use DD_TRACE_HTTP_SERVER_ERROR_STATUSES environment variable instead.trace_headers, this attribute is internal to the tracer.header_is_traced, this attribute is internal to the tracer.convert_rc_trace_sampling_rules, this attribute is internal to the tracer.enable_remote_configuration, use DD_REMOTE_CONFIGURATION_ENABLED environment variable instead.get_from, use ddtrace.trace.Pin to set instance level configurations.ddtrace.config.Removes code and tests related to Python 3.7, breaking the library's compatibility with this Python version.
ASM
Code Security (IAST)
LLM Observability
ASM
CI Visibility
Code security (IAST)
LLM Observability
LLMObs.annotate() on the same span caused overwriting of metrics, metadata, and prompt dictionaries. Now, calling LLMObs.annotate() will merge new metrics/metadata/prompt values with any existing ones.LLMObs.enable() to configure LLM Observability without environment variables would not automatically propagate distributed tracing headers.openai: Fixes a patching issue where asynchronous moderation endpoint calls resulted in coroutine scheduling errors.vertexai: Resolves an issue with chat.send_message() where the content keyword argument was not parsed correctly.Profiling
DD_PROFILING_TIMELINE_ENABLED and DD_PROFILING_STACK_V2_ENABLED.cassandra , cherrypy, flask_cache, starlette: Ensures a deprecation warning is not raised when patching these integrations via ddtrace-run and import ddtrace.auto.ddtrace package.Pin to wrapt<2 until we can ensure full compatibility with the breaking changes.
Estimated end-of-life date, accurate to within three months: 11-2025 See the support level definitions for more information.
wrapt<2 until we can ensure full compatibility with the breaking changes.Estimated end-of-life date: 10-2025 See the support level definitions for more information.
Estimated end-of-life date: 10-2025 See the support level definitions for more information.
Estimated end-of-life date: 10-2025 See the support level definitions for more information.
Estimated end-of-life date: 10-2025 See the support level definitions for more information.
TypeError in encoding when truncating a large bytes object.unicode string is too large error.Resolved an issue where the CODEOWNERS file of the project being tested was looked up in the current working directory rather than the repository root
CODEOWNERS file of the project being tested was looked up in the current working directory rather than the repository root.TypeError, causing a failed connection. confluent-kafka supports both unpacked and packed config; this change allows initialization with either.SynchronizedSamplePool where pool could be null when calling into ddog_ArrayQueue_ functions, leading to segfaults in the uWSGI shutdown?* was not being matched correctly for DD_TRACE_SAMPLING_RULES tags, due to it matching on spans with no tag matching the specified key.Avoid excessive filtering of stacktrace locations when finding vulnerabilities. After this change, vulnerabilities that were previously discarded will…
Fixes incompatibility between code origin and dynamic instrumentation probe on exit span functions.
Fixes an issue with PosixPath handling in path operations that could cause errors during taint tracking. This fix improves stability and slightly redu
Code Security
Profiling
_self_acquired_at attribute.Tracing
Fixes a NotImplementedError that occurred when trying to deepcopy wrapped builtin functions (likeopen) while ASM or IAST were enabled. The error was c
ASM
NotImplementedError that occurred when trying to deepcopy wrapped builtin functions (likeopen) while ASM or IAST were enabled. The error was caused by the wrapper not implementing the __deepcopy__ method.LLM Observability
agentless_enabled=True or DD_LLMOBS_AGENTLESS_ENABLED=true.Profiling
AttributeError: '_ProfiledThreadingLock' object has no attribute '_self_acquired_at'.Tracing
pylibmc: Fixes an issue where using Client(server=[url]) would throw the error __init__() got multiple values for argument 'servers'Nothing published for this version
This release is missing the OCI images due to issues with the release pipeline. If you need an OCI image with the features introduced in the 2.14 rele
This release is missing the OCI images due to issues with the release pipeline. If you need an OCI image with the features introduced in the 2.14 release line, please use v2.14.5 instead.
configurations: Resolves an issue where DD_TRACE_CLOUD_PAYLOAD_TAGGING_SERVICES env variable was not parsed correctly
DD_TRACE_CLOUD_PAYLOAD_TAGGING_SERVICES env variable was not parsed correctlyFix performance overhead of Python distribution parsing for internal telemetry.
Internal:
ASM:
Library Injection:
boto3 from the minimum versions list to avoid blocking SSI on boto3 version.boto3 from the minimum versions list to avoid blocking SSI on boto3 version.Ensure that no module from ASM are loaded when ASM is disabled or unavailable.
ASM
Code Security
Profiling
Tracing
DD_AIOHTTP_CLIENT_DISABLE_STREAM_TIMING_FOR_MEM_LEAK to address a potential memory leak in the aiohttp integration. When set to true, this flag may cause streamed response span timing to be inaccurate. The flag defaults to false.__str__.Moves ddtrace.provider.BaseContextProvider to ddtrace.trace.BaseContextProvider. The ddtrace.provider module is deprecated and will be removed in v3.0…
openai: Introduces tracing support to the OpenAI integration for Python versions 3.12 and 3.13.ddtrace.provider.BaseContextProvider to ddtrace.trace.BaseContextProvider. The ddtrace.provider module is deprecated and will be removed in v3.0.0.ddtrace.constants module:
ANALYTICS_SAMPLE_RATE_KEYSAMPLE_RATE_METRIC_KEYSAMPLING_PRIORITY_KEYSAMPLING_AGENT_DECISIONSAMPLING_RULE_DECISIONSAMPLING_LIMIT_DECISIONORIGIN_KEYUSER_ID_KEYHOSTNAME_KEYRUNTIME_FAMILYBASE_SERVICE_KEYSPAN_MEASURED_KEYKEEP_SPANS_RATE_KEYMULTIPLE_IP_HEADERSCONFIG_ENDPOINT_ENVCONFIG_ENDPOINT_RETRIES_ENVCONFIG_ENDPOINT_TIMEOUT_ENVddtrace.settings.config module and deprecates the following ddtrace.config attributes:
http, use DD_TRACE_HEADER_TAGS environment variable instead.http_server, use DD_TRACE_HTTP_SERVER_ERROR_STATUSES environment variable instead.trace_headers, this attribute is internal to the tracer.header_is_traced, this attribute is internal to the tracer.convert_rc_trace_sampling_rules, this attribute is internal to the tracer.enable_remote_configuration, use DD_REMOTE_CONFIGURATION_ENABLED environment variable instead.get_from, use ddtrace.trace.Pin to set instance level configurations.CI Visibility
Code Security
callonce decorator could trigger an import loopLLM Observability
LLMObs.enable() to configure LLM Observability without environment variables would not automatically propagate distributed tracing headers.openai: Fixes a patching issue where asynchronous moderation endpoint calls resulted in coroutine scheduling errors.vertexai: Resolves an issue with chat.send_message() where the content keyword argument was not parsed correctly.Profiling
SystemError from the memory profiler returning NULL when collecting eventscassandra, cherrypy, flask_cache, starlette: Ensures a deprecation warning is not raised when patching these integrations via ddtrace-run and import ddtrace.auto.Moves ddtrace.provider.BaseContextProvider to ddtrace.trace.BaseContextProvider. The ddtrace.provider module is deprecated and will be removed in v3.0…
openai: Introduces tracing support to the OpenAI integration for Python versions 3.12 and 3.13.ddtrace.provider.BaseContextProvider to ddtrace.trace.BaseContextProvider. The ddtrace.provider module is deprecated and will be removed in v3.0.0.ddtrace.constants module:
ANALYTICS_SAMPLE_RATE_KEYSAMPLE_RATE_METRIC_KEYSAMPLING_PRIORITY_KEYSAMPLING_AGENT_DECISIONSAMPLING_RULE_DECISIONSAMPLING_LIMIT_DECISIONORIGIN_KEYUSER_ID_KEYHOSTNAME_KEYRUNTIME_FAMILYBASE_SERVICE_KEYSPAN_MEASURED_KEYKEEP_SPANS_RATE_KEYMULTIPLE_IP_HEADERSCONFIG_ENDPOINT_ENVCONFIG_ENDPOINT_RETRIES_ENVCONFIG_ENDPOINT_TIMEOUT_ENVddtrace.settings.config module and deprecates the following ddtrace.config attributes:
http, use DD_TRACE_HEADER_TAGS environment variable instead.http_server, use DD_TRACE_HTTP_SERVER_ERROR_STATUSES environment variable instead.trace_headers, this attribute is internal to the tracer.header_is_traced, this attribute is internal to the tracer.convert_rc_trace_sampling_rules, this attribute is internal to the tracer.enable_remote_configuration, use DD_REMOTE_CONFIGURATION_ENABLED environment variable instead.get_from, use ddtrace.trace.Pin to set instance level configurations.CI Visibility
Code security
LLM Observability
LLMObs.enable() to configure LLM Observability without environment variables would not automatically propagate distributed tracing headers.openai: Fixes a patching issue where asynchronous moderation endpoint calls resulted in coroutine scheduling errors.vertexai: Resolves an issue with chat.send_message() where the content keyword argument was not parsed correctly.Profiling
cassandra,cherrypy,flask_cache,starlette: Ensures a deprecation warning is not raised when patching these integrations via ddtrace-run and import ddtrace.auto.Ensures that no module from ASM are loaded when ASM is disabled or unavailable.
ASM
ImportError exception on Windows.Code Security
Tracing
Moves ddtrace.provider.BaseContextProvider to ddtrace.trace.BaseContextProvider. The ddtrace.provider module is deprecated and will be removed in v3.0…
ddtrace.provider.BaseContextProvider to ddtrace.trace.BaseContextProvider. The ddtrace.provider module is deprecated and will be removed in v3.0.0.ddtrace.constants module:
ANALYTICS_SAMPLE_RATE_KEYSAMPLE_RATE_METRIC_KEYSAMPLING_PRIORITY_KEYSAMPLING_AGENT_DECISIONSAMPLING_RULE_DECISIONSAMPLING_LIMIT_DECISIONORIGIN_KEYUSER_ID_KEYHOSTNAME_KEYRUNTIME_FAMILYBASE_SERVICE_KEYSPAN_MEASURED_KEYKEEP_SPANS_RATE_KEYMULTIPLE_IP_HEADERSCONFIG_ENDPOINT_ENVCONFIG_ENDPOINT_RETRIES_ENVCONFIG_ENDPOINT_TIMEOUT_ENVddtrace.settings.config module and deprecates the following ddtrace.config attributes:
http, use DD_TRACE_HEADER_TAGS environment variable instead.http_server, use DD_TRACE_HTTP_SERVER_ERROR_STATUSES environment variable instead.trace_headers, this attribute is internal to the tracer.header_is_traced, this attribute is internal to the tracer.convert_rc_trace_sampling_rules, this attribute is internal to the tracer.enable_remote_configuration, use DD_REMOTE_CONFIGURATION_ENABLED environment variable instead.get_from, use ddtrace.trace.Pin to set instance level configurations.LLMObs.enable() to configure LLM Observability without environment variables would not automatically propagate distributed tracing headers.vertexai: Resolves an issue with chat.send_message() where the content keyword argument was not parsed correctly.cassandra, cherrypy ,flask_cache, starlette: Ensures a deprecation warning is not raised when patching these integrations via ddtrace-run and import ddtrace.auto.Adds support for Header Injection vulnerability sink point.
ddtrace-run entrypoint.CI Visibility
ddtrace.contrib.<integration> to ddtrace.contrib.internal.<integration>.Tracing
ddtrace.filters.FilterRequestsOnUrl. Spans should be filtered/sampled using DD_TRACE_SAMPLING_RULES configuration.ddtrace.tracer) `should be used instead.ddtrace.configure(...) function. These parameters will be removed in ddtrace>=3.0.0: - enabled - hostname - port - uds_path - https - sampler - settings - priority_sampling - settings - dogstatsd_url - writer - partial_flush_enabled - partial_flush_min_spans - api_version - compute_stats_enabled - wrap_executorddtrace.pin module and moves the Pin class to ddtrace.trace package. In v3.0.0 the ddtrace/pin.py will be removed.ddtrace.filters module and moves the TraceFilter and FilterRequestsOnUrl classes to ddtrace.trace package. In v3.0.0 the ddtrace/filters.py will be removed.ddtrace.contrib.trace_utils_async and ddtrace.contrib.redis_utils. Replaces ddtrace.contrib.trace_utils_async.with_traced_module(...) with ddtrace.contrib.trace_utils.with_traced_module_async(...). Moves public attributes defined in ddtrace.contrib.redis_utils.* to ddtrace.contrib.trace_utils.ddtrace>=3.0.0 integrations should only be enabled and configured via ddtrace.patch(..), import ddtrace.auto or the ddtrace-run command. Unpatching integrations or getting the version of an integration is no longer supported.rq: Ensures the implementation details of the rq integration are internal to ddtrace library. In ddtrace>=3.0.0 this integration should only be enabled and configured via ddtrace.patch(..), import ddtrace.auto or the ddtrace-run commandASM
Code Security
LLM Observability
openai: Introduces automatic extraction of token usage from streamed chat completions. Unless stream_options: {"include_usage": False} is explicitly set on your streamed chat completion request, the OpenAI integration will add stream_options: {"include_usage": True} to your request and automatically extract the token usage chunk from the streamed response.Profiling
gunicorn with Stack V2 results in performance degradationgevent is lackingDD_PROFILING_STACK_V2_ENABLED=0. If you find any other issue, then please proceed to escalate using appropriate support channels or file an issue on the repository.Tracing
DD_TRACE_PROPAGATION_BEHAVIOR_EXTRACT to control the behavior of the extraction of distributed tracing headers. The values, continue (default), ignore, and restart, are supported. The default value is continue which has no change from the current behavior of always propagating valid headers. ignore ignores all incoming headers, never propagating the incoming trace information and restart turns the first extracted propagation style into a span link and propagates baggage if extracted.Code Security
Exception Replay
Lib Injection/SSI
LLM Observability
botocore: Resolves formatting errors in the bedrock integration when parsing request model IDs, which can now accept AWS ARNs.Profiling
AttributeError: _Profiled_ThreadingLock object has no attribute self_acquired_atTracing
asgi: Resolves an issue parsing response cookies in FastAPI and awsgiasyncio: Resolves an issue where asyncio event loops fail to register when ddtrace-run/import ddtrace.auto is used and gevent is installed.celery: Fixes an issue where celery.apply spans from Celery prerun got closed too soon leading to span tags being missing.django: Fixes issue where django cache is represented as a django service rather than the third party service.Adds support for Header Injection vulnerability sink point.
ddtrace-run entrypoint.CI Visibility
ddtrace.contrib.<integration> to ddtrace.contrib.internal.<integration>.Tracing
ddtrace.filters.FilterRequestsOnUrl. Spans should be filtered/sampled using DD_TRACE_SAMPLING_RULES configuration.ddtrace.tracer) `should be used instead.ddtrace.configure(...) function. These parameters will be removed in ddtrace>=3.0.0: - enabled - hostname - port - uds_path - https - sampler - settings - priority_sampling - settings - dogstatsd_url - writer - partial_flush_enabled - partial_flush_min_spans - api_version - compute_stats_enabled - wrap_executorddtrace.pin module and moves the Pin class to ddtrace.trace package. In v3.0.0 the ddtrace/pin.py will be removed.ddtrace.filters module and moves the TraceFilter and FilterRequestsOnUrl classes to ddtrace.trace package. In v3.0.0 the ddtrace/filters.py will be removed.ddtrace.contrib.trace_utils_async and ddtrace.contrib.redis_utils. Replaces ddtrace.contrib.trace_utils_async.with_traced_module(...) with ddtrace.contrib.trace_utils.with_traced_module_async(...). Moves public attributes defined in ddtrace.contrib.redis_utils.* to ddtrace.contrib.trace_utils.ddtrace>=3.0.0 integrations should only be enabled and configured via ddtrace.patch(..), import ddtrace.auto or the ddtrace-run command. Unpatching integrations or getting the version of an integration is no longer supported.rq: Ensures the implementation details of the rq integration are internal to ddtrace library. In ddtrace>=3.0.0 this integration should only be enabled and configured via ddtrace.patch(..), import ddtrace.auto or the ddtrace-run commandASM
Code Security
LLM Observability
openai: Introduces automatic extraction of token usage from streamed chat completions. Unless stream_options: {"include_usage": False} is explicitly set on your streamed chat completion request, the OpenAI integration will add stream_options: {"include_usage": True} to your request and automatically extract the token usage chunk from the streamed response.Profiling
gunicorn with Stack V2 results in performance degradationgevent is lackingDD_PROFILING_STACK_V2_ENABLED=0. If you find any other issue, then please proceed to escalate using appropriate support channels or file an issue on the repository.Tracing
DD_TRACE_PROPAGATION_BEHAVIOR_EXTRACT to control the behavior of the extraction of distributed tracing headers. The values, continue (default), ignore, and restart, are supported. The default value is continue which has no change from the current behavior of always propagating valid headers. ignore ignores all incoming headers, never propagating the incoming trace information and restart turns the first extracted propagation style into a span link and propagates baggage if extracted.Code Security
Exception Replay
Lib Injection/SSI
LLM Observability
botocore: Resolves formatting errors in the bedrock integration when parsing request model IDs, which can now accept AWS ARNs.Profiling
AttributeError: _Profiled_ThreadingLock object has no attribute self_acquired_atTracing
asgi: Resolves an issue parsing response cookies in FastAPI and awsgiasyncio: Resolves an issue where asyncio event loops fail to register when ddtrace-run/import ddtrace.auto is used and gevent is installed.celery: Fixes an issue where celery.apply spans from Celery prerun got closed too soon leading to span tags being missing.django: Fixes issue where django cache is represented as a django service rather than the third party service.ASM: This ensures that no module from ASM are loaded when ASM is disabled or unavailable.
ASM: This ensures that no module from ASM are loaded when ASM is disabled or unavailable.
SCA: This ensures that no module from IAST are loaded when IAST is disabled or unavailable.
Runtime Code Analysis (IAST): Avoid imports of IAST native module when IAST is not enabled.
ASM: This fix resolves an issue where IAST modules could be loaded, even if disabled,
which could create an ImportError exception on Windows.
CI Visibility: fixes an issue where Auto Test Retries with pytest would always consider retries of tests defined inside unittest classes to be success
celery: Fixes an issue where celery.apply spans from Celery prerun got closed too soon leading to span tags being missing.
celery.apply spans from Celery prerun got closed too soon leading to span tags being missing.chat.send_message() where the content keyword argument was not parsed correctly.Fixes an issue where the memory allocation profiler can cause a segmentation fault due to data races when accessing its own global data structures fro
Profiling
Tracing
django: Fixes issue where django cache is represented as a django service rather than the third party service.botocore: Resolves formatting errors in the bedrock integration when parsing request model IDs, which can now accept AWS ARNs.Introduces "Standalone SCA billing", opting out for APM billing and applying to only SCA. Enable this by setting these two environment variables: DD_A
ASM
DD_APPSEC_SCA_ENABLED and DD_EXPERIMENTAL_APPSEC_STANDALONE_ENABLEDCode Security
Profiling
DD_PROFILING_PYTORCH_ENABLED=true. This feature instruments the PyTorch profiler API (https://pytorch.org/docs/stable/_modules/torch/profiler/profiler.html) so that GPU profiling data can be sent to Datadog for visualization. This feature supports torch version >= 1.8.1.Tracing
azure_functions: Introduces support for Azure Functions.ASM
Code Security
Lib-Injection
LLM Observability
LLMObs.enable() ignored global patch configurations, specificallyDD_TRACE_<INTEGRATION>_ENABLED and DD_PATCH_MODULES environment variables.Telemetry
Tracing
ASGI: This fix resolves an issue parsing response cookies in FastAPI and awsgiasyncio: Resolves an issue where asyncio event loops fail to register when ddtrace-run/import ddtrace.auto is used and gevent is installed.datastreams: Logs at warning level for Kinesis errors that break the Data Streams Monitoring map.Adds more modules to the IAST patching denylist to improve startup time
Code Security
Tracing
ASGI: Resolves an issue parsing response cookies in FastAPI and awsgiasyncio: Resolves an issue where asyncio event loops fail to register when ddtrace-run/import ddtrace.auto is used and gevent is installed.Makes the library compatible with Python 3.13.
ASM
DD_APPSEC_SCA_ENABLED and DD_EXPERIMENTAL_APPSEC_STANDALONE_ENABLEDCode Security
Profiling
DD_PROFILING_PYTORCH_ENABLED=true. This feature instruments the PyTorch profiler API so that GPU profiling data can be sent to Datadog for visualization. This feature supports torch version >= 1.8.1.Tracing
azure_functions: Introduces support for Azure Functions.ASM
Lib-Injection
LLM Observability
LLMObs.enable() ignored global patch configurations, specificallyDD_TRACE_<INTEGRATION>_ENABLED and DD_PATCH_MODULES environment variables.Telemetry
Tracing
datastreams: Logs at warning level for Kinesis errors that break the Data Streams Monitoring map.Adds more modules to the IAST patching denylist to improve startup time
Code Security
Profiling
Tracing
ASGI: Resolves an issue parsing response cookies in FastAPI and awsgiFixes an issue where the memory allocation profiler can cause a segmentation fault due to data races when accessing its own global data structures fro
Profiling:
Tracing:
celery.apply spans from Celery prerun got closed too soon leading to span tags being missing.Ensures IAST SSRF vulnerability redacts the url query parameters correctly.
With this upgrade, you can now control how the stack trace report are cropped when reported for exploit prevention or IAST.
DD_APPSEC_MAX_STACK_TRACE_DEPTH allowed to control the maximum stack trace size reported (default 32)DD_APPSEC_MAX_STACK_TRACE_DEPTH_TOP_PERCENT allows now to specify how the stack trace is cropped as a percentage.For example, a value of 100 will report the top DD_APPSEC_MAX_STACK_TRACE_DEPTH frames from the stack, while a value of 0 will report the bottom DD_APPSEC_MAX_STACK_TRACE_DEPTH frames of the trace. A value of 50 will report half of DD_APPSEC_MAX_STACK_TRACE_DEPTH (rounded down) frames from the top of the stack and the rest from bottom. Default value is 75.
Upgrades libddwaf to 1.22.0
Upgrades libddwaf to 1.21.0 and security rule file to 1.13.3
CI Visibility
Beta release of the new version of the pytest plugin, introducing the following features:
Set the DD_PYTEST_USE_NEW_PLUGIN_BETA environment variable to true to use this new version.
NOTE: this new version of the plugin introduces breaking changes:
module, suite, and test names are now parsed from the item.nodeid attributeAdds support for Selenium and RUM integration
Code Security
DD_IAST_ENABLED=1 and DD_EXPERIMENTAL_APPSEC_STANDALONE_ENABLED=1.LLM Observability
vertexai: Introduces tracing support for Google's Vertex AI SDK for Python's generate_content and send_message calls. See the docs for more information.Profiling
tracer.configure()ASM
CI Visibility
None in EVP proxy mode if DD_ENV was not specified but the agent had a default environment set to a value other than none (eg: using DD_APM_ENV in the agent's environment).--ddtrace are no longer skipped when executing tests with pytest. Previously, the algorithm misinterpreted these arguments as standard flags, overlooking possible test paths that may contribute to the inferred service name.Code Security
psycopg3.umap, numba and pynndescent to the Code Security denylist.Crashtracking
Lib-Injection
sitecustomize.py are preserved when auto-injecting.click and jinja2 installed on 3.8 were outside of the allowed minimum versions for autoinstrumentation.LLM Observability
langchain: Resolves a JSON decoding issue resulting from tagging streamed outputs from chains ending with a PydanticOutputParser.Profiling
asyncio task names are not properly propagated when using stack v2, i.e. when DD_PROFILING_STACK_V2_ENABLED is set. Fixes an issue where asyncio tasks are not associated with spans when using stack v2, i.e. when DD_PROFILING_STACK_V2_ENABLED is set.Telemetry
Tracing
botocore: This fix resolves an issue in the Bedrock integration where not consuming the full response stream would prevent spans from finishing.botocore: This fix resolves the issue where the span pointer for deserialized DynamoDB requests (through the resource-based API) were not being generated.botocore: This fix resolves an issue where our span pointer calculation code added recently logged unactionable messages.celery: This fix resolves two issues with context propagation in celery
celery: Changes celery out.host span tag to point towards broker host url instead of local celery process hostname. Fixes inferred service representation issues when using celery.grpcaio: Resolves a concurrency bug where distributed tracing headers were overwritten resulting in spans being assigned to the wrong trace.kafka: Fixes an issue with Kafka consumer spans not using the active trace context when distributed tracing was enabled and no valid distributed context found was found within a consumed message.Ensures IAST SSRF vulnerability redacts the url query parameters correctly.
With this upgrade, you can now control how the stack trace report are cropped when reported for exploit prevention or IAST.
DD_APPSEC_MAX_STACK_TRACE_DEPTH allowed to control the maximum stack trace size reported (default 32)DD_APPSEC_MAX_STACK_TRACE_DEPTH_TOP_PERCENT allows now to specify how the stack trace is cropped as a percentage.For example, a value of 100 will report the top DD_APPSEC_MAX_STACK_TRACE_DEPTH frames from the stack, while a value of 0 will report the bottom DD_APPSEC_MAX_STACK_TRACE_DEPTH frames of the trace. A value of 50 will report half of DD_APPSEC_MAX_STACK_TRACE_DEPTH (rounded down) frames from the top of the stack and the rest from bottom. Default value is 75.
Upgrades libddwaf to 1.22.0
Upgrades libddwaf to 1.21.0 and security rule file to 1.13.3
CI Visibility
Beta release of the new version of the pytest plugin, introducing the following features:
Set the DD_PYTEST_USE_NEW_PLUGIN_BETA environment variable to true to use this new version.
NOTE: this new version of the plugin introduces breaking changes:
module, suite, and test names are now parsed from the item.nodeid attributeAdds support for Selenium and RUM integration
Code Security
-Introduces "Standalone Code Security", a feature that disables APM in the tracer but keeps Code Security (IAST) enabled. In order to enable it, set the environment variables DD_IAST_ENABLED=1 and DD_EXPERIMENTAL_APPSEC_STANDALONE_ENABLED=1.
LLM Observability
vertexai: Introduces tracing support for Google's Vertex AI SDK for Python's generate_content and send_message calls. See the docs for more information.Profiling
tracer.configure()ASM
CI Visibility
None in EVP proxy mode if DD_ENV was not specified but the agent had a default environment set to a value other than none (eg: using DD_APM_ENV in the agent's environment).--ddtrace are no longer skipped when executing tests with pytest. Previously, the algorithm misinterpreted these arguments as standard flags, overlooking possible test paths that may contribute to the inferred service name.Code Security
psycopg3.umap, numba and pynndescent to the Code Security denylist.Crashtracking
Lib-Injection
sitecustomize.py are preserved when auto-injecting.click and jinja2 installed on 3.8 were outside of the allowed minimum versions for autoinstrumentation.LLM Observability
langchain: Resolves a JSON decoding issue resulting from tagging streamed outputs from chains ending with a PydanticOutputParser.Profiling
asyncio task names are not properly propagated when using stack v2, i.e. when DD_PROFILING_STACK_V2_ENABLED is set. Fixes an issue where asyncio tasks are not associated with spans when using stack v2, i.e. when DD_PROFILING_STACK_V2_ENABLED is set.Telemetry
Ensures that Telemetry heartbeats are not skipped for forked processes, as doing so could result in the dependency list being lost over time.
Tracing
botocore: This fix resolves an issue in the Bedrock integration where not consuming the full response stream would prevent spans from finishing.botocore: This fix resolves the issue where the span pointer for deserialized DynamoDB requests (through the resource-based API) were not being generated.botocore: This fix resolves an issue where our span pointer calculation code added recently logged unactionable messages.celery: This fix resolves two issues with context propagation in celery
celery: Changes celery out.host span tag to point towards broker host url instead of local celery process hostname. Fixes inferred service representation issues when using celery.grpcaio: Resolves a concurrency bug where distributed tracing headers were overwritten resulting in spans being assigned to the wrong trace.kafka: Fixes an issue with Kafka consumer spans not using the active trace context when distributed tracing was enabled and no valid distributed context found was found within a consumed message.celery: Fixes an issue where celery.apply spans from Celery pre-run got closed too soon leading to span tags being missing.
celery: Fixes an issue where celery.apply spans from Celery pre-run got closed too soon leading to span tags being missing.Adds more modules to the IAST patching denylist to improve startup time
Code Security
ASM
LLM Observability
LLMObs.enable() ignored global patch configurations, specifically the DD_TRACE_<INTEGRATION>_ENABLED and DD_PATCH_MODULES environment variables.langchain: Resolves a JSON decoding issue resulting from tagging streamed outputs from chains ending with a PydanticOutputParser.Profiling
Tracing
ASGI: Resolves an issue parsing response cookies in FastAPI and awsgiEnsure IAST SSRF vulnerability redacts the url query parameters correctly.
SCA:
Celery:
Code Security:
Profiling:
Your coding agent can read these notes before it upgrades. Set up the MCP server →