NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
PyPI · #494 most downloaded on PyPI
Datadog APM client library
Last release 3 days ago
01 Oct 2026
Ships on a steady schedule
a new release about every 2 weeks
Nearly every release is documented
notes for 60 of the last 60 stable releases
5 versions withdrawn
withdrawn after publishing
10 years old
853 releases · first in 2017
internal: Fixes an issue on macOS where starting a subprocess can deadlock while a hostname lookup is in progress.
DD_TRACE_OTEL_CTX_ENABLED is set to true and a thread exits while an OpenTelemetry thread context is still attached.Resolves an issue where a streamed Bedrock InvokeModelWithResponseStream request that ended in an error or was cancelled part-way produced a span with
InvokeModelWithResponseStream request that ended in an error or was cancelled part-way produced a span with no output annotation. The text received before the failure is now tagged on the span.ConverseStream request that ended in an error or was cancelled part-way produced a span that was not marked as an error, making an interrupted request indistinguishable from a successful one.DD_RUNTIME_METRICS_ENABLED is true.One column per quarter.
tracing: DD_TRACE_OTEL_CTX_ENABLED now defaults to false until a Linux-native memory leak caused by setting it to true is fixed.
pytest-timeout thread mode and terminate a pytest-xdist worker.Resolves an issue where gen_ai.* attributes were added to APM spans when LLM Observability was disabled. These attributes are now only emitted when LL
gen_ai.* attributes were added to APM spans when LLM Observability was disabled. These attributes are now only emitted when LLM Observability is enabled.…and reported by dependency scanners as CVE-2025-56005 . Existing redaction paths continue to work.
Estimated end-of-life date, accurate to within three months: 05-2027
See the support level definitions for more information.
pydantic_ai integration now reports agent configuration using the shared agent manifest schema. framework, name, model, model_settings, instructions, system_prompts and tools keep their names and meaning. New keys are metadata, extra_instructions, capabilities, data_contracts, memory_policies, guardrails and agent_settings.discard boolean field to DD_TRACE_SAMPLING_RULES rules to fully drop trace chunks which were rejected during sampling. See the DD_TRACE_SAMPLING_RULES configuration docs for details.ot.th and ot.rv) to spans exported through OTLP when OTEL_TRACES_EXPORTER=otlp is configured, allowing downstream collectors to preserve and extrapolate sampling decisions.DD_RUNTIME_METRICS_ENABLED is true.DD_TRACE_OTEL_CTX_ENABLED=false or DD_TRACE_ASYNCIO_ENABLED=false.OTEL_TRACES_SPAN_METRICS_ENABLED=true to enable it.DD_TRACE_STATS_ADDITIONAL_TAGS, which adds configured span tags to native v0.6 trace stats and OTLP trace metrics.Span.remove_tag(key) and Span.remove_metric(key) for removing a previously set tag or metric from a span. Both are no-ops if the key is not set.DD_AGENTLESS_ENABLED (default false), a single switch that submits all telemetry, traces, Remote Configuration, Dynamic Instrumentation, crash reports, Test Optimization and LLM Observability data directly to the Datadog intake instead of through a local Datadog Agent. It becomes the default for the per-product agentless settings, which can still be set individually to override it. DD_API_KEY is required when it is enabled. Note that tracer flares, runtime metrics and profiling are notably not supported in this mode.DD_TRACE_OTEL_CTX_ENABLED=false.DD_TRACE_OTEL_CTX_ENABLED is true (the default), propagated trace contexts are published to OpenTelemetry thread context records. This lets external readers retain trace and span correlation when no active span object is available.DD_TRACE_OTEL_CTX_ENABLED to false.DD_TRACE_PROPAGATION_AS_SPAN_LINKS environment variable, a comma-separated list of integration names (e.g. google_cloud_pubsub,kafka) for which propagated context is attached as span links instead of a parent-child relationship.InvokeModel spans for Anthropic models, matching the Converse API.openai, litellm, and anthropic integrations.gen_ai.* attributes, so model, provider, application, conversation, and token usage can be searched, faceted, and monitored directly in APM.llm span.image_parts on LLM span messages, so they can be rendered in LLM Observability. Note: images referenced by a URL or a file ID are not captured, and a very large inline image is replaced with a text placeholder.DD_APP_KEY on the application when a Datadog Agent is used.LLMObs.list_experiments() to query experiments by logical name, metadata containment (e.g. {"tags": ["git.commit.sha:abc123"]}). Returns run status and aggregate data (evals, token costs, error rates), so CI/CD workflows can compare a run against a baseline commit and gate on the result.version when starting an agent span, either with LLMObs.agent(name="my-agent", version="v3") or with the @agent(version="v3") decorator. For agents created by an auto-instrumented framework, use LLMObs.annotation_context(agent={"version": "v3"}). This sets the agent_version tag on agent spans. The agent argument accepts a plain dictionary or the new ddtrace.llmobs.Agent TypedDict class.image_parts on LLM span messages, for both Chat Completions and the Responses API, so image inputs render in LLM Observability. This also applies to the OpenAI Agents SDK and to the LiteLLM integration, which share the same message extraction. Images referenced by a remote URL or file_id are not fetched and keep their existing text reference. A single inline image whose base64 payload exceeds 4 MiB is left as an [image omitted: too large] marker instead. Note that this budget is per image, not per request: several inline images (or an image alongside inline audio) can still take a span event past the 5 MB per-event limit, at which point the event's input and output are replaced with a placeholder. To stop image bytes from being recorded, register a span processor with LLMObs.register_processor and remove the image_parts key from the messages on span.input.LLMObs.submit_feedback() for submitting end-user feedback associated with a span, trace, session, or customer-defined feedback join key.DD_API_KEY; set DD_FEATURE_FLAGS_CONFIGURATION_SOURCE=remote_config to keep loading configuration through the Datadog Agent's Remote Configuration.DD_CIVISIBILITY_DYNAMIC_ATR_ENABLED to enable dynamic Auto Test Retries budgets based on test duration, instead of the flat per-test retry limit. Optionally configure the five duration-based budgets with DD_CIVISIBILITY_DYNAMIC_ATR_BUCKETS (five comma-separated integers in [1, 20]), corresponding to the Early Flake Detection duration buckets.ci.pipeline.display_name tag for the Buildkite provider, populated from the BUILDKITE_PIPELINE_NAME environment variable.DD_TRACE_ANYIO_ENABLED=false.kafka in the DD_TRACE_PROPAGATION_AS_SPAN_LINKS environment variable. When enabled together with distributed tracing, the kafka.consume span starts a new trace and relates to every consumed message's producer via span links, instead of continuing a single producer's trace. This is disabled by default.DD_PROFILING_MEMORY_MEM_DOMAIN_ENABLED now defaults to true, so the heap profiler also tracks allocations in the Mem domain on Python 3.12+.DD_PROFILING_NATIVE_HEAP_ENABLED=true.DD_PROFILING_STACK_GC_ENABLED=true.build_py option --no-bundle-libddwaf, set through the [build_py] section of setup.cfg, which builds the package without a bundled libddwaf. The library is then loaded from the system at import time (libddwaf.so.2, then libddwaf.so, resolved by the dynamic linker, and required to be 2.x). Intended for Linux distribution packages; default builds bundle libddwaf as before.httpx2 client.httpx2>=2.0.0aiomysql, aiopg, psycopg async, and MySQL Connector/Python asyncio database drivers.tags argument to ddtrace.data_streams.set_produce_checkpoint and ddtrace.data_streams.set_consume_checkpoint. Pass a list of "key:value" strings to attach additional edge tags to a manual checkpoint alongside the automatically set type, topic, and direction tags, for example set_produce_checkpoint("eventbridge", detail_type, carrier_set, tags=["exchange:my-bus"]). Existing calls that do not pass tags are unaffected.http/protobuf protocol (the default) and http/json, selected via the OTEL_EXPORTER_OTLP_TRACES_PROTOCOL environment variable.urllib3 requests include an Exploit Prevention frame in their traceback, which caused ordinary application errors to be misattributed to Datadog. Enabling urllib3 APM tracing, which is off by default, still contributes a frame of its own.DD_APPSEC_RULES to an empty value prevents AppSec rules from being updated through remote configuration.subprocess commands are passed as tuples.DD_AGENTLESS_ENABLED is set. The errors-intake upload was incorrectly reusing the same endpoint configured for the telemetry crash-report path, sending it to the telemetry intake host instead of the dedicated errors-intake host.InvokeModel integration only captured the first content block of an Anthropic response, so output messages were empty or truncated when a response began with a tool_use or thinking block.DD_APM_TRACING_ENABLED=false discarded every trace, so AI Guard, App and API Protection, Code Security and Software Composition Analysis reported no spans while running in standalone mode. These traces are now sent, and APM billing stays opted out.google_genai responses was merged into the output message instead of being captured as a separate reasoning message._DD_TRACE_WRITER_ADDITIONAL_HEADERS header values containing a whitespace (for example Bearer <token>) were incorrectly parsed, resulting in a truncated header value that caused LLM Observability spans to be dropped.claude-agent-sdk versions prior to 0.1.49, which do not report usage on the assistant message. Note that explicitly setting include_partial_messages=False on ClaudeAgentOptions opts out of this behavior and will result in inaccurate per-turn output token counts.agent_attribution missing or pointing at the wrong ancestor.TypeError raised during streamed tool-call reconstruction when an OpenAI-compatible backend (e.g. DashScope/Qwen) emits a tool-call delta with function.arguments or custom.input set to None instead of "".DD_FEATURE_FLAGS_CONFIGURATION_SOURCE is set to agentless or remote_config.LLMObs.annotate(agent=...) are also now coerced to strings.[image]; remote URL and file_id references are unchanged.[image] marker, or as the screenshot's URL when it is a remote reference.pytest plugin entered a code coverage collection context around every test even when code coverage was disabled, which reported coverage as active while no coverage was being collected and, on Python 3.12 and later, restarted sys.monitoring events once per test.TypeError.ValueError: I/O operation on closed file logging tracebacks at the end of pytest sessions when standard stream handlers point to streams closed by pytest. Tracer logs continue to reach healthy root handlers, including file output and log forwarding.site-packages or dist-packages directories.pass instead of fail when pytest exits with a non-OK, non-NO_TESTS_COLLECTED exit code (e.g. collection errors, interruptions, internal errors).PutEvents now propagates Data Streams Monitoring (DSM) context when DD_DATA_STREAMS_ENABLED is enabled.invoke_model_with_response_stream calls reported all token usage metrics as 0. Token counts are now extracted from the streamed response and set on the span.client_error, making them difficult to diagnose.webbrowser.open, or for calls to urllib.request.urlopen that pass the URL positionally or go through a custom opener installed with urllib.request.install_opener.urllib.request.urlopen and http.client requests include an internal Datadog frame in their traceback when Exploit Prevention is enabled, which caused ordinary application errors to be misattributed to the tracer.open() or pathlib.Path.open() are reported with a corrupted traceback when Exploit Prevention is enabled, showing a duplicated caller frame and omitting the frame that actually raised.OTEL_* environment variables override ddtrace defaults instead of being treated as unset.http.client.request span unfinished and active, so it is never sent and every later span on that thread is parented to it.@tracer.wrap on functions created at run time has been fixed.mysql.connector.Connect().rq.worker.perform_job span was not created when jobs were processed by rq.SimpleWorker on rq >= 2.7.WebDriver.get, WebDriver.quit and WebDriver.close calls made.tracer.configure(apm_tracing_disabled=True) did not apply the standalone sampling settings, so traces were sent without the one trace per minute limit and without the _dd.apm.enabled=0 metric that opts them out of APM billing.datadog.runtime_id resource attribute on OTLP trace and trace-metrics exports was a randomly generated value instead of the tracer's runtime id, preventing those exports from being correlated with the other signals emitted by the same process.aiohttp, httpx, requests, urllib3) where a strong reference to each response object was retained, preventing garbage collection of response bodies and causing linear RSS growth on sequential requests.DD_TRACE_OTEL_SEMANTICS_ENABLED=true, HTTP client spans now use the HTTP method as their resource name. This keeps the resource name distinct from the operation name and adds method-level information that better aligns with OpenTelemetry semantics; default resource names are unchanged.DD_TRACE_OTEL_CTX_ENABLED environment variable to false.runtime.python.* metrics stopped carrying the dd.internal.entity_id tag after the first flush, even when DD_ENTITY_ID was set. This prevented the Datadog Agent from attaching pod_name and kube_namespace to runtime metrics in Kubernetes.v0.5 trace API (DD_TRACE_API_VERSION=v0.5) drops traces when a single payload contains between 65,536 and 131,071 unique tag keys and values.deployment.environment.name attribute in OTEL_RESOURCE_ATTRIBUTES.mcp is importable but is not the MCP SDK. Patching is now skipped instead of raising.--lazy-apps and --skip-atexit can crash while workers shut down on Python 3.12 or later.get_final_message() or get_final_text().django.setup().ddtrace-run (or single-step instrumentation) could raise a PydanticSchemaGenerationError at import time for models that use forward references, when gevent is installed and the application runs on CPython 3.14 or later. The annotationlib module, which now owns the ForwardRef class on CPython 3.14, is preserved during module cloning so that pydantic resolves forward references correctly.AttributeError on google-adk >= 2.7.0 caused by the removal of __call_tool_live. Applications with LLM Observability enabled failed to start.google-adk < 2.7.0 where streaming tool call spans recorded no output.SystemError raised inside an instrumented request when an operation handled a container holding an object that cannot be hashed, such as the cookie objects Django builds in HttpResponse.set_cookie, and fixes the same error when a regular expression matched against a bytearray.NameError raised from application code that calls eval() when the wrapt C extension is unavailable and its pure-Python implementation is used, which caused Code Security to resolve the wrong caller scope.gunicorn or uWSGI) with no configuration, so every subsequent flag evaluation returned the caller-provided default with the PROVIDER_NOT_READY error code.ddtrace.openfeature, preventing the application from starting. Such values are now logged and the documented default is used instead.opentelemetry-exporter-otlp-proto-grpc 1.34.0 or later.DD_LOGS_OTEL_ENABLED=true) could cause the tracer to export its own internal log records, producing a growing volume of exported logs. The tracer's own log records and the OpenTelemetry exporter's log records are now excluded from OpenTelemetry logs export, while application logs continue to be collected and exported as before.pytorch.rank span could end too soon.requests calls pointed at the wrong span when the urllib3 integration was also enabled (for example via DD_TRACE_URLLIB3_ENABLED=true), causing those requests to be missing or show incorrect latency in downstream traces.command is passed as a keyword argument to execute or executemany.ModuleNotFoundError: No module named 'vllm.v1.engine.processor', silently disabling all vLLM tracing and metrics.pytest plugin performs once per test, which is most noticeable on large suites of fast tests.python-jsonpath in place of the vendored jsonpath-ng and ply, so ddtrace no longer ships ply, which is unmaintained and reported by dependency scanners as CVE-2025-56005. Existing redaction paths continue to work.tracer parameter of ddtrace.runtime.RuntimeMetrics.enable is deprecated and will be removed in a future version. It has no effect.DD_GOOGLE_CLOUD_PUBSUB_PROPAGATION_AS_SPAN_LINKS environment variable is deprecated and will be removed in a future release. Add google_cloud_pubsub to the DD_TRACE_PROPAGATION_AS_SPAN_LINKS list instead.Estimated end-of-life date, accurate to within three months: 05-2027
Estimated end-of-life date, accurate to within three months: 05-2027
See the support level definitions for more information.
discard boolean field to DD_TRACE_SAMPLING_RULES rules to fully drop trace chunks which were rejected during sampling. See the DD_TRACE_SAMPLING_RULES configuration docs for details.ot.th and ot.rv) to spans exported through OTLP when OTEL_TRACES_EXPORTER=otlp is configured, allowing downstream collectors to preserve and extrapolate sampling decisions.DD_RUNTIME_METRICS_ENABLED is true.DD_TRACE_OTEL_CTX_ENABLED=false or DD_TRACE_ASYNCIO_ENABLED=false.InvokeModel spans for Anthropic models, matching the Converse API.openai, litellm, and anthropic integrations.gen_ai.* attributes, so model, provider, application, conversation, and token usage can be searched, faceted, and monitored directly in APM.llm span.DD_CIVISIBILITY_DYNAMIC_ATR_ENABLED to enable dynamic Auto Test Retries budgets based on test duration, instead of the flat per-test retry limit. Optionally configure the five duration-based budgets with DD_CIVISIBILITY_DYNAMIC_ATR_BUCKETS (five comma-separated integers in [1, 20]), corresponding to the Early Flake Detection duration buckets.DD_TRACE_ANYIO_ENABLED=false.kafka in the DD_TRACE_PROPAGATION_AS_SPAN_LINKS environment variable. When enabled together with distributed tracing, the kafka.consume span starts a new trace and relates to every consumed message's producer via span links, instead of continuing a single producer's trace. This is disabled by default.DD_PROFILING_MEMORY_MEM_DOMAIN_ENABLED now defaults to true, so the heap profiler also tracks allocations in the Mem domain on Python 3.12+.build_py option --no-bundle-libddwaf, set through the [build_py] section of setup.cfg, which builds the package without a bundled libddwaf. The library is then loaded from the system at import time (libddwaf.so.2, then libddwaf.so, resolved by the dynamic linker, and required to be 2.x). Intended for Linux distribution packages; default builds bundle libddwaf as before.urllib3 requests include an Exploit Prevention frame in their traceback, which caused ordinary application errors to be misattributed to Datadog. Enabling urllib3 APM tracing, which is off by default, still contributes a frame of its own.DD_AGENTLESS_ENABLED is set. The errors-intake upload was incorrectly reusing the same endpoint configured for the telemetry crash-report path, sending it to the telemetry intake host instead of the dedicated errors-intake host.InvokeModel integration only captured the first content block of an Anthropic response, so output messages were empty or truncated when a response began with a tool_use or thinking block.DD_APM_TRACING_ENABLED=false discarded every trace, so AI Guard, App and API Protection, Code Security and Software Composition Analysis reported no spans while running in standalone mode. These traces are now sent, and APM billing stays opted out.google_genai responses was merged into the output message instead of being captured as a separate reasoning message.pytest plugin entered a code coverage collection context around every test even when code coverage was disabled, which reported coverage as active while no coverage was being collected and, on Python 3.12 and later, restarted sys.monitoring events once per test.TypeError.ValueError: I/O operation on closed file logging tracebacks at the end of pytest sessions when standard stream handlers point to streams closed by pytest. Tracer logs continue to reach healthy root handlers, including file output and log forwarding.PutEvents now propagates Data Streams Monitoring (DSM) context when DD_DATA_STREAMS_ENABLED is enabled.client_error, making them difficult to diagnose.webbrowser.open, or for calls to urllib.request.urlopen that pass the URL positionally or go through a custom opener installed with urllib.request.install_opener.urllib.request.urlopen and http.client requests include an internal Datadog frame in their traceback when Exploit Prevention is enabled, which caused ordinary application errors to be misattributed to the tracer.OTEL_* environment variables override ddtrace defaults instead of being treated as unset.http.client.request span unfinished and active, so it is never sent and every later span on that thread is parented to it.mysql.connector.Connect().rq.worker.perform_job span was not created when jobs were processed by rq.SimpleWorker on rq >= 2.7.WebDriver.get, WebDriver.quit and WebDriver.close calls made.tracer.configure(apm_tracing_disabled=True) did not apply the standalone sampling settings, so traces were sent without the one trace per minute limit and without the _dd.apm.enabled=0 metric that opts them out of APM billing.datadog.runtime_id resource attribute on OTLP trace and trace-metrics exports was a randomly generated value instead of the tracer's runtime id, preventing those exports from being correlated with the other signals emitted by the same process.mcp is importable but is not the MCP SDK. Patching is now skipped instead of raising.pytest plugin performs once per test, which is most noticeable on large suites of fast tests.The tracer parameter of ddtrace.runtime.RuntimeMetrics.enable is deprecated and will be removed in a future version. It has no effect.
Estimated end-of-life date, accurate to within three months: 05-2027
See the support level definitions for more information.
pydantic_ai integration now reports agent configuration using the shared agent manifest schema. framework, name, model, model_settings, instructions, system_prompts and tools keep their names and meaning. New keys are metadata, extra_instructions, capabilities, data_contracts, memory_policies, guardrails and agent_settings.tracer parameter of ddtrace.runtime.RuntimeMetrics.enable is deprecated and will be removed in a future version. It has no effect.DD_GOOGLE_CLOUD_PUBSUB_PROPAGATION_AS_SPAN_LINKS environment variable is deprecated and will be removed in a future release. Add google_cloud_pubsub to the DD_TRACE_PROPAGATION_AS_SPAN_LINKS list instead.httpx2>=2.0.0OTEL_TRACES_SPAN_METRICS_ENABLED=true to enable it.DD_TRACE_STATS_ADDITIONAL_TAGS, which adds configured span tags to native v0.6 trace stats and OTLP trace metrics.DD_TRACE_OTEL_CTX_ENABLED=false.DD_TRACE_OTEL_CTX_ENABLED is true (the default), propagated trace contexts are published to OpenTelemetry thread context records. This lets external readers retain trace and span correlation when no active span object is available.DD_TRACE_OTEL_CTX_ENABLED to false.DD_TRACE_PROPAGATION_AS_SPAN_LINKS environment variable, a comma-separated list of integration names (e.g. google_cloud_pubsub,kafka) for which propagated context is attached as span links instead of a parent-child relationship.DD_PROFILING_NATIVE_HEAP_ENABLED=true.DD_PROFILING_STACK_GC_ENABLED=true.httpx2 client.aiomysql, aiopg, psycopg async, and MySQL Connector/Python asyncio database drivers.image_parts on LLM span messages, so they can be rendered in LLM Observability. Note: images referenced by a URL or a file ID are not captured, and a very large inline image is replaced with a text placeholder.DD_APP_KEY on the application when a Datadog Agent is used.image_parts on LLM span messages, for both Chat Completions and the Responses API, so image inputs render in LLM Observability. This also applies to the OpenAI Agents SDK and to the LiteLLM integration, which share the same message extraction. Images referenced by a remote URL or file_id are not fetched and keep their existing text reference. A single inline image whose base64 payload exceeds 4 MiB is left as an [image omitted: too large] marker instead. Note that this budget is per image, not per request: several inline images (or an image alongside inline audio) can still take a span event past the 5 MB per-event limit, at which point the event's input and output are replaced with a placeholder. To stop image bytes from being recorded, register a span processor with LLMObs.register_processor and remove the image_parts key from the messages on span.input.aiohttp, httpx, requests, urllib3) where a strong reference to each response object was retained, preventing garbage collection of response bodies and causing linear RSS growth on sequential requests.DD_TRACE_OTEL_SEMANTICS_ENABLED=true, HTTP client spans now use the HTTP method as their resource name. This keeps the resource name distinct from the operation name and adds method-level information that better aligns with OpenTelemetry semantics; default resource names are unchanged.runtime.python.* metrics stopped carrying the dd.internal.entity_id tag after the first flush, even when DD_ENTITY_ID was set. This prevented the Datadog Agent from attaching pod_name and kube_namespace to runtime metrics in Kubernetes.deployment.environment.name attribute in OTEL_RESOURCE_ATTRIBUTES._DD_TRACE_WRITER_ADDITIONAL_HEADERS header values containing a whitespace (for example Bearer <token>) were incorrectly parsed, resulting in a truncated header value that caused LLM Observability spans to be dropped.claude-agent-sdk versions prior to 0.1.49, which do not report usage on the assistant message. Note that explicitly setting include_partial_messages=False on ClaudeAgentOptions opts out of this behavior and will result in inaccurate per-turn output token counts.agent_attribution missing or pointing at the wrong ancestor.DD_FEATURE_FLAGS_CONFIGURATION_SOURCE is set to agentless or remote_config.LLMObs.annotate(agent=...) are also now coerced to strings.[image]; remote URL and file_id references are unchanged.[image] marker, or as the screenshot's URL when it is a remote reference.DD_APPSEC_RULES to an empty value prevents AppSec rules from being updated through remote configuration.subprocess commands are passed as tuples.open() or pathlib.Path.open() are reported with a corrupted traceback when Exploit Prevention is enabled, showing a duplicated caller frame and omitting the frame that actually raised.AttributeError on google-adk >= 2.7.0 caused by the removal of __call_tool_live. Applications with LLM Observability enabled failed to start.google-adk < 2.7.0 where streaming tool call spans recorded no output.SystemError raised inside an instrumented request when an operation handled a container holding an object that cannot be hashed, such as the cookie objects Django builds in HttpResponse.set_cookie, and fixes the same error when a regular expression matched against a bytearray.NameError raised from application code that calls eval() when the wrapt C extension is unavailable and its pure-Python implementation is used, which caused Code Security to resolve the wrong caller scope.gunicorn or uWSGI) with no configuration, so every subsequent flag evaluation returned the caller-provided default with the PROVIDER_NOT_READY error code.ddtrace.openfeature, preventing the application from starting. Such values are now logged and the documented default is used instead.opentelemetry-exporter-otlp-proto-grpc 1.34.0 or later.command is passed as a keyword argument to execute or executemany.ModuleNotFoundError: No module named 'vllm.v1.engine.processor', silently disabling all vLLM tracing and metrics.tracing: DD_TRACE_OTEL_CTX_ENABLED now defaults to false until a Linux-native memory leak caused by setting it to true is fixed.
pytest-timeout thread mode and terminate a pytest-xdist worker.DD_APM_TRACING_ENABLED=false discarded every trace, so AI Guard, App and API Protection, Code Security and Software Composition Analysis reported no spans while running in standalone mode. These traces are now sent, and APM billing stays opted out.tracer.configure(apm_tracing_disabled=True) did not apply the standalone sampling settings, so traces were sent without the one trace per minute limit and without the _dd.apm.enabled=0 metric that opts them out of APM billing.Estimated end-of-life date, accurate to within three months: 05-2027
Estimated end-of-life date, accurate to within three months: 05-2027
See the support level definitions for more information.
google-adk: Fixes an AttributeError on google-adk >= 2.7.0 caused by the removal of __call_tool_live. Applications with LLM Observability enabled failed to start.
google-adk: Fixes an issue on google-adk < 2.7.0 where streaming tool call spans recorded no output.
vllm: Fixes an issue where the integration failed to instrument vLLM >= 0.14.0 with ModuleNotFoundError: No module named 'vllm.v1.engine.processor', silently disabling all vLLM tracing and metrics.
CI Visibility: Fixes an issue where test sessions running with pytest-xdist on shallow Git checkouts can spend excessive time unshallowing the repository and may fail or time out during collection.
LLM Observability: Fixes an issue where reasoning content from streamed google_genai responses was merged into the output message instead of being captured as a separate reasoning message.
LLM Observability: Fixes an issue where the agentless exporter dropped traces due to non-JSON-serializable objects being stored on a span. All LLM Observability span data is now made JSON-serializable before submission. Non-string span tag keys, span metric keys, and agent versions supplied through LLMObs.annotate(agent=...) are also coerced to strings.
MCP: Resolves an error logged when a module named mcp is importable but is not the MCP SDK. Patching is now skipped instead of raising an error.
Profiling: Fast memory copy is now automatically disabled when Python is running as an embedded interpreter.
As a result, ddtrace no longer ships ply , which is unmaintained and reported by dependency scanners as CVE-2025-56005 . Existing redaction paths cont…
Estimated end-of-life date, accurate to within three months: 05-2027
See the support level definitions for more information.
AI Guard:
CI Visibility:
ci.pipeline.display_name tag for the Buildkite provider, populated from the BUILDKITE_PIPELINE_NAME environment variable.Data Streams Monitoring:
tags argument to ddtrace.data_streams.set_produce_checkpoint and ddtrace.data_streams.set_consume_checkpoint. Pass a list of "key:value" strings to attach additional edge tags to a manual checkpoint alongside the automatically set type, topic, and direction tags. For example: set_produce_checkpoint("eventbridge", detail_type, carrier_set, tags=["exchange:my-bus"]). Existing calls that do not pass tags are unaffected.LLM Observability:
LLMObs.list_experiments() to query experiments by logical name and metadata containment, such as {"tags": ["git.commit.sha:abc123"]}. It returns run status and aggregate data—including evaluations, token costs, and error rates—so CI/CD workflows can compare a run against a baseline commit and gate on the result.version when starting an agent span, either with LLMObs.agent(name="my-agent", version="v3") or with the @agent(version="v3") decorator. For agents created by an auto-instrumented framework, use LLMObs.annotation_context(agent={"version": "v3"}). This sets the agent_version tag on agent spans. The agent argument accepts a plain dictionary or the new ddtrace.llmobs.Agent TypedDict class.LLMObs.submit_feedback() for submitting end-user feedback associated with a span, trace, session, or customer-defined feedback join key.OpenFeature:
DD_API_KEY. Set DD_FEATURE_FLAGS_CONFIGURATION_SOURCE=remote_config to continue loading configuration through the Datadog Agent's Remote Configuration.OpenTelemetry:
http/protobuf protocol—the default—and http/json, selected through the OTEL_EXPORTER_OTLP_TRACES_PROTOCOL environment variable.Tracing:
Span.remove_tag(key) and Span.remove_metric(key) for removing a previously set tag or metric from a span. Both methods are no-ops if the key is not set.AAP:
Botocore:
invoke_model_with_response_stream calls reported all token usage metrics as 0. Token counts are now extracted from the streamed response and set on the span.CI Visibility:
site-packages or dist-packages directories.pass instead of fail when pytest exited with a non-OK, non-NO_TESTS_COLLECTED exit code, such as for collection errors, interruptions, or internal errors.Code Origin for Spans:
Code Security:
Django:
django.setup().Gevent:
ddtrace-run or single-step instrumentation could raise a PydanticSchemaGenerationError at import time for models that use forward references when gevent is installed and the application runs on CPython 3.14 or later. The annotationlib module, which now owns the ForwardRef class on CPython 3.14, is preserved during module cloning so that Pydantic resolves forward references correctly.Kombu:
OpenTelemetry Logs:
DD_LOGS_OTEL_ENABLED=true could cause the tracer to export its own internal log records, producing a growing volume of exported logs. The tracer's own log records and the OpenTelemetry exporter's log records are now excluded, while application logs continue to be collected and exported.Profiling:
--lazy-apps and --skip-atexit could crash while workers shut down on Python 3.12 or later.PyTorch:
pytorch.rank span could end too soon.Requests:
requests calls pointed to the wrong span when the urllib3 integration was also enabled—for example, through DD_TRACE_URLLIB3_ENABLED=true—causing those requests to be missing or to show incorrect latency in downstream traces.Snowflake:
command was passed as a keyword argument to execute or executemany.Tracing:
DD_TRACE_OTEL_CTX_ENABLED environment variable to false._dd.svc_src was incorrectly set to either m or the service name.Botocore:
python-jsonpath instead of the vendored jsonpath-ng and ply. As a result, ddtrace no longer ships ply, which is unmaintained and reported by dependency scanners as CVE-2025-56005. Existing redaction paths continue to work.code origin for spans: prevent view or traced functions from raising an exception in situations where their number is large.
tracing: avoid reentrant SpanData borrow panic.
tracing: avoid reentrant SpanData borrow panic.
snowflake: Fixes an issue where traced cursor calls fail when command is passed as a keyword argument to execute or executemany .
command is passed as a keyword argument to execute or executemany.…and flagged by dependency scanners for CVE-2025-56005 . Existing redaction paths continue to work.
ci.pipeline.display_name tag for the Buildkite provider, populated from the BUILDKITE_PIPELINE_NAME environment variable.tags argument to ddtrace.data_streams.set_produce_checkpoint and ddtrace.data_streams.set_consume_checkpoint. Pass a list of "key:value" strings to attach additional edge tags to a manual checkpoint alongside the automatically set type, topic, and direction tags—for example, set_produce_checkpoint("eventbridge", detail_type, carrier_set, tags=["exchange:my-bus"]). Existing calls that do not pass tags are unaffected.LLMObs.list_experiments() to query experiments by logical name and metadata containment (for example, {"tags": ["git.commit.sha:abc123"]}). It returns run status and aggregate data (evaluations, token costs, and error rates), allowing CI/CD workflows to compare a run against a baseline commit and gate on the result.version when starting an agent span, either with LLMObs.agent(name="my-agent", version="v3") or with the @agent(version="v3") decorator. For agents created by an auto-instrumented framework, use LLMObs.annotation_context(agent={"version": "v3"}). This sets the agent_version tag on agent spans. The agent argument accepts either a plain dictionary or the new ddtrace.llmobs.Agent TypedDict class.LLMObs.submit_feedback() for submitting end-user feedback associated with a span, trace, session, or customer-defined feedback join key.DD_API_KEY; set DD_FEATURE_FLAGS_CONFIGURATION_SOURCE=remote_config to continue loading configuration through the Datadog Agent's Remote Configuration.http/protobuf protocol (the default) and http/json, selected via the OTEL_EXPORTER_OTLP_TRACES_PROTOCOL environment variable.Span.remove_tag(key) and Span.remove_metric(key) for removing a previously set tag or metric from a span. Both are no-ops if the key is not set.invoke_model_with_response_stream calls reported all token usage metrics as 0. Token counts are now extracted from the streamed response and set on the span.site-packages or dist-packages directories.pass instead of fail when pytest exited with a non-OK, non-NO_TESTS_COLLECTED exit code (for example, because of collection errors, interruptions, or internal errors).django.setup().ddtrace-run (or single-step instrumentation) could raise a PydanticSchemaGenerationError at import time for models that use forward references when gevent is installed and the application runs on CPython 3.14 or later. The annotationlib module, which now owns the ForwardRef class on CPython 3.14, is preserved during module cloning so that pydantic resolves forward references correctly.DD_LOGS_OTEL_ENABLED=true) could cause the tracer to export its own internal log records, producing a growing volume of exported logs. The tracer's own log records and the OpenTelemetry exporter's log records are now excluded from OpenTelemetry logs export, while application logs continue to be collected and exported as before.--lazy-apps and --skip-atexit could crash while workers shut down on Python 3.12 or later.pytorch.rank span could end too soon.requests calls pointed to the wrong span when the urllib3 integration was also enabled (for example, via DD_TRACE_URLLIB3_ENABLED=true), causing those requests to be missing or to show incorrect latency in downstream traces.DD_TRACE_OTEL_CTX_ENABLED environment variable to false._dd.svc_src was incorrectly set to either "m" or the service name.python-jsonpath in place of the vendored jsonpath-ng and ply, so ddtrace no longer ships ply, which is unmaintained and flagged by dependency scanners for CVE-2025-56005. Existing redaction paths continue to work.tracing: DD_TRACE_OTEL_CTX_ENABLED now defaults to false until a Linux-native memory leak caused by setting it to true is fixed.
CI visibility: Fixes an issue where test sessions running with pytest-xdist on shallow Git checkouts can spend excessive time unshallowing the repository and may fail or time out during collection.
OpenTelemetry logs: Fixes an issue where enabling OpenTelemetry logs (DD_LOGS_OTEL_ENABLED=true, or OTEL_SDK_DISABLED=false) could make the process stop responding and export an ever-growing volume of its own log records. Application logs are still collected and exported.
code origin for spans: prevent view or traced functions from raising an exception in situations where their number is large.
code origin for spans: prevent view or traced functions from raising an exception in situations where their number is large.
Profiling: Fast memory copy is now automatically disabled when Python is running as an embedded interpreter.
snowflake: Fixes an issue where traced cursor calls fail when command is passed as a keyword argument to execute or executemany .
command is passed as a keyword argument to execute or executemany.Deprecates the ml_app argument and DD_LLMOBS_ML_APP environment variable, which will be removed in the next major version of ddtrace. Use agent_servic…
ml_app argument and DD_LLMOBS_ML_APP environment variable, which will be removed in the next major version of ddtrace. Use agent_service and DD_SERVICE instead to group traces belonging to a root agentic service (i.e. previously known as ML application).ddtrace.appsec.ai_guard package to the top-level ddtrace.aiguard package. Importing from ddtrace.appsec.ai_guard is deprecated, now emits a ddtrace.DDTraceDeprecationWarning, and will be removed in 5.0.0. As an alternative, update imports such as from ddtrace.appsec.ai_guard import AIGuardClient to from ddtrace.aiguard import AIGuardClient. The Strands Agents classes are the exception: import them from ddtrace.aiguard.integrations.strands (AIGuardStrandsPlugin and AIGuardStrandsHookProvider are not re-exported from the top-level package).ddtrace/contrib/internal/pytest) is deprecated and will be removed in ddtrace 5.0.0. This plugin is active only when DD_PYTEST_USE_NEW_PLUGIN is explicitly set to false or 0. The new pytest plugin (ddtrace/testing/internal/pytest) is already the default. To migrate, remove the DD_PYTEST_USE_NEW_PLUGIN environment variable from your configuration.label parameter of LLMObs.get_prompt() and LLMObs.refresh_prompt() is deprecated. Set DD_ENV instead; the prompt version is resolved for that environment.ManagedPrompt.label property is deprecated. #19399DD_TRACE_OTEL_SEMANTICS_ENABLED to opt in to OpenTelemetry trace semantics for spans created through OpenTelemetry APIs and for OpenTelemetry Protocol (OTLP) trace export.DD_AI_GUARD_OPENAI_ENABLED switch to the Anthropic and LangChain auto-instrumentations: DD_AI_GUARD_ANTHROPIC_ENABLED and DD_AI_GUARD_LANGCHAIN_ENABLED (both true by default). Set either to false to disable AI Guard for that specific provider/framework without affecting other integrations or requiring a tracer version rollback.DD_APPSEC_AGENTIC_ONBOARDING environment variable. Its value is reported verbatim in configuration telemetry, letting Datadog record that a service was onboarded through the agentic onboarding solution.DD_BOTOCORE_BEDROCK_RESOLVE_INFERENCE_PROFILE (default false). When enabled, Bedrock calls using an application-inference-profile ARN as modelId resolve the underlying foundation model, so the span reports the real model and cost populates. Requires DD_LLMOBS_ENABLED=true and bedrock:GetInferenceProfile IAM permission.Session object on LLM Observability spans.agent_service as a preferred alias for the ml_app argument across SDK methods (LLMObs.enable(...), span constructors, decorators, submit_evaluation(...), get_spans(...), and publish_evaluator(...)). When provided, agent_service takes precedence over the deprecated ml_app argument and over service. When neither agent_service nor ml_app is set, application identity continues to resolve from a propagated parent value, then DD_LLMOBS_ML_APP, then DD_SERVICE.LLMObs.get_prompt() and rendered with .format() to the resulting LLM span.LLMObs.get_prompt() now resolves environment-scoped prompt versions and supports targeting for A/B testing. Set DD_ENV to select the version for that environment. The signature is now keyword-only after prompt_id and accepts targeting_key (sticky bucketing) and **attributes (targeting rules). #18127version argument to LLMObs.get_prompt() for retrieving an exact prompt version. #19050LLMObs.annotate now accept an optional image_parts field, a list of images each with a mime_type and either inline base64 content or an attachment_key. This lets images be collected and rendered in LLM Observability.client.realtime.connect(...), both sync and async). Each conversation turn is captured as an llm span with the user and assistant transcripts, audio, token usage, and any tool calls (function and MCP), grouped per connection so the whole conversation is traced end to end. Realtime instrumentation can be disabled on its own, without affecting the rest of the OpenAI integration, by setting DD_OPENAI_REALTIME_ENABLED=false.LLMObs for creating, updating, deleting, and listing prompts and prompt versions in the prompt registry. #18186env_ids argument to prompt creation and version update methods, allowing prompt versions to be deployed to feature-flag environments.session_id now propagates to following spans in a trace. If session_id is set explicitly on a span, this will take precedence over the inherited or propagated value.mistralai client.mistralai client.DD_CODE_COVERAGE_FLAGS to group and filter uploaded code coverage reports.DD_EXPERIMENTAL_FLAGGING_PROVIDER_SPAN_ENRICHMENT_ENABLED=true, the provider attaches feature-flag evaluation metadata (ffe_flags_enc, ffe_subjects_enc, and ffe_runtime_defaults) to the root APM span so traces and errors can be filtered by active flag variant. This is off by default and, when disabled, adds no per-span overhead.flagevaluation track), lighting up server-side flag-evaluation observability. The existing feature_flag.evaluations OpenTelemetry metric behavior is unchanged. The new emission is gated by DD_FLAGGING_EVALUATION_COUNTS_ENABLED (default enabled).get_final_message() or get_final_text().yaml/_yaml loaded during module cleanup, fixing an issue that broke PyYAML consumers such as Airflow.DD_PROFILING_MAX_FRAMES by clamping it before stack collection so samples stay within the backend's 600-location limit.SIGSEGV/SIGBUS handler that the profiler cannot safely chain with (e.g. CUDA, PyTorch, abseil via vLLM/gRPC; see sampler.cpp). The sampler upgrades to the faster fault-recovery copy if it still owns both fault handlers afterwards. Otherwise, it permanently falls back to the syscall-based copy.v0.5 trace API (DD_TRACE_API_VERSION=v0.5) drops traces when a single payload contains between 65,536 and 131,071 unique tag keys and values.@tracer.wrap on functions created at run time has been fixed.DD_TRACE_OTEL_CTX_ENABLED environment variable to false.TypeError raised during streamed tool-call reconstruction when an OpenAI-compatible backend (e.g. DashScope/Qwen) emits a tool-call delta with function.arguments or custom.input set to None instead of "".AssistantMessage chunks that share a message_id, the integration emitted one llm span per chunk and counted the turn's token usage multiple times. Chunks sharing a message_id are now merged into a single llm span with the usage counted once. This applies to claude-agent-sdk 0.1.51 and later, which expose message_id; earlier versions retain the previous behavior.google_adk integration dropped an agent span (logging missing span kind in span context) and orphaned its child spans when a Gemini response part could not be parsed. The span kind is now set before input/output extraction, so a parsing failure degrades to empty input/output instead of dropping the span.google_adk integrations rendered unhandled Gemini response parts (such as inline_data, file_data, empty parts, or thought-signature parts) as a confusing Unsupported file type placeholder. These parts now produce a concise summary or empty content. - IAST: A crash occurring when using uvloop has been fixed.DD_LLMOBS_OVERRIDE_ORIGIN had no effect when the resolved export mode routed events alongside the APM trace (APM_AGENTLESS or APM_AGENT). Setting this variable now always routes LLM Observability events directly through the LLM Observability writer so the override is respected.model_settings contained non-JSON-serializable provider sentinel values.TypeError: 'tuple' object does not support item assignment raised when passing headers as a sequence of (key, value) tuples (aiohttp's LooseHeaders type) to request methods such as get or post.aws.durable.replayed value and an aws.durable.operation_attempt one higher than its original execution.No extra information available when viewing details about why a test was skipped by Intelligent Test Runner, due to missing ITR correlation ID on test or suite events.DD_CIVISIBILITY_CODE_COVERAGE_REPORT_UPLOAD_ENABLED=true) alongside Test Impact Analysis caused per-test coverage data to be silently empty. The two features can now be enabled together.resource remaining as the raw URL path when a worker is killed before start_response runs (e.g. gunicorn --timeout). The route pattern is now populated from request.url_rule during preprocess_request.% string formatting handled tainted text containing literal IAST evidence marker delimiters.copy_ranges_from_strings and copy_and_shift_ranges_from_strings to the active request slot, matching the scoped taint read path. Previously these copy helpers resolved the taint map by scanning all request slots, so a concurrent or still-open request could capture the derived taint and the current request would miss the transformed tainted input.llm-kind LLM Observability spans for a single call when the langchain and google_genai integrations were both enabled, causing downstream cost and token metrics to be double-counted.openfeature-sdk 0.10.0 or later.break-ing out of astream() or stream() when an __interrupt__ is received in a human-in-the-loop workflow) left the graph's root span unfinished, so the trace failed to assemble and did not appear in Datadog. The root span is now finished when the stream is torn down, so interrupted runs are traced. - LLM Observability: This fix resolves an issue where annotating a span with a metric key containing a dot (e.g. anomaly.query_count) prevented the span from being ingested. Dots in metric keys are now replaced with underscores.openai-agents >= 0.8.0. The integration now instruments those versions, restoring agent manifest capture.DD_TRACE_CLOUD_REQUEST_PAYLOAD_TAGGING or DD_TRACE_CLOUD_RESPONSE_PAYLOAD_TAGGING is enabled.LLM Observability: Resolves a TypeError raised during streamed tool-call reconstruction when an OpenAI-compatible backend (e.g. DashScope/Qwen) e mits
TypeError raised during streamed tool-call reconstruction when an OpenAI-compatible backend (e.g. DashScope/Qwen) e
mits a tool-call delta with function.arguments or custom.input set to None instead of "".<!-- -->
Nothing published for this version
Deprecates the ml_app argument and DD_LLMOBS_ML_APP environment variable, which will be removed in the next major version of ddtrace. Use agent_servic…
ml_app argument and DD_LLMOBS_ML_APP environment variable, which will be removed in the next major version of ddtrace. Use agent_service and DD_SERVICE instead to group traces belonging to a root agentic service (i.e. previously known as ML application).AI Guard:
ddtrace.appsec.ai_guard package to the top-level ddtrace.aiguard package. Importing from ddtrace.appsec.ai_guard is deprecated, now emits a ddtrace.DDTraceDeprecationWarning, and will be removed in 5.0.0. As an alternative, update imports such as from ddtrace.appsec.ai_guard import AIGuardClient to from ddtrace.aiguard import AIGuardClient. The Strands Agents classes are the exception: import them from ddtrace.aiguard.integrations.strands (AIGuardStrandsPlugin and AIGuardStrandsHookProvider are not re-exported from the top-level package).pytest:
ddtrace/contrib/internal/pytest) is deprecated and will be removed in ddtrace 5.0.0. This plugin is active only when DD_PYTEST_USE_NEW_PLUGIN is explicitly set to false or 0. The new pytest plugin (ddtrace/testing/internal/pytest) is already the default. To migrate, remove the DD_PYTEST_USE_NEW_PLUGIN environment variable from your configuration.LLM Observability:
label parameter of LLMObs.get_prompt() and LLMObs.refresh_prompt() is deprecated. Set DD_ENV instead; the prompt version is resolved for that environment.Tracing:
DD_TRACE_OTEL_SEMANTICS_ENABLED to opt in to OpenTelemetry trace semantics for spans created through OpenTelemetry APIs and for OpenTelemetry Protocol (OTLP) trace export.AI Guard:
DD_AI_GUARD_OPENAI_ENABLED switch to the Anthropic and LangChain auto-instrumentations: DD_AI_GUARD_ANTHROPIC_ENABLED and DD_AI_GUARD_LANGCHAIN_ENABLED (both true by default). Set either to false to disable AI Guard for that specific provider/framework without affecting other integrations or requiring a tracer version rollback.AAP:
DD_APPSEC_AGENTIC_ONBOARDING environment variable. Its value is reported verbatim in configuration telemetry, letting Datadog record that a service was onboarded through the agentic onboarding solution.LLM Observability:
DD_BOTOCORE_BEDROCK_RESOLVE_INFERENCE_PROFILE (default false). When enabled, Bedrock calls using an application-inference-profile ARN as modelId resolve the underlying foundation model, so the span reports the real model and cost populates. Requires DD_LLMOBS_ENABLED=true and bedrock:GetInferenceProfile IAM permission.Session object on LLM Observability spans.agent_service as a preferred alias for the ml_app argument across SDK methods (LLMObs.enable(...), span constructors, decorators, submit_evaluation(...), get_spans(...), and publish_evaluator(...)). When provided, agent_service takes precedence over the deprecated ml_app argument and over service. When neither agent_service nor ml_app is set, application identity continues to resolve from a propagated parent value, then DD_LLMOBS_ML_APP, then DD_SERVICE.LLMObs.get_prompt() and rendered with .format() to the resulting LLM span.LLMObs.get_prompt() now resolves environment-scoped prompt versions and supports targeting for A/B testing. Set DD_ENV to select the version for that environment. The signature is now keyword-only after prompt_id and accepts targeting_key (sticky bucketing) and **attributes (targeting rules). #18127version argument to LLMObs.get_prompt() for retrieving an exact prompt version. #19050LLMObs.annotate now accept an optional image_parts field, a list of images each with a mime_type and either inline base64 content or an attachment_key. This lets images be collected and rendered in LLM Observability.client.realtime.connect(...), both sync and async). Each conversation turn is captured as an llm span with the user and assistant transcripts, audio, token usage, and any tool calls (function and MCP), grouped per connection so the whole conversation is traced end to end. Realtime instrumentation can be disabled on its own, without affecting the rest of the OpenAI integration, by setting DD_OPENAI_REALTIME_ENABLED=false.LLMObs for creating, updating, deleting, and listing prompts and prompt versions in the prompt registry. #18186env_ids argument to prompt creation and version update methods, allowing prompt versions to be deployed to feature-flag environments.session_id now propagates to following spans in a trace. If session_id is set explicitly on a span, this will take precedence over the inherited or propagated value.mistralai client.CI visibility:
DD_CODE_COVERAGE_FLAGS to group and filter uploaded code coverage reports.Crashtracking:
Stats computation:
Profiling:
openfeature:
DD_EXPERIMENTAL_FLAGGING_PROVIDER_SPAN_ENRICHMENT_ENABLED=true, the provider attaches feature-flag evaluation metadata (ffe_flags_enc, ffe_subjects_enc, and ffe_runtime_defaults) to the root APM span so traces and errors can be filtered by active flag variant. This is off by default and, when disabled, adds no per-span overhead.flagevaluation track), lighting up server-side flag-evaluation observability. The existing feature_flag.evaluations OpenTelemetry metric behavior is unchanged. The new emission is gated by DD_FLAGGING_EVALUATION_COUNTS_ENABLED (default enabled).mistralai:
mistralai client.Profiling:
aiohttp:
TypeError: 'tuple' object does not support item assignment raised when passing headers as a sequence of (key, value) tuples (aiohttp's LooseHeaders type) to request methods such as get or post.aws_durable_execution_sdk_python:
aws.durable.replayed value and an aws.durable.operation_attempt one higher than its original execution.CI Visibility:
No extra information available when viewing details about why a test was skipped by Intelligent Test Runner, due to missing ITR correlation ID on test or suite events.DD_CIVISIBILITY_CODE_COVERAGE_REPORT_UPLOAD_ENABLED=true) alongside Test Impact Analysis caused per-test coverage data to be silently empty. The two features can now be enabled together.LLM Observability:
AssistantMessage chunks that share a message_id, the integration emitted one llm span per chunk and counted the turn's token usage multiple times. Chunks sharing a message_id are now merged into a single llm span with the usage counted once. This applies to claude-agent-sdk 0.1.51 and later, which expose message_id; earlier versions retain the previous behavior.google_adk integration dropped an agent span (logging missing span kind in span context) and orphaned its child spans when a Gemini response part could not be parsed. The span kind is now set before input/output extraction, so a parsing failure degrades to empty input/output instead of dropping the span.google_adk integrations rendered unhandled Gemini response parts (such as inline_data, file_data, empty parts, or thought-signature parts) as a confusing Unsupported file type placeholder. These parts now produce a concise summary or empty content.LLMObs.submit_evaluation() incorrectly tagging evaluations with source:otel based on the global DD_TRACE_OTEL_ENABLED flag. The tag is now controlled by the is_otel field on the ExportedLLMObsSpan dictionary: True for OTel gen.ai instrumented spans, False for Datadog LLMObs spans. LLMObs.export_span() sets this field automatically.model_settings contained non-JSON-serializable provider sentinel values.Tracing:
flask:
resource remaining as the raw URL path when a worker is killed before start_response runs (e.g. gunicorn --timeout). The route pattern is now populated from request.url_rule during preprocess_request.Code Security (IAST):
% string formatting handled tainted text containing literal IAST evidence marker delimiters.copy_ranges_from_strings and copy_and_shift_ranges_from_strings to the active request slot, matching the scoped taint read path. Previously these copy helpers resolved the taint map by scanning all request slots, so a concurrent or still-open request could capture the derived taint and the current request would miss the transformed tainted input.uvloop has been fixed.langchain:
llm-kind LLM Observability spans for a single call when the langchain and google_genai integrations were both enabled, causing downstream cost and token metrics to be double-counted.openfeature:
openfeature-sdk 0.10.0 or later.AAP:
Fixed an issue that could have caused the internal symbol DB component to restart in nested fork child processes, potentially leading to high memory usage.
Fix crashes in uwsgi worker when exiting from SIGTERM.
openai_agents:
openai-agents >= 0.8.0. The integration now instruments those versions, restoring agent manifest capture.profiling:
DD_PROFILING_MAX_FRAMES by clamping it before stack collection so samples stay within the backend's 600-location limit.DD_TRACE_CLOUD_REQUEST_PAYLOAD_TAGGING or DD_TRACE_CLOUD_RESPONSE_PAYLOAD_TAGGING is enabled.anthropic: Fixes an issue where spans were not exported when streamed responses were consumed using get_final_message() or get_final_text() .
get_final_message() or get_final_text().command is passed as a keyword argument to execute or executemany.v0.5 trace API (DD_TRACE_API_VERSION=v0.5) drops traces when a single payload contains between 65,536 and 131,071 unique tag keys and values.@tracer.wrap on functions created at run time has been fixed.aws_durable_execution_sdk_python: This fix resolves an issue where a replayed durable operation that had failed permanently reported an incorrect aws.
aws.durable.replayed value and an aws.durable.operation_attempt one higher than its original execution.CI Visibility: This fix resolves an issue where the Datadog UI showed No extra information available when viewing details about why a test was skipped
CI Visibility: This fix resolves an issue where the Datadog UI showed No extra information available when viewing details about why a test was
skipped by Intelligent Test Runner, due to missing ITR correlation ID on test or suite events.
Code Security (IAST): fixes an issue where taint tracking could abort the Python process when old-style % string formatting handled tainted
text containing literal IAST evidence marker delimiters.
Code Security (IAST): This fix scopes copy_ranges_from_strings and copy_and_shift_ranges_from_strings to the active request slot, matching
the scoped taint read path. Previously these copy helpers resolved the taint map by scanning all request slots, so a concurrent or still-open
request could capture the derived taint and the current request would miss the transformed tainted input.
LLM Observability: Resolves a TypeError raised during streamed tool-call reconstruction when an OpenAI-compatible backend (e.g.
DashScope/Qwen) emits a tool-call delta with function.arguments or custom.input set to None instead of "".
Fix crashes in uwsgi worker when exiting from SIGTERM.
IAST: A crash occurring when using uvloop has been fixed.
LLM Observability: Resolves an issue where the OpenAI and LiteLLM integrations reported zero cost for OpenRouter requests using a
bring-your-own-key setup.
Fixed an issue where sensitive content could be left unredacted in vulnerability reports when a sensitive value overlapped a tainted range.
v0.4 trace API version (v0.5 cannot carry the LLMObs span data). Setting DD_TRACE_API_VERSION=v0.5 with LLMObs enabled logs a warning and downgrades to v0.4. No user action is required.<!-- -->
AI Guard:
create_agent.DD_AI_GUARD_OPENAI_ENABLED environment variable (true by default) as a per-provider kill switch. Set it to false to disable AI Guard auto-instrumentation of the OpenAI SDK without affecting other providers or requiring a tracer version rollback.DD_AI_GUARD_ANALYZE_STREAM_RESPONSES_ENABLED (default off); streams are buffered and evaluated before any chunk is delivered, raising AIGuardAbortError on a block.DD_APM_TRACING_ENABLED=false), traces produced by AI Guard are still sent to Datadog, kept with USER_KEEP sampling priority and the AI Guard decision maker so that they can be attributed to AI Guard, while APM host billing is not triggered.DD_AI_GUARD_ANALYZE_STREAM_RESPONSES_ENABLED=true (default false), all streamed chunks are buffered, the full assistant response is evaluated by AI Guard, and chunks are replayed to the caller only after an ALLOW verdict. On DENY/ABORT, AIGuardAbortError is raised and zero chunks are delivered. When the flag is off (the default), streaming behavior is unchanged.AAP:
_dd.appsec.normalized_route) for Tornado, following RFC-1103 and the existing FastAPI, Starlette, Django, and Flask implementations. The tag is emitted on every request span that already carries http.route when the API Security feature is active. Tornado routes using named capturing groups ((?P<name>...)) produce named parameters (e.g. /users/{id}); positional groups produce auto-numbered placeholders (e.g. {param1}). Optional trailing-slash patterns (/?) are treated as not declaring a trailing slash per RFC-1103 rule 1. #18398AppSec:
chat.completions, completions, or responses call is made during a web request, the WAF address server.business_logic.llm.event is emitted with the provider and model name. This enables WAF rules to tag the trace with appsec.events.llm.call.provider and appsec.events.llm.call.model, surfacing LLM-backed endpoints in the API Endpoint Catalog. Supports both sync and async OpenAI clients, Azure OpenAI (engine kwarg), and streaming. Only the first LLM call per request is recorded (endpoint-level detection, not per-invocation).aws_durable_execution_sdk_python:
aws.durable.operation_attempt tag to aws.durable.step and aws.durable.wait_for_condition spans. The tag is 0 for the original attempt and 1, 2, etc. for each subsequent retry.ray_serve:
ray[serve]>=2.47.1. When Ray is instrumented, ddtrace now traces Serve HTTP and gRPC requests, deployment routing, deployment method execution, and calls made with DeploymentHandle.remote(), with distributed trace context propagated across Serve deployments. See the Ray documentation for more information. #18076LLM Observability:
LLMObs.annotate now accept an optional audio_parts field, a list of audio segments each with a mime_type and either inline base64 content or an attachment_key. This lets audio (for example, speech-to-text and text-to-speech payloads) be collected and rendered in LLM Observability.sample_rate argument to LLMObs.enable() to configure the proportion of LLMObs traces to sample (between 0.0 and 1.0) in code. This takes precedence over the DD_LLMOBS_SAMPLE_RATE environment variable.SyncExperiment.rerun_evaluators() to re-run evaluators on the stored task outputs from a previous experiment run without re-executing the task function. Reads from experiment.result (set by run() or pull()) and returns a new SyncExperiment with fresh evaluations while preserving original span IDs, trace IDs, inputs, and outputs via replay span copies linked back to the originals via parent_experiment_span_id. The missing_task_strategy parameter controls behavior when prior rows contain errors: "raise" (default) raises immediately, "skip" omits the row, and "retry" re-executes the task for that row.gpt-audio, gpt-audio-mini, gpt-4o-audio-preview) as audio_parts on LLM span messages. Note: audio in streamed responses (stream=True) is not captured yet.DD_LLMOBS_SAMPLE_RATE, which controls the proportion of LLM Observability spans (between 0.0 and 1.0, defaults to 1.0) that are retained. This does not affect APM span retention or the accuracy of LLM Observability metrics such as token usage, and cost.metadata to the data exposed to the span processor registered with LLMObs.enable(span_processor=...) or LLMObs.register_processor(...). Previously only input and output were exposed; the processor can now read and modify metadata to redact or remove sensitive values that integrations record there.mistralai client chat and embedding calls.profiling:
mistralai:
mistralai client chat and embedding calls.pytorch:
pytorch.rank lifetime span for PyTorch distributed training. The span opens at init_process_group and closes at destroy_process_group or process exit. Tags include rank, world_size, framework (DDP / FSDP / DeepSpeed), launcher, torch.distributed.backend, and training_job_id (resolved from launcher environment variables). When running under Ray Train, ray.train.run_name, ray.submission_id, and ray.metadata.* are also applied. Enable with DD_PATCH_MODULES=pytorch:true.CI Visibility:
DD_TEST_MANAGEMENT_ATTEMPT_TO_FIX_RETRIES environment variable is now respected by the ddtrace.testing internal pytest plugin. When set to a valid integer, it overrides the retry count returned by the Datadog API for "attempt to fix" tests.Library Injection:
DD_INJECT_EXPERIMENTAL_OVERRIDE_USER_DDTRACE=true can be added to the environment.LLM Observability:
us3, us5, ap1, and ap2 Datadog sites. This affected customers on these sites when no Datadog Agent was running or agentless export was explicitly enabled (DD_LLMOBS_AGENTLESS_ENABLED=1).openai integration where streamed chat completion spans under-reported output_tokens and total_tokens for OpenAI-compatible providers that emit a cumulative usage object on every streamed chunk._DD_LLMOBS_WRITER_TIMEOUT, default 5 seconds) and instead used the 2 second connection default, causing intermittent TimeoutError and dropped span events on high-latency connections to the agent or intake.model_name with the request deployment name (e.g. my-deployment) instead of the canonical model returned by the API (e.g. gpt-4o-2024-08-06). The integration now prefers the response model for the azure and azure_text providers.metadata containing non-string keys (e.g. int/float/bool) could be dropped during ingestion. Metadata keys are now stringified before encoding.Omit/NotGiven sentinel values) as noise in LLM span metadata.anomaly.query_count) prevented the span from being ingested. Dots in metric keys are now replaced with underscores.DD_LLMOBS_OVERRIDE_ORIGIN had no effect when the resolved export mode routed events alongside the APM trace (APM_AGENTLESS or APM_AGENT). Setting this variable now always routes LLM Observability events directly through the LLM Observability writer so the override is respected.LLMObs.annotate(tags=...) could cause spans to be dropped during ingestion.AI Guard:
document content blocks, which could cause evaluation to be skipped for prompts whose only content was a document. Document text is now scanned and binary documents leave a placeholder so evaluation still runs.block dynamic parameter, supplied in the incoming request body, to disable the operator-configured blocking policy and let a denied request proceed in monitor mode. Request-supplied parameters can now only strengthen blocking (turning monitor mode into blocking); they can no longer weaken an enabled blocking policy.langchain 1.x:
aiokafka:
kafka.partition and kafka.message_offset span tags being incorrectly set to -1. Consumer spans no longer misreport 0 (e.g. single-partition topics or the first message in a partition) due to falsy evaluation. Producer spans and any span where the value is genuinely unknown now omit these tags entirely rather than emitting a misleading -1.anthropic:
model_provider as unknown for the AnthropicVertex and AnthropicBedrock clients. The provider is now resolved to google and amazon respectively.AAP:
Content-Type header for the Django, Tornado, Flask, and FastAPI integrations.urllib3 was not inspected for SSRF and API Security (API10).ASM:
ddtrace.contrib.trace_utils.set_user if neither ddtrace.appsec.trace_utils nor the user tracking SDK had been imported. The user blocking decision is now always enforced when AppSec is enabled.bytes commands passed to os.system, os.spawn*, and subprocess (for example os.system(b"...") or subprocess.run(b"...", shell=True)) bypassed the Exploit Prevention (RASP) shell and command injection checks. bytes commands are valid on POSIX systems and are now passed to the security checks like str commands.anonymization mode. It is now hashed, consistently with the user id and login.aws_durable_execution_sdk_python:
aws.durable.operation_attempt span tag was reported off by one for replayed durable operations, causing a resumed (replayed) execution to record a different attempt number than the original execution for the same operation. Replayed operations now report the same 0-indexed attempt number as a fresh execution.CI visibility:
ValueError: tool 1 is already in use error that occurred on Python 3.12+ when running pytest with coverage.py or pytest-cov alongside ddtrace. Both tools previously competed for sys.monitoring.COVERAGE_ID (slot 1). ddtrace now tries tool slots in priority order (4, 3, 1), falling back gracefully if all candidate slots are occupied.ResourceWarning: unclosed socket that occurred when using the pytest plugin. The background writer thread's thread-local HTTP connection was not closed on thread exit.database monitoring:
DD_DBM_PROPAGATION_MODE was set to full, service, or dynamic_service.elasticsearch:
coro.close() immediately on the error path. Affects Elasticsearch and OpenSearch integrations.Tracing:
TypeError: object NoneType can't be used in 'await' expression error. This occurs when the generator body awaits a coroutine that suspends to the event loop before its first yield. Python 3.9 and 3.10 are not affected.os.fork() could be sent twice, once by the parent and once by the child./v0.7/config payloads in multi-threaded applications (e.g. uWSGI).@tracer.wrap() to an async generator now forwards sent values, thrown exceptions, and close requests to the underlying generator, so it behaves like the unwrapped generator in all cases. Previously the wrapper only relayed values during forward iteration, so sent values were dropped and try/finally cleanup was skipped whenever the generator was closed early or received a thrown exception.TypeError: cannot pickle '_thread.lock' raised when serializing ddtrace-instrumented objects with cloudpickle. Fork-safe locks and events (ddtrace.internal.forksafe.Lock / Event) are now pickle-safe, and unpickling reconstructs a fresh instance.network.client.ip to contain the actual TCP peer IP address (i.e. REMOTE_ADDR) rather than the resolved HTTP client IP (which could originate from forwarding headers such as X-Forwarded-For). This affects the Flask, Django, ASGI (FastAPI/Starlette), and Tornado integrations. The resolved HTTP client IP continues to be reported under http.client_ip, which is the value used by App and API Protection (AAP) for IP-based threat detection and blocking.botocore:
_datadog message attribute is now read from the top-level SQS MessageAttributes first, and the message body is only parsed for SNS-to-SQS notifications.PutRecords now propagates trace context to each record in the batch. This enables trace context propagation to correctly map to each individual record, as well as enabling the correct tracking of messages through Data Streams Monitoring. Previously, only the first record would ever have been tracked.botocore, aiobotocore:
The request signature we calculated does not match the signature you provided when distributed tracing was enabled.celery:
DD_TRACE_REMOVE_INTEGRATION_SERVICE_NAMES_ENABLED, causing spans to always use celery-worker/celery-producer instead of the configured global service name.bootstrap:
yaml/_yaml loaded during module cleanup, fixing an issue that broke PyYAML consumers such as Airflow.Django:
process_exception, were not preserved, causing Django to skip exception handlers under ASGI.fastapi:
grpc:
grpc.aio client spans intermittently reporting error.message as "Internal error from Core" instead of the application-set abort details when a server aborts the RPC. #18015iast:
litellm:
AttributeError when metadata=None is passed to traced litellm.completion() calls. #18033MCP:
code origin:
sqlalchemy:
trace_engine() is called repeatedly for the same engine.core:
google_genai:
TypeError when Google GenAI returns None for optional response fields (token counts or candidates) introduced in google-genai>=2.6.0.Code Security (IAST):
re.finditer eagerly consumed the entire match iterator to taint every match, which removed the laziness of re.finditer and allowed a request-controlled input with many matches to allocate excess memory. Matches are now tainted lazily as the application iterates over them.kafka:
langgraph:
break-ing out of astream() or stream() when an __interrupt__ is received in a human-in-the-loop workflow) left the graph's root span unfinished, so the trace failed to assemble and did not appear in Datadog. The root span is now finished when the stream is torn down, so interrupted runs are traced.otel:
profiling:
SIGSEGV/SIGBUS handler that the profiler cannot wrap (e.g. CUDA, PyTorch, etc.). The sampler upgrades to the faster fault-recovery copy if it still owns both fault handlers afterwards. Otherwise, it permanently falls back to the syscall-based copy.protobuf:
redis:
out.host and server.address tags on spans produced by redis.cluster.RedisCluster and redis.asyncio.cluster.RedisCluster (redis-py >=4.1). These clients do not expose a connection_pool attribute, so connection metadata was never extracted and the Datadog inferred-entity feature fell back to the generic peer.db.system: redis identifier instead of resolving to the specific cluster hostname. Spans now include out.host and server.address derived from the cluster's startup node, enabling the service map to display a host-specific Redis entity.remoteconfig:
acknowledged before the product received it. The state is now set to acknowledged only after the configuration has been delivered to the product, and to error only when the payload is malformed and cannot be deserialized.starlette:
CORSMiddleware handled an OPTIONS request, the span resource was set to the raw URL path (e.g. OPTIONS /users/123) instead of the route template (e.g. OPTIONS /users/{user_id}). The route pattern is now resolved via the app's route tree when CORSMiddleware short-circuits before the Starlette router runs. Fixes #14321.Fixed an issue where sensitive content could be left unredacted in vulnerability reports when a sensitive value overlapped a tainted range.
Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
v0.4 trace API version (v0.5 cannot carry the LLMObs span data). Setting DD_TRACE_API_VERSION=v0.5 with LLMObs enabled logs a warning and downgrades to v0.4. No user action is required.DD_AI_GUARD_OPENAI_ENABLED environment variable (true by default) as a per-provider kill switch. Set it to false to disable AI Guard auto-instrumentation of the OpenAI SDK without affecting other providers or requiring a tracer version rollback.create_agent.DD_AI_GUARD_ANALYZE_STREAM_RESPONSES_ENABLED (default off); streams are buffered and evaluated before any chunk is delivered, raising AIGuardAbortError on a block.DD_APM_TRACING_ENABLED=false), traces produced by AI Guard are still sent to Datadog, kept with USER_KEEP sampling priority and the AI Guard decision maker so that they can be attributed to AI Guard, while APM host billing is not triggered.DD_AI_GUARD_ANALYZE_STREAM_RESPONSES_ENABLED=true (default false), all streamed chunks are buffered, the full assistant response is evaluated by AI Guard, and chunks are replayed to the caller only after an ALLOW verdict. On DENY/ABORT, AIGuardAbortError is raised and zero chunks are delivered. When the flag is off (the default), streaming behavior is unchanged._dd.appsec.normalized_route) for Tornado, following RFC-1103 and the existing FastAPI, Starlette, Django, and Flask implementations. The tag is emitted on every request span that already carries http.route when the API Security feature is active. Tornado routes using named capturing groups ((?P<name>...)) produce named parameters (e.g. /users/{id}); positional groups produce auto-numbered placeholders (e.g. {param1}). Optional trailing-slash patterns (/?) are treated as not declaring a trailing slash per RFC-1103 rule 1. #18398chat.completions, completions, or responses call is made during a web request, the WAF address server.business_logic.llm.event is emitted with the provider and model name. This enables WAF rules to tag the trace with appsec.events.llm.call.provider and appsec.events.llm.call.model, surfacing LLM-backed endpoints in the API Endpoint Catalog. Supports both sync and async OpenAI clients, Azure OpenAI (engine kwarg), and streaming. Only the first LLM call per request is recorded (endpoint-level detection, not per-invocation).aws.durable.operation_attempt tag to aws.durable.step and aws.durable.wait_for_condition spans. The tag is 0 for the original attempt and 1, 2, etc. for each subsequent retry.ray[serve]>=2.47.1. When Ray is instrumented, ddtrace now traces Serve HTTP and gRPC requests, deployment routing, deployment method execution, and calls made with DeploymentHandle.remote(), with distributed trace context propagated across Serve deployments. See the Ray documentation for more information. #18076LLMObs.annotate now accept an optional audio_parts field, a list of audio segments each with a mime_type and either inline base64 content or an attachment_key. This lets audio (for example, speech-to-text and text-to-speech payloads) be collected and rendered in LLM Observability.sample_rate argument to LLMObs.enable() to configure the proportion of LLMObs traces to sample (between 0.0 and 1.0) in code. This takes precedence over the DD_LLMOBS_SAMPLE_RATE environment variable.SyncExperiment.rerun_evaluators() to re-run evaluators on the stored task outputs from a previous experiment run without re-executing the task function. Reads from experiment.result (set by run() or pull()) and returns a new SyncExperiment with fresh evaluations while preserving original span IDs, trace IDs, inputs, and outputs via replay span copies linked back to the originals via parent_experiment_span_id. The missing_task_strategy parameter controls behavior when prior rows contain errors: "raise" (default) raises immediately, "skip" omits the row, and "retry" re-executes the task for that row.gpt-audio, gpt-audio-mini, gpt-4o-audio-preview) as audio_parts on LLM span messages. Note: audio in streamed responses (stream=True) is not captured yet.DD_LLMOBS_SAMPLE_RATE, which controls the proportion of LLM Observability spans (between 0.0 and 1.0, defaults to 1.0) that are retained. This does not affect APM span retention or the accuracy of LLM Observability metrics such as token usage, and cost.metadata to the data exposed to the span processor registered with LLMObs.enable(span_processor=...) or LLMObs.register_processor(...). Previously only input and output were exposed; the processor can now read and modify metadata to redact or remove sensitive values that integrations record there.mistralai client chat and embedding calls.mistralai client chat and embedding calls.pytorch.rank lifetime span for PyTorch distributed training. The span opens at init_process_group and closes at destroy_process_group or process exit. Tags include rank, world_size, framework (DDP / FSDP / DeepSpeed), launcher, torch.distributed.backend, and training_job_id (resolved from launcher environment variables). When running under Ray Train, ray.train.run_name, ray.submission_id, and ray.metadata.* are also applied. Enable with DD_PATCH_MODULES=pytorch:true.DD_INJECT_EXPERIMENTAL_OVERRIDE_USER_DDTRACE=true can be added to the environment.DD_TEST_MANAGEMENT_ATTEMPT_TO_FIX_RETRIES environment variable is now respected by the ddtrace.testing internal pytest plugin. When set to a valid integer, it overrides the retry count returned by the Datadog API for "attempt to fix" tests.us3, us5, ap1, and ap2 Datadog sites. This affected customers on these sites when no Datadog Agent was running or agentless export was explicitly enabled (DD_LLMOBS_AGENTLESS_ENABLED=1).openai integration where streamed chat completion spans under-reported output_tokens and total_tokens for OpenAI-compatible providers that emit a cumulative usage object on every streamed chunk._DD_LLMOBS_WRITER_TIMEOUT, default 5 seconds) and instead used the 2 second connection default, causing intermittent TimeoutError and dropped span events on high-latency connections to the agent or intake.model_name with the request deployment name (e.g. my-deployment) instead of the canonical model returned by the API (e.g. gpt-4o-2024-08-06). The integration now prefers the response model for the azure and azure_text providers.metadata containing non-string keys (e.g. int/float/bool) could be dropped during ingestion. Metadata keys are now stringified before encoding.Omit/NotGiven sentinel values) as noise in LLM span metadata.LLMObs.annotate(tags=...) could cause spans to be dropped during ingestion.document content blocks, which could cause evaluation to be skipped for prompts whose only content was a document. Document text is now scanned and binary documents leave a placeholder so evaluation still runs.block dynamic parameter, supplied in the incoming request body, to disable the operator-configured blocking policy and let a denied request proceed in monitor mode. Request-supplied parameters can now only strengthen blocking (turning monitor mode into blocking); they can no longer weaken an enabled blocking policy.kafka.partition and kafka.message_offset span tags being incorrectly set to -1. Consumer spans no longer misreport 0 (e.g. single-partition topics or the first message in a partition) due to falsy evaluation. Producer spans and any span where the value is genuinely unknown now omit these tags entirely rather than emitting a misleading -1.model_provider as unknown for the AnthropicVertex and AnthropicBedrock clients. The provider is now resolved to google and amazon respectively.Content-Type header for the Django, Tornado, Flask, and FastAPI integrations.urllib3 was not inspected for SSRF and API Security (API10).ddtrace.contrib.trace_utils.set_user if neither ddtrace.appsec.trace_utils nor the user tracking SDK had been imported. The user blocking decision is now always enforced when AppSec is enabled.bytes commands passed to os.system, os.spawn*, and subprocess (for example os.system(b"...") or subprocess.run(b"...", shell=True)) bypassed the Exploit Prevention (RASP) shell and command injection checks. bytes commands are valid on POSIX systems and are now passed to the security checks like str commands.anonymization mode. It is now hashed, consistently with the user id and login.aws.durable.operation_attempt span tag was reported off by one for replayed durable operations, causing a resumed (replayed) execution to record a different attempt number than the original execution for the same operation. Replayed operations now report the same 0-indexed attempt number as a fresh execution.ValueError: tool 1 is already in use error that occurred on Python 3.12+ when running pytest with coverage.py or pytest-cov alongside ddtrace. Both tools previously competed for sys.monitoring.COVERAGE_ID (slot 1). ddtrace now tries tool slots in priority order (4, 3, 1), falling back gracefully if all candidate slots are occupied.ResourceWarning: unclosed socket that occurred when using the pytest plugin. The background writer thread's thread-local HTTP connection was not closed on thread exit.DD_DBM_PROPAGATION_MODE was set to full, service, or dynamic_service.coro.close() immediately on the error path. Affects Elasticsearch and OpenSearch integrations.TypeError: object NoneType can't be used in 'await' expression error. This occurs when the generator body awaits a coroutine that suspends to the event loop before its first yield. Python 3.9 and 3.10 are not affected.os.fork() could be sent twice, once by the parent and once by the child./v0.7/config payloads in multi-threaded applications (e.g. uWSGI).@tracer.wrap() to an async generator now forwards sent values, thrown exceptions, and close requests to the underlying generator, so it behaves like the unwrapped generator in all cases. Previously the wrapper only relayed values during forward iteration, so sent values were dropped and try/finally cleanup was skipped whenever the generator was closed early or received a thrown exception.TypeError: cannot pickle '_thread.lock' raised when serializing ddtrace-instrumented objects with cloudpickle. Fork-safe locks and events (ddtrace.internal.forksafe.Lock / Event) are now pickle-safe, and unpickling reconstructs a fresh instance.network.client.ip to contain the actual TCP peer IP address (i.e. REMOTE_ADDR) rather than the resolved HTTP client IP (which could originate from forwarding headers such as X-Forwarded-For). This affects the Flask, Django, ASGI (FastAPI/Starlette), and Tornado integrations. The resolved HTTP client IP continues to be reported under http.client_ip, which is the value used by App and API Protection (AAP) for IP-based threat detection and blocking._datadog message attribute is now read from the top-level SQS MessageAttributes first, and the message body is only parsed for SNS-to-SQS notifications.PutRecords now propagates trace context to each record in the batch. This enables trace context propagation to correctly map to each individual record, as well as enabling the correct tracking of messages through Data Streams Monitoring. Previously, only the first record would ever have been tracked.The request signature we calculated does not match the signature you provided when distributed tracing was enabled.DD_TRACE_REMOVE_INTEGRATION_SERVICE_NAMES_ENABLED, causing spans to always use celery-worker/celery-producer instead of the configured global service name.process_exception, were not preserved, causing Django to skip exception handlers under ASGI.grpc.aio client spans intermittently reporting error.message as "Internal error from Core" instead of the application-set abort details when a server aborts the RPC. #18015re.finditer eagerly consumed the entire match iterator to taint every match, which removed the laziness of re.finditer and allowed a request-controlled input with many matches to allocate excess memory. Matches are now tainted lazily as the application iterates over them.AttributeError when metadata=None is passed to traced litellm.completion() calls. #18033trace_engine() is called repeatedly for the same engine.TypeError when Google GenAI returns None for optional response fields (token counts or candidates) introduced in google-genai>=2.6.0.out.host and server.address tags on spans produced by redis.cluster.RedisCluster and redis.asyncio.cluster.RedisCluster (redis-py >=4.1). These clients do not expose a connection_pool attribute, so connection metadata was never extracted and the Datadog inferred-entity feature fell back to the generic peer.db.system: redis identifier instead of resolving to the specific cluster hostname. Spans now include out.host and server.address derived from the cluster's startup node, enabling the service map to display a host-specific Redis entity.acknowledged before the product received it. The state is now set to acknowledged only after the configuration has been delivered to the product, and to error only when the payload is malformed and cannot be deserialized.CORSMiddleware handled an OPTIONS request, the span resource was set to the raw URL path (e.g. OPTIONS /users/123) instead of the route template (e.g. OPTIONS /users/{user_id}). The route pattern is now resolved via the app's route tree when CORSMiddleware short-circuits before the Starlette router runs. Fixes #14321.Fixed an issue where sensitive content could be left unredacted in vulnerability reports when a sensitive value overlapped a tainted range.
Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
v0.4 trace API version (v0.5 cannot carry the LLMObs span data). Setting DD_TRACE_API_VERSION=v0.5 with LLMObs enabled logs a warning and downgrades to v0.4. No user action is required.DD_AI_GUARD_OPENAI_ENABLED environment variable (true by default) as a per-provider kill switch. Set it to false to disable AI Guard auto-instrumentation of the OpenAI SDK without affecting other providers or requiring a tracer version rollback.create_agent.DD_AI_GUARD_ANALYZE_STREAM_RESPONSES_ENABLED (default off); streams are buffered and evaluated before any chunk is delivered, raising AIGuardAbortError on a block.DD_APM_TRACING_ENABLED=false), traces produced by AI Guard are still sent to Datadog, kept with USER_KEEP sampling priority and the AI Guard decision maker so that they can be attributed to AI Guard, while APM host billing is not triggered.DD_AI_GUARD_ANALYZE_STREAM_RESPONSES_ENABLED=true (default false), all streamed chunks are buffered, the full assistant response is evaluated by AI Guard, and chunks are replayed to the caller only after an ALLOW verdict. On DENY/ABORT, AIGuardAbortError is raised and zero chunks are delivered. When the flag is off (the default), streaming behavior is unchanged._dd.appsec.normalized_route) for Tornado, following RFC-1103 and the existing FastAPI, Starlette, Django, and Flask implementations. The tag is emitted on every request span that already carries http.route when the API Security feature is active. Tornado routes using named capturing groups ((?P<name>...)) produce named parameters (e.g. /users/{id}); positional groups produce auto-numbered placeholders (e.g. {param1}). Optional trailing-slash patterns (/?) are treated as not declaring a trailing slash per RFC-1103 rule 1. #18398chat.completions, completions, or responses call is made during a web request, the WAF address server.business_logic.llm.event is emitted with the provider and model name. This enables WAF rules to tag the trace with appsec.events.llm.call.provider and appsec.events.llm.call.model, surfacing LLM-backed endpoints in the API Endpoint Catalog. Supports both sync and async OpenAI clients, Azure OpenAI (engine kwarg), and streaming. Only the first LLM call per request is recorded (endpoint-level detection, not per-invocation).aws.durable.operation_attempt tag to aws.durable.step and aws.durable.wait_for_condition spans. The tag is 0 for the original attempt and 1, 2, etc. for each subsequent retry.ray[serve]>=2.47.1. When Ray is instrumented, ddtrace now traces Serve HTTP and gRPC requests, deployment routing, deployment method execution, and calls made with DeploymentHandle.remote(), with distributed trace context propagated across Serve deployments. See the Ray documentation for more information. #18076LLMObs.annotate now accept an optional audio_parts field, a list of audio segments each with a mime_type and either inline base64 content or an attachment_key. This lets audio (for example, speech-to-text and text-to-speech payloads) be collected and rendered in LLM Observability.sample_rate argument to LLMObs.enable() to configure the proportion of LLMObs traces to sample (between 0.0 and 1.0) in code. This takes precedence over the DD_LLMOBS_SAMPLE_RATE environment variable.SyncExperiment.rerun_evaluators() to re-run evaluators on the stored task outputs from a previous experiment run without re-executing the task function. Reads from experiment.result (set by run() or pull()) and returns a new SyncExperiment with fresh evaluations while preserving original span IDs, trace IDs, inputs, and outputs via replay span copies linked back to the originals via parent_experiment_span_id. The missing_task_strategy parameter controls behavior when prior rows contain errors: "raise" (default) raises immediately, "skip" omits the row, and "retry" re-executes the task for that row.gpt-audio, gpt-audio-mini, gpt-4o-audio-preview) as audio_parts on LLM span messages. Note: audio in streamed responses (stream=True) is not captured yet.DD_LLMOBS_SAMPLE_RATE, which controls the proportion of LLM Observability spans (between 0.0 and 1.0, defaults to 1.0) that are retained. This does not affect APM span retention or the accuracy of LLM Observability metrics such as token usage, and cost.metadata to the data exposed to the span processor registered with LLMObs.enable(span_processor=...) or LLMObs.register_processor(...). Previously only input and output were exposed; the processor can now read and modify metadata to redact or remove sensitive values that integrations record there.mistralai client chat and embedding calls.mistralai client chat and embedding calls.pytorch.rank lifetime span for PyTorch distributed training. The span opens at init_process_group and closes at destroy_process_group or process exit. Tags include rank, world_size, framework (DDP / FSDP / DeepSpeed), launcher, torch.distributed.backend, and training_job_id (resolved from launcher environment variables). When running under Ray Train, ray.train.run_name, ray.submission_id, and ray.metadata.* are also applied. Enable with DD_PATCH_MODULES=pytorch:true.DD_INJECT_EXPERIMENTAL_OVERRIDE_USER_DDTRACE=true can be added to the environment.DD_TEST_MANAGEMENT_ATTEMPT_TO_FIX_RETRIES environment variable is now respected by the ddtrace.testing internal pytest plugin. When set to a valid integer, it overrides the retry count returned by the Datadog API for "attempt to fix" tests.us3, us5, ap1, and ap2 Datadog sites. This affected customers on these sites when no Datadog Agent was running or agentless export was explicitly enabled (DD_LLMOBS_AGENTLESS_ENABLED=1).openai integration where streamed chat completion spans under-reported output_tokens and total_tokens for OpenAI-compatible providers that emit a cumulative usage object on every streamed chunk._DD_LLMOBS_WRITER_TIMEOUT, default 5 seconds) and instead used the 2 second connection default, causing intermittent TimeoutError and dropped span events on high-latency connections to the agent or intake.model_name with the request deployment name (e.g. my-deployment) instead of the canonical model returned by the API (e.g. gpt-4o-2024-08-06). The integration now prefers the response model for the azure and azure_text providers.metadata containing non-string keys (e.g. int/float/bool) could be dropped during ingestion. Metadata keys are now stringified before encoding.Omit/NotGiven sentinel values) as noise in LLM span metadata.LLMObs.annotate(tags=...) could cause spans to be dropped during ingestion.document content blocks, which could cause evaluation to be skipped for prompts whose only content was a document. Document text is now scanned and binary documents leave a placeholder so evaluation still runs.block dynamic parameter, supplied in the incoming request body, to disable the operator-configured blocking policy and let a denied request proceed in monitor mode. Request-supplied parameters can now only strengthen blocking (turning monitor mode into blocking); they can no longer weaken an enabled blocking policy.kafka.partition and kafka.message_offset span tags being incorrectly set to -1. Consumer spans no longer misreport 0 (e.g. single-partition topics or the first message in a partition) due to falsy evaluation. Producer spans and any span where the value is genuinely unknown now omit these tags entirely rather than emitting a misleading -1.model_provider as unknown for the AnthropicVertex and AnthropicBedrock clients. The provider is now resolved to google and amazon respectively.Content-Type header for the Django, Tornado, Flask, and FastAPI integrations.urllib3 was not inspected for SSRF and API Security (API10).ddtrace.contrib.trace_utils.set_user if neither ddtrace.appsec.trace_utils nor the user tracking SDK had been imported. The user blocking decision is now always enforced when AppSec is enabled.bytes commands passed to os.system, os.spawn*, and subprocess (for example os.system(b"...") or subprocess.run(b"...", shell=True)) bypassed the Exploit Prevention (RASP) shell and command injection checks. bytes commands are valid on POSIX systems and are now passed to the security checks like str commands.anonymization mode. It is now hashed, consistently with the user id and login.aws.durable.operation_attempt span tag was reported off by one for replayed durable operations, causing a resumed (replayed) execution to record a different attempt number than the original execution for the same operation. Replayed operations now report the same 0-indexed attempt number as a fresh execution.ValueError: tool 1 is already in use error that occurred on Python 3.12+ when running pytest with coverage.py or pytest-cov alongside ddtrace. Both tools previously competed for sys.monitoring.COVERAGE_ID (slot 1). ddtrace now tries tool slots in priority order (4, 3, 1), falling back gracefully if all candidate slots are occupied.ResourceWarning: unclosed socket that occurred when using the pytest plugin. The background writer thread's thread-local HTTP connection was not closed on thread exit.DD_DBM_PROPAGATION_MODE was set to full, service, or dynamic_service.coro.close() immediately on the error path. Affects Elasticsearch and OpenSearch integrations.TypeError: object NoneType can't be used in 'await' expression error. This occurs when the generator body awaits a coroutine that suspends to the event loop before its first yield. Python 3.9 and 3.10 are not affected.os.fork() could be sent twice, once by the parent and once by the child./v0.7/config payloads in multi-threaded applications (e.g. uWSGI).@tracer.wrap() to an async generator now forwards sent values, thrown exceptions, and close requests to the underlying generator, so it behaves like the unwrapped generator in all cases. Previously the wrapper only relayed values during forward iteration, so sent values were dropped and try/finally cleanup was skipped whenever the generator was closed early or received a thrown exception.TypeError: cannot pickle '_thread.lock' raised when serializing ddtrace-instrumented objects with cloudpickle. Fork-safe locks and events (ddtrace.internal.forksafe.Lock / Event) are now pickle-safe, and unpickling reconstructs a fresh instance.network.client.ip to contain the actual TCP peer IP address (i.e. REMOTE_ADDR) rather than the resolved HTTP client IP (which could originate from forwarding headers such as X-Forwarded-For). This affects the Flask, Django, ASGI (FastAPI/Starlette), and Tornado integrations. The resolved HTTP client IP continues to be reported under http.client_ip, which is the value used by App and API Protection (AAP) for IP-based threat detection and blocking._datadog message attribute is now read from the top-level SQS MessageAttributes first, and the message body is only parsed for SNS-to-SQS notifications.PutRecords now propagates trace context to each record in the batch. This enables trace context propagation to correctly map to each individual record, as well as enabling the correct tracking of messages through Data Streams Monitoring. Previously, only the first record would ever have been tracked.The request signature we calculated does not match the signature you provided when distributed tracing was enabled.DD_TRACE_REMOVE_INTEGRATION_SERVICE_NAMES_ENABLED, causing spans to always use celery-worker/celery-producer instead of the configured global service name.process_exception, were not preserved, causing Django to skip exception handlers under ASGI.grpc.aio client spans intermittently reporting error.message as "Internal error from Core" instead of the application-set abort details when a server aborts the RPC. #18015re.finditer eagerly consumed the entire match iterator to taint every match, which removed the laziness of re.finditer and allowed a request-controlled input with many matches to allocate excess memory. Matches are now tainted lazily as the application iterates over them.AttributeError when metadata=None is passed to traced litellm.completion() calls. #18033trace_engine() is called repeatedly for the same engine.TypeError when Google GenAI returns None for optional response fields (token counts or candidates) introduced in google-genai>=2.6.0.out.host and server.address tags on spans produced by redis.cluster.RedisCluster and redis.asyncio.cluster.RedisCluster (redis-py >=4.1). These clients do not expose a connection_pool attribute, so connection metadata was never extracted and the Datadog inferred-entity feature fell back to the generic peer.db.system: redis identifier instead of resolving to the specific cluster hostname. Spans now include out.host and server.address derived from the cluster's startup node, enabling the service map to display a host-specific Redis entity.acknowledged before the product received it. The state is now set to acknowledged only after the configuration has been delivered to the product, and to error only when the payload is malformed and cannot be deserialized.CORSMiddleware handled an OPTIONS request, the span resource was set to the raw URL path (e.g. OPTIONS /users/123) instead of the route template (e.g. OPTIONS /users/{user_id}). The route pattern is now resolved via the app's route tree when CORSMiddleware short-circuits before the Starlette router runs. Fixes #14321.CI visibility: Fixes an issue where test sessions running with pytest-xdist on shallow Git checkouts can spend excessive time unshallowing the reposit
code origin for spans: prevent view or traced functions from raising an exception in situations where their number is large.
profiling: Fixes potential process memory growth when a stack profiling sampling cycle exits before completing a sample.
openfeature: This fix resolves an issue where the feature flagging provider failed to load with openfeature-sdk 0.10.0 or later.
openfeature-sdk 0.10.0 or later.<!-- -->
<!-- -->
@tracer.wrap on functions created at run time has been fixed.Fix crashes in uwsgi worker when exiting from SIGTERM.
Fix a potential memory leak triggered by having code origin for span explicitly disabled via environment variables.
<!-- -->
This fix resolves an issue where the Datadog UI showed No extra information available when viewing details about why a test was skipped by Intelligent
CI Visibility:
No extra information available when viewing details about why a test was skipped by Intelligent Test Runner, due to missing ITR correlation ID on test or suite events.Code Security (IAST):
copy_ranges_from_strings and copy_and_shift_ranges_from_strings to the active request slot, matching the scoped taint read path. Previously these copy helpers resolved the taint map by scanning all request slots, so a concurrent or still-open request could capture the derived taint and the current request would miss the transformed tainted input.MCP:
code origin:
IAST:
uvloop has been fixed.LLM Observability:
bootstrap: keep yaml/_yaml loaded during module cleanup, fixing an issue that broke PyYAML consumers such as Airflow.
yaml/_yaml loaded during module cleanup, fixing an issue that broke PyYAML consumers such as Airflow.<!-- -->
% string formatting handled tainted text containing literal IAST evidence marker delimiters.<!-- -->
<!-- -->
anomaly.query_count) prevented the span from being ingested. Dots in metric keys are now replaced with underscores.<!-- -->
IAST: This fix resolves an issue where IAST could report a false positive vulnerability against a request whose input did not actually contain tainted…
SCRIPT_NAME now expose the client-hit resource on a new flask.resource.full tag (e.g. GET /api/v2/users). The span resource and flask.url_rule tag are unchanged. The tag is only set when its value would differ from span.resource.<!-- -->
HEAD for any GET route and Flask's auto-handled OPTIONS for every route — not just the methods listed in methods=[...].<!-- -->
v0.4 trace API version (v0.5 cannot carry the LLMObs span data). Setting DD_TRACE_API_VERSION=v0.5 with LLMObs enabled logs a warning and downgrades to v0.4. No user action is required.x-datadog-endpoint-scan and x-datadog-security-test HTTP request headers on service entry spans unconditionally as http.request.headers.x-datadog-endpoint-scan and http.request.headers.x-datadog-security-test tags. These markers identify Datadog-originated endpoint scans and security tests so the API inventory pipeline can distinguish scan/test traffic from real user traffic. The headers are tagged regardless of DD_TRACE_HEADER_TAGS configuration or AppSec enablement, and are not propagated to downstream services.<!-- -->
dynamic_service as a new DD_DBM_PROPAGATION_MODE value. Set DD_DBM_PROPAGATION_MODE=dynamic_service to inject DBM service metadata and the SQL base hash without injecting trace context.<!-- -->
Messages.create / Messages.stream and their async and Beta variants), and evaluation is automatically skipped when a framework integration (LangChain) is already evaluating the same call.<!-- -->
kafka.partition and kafka.message_offset tags to the producer span once the broker acknowledges the send. The partition reflects the partition the broker actually assigned to the message, which may differ from the partition the caller requested. Mirrors the behavior added to the Java tracer in DataDog/dd-trace-java#11107.<!-- -->
_dd.appsec.normalized_route span tag for FastAPI and Starlette request spans when API Security is enabled. The tag follows RFC-1103 and provides a per-request, framework-agnostic representation of the matched route — converter types are stripped, multi-parameter URL segments are combined with +, path catch-all parameters are emitted as a single tail element, and trailing slashes are preserved as declared. Mount-prefixed sub-application routes are reported with their full assembled path.<!-- -->
_dd.appsec.normalized_route span tag support to Flask request spans when API Security is enabled. Flask / Werkzeug <converter:name> route syntax is normalized following RFC-1103: converter types are stripped, multi-parameter URL segments (e.g. /<first>.<last>/) are combined with +, <path:name> catch-all parameters are emitted as a single tail element, and trailing slashes are preserved as declared. Routes served through DispatcherMiddleware sub-apps are reported with their full assembled path (mount prefix included).<!-- -->
claude_agent_sdk integration now emits span links between step, LLM, and tool spans so multi-step traces render the sequencing between LLM calls and tool calls.<!-- -->
aws_durable_execution_sdk_python for details and opt-out configuration.<!-- -->
werkzeug.middleware.dispatcher.DispatcherMiddleware. Sub-app routes are reported with their full mounted path.<!-- -->
DD_DATA_STREAMS_ENABLED=true.<!-- -->
git.commit.sha and git.repository_url. Values come from DD_GIT_COMMIT_SHA / DD_GIT_REPOSITORY_URL or the main package's Project-URL metadata, falling back to running git against the current working directory. Honors DD_TRACE_GIT_METADATA_ENABLED and user-supplied tags with the same keys.<!-- -->
<!-- -->
MultiEvaluatorResult to emit multiple named evaluation metrics from a single evaluator call. Each sub-value may itself be an EvaluatorResult carrying its own reasoning, assessment, metadata, and tags. By default emitted metric labels are prefixed with the evaluator's name ("<evaluator_name>-<key>"); pass prefix=False to emit raw keys.<!-- -->
LLMObs.pull_experiment(experiment_id) to fetch a previously-run experiment from the Datadog backend by UUID. The returned SyncExperiment has its .result populated and is ready for downstream inspection (e.g. .as_dataframe()) without re-executing the original task.<!-- -->
task and dataset are now optional when creating an experiment via LLMObs.experiment() / LLMObs.async_experiment(), supporting pull-based workflows. Calling run() without providing task and dataset raises a ValueError.<!-- -->
DD_LLMOBS_SAMPLE_RATE, which controls the proportion of LLM Observability spans (between 0.0 and 1.0, defaults to 1.0) that are retained. This does not affect APM span retention or the accuracy of LLM Observability metrics such as token usage, and cost.<!-- -->
version field is set on a parent LLM span via tool_definitions, the resolved tool version is now also written onto manually-started child tool spans (created via LLMObs.tool()) as meta.tool.version. Previously, this propagation only occurred for tool spans produced by auto-instrumented integrations (OpenAI, LangChain, OpenAI Agents, ReAct) through the LinkTracker dispatch mechanism.<!-- -->
version field is set on a parent LLM span via tool_definitions, the resolved tool version is now also written onto the corresponding child tool span as meta.tool.version. This allows the LLM Observability UI to aggregate tool version counts and filter tool spans by version without correlating to the parent LLM span.<!-- -->
<!-- -->
PYMEM_DOMAIN_MEM (PyMem_Malloc/Calloc/Realloc) in addition to the existing PYMEM_DOMAIN_OBJ coverage when DD_PROFILING_MEMORY_MEM_DOMAIN_ENABLED=true is set. This support is effective on Python 3.12 and newer, making some previously invisible allocations visible in heap profiles in those environments.<!-- -->
DD_ML_JOB_ENV — a single opt-in env var for forwarding configuration into Ray job workers. Set it on the dashboard process as a semicolon-separated list of KEY:VALUE pairs, e.g. DD_SERVICE:my-svc;DD_AGENT_HOST:10.0.0.1, and workers receive them verbatim as DD_SERVICE=my-svc, DD_AGENT_HOST=10.0.0.1, etc.<!-- -->
DD_INJECT_EXPERIMENTAL_OVERRIDE_USER_DDTRACE=true can be added to the environment.<!-- -->
CI Visibility: Adds ddtrace.testing.logs.DDTestLogsHandler, a public logging.Handler for shipping log records to the Datadog logs intake correlated with CI Visibility test traces, intended for use in subprocesses that execute tests outside of the main pytest process. Also adds ddtrace.testing.logs.CorrelationFilter, a base class for stamping dd.trace_id / dd.span_id onto log records, and a ready-made ddtrace.testing.logs.ThreadLocalCorrelationFilter for the common thread-local case. Subclass CorrelationFilter to plug in other concurrency models (asyncio ContextVar, global state, etc.).
Example:
from pydantic_evals.evaluators import EqualsExpected import logging
from ddtrace.testing.logs import DDTestLogsHandler from ddtrace.testing.logs import ThreadLocalCorrelationFilter
handler = DDTestLogsHandler(service="my-service") correlation = ThreadLocalCorrelationFilter() handler.addFilter(correlation) logging.getLogger().addHandler(handler)
while True:
job = queue.get() correlation.set_context(trace_id=job.trace_id, span_id=job.span_id) run_test(job.item)handler.close()
websocket.connect ASGI message, causing ASGI/FastAPI WebSocket connections to fail with HTTP 500 when AppSec was enabled.<!-- -->
us3, us5, ap1, and ap2 Datadog sites. This affected customers on these sites when no Datadog Agent was running or agentless export was explicitly enabled (DD_LLMOBS_AGENTLESS_ENABLED=1).<!-- -->
confluent_kafka integration.<!-- -->
waf.duration, waf.duration_ext, rasp.duration, and rasp.duration_ext.<!-- -->
<!-- -->
cosmosdb.query span resource names included per-item identifiers (such as document, user, and permission ids) from the request URI, which caused unbounded resource cardinality. The integration now redacts those ids by replacing them with ? while keeping the database (dbs) and collection (colls) names intact. For example, Read /dbs/myDb/colls/myColl/docs/item1 is now reported as Read /dbs/myDb/colls/myColl/docs/?.<!-- -->
DD_CIVISIBILITY_BACKEND_API_TIMEOUT_MILLIS environment variable (previously missing) to override it. The value is expressed in milliseconds (e.g. 60000 for 60 seconds), consistent with the Java tracer. The same timeout now applies uniformly to all backend requests, including skippable test fetches.<!-- -->
DD_DBM_PROPAGATION_MODE was set to full, service, or dynamic_service.<!-- -->
wrapt==2.2.0 from the supported dependency range to avoid a regression that breaks wrapped C descriptors.<!-- -->
DD_KOMBU_SERVICE, DD_JINJA2_SERVICE, DD_DJANGO_DATABASE_SERVICE, DD_CELERY_PRODUCER_SERVICE, DD_CELERY_WORKER_SERVICE, and DD_DJANGO_CACHE_SERVICE now override the service tag on their respective spans. Previously they were ignored in favor of the DD_<INTEGRATION>_SERVICE_NAME form.<!-- -->
config.kombu["service"] before patching was applied to producer spans but not consumer spans. Both now read from the same service config source.<!-- -->
None value for req.uri_template caused a TypeError when constructing the route in the Falcon tracing middleware.<!-- -->
Content-Type header carried a parameter (most commonly charset=utf-8).<!-- -->
TypeError: object NoneType can't be used in 'await' expression error. This occurs when the generator body awaits a coroutine that suspends to the event loop before its first yield. Python 3.9 and 3.10 are not affected.<!-- -->
_datadog message attribute is now read from the top-level SQS MessageAttributes first, and the message body is only parsed for SNS-to-SQS notifications.<!-- -->
<!-- -->
DD_TRACE_REMOVE_INTEGRATION_SERVICE_NAMES_ENABLED, causing spans to always use celery-worker/celery-producer instead of the configured global service name.<!-- -->
<!-- -->
unknown error category from the upstream Claude Agent SDK instead of a descriptive API error. The integration now surfaces the detailed error message from the assistant message content.<!-- -->
ClaudeSDKClient.query() calls produced nested agent spans instead of separate root traces. Agent spans are now finalized eagerly when a ResultMessage is observed, rather than relying on the wrapping async generator's finally block to execute.<!-- -->
*args / **kwargs for any function that declares keyword-only parameters alongside variadic arguments.<!-- -->
process_exception, were not preserved, causing Django to skip exception handlers under ASGI.<!-- -->
os.fork() could be sent twice, once by the parent and once by the child.<!-- -->
/v0.7/config payloads in multi-threaded applications (e.g. uWSGI).<!-- -->
<!-- -->
<!-- -->
langchain_aws.ChatBedrockConverse spans reported an opaque inference-profile ARN identifier as the model name when an inference profile was used. base_model_id which represents the underlying foundation model is now checked first when extracting model names, and the botocore Bedrock integration reads the resolved base model from a shared in-process cache populated by langchain so the same resolution applies to the underlying bedrock-runtime span.<!-- -->
openai integration where streamed chat completion spans under-reported output_tokens and total_tokens for OpenAI-compatible providers that emit a cumulative usage object on every streamed chunk.<!-- -->
reasoning_content was missing from streamed chat completions in the OpenAI and LiteLLM integrations when an OpenAI-compatible reasoning provider (e.g. DeepSeek, Qwen) emitted delta.reasoning_content chunks. The aggregated message now captures reasoning text in the output message, matching non-streaming behavior.<!-- -->
_DD_LLMOBS_WRITER_TIMEOUT, default 5 seconds) and instead used the 2 second connection default, causing intermittent TimeoutError and dropped span events on high-latency connections to the agent or intake.<!-- -->
gevent monkey-patching threading lock primitives in place after the profiler had already patched them.<!-- -->
Content-Type header carried parameters being dropped.<!-- -->
<!-- -->
awake() on a PeriodicThread after it had been stopped would block forever. awake() now returns immediately in that case.<!-- -->
PeriodicThread callbacks were invisible to Python's cyclic garbage collector and could accumulate when threads used bound methods as targets.<!-- -->
<!-- -->
pydantic_ai.tool / pydantic_ai.agent) and the resource name is the specific tool or agent name, matching Datadog APM convention. This restores per-tool and per-agent grouping on APM service and resource pages. LLM Observability views are unaffected.<!-- -->
pod_name) on hosts using cgroup v2.<!-- -->
<!-- -->
= in the value. tracing: The subprocess integration now correctly scrubs env-var values for env-var names containing digits.<!-- -->
<!-- -->
@tracer.wrap() to an async generator now forwards sent values, thrown exceptions, and close requests to the underlying generator, so it behaves like the unwrapped generator in all cases. Previously the wrapper only relayed values during forward iteration, so sent values were dropped and try/finally cleanup was skipped whenever the generator was closed early or received a thrown exception.<!-- -->
<!-- -->
git warnings when running tests with pytest-xdist. Simultaneous git fetch --update-shallow calls now retry with exponential back-off on .git/shallow.lock contention, skip the fetch entirely once a sibling worker has already unshallowed the repository, and run under the C locale so the contention check is robust against non-English git messages. git merge-base is deferred until after unshallowing so the required commits are available locally.<!-- -->
<!-- -->
<!-- -->
<!-- -->
<!-- -->
<!-- -->
metadata containing non-string keys (e.g. int/float/bool) could be dropped during ingestion. Metadata keys are now stringified before encoding.<!-- -->
Omit/NotGiven sentinel values) as noise in LLM span metadata.<!-- -->
_current_trace_context did not propagate _dd.p.llmobs_parent_id into the trace context metadata. Without this value, downstream consumers (e.g. dd-trace-go) could not locate the active LLMObs parent span and would create orphaned child traces instead of correctly continuing the parent trace.<!-- -->
LLMObsSpan now exposes input_documents and output_documents for mutation or removal before export.<!-- -->
openai spans when an OpenAI (or AsyncOpenAI) client and an AzureOpenAI (or AsyncAzureOpenAI) client are instantiated at the same time. Provider is now determined per-call rather than from the most recently constructed client.<!-- -->
<!-- -->
<!-- -->
<!-- -->
<!-- -->
SIGINT and SIGTERM signals.<!-- -->
<!-- -->
<!-- -->
ddtrace-run caused a traceback on keyboard interrupt.<!-- -->
confluent-kafka integration where internal list_topics calls triggered librdkafka background metadata-refresh tasks.OTEL_METRIC_EXPORT_TIMEOUT and OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE to the allowlist of recognized OpenTelemetry environment variables. Previously, setting any of these variables produced an inaccurate OpenTelemetry configuration ... is not supported by Datadog warning at startup, even though the variables are valid OpenTelemetry SDK settings.<!-- -->
DD_LLMOBS_AGENTLESS_ENABLED=1), APM traces are now exported agentlessly to Datadog's intake. This should not change user-facing behavior: both APM and LLMObs spans remain visible in the UI; LLMObs spans are simply no longer shipped separately for agentless users. Note that setting DD_APM_TRACING_ENABLED=false takes higher precedence and will result in LLMObs span events shipping separately as existing behavior.<!-- -->
info.profiler.settings on the upload event, so profiles can be filtered by individual settings.<!-- -->
socket.gethostname(), ray.get_runtime_context().get_*()) are amortized across the thread lifetime, reducing overhead for high-frequency Ray task and actor-method instrumentation.AAP: Fix an issue which could make the Threat Protection panel wrongly report that a library update was required.
<!-- -->
DD_DBM_PROPAGATION_MODE was set to full, service, or dynamic_service.<!-- -->
_datadog message attribute is now read from the top-level SQS MessageAttributes first, and the message body is only parsed for SNS-to-SQS notifications.<!-- -->
process_exception, were not preserved, causing Django to skip exception handlers under ASGI.<!-- -->
<!-- -->
Nothing published for this version
tracing: Resolves an issue where wrapping an async generator on Python 3.11 through 3.14 raises a TypeError: object NoneType can't be used in 'await'
tracing: Resolves an issue where wrapping an async generator on Python 3.11 through 3.14 raises a TypeError: object NoneType can't be used in 'await' expression error. This occurs when the generator body awaits a coroutine that suspends to the event loop before its first yield. Python 3.9 and 3.10 are not affected.
LLM Observability: Fixes an issue where the span writer ignored its configured request timeout (_DD_LLMOBS_WRITER_TIMEOUT, default 5 seconds) and instead used the 2 second connection default, causing intermittent TimeoutError and dropped span events on high-latency connections to the agent or intake.
celery: Fixes Celery worker and producer spans ignoring DD_TRACE_REMOVE_INTEGRATION_SERVICE_NAMES_ENABLED, causing spans to always use celery-worker/c
celery: Fixes Celery worker and producer spans ignoring DD_TRACE_REMOVE_INTEGRATION_SERVICE_NAMES_ENABLED, causing spans to always use celery-worker/celery-producer instead of the configured global service name.
tracing: This fix resolves an issue where traces buffered before os.fork() could be sent twice, once by the parent and once by the child.
tracing: Applying @tracer.wrap() to an async generator now forwards sent values, thrown exceptions, and close requests to the underlying generator, so it behaves like the unwrapped generator in all cases. Previously the wrapper only relayed values during forward iteration, so sent values were dropped and try/finally cleanup was skipped whenever the generator was closed early or received a thrown exception.
tracing: A rare crash happening on versions of CPython prior to 3.12 has been fixed.
LLM Observability: Resolves an issue where spans annotated with metadata containing non-string keys (e.g. int/float/bool) could be dropped during ingestion. Metadata keys are now stringified before encoding.
LLM Observability: Resolves an issue where the OpenAI integration recorded unset request parameters (OpenAI SDK's Omit/NotGiven sentinel values) as noise in LLM span metadata.
profiling: Rare crashes that could happen post-fork in fork-based applications have been fixed.
LLM Observability: Adds support for DD_LLMOBS_SAMPLE_RATE, which controls the proportion of LLM Observability spans (between 0.0 and 1.0, defaults to
DD_LLMOBS_SAMPLE_RATE, which controls the proportion of LLM Observability spans (between 0.0 and 1.0, defaults to 1.0) that are retained. This does not affect APM span retention or the accuracy of LLM Observability metrics such as token usage, and cost.LLM Observability: when APM and LLMObs are both enabled, the APM trace writer now uses the v0.4 trace API version (v0.5 cannot carry the LLMObs span d
v0.4 trace API version (v0.5 cannot carry the LLMObs span data). Setting DD_TRACE_API_VERSION=v0.5 with LLMObs enabled logs a warning and downgrades to v0.4. No user action is required.<!-- -->
DD_INJECT_EXPERIMENTAL_OVERRIDE_USER_DDTRACE=true can be added to the environment.us3, us5, ap1, and ap2 Datadog sites. This affected customers on these sites when no Datadog Agent was running or agentless export was explicitly enabled (DD_LLMOBS_AGENTLESS_ENABLED=1).<!-- -->
/v0.7/config payloads in multi-threaded applications (e.g. uWSGI).<!-- -->
<!-- -->
<!-- -->
<!-- -->
IAST: This fix resolves an issue where IAST could report a false positive vulnerability against a request whose input did not actually contain tainted…
SCRIPT_NAME now expose the client-hit resource on a new flask.resource.full tag (e.g. GET /api/v2/users). The span resource and flask.url_rule tag are unchanged. The tag is only set when its value would differ from span.resource.<!-- -->
HEAD for any GET route and Flask's auto-handled OPTIONS for every route — not just the methods listed in methods=[...].x-datadog-endpoint-scan and x-datadog-security-test HTTP request headers on service entry spans unconditionally as http.request.headers.x-datadog-endpoint-scan and http.request.headers.x-datadog-security-test tags. These markers identify Datadog-originated endpoint scans and security tests so the API inventory pipeline can distinguish scan/test traffic from real user traffic. The headers are tagged regardless of DD_TRACE_HEADER_TAGS configuration or AppSec enablement, and are not propagated to downstream services.<!-- -->
dynamic_service as a new DD_DBM_PROPAGATION_MODE value. Set DD_DBM_PROPAGATION_MODE=dynamic_service to inject DBM service metadata and the SQL base hash without injecting trace context.<!-- -->
Messages.create / Messages.stream and their async and Beta variants), and evaluation is automatically skipped when a framework integration (LangChain) is already evaluating the same call.<!-- -->
kafka.partition and kafka.message_offset tags to the producer span once the broker acknowledges the send. The partition reflects the partition the broker actually assigned to the message, which may differ from the partition the caller requested. Mirrors the behavior added to the Java tracer in DataDog/dd-trace-java#11107.<!-- -->
_dd.appsec.normalized_route span tag for FastAPI and Starlette request spans when API Security is enabled. The tag follows RFC-1103 and provides a per-request, framework-agnostic representation of the matched route — converter types are stripped, multi-parameter URL segments are combined with +, path catch-all parameters are emitted as a single tail element, and trailing slashes are preserved as declared. Mount-prefixed sub-application routes are reported with their full assembled path.<!-- -->
_dd.appsec.normalized_route span tag support to Flask request spans when API Security is enabled. Flask / Werkzeug <converter:name> route syntax is normalized following RFC-1103: converter types are stripped, multi-parameter URL segments (e.g. /<first>.<last>/) are combined with +, <path:name> catch-all parameters are emitted as a single tail element, and trailing slashes are preserved as declared. Routes served through DispatcherMiddleware sub-apps are reported with their full assembled path (mount prefix included).<!-- -->
claude_agent_sdk integration now emits span links between step, LLM, and tool spans so multi-step traces render the sequencing between LLM calls and tool calls.<!-- -->
aws_durable_execution_sdk_python for details and opt-out configuration.<!-- -->
werkzeug.middleware.dispatcher.DispatcherMiddleware. Sub-app routes are reported with their full mounted path.<!-- -->
DD_DATA_STREAMS_ENABLED=true.<!-- -->
git.commit.sha and git.repository_url. Values come from DD_GIT_COMMIT_SHA / DD_GIT_REPOSITORY_URL or the main package's Project-URL metadata, falling back to running git against the current working directory. Honors DD_TRACE_GIT_METADATA_ENABLED and user-supplied tags with the same keys.<!-- -->
MultiEvaluatorResult to emit multiple named evaluation metrics from a single evaluator call. Each sub-value may itself be an EvaluatorResult carrying its own reasoning, assessment, metadata, and tags. By default emitted metric labels are prefixed with the evaluator's name ("<evaluator_name>-<key>"); pass prefix=False to emit raw keys.<!-- -->
LLMObs.pull_experiment(experiment_id) to fetch a previously-run experiment from the Datadog backend by UUID. The returned SyncExperiment has its .result populated and is ready for downstream inspection (e.g. .as_dataframe()) without re-executing the original task.<!-- -->
task and dataset are now optional when creating an experiment via LLMObs.experiment() / LLMObs.async_experiment(), supporting pull-based workflows. Calling run() without providing task and dataset raises a ValueError.<!-- -->
version field is set on a parent LLM span via tool_definitions, the resolved tool version is now also written onto manually-started child tool spans (created via LLMObs.tool()) as meta.tool.version. Previously, this propagation only occurred for tool spans produced by auto-instrumented integrations (OpenAI, LangChain, OpenAI Agents, ReAct) through the LinkTracker dispatch mechanism.<!-- -->
version field is set on a parent LLM span via tool_definitions, the resolved tool version is now also written onto the corresponding child tool span as meta.tool.version. This allows the LLM Observability UI to aggregate tool version counts and filter tool spans by version without correlating to the parent LLM span.<!-- -->
<!-- -->
PYMEM_DOMAIN_MEM (PyMem_Malloc/Calloc/Realloc) in addition to the existing PYMEM_DOMAIN_OBJ coverage when DD_PROFILING_MEMORY_MEM_DOMAIN_ENABLED=true is set. This support is effective on Python 3.12 and newer, making some previously invisible allocations visible in heap profiles in those environments.<!-- -->
DD_ML_JOB_ENV — a single opt-in env var for forwarding configuration into Ray job workers. Set it on the dashboard process as a semicolon-separated list of KEY:VALUE pairs, e.g. DD_SERVICE:my-svc;DD_AGENT_HOST:10.0.0.1, and workers receive them verbatim as DD_SERVICE=my-svc, DD_AGENT_HOST=10.0.0.1, etc.<!-- -->
CI Visibility: Adds ddtrace.testing.logs.DDTestLogsHandler, a public logging.Handler for shipping log records to the Datadog logs intake correlated with CI Visibility test traces, intended for use in subprocesses that execute tests outside of the main pytest process. Also adds ddtrace.testing.logs.CorrelationFilter, a base class for stamping dd.trace_id / dd.span_id onto log records, and a ready-made ddtrace.testing.logs.ThreadLocalCorrelationFilter for the common thread-local case. Subclass CorrelationFilter to plug in other concurrency models (asyncio ContextVar, global state, etc.).
Example:
from pydantic_evals.evaluators import EqualsExpected import logging
from ddtrace.testing.logs import DDTestLogsHandler from ddtrace.testing.logs import ThreadLocalCorrelationFilter
handler = DDTestLogsHandler(service="my-service") correlation = ThreadLocalCorrelationFilter() handler.addFilter(correlation) logging.getLogger().addHandler(handler)
while True:
job = queue.get() correlation.set_context(trace_id=job.trace_id, span_id=job.span_id) run_test(job.item)handler.close()
websocket.connect ASGI message, causing ASGI/FastAPI WebSocket connections to fail with HTTP 500 when AppSec was enabled.<!-- -->
confluent_kafka integration.<!-- -->
waf.duration, waf.duration_ext, rasp.duration, and rasp.duration_ext.<!-- -->
cosmosdb.query span resource names included per-item identifiers (such as document, user, and permission ids) from the request URI, which caused unbounded resource cardinality. The integration now redacts those ids by replacing them with ? while keeping the database (dbs) and collection (colls) names intact. For example, Read /dbs/myDb/colls/myColl/docs/item1 is now reported as Read /dbs/myDb/colls/myColl/docs/?.<!-- -->
DD_CIVISIBILITY_BACKEND_API_TIMEOUT_MILLIS environment variable (previously missing) to override it. The value is expressed in milliseconds (e.g. 60000 for 60 seconds), consistent with the Java tracer. The same timeout now applies uniformly to all backend requests, including skippable test fetches.<!-- -->
wrapt==2.2.0 from the supported dependency range to avoid a regression that breaks wrapped C descriptors.<!-- -->
DD_KOMBU_SERVICE, DD_JINJA2_SERVICE, DD_DJANGO_DATABASE_SERVICE, DD_CELERY_PRODUCER_SERVICE, DD_CELERY_WORKER_SERVICE, and DD_DJANGO_CACHE_SERVICE now override the service tag on their respective spans. Previously they were ignored in favor of the DD_<INTEGRATION>_SERVICE_NAME form.<!-- -->
config.kombu["service"] before patching was applied to producer spans but not consumer spans. Both now read from the same service config source.<!-- -->
None value for req.uri_template caused a TypeError when constructing the route in the Falcon tracing middleware.<!-- -->
Content-Type header carried a parameter (most commonly charset=utf-8).<!-- -->
<!-- -->
<!-- -->
unknown error category from the upstream Claude Agent SDK instead of a descriptive API error. The integration now surfaces the detailed error message from the assistant message content.<!-- -->
ClaudeSDKClient.query() calls produced nested agent spans instead of separate root traces. Agent spans are now finalized eagerly when a ResultMessage is observed, rather than relying on the wrapping async generator's finally block to execute.<!-- -->
*args / **kwargs for any function that declares keyword-only parameters alongside variadic arguments.<!-- -->
<!-- -->
langchain_aws.ChatBedrockConverse spans reported an opaque inference-profile ARN identifier as the model name when an inference profile was used. base_model_id which represents the underlying foundation model is now checked first when extracting model names, and the botocore Bedrock integration reads the resolved base model from a shared in-process cache populated by langchain so the same resolution applies to the underlying bedrock-runtime span.<!-- -->
reasoning_content was missing from streamed chat completions in the OpenAI and LiteLLM integrations when an OpenAI-compatible reasoning provider (e.g. DeepSeek, Qwen) emitted delta.reasoning_content chunks. The aggregated message now captures reasoning text in the output message, matching non-streaming behavior.<!-- -->
gevent monkey-patching threading lock primitives in place after the profiler had already patched them.<!-- -->
Content-Type header carried parameters being dropped.<!-- -->
<!-- -->
awake() on a PeriodicThread after it had been stopped would block forever. awake() now returns immediately in that case.<!-- -->
PeriodicThread callbacks were invisible to Python's cyclic garbage collector and could accumulate when threads used bound methods as targets.<!-- -->
<!-- -->
pydantic_ai.tool / pydantic_ai.agent) and the resource name is the specific tool or agent name, matching Datadog APM convention. This restores per-tool and per-agent grouping on APM service and resource pages. LLM Observability views are unaffected.<!-- -->
<!-- -->
= in the value. tracing: The subprocess integration now correctly scrubs env-var values for env-var names containing digits.<!-- -->
<!-- -->
git warnings when running tests with pytest-xdist. Simultaneous git fetch --update-shallow calls now retry with exponential back-off on .git/shallow.lock contention, skip the fetch entirely once a sibling worker has already unshallowed the repository, and run under the C locale so the contention check is robust against non-English git messages. git merge-base is deferred until after unshallowing so the required commits are available locally.<!-- -->
<!-- -->
_current_trace_context did not propagate _dd.p.llmobs_parent_id into the trace context metadata. Without this value, downstream consumers (e.g. dd-trace-go) could not locate the active LLMObs parent span and would create orphaned child traces instead of correctly continuing the parent trace.<!-- -->
LLMObsSpan now exposes input_documents and output_documents for mutation or removal before export.<!-- -->
openai spans when an OpenAI (or AsyncOpenAI) client and an AzureOpenAI (or AsyncAzureOpenAI) client are instantiated at the same time. Provider is now determined per-call rather than from the most recently constructed client.<!-- -->
<!-- -->
<!-- -->
<!-- -->
<!-- -->
SIGINT and SIGTERM signals.<!-- -->
ddtrace-run caused a traceback on keyboard interrupt.<!-- -->
confluent-kafka integration where internal list_topics calls triggered librdkafka background metadata-refresh tasks.OTEL_METRIC_EXPORT_TIMEOUT and OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE to the allowlist of recognized OpenTelemetry environment variables. Previously, setting any of these variables produced an inaccurate OpenTelemetry configuration ... is not supported by Datadog warning at startup, even though the variables are valid OpenTelemetry SDK settings.<!-- -->
DD_LLMOBS_AGENTLESS_ENABLED=1), APM traces are now exported agentlessly to Datadog's intake. This should not change user-facing behavior: both APM and LLMObs spans remain visible in the UI; LLMObs spans are simply no longer shipped separately for agentless users. Note that setting DD_APM_TRACING_ENABLED=false takes higher precedence and will result in LLMObs span events shipping separately as existing behavior.<!-- -->
info.profiler.settings on the upload event, so profiles can be filtered by individual settings.<!-- -->
socket.gethostname(), ray.get_runtime_context().get_*()) are amortized across the thread lifetime, reducing overhead for high-frequency Ray task and actor-method instrumentation.internal: Fixes an issue on Python 3.9 through 3.12.3 where applications can crash when background worker threads clean up thread-local or context-loc
Fix crashes in uwsgi worker when exiting from SIGTERM.
This fix resolves an issue where the Datadog UI showed No extra information available when viewing details about why a test was skipped by Intelligent
CI Visibility:
No extra information available when viewing details about why a test was skipped by Intelligent Test Runner, due to missing ITR correlation ID on test or suite events.Code Security (IAST):
copy_ranges_from_strings and copy_and_shift_ranges_from_strings to the active request slot, matching the scoped taint read path. Previously these copy helpers resolved the taint map by scanning all request slots, so a concurrent or still-open request could capture the derived taint and the current request would miss the transformed tainted input.MCP:
IAST:
uvloop has been fixed.IAST: This fix resolves an issue where IAST could report a false positive vulnerability against a request whose input did not actually contain tainted…
yaml/_yaml loaded during module cleanup, fixing an issue that broke PyYAML consumers such as Airflow.<!-- -->
<!-- -->
% string formatting handled tainted text containing literal IAST evidence marker delimiters.<!-- -->
<!-- -->
SIGSEGV/SIGBUS handler that the profiler cannot wrap (e.g. CUDA, PyTorch, etc.). The sampler upgrades to the faster fault-recovery copy if it still owns both fault handlers afterwards. Otherwise, it permanently falls back to the syscall-based copy.AAP: Fix an issue which could make the Threat Protection panel wrongly report that a library update was required.
<!-- -->
<!-- -->
google_adk integration dropped an agent span (logging missing span kind in span context) and orphaned its child spans when a Gemini response part could not be parsed. The span kind is now set before input/output extraction, so a parsing failure degrades to empty input/output instead of dropping the span.<!-- -->
google_adk integrations rendered unhandled Gemini response parts (such as inline_data, file_data, empty parts, or thought-signature parts) as a confusing Unsupported file type placeholder. These parts now produce a concise summary or empty content.<!-- -->
database monitoring: Resolved a memory leak that grew with query volume when DD_DBM_PROPAGATION_MODE was set to full, service, or dynamic_service.
DD_DBM_PROPAGATION_MODE was set to full, service, or dynamic_service.<!-- -->
_datadog message attribute is now read from the top-level SQS MessageAttributes first, and the message body is only parsed for SNS-to-SQS notifications.<!-- -->
process_exception, were not preserved, causing Django to skip exception handlers under ASGI.tracing: Resolves an issue where wrapping an async generator on Python 3.11 through 3.14 raises a TypeError: object NoneType can't be used in 'await'
TypeError: object NoneType can't be used in 'await' expression error. This occurs when the generator body awaits a coroutine that suspends to the event loop before its first yield. Python 3.9 and 3.10 are not affected.<!-- -->
openai integration where streamed chat completion spans under-reported output_tokens and total_tokens for OpenAI-compatible providers that emit a cumulative usage object on every streamed chunk.<!-- -->
@tracer.wrap() to an async generator now forwards sent values, thrown exceptions, and close requests to the underlying generator, so it behaves like the unwrapped generator in all cases. Previously the wrapper only relayed values during forward iteration, so sent values were dropped and try/finally cleanup was skipped whenever the generator was closed early or received a thrown exception.<!-- -->
<!-- -->
tracing: This fix resolves an issue where traces buffered before os.fork() could be sent twice, once by the parent and once by the child.
tracing: This fix resolves an issue where traces buffered before os.fork() could be sent twice, once by the parent and once by the child.
LLM Observability: Resolves an issue where the OpenAI integration recorded unset request parameters (OpenAI SDK's Omit/NotGiven sentinel values) as noise in LLM span metadata.
profiling: Rare crashes that could happen post-fork in fork-based applications have been fixed.
LLM Observability: Fixes agentless export dropping data on the us3, us5, ap1, and ap2 Datadog sites. This affected customers on these sites when no Da
LLM Observability: Fixes agentless export dropping data on the us3, us5, ap1, and ap2 Datadog sites. This affected customers on these sites when no Datadog Agent was running or agentless export was explicitly enabled (DD_LLMOBS_AGENTLESS_ENABLED=1).
runtime metrics: Fixes an issue where runtime metrics were missing container and orchestrator tags (such as pod_name) on hosts using cgroup v2.
IAST: A crash that could happen at interpreter teardown has been fixed.
profiling: A rare crash that could happen after fork in fork-based applications has been fixed.
tracing: Fixes a bug where running ddtrace-run caused a traceback on keyboard interrupt.
CI Visibility: fix the default HTTP timeout for backend requests from 15 seconds to 30 seconds, and add the DD_CIVISIBILITY_BACKEND_API_TIMEOUT_MILLIS
DD_CIVISIBILITY_BACKEND_API_TIMEOUT_MILLIS environment variable (previously missing) to override it. The value is expressed in milliseconds (e.g. 60000 for 60 seconds), consistent with the Java tracer. The same timeout now applies uniformly to all backend requests, including skippable test fetches.<!-- -->
pydantic_ai.tool / pydantic_ai.agent) and the resource name is the specific tool or agent name, matching Datadog APM convention. This restores per-tool and per-agent grouping on APM service and resource pages. LLM Observability views are unaffected.<!-- -->
trace_engine() is called repeatedly for the same engine.<!-- -->
LLM Observability: This fix resolves an issue in the Claude Agent SDK integration where a span's error message showed an uncategorized unknown error c
unknown error category from the upstream Claude Agent SDK instead of a descriptive API error. The integration now surfaces the detailed error message from the assistant message content.<!-- -->
/v0.7/config payloads in multi-threaded applications (e.g. uWSGI).<!-- -->
<!-- -->
LLMObs.annotate(tags=...) could cause spans to be dropped during ingestion.<!-- -->
openai spans when an OpenAI (or AsyncOpenAI) client and an AzureOpenAI (or AsyncAzureOpenAI) client are instantiated at the same time. Provider is now determined per-call rather than from the most recently constructed client.internal: Fixed an issue that could have caused some timers, like the one responsible for Symbol Database uploads, to fire repeatedly after the first
<!-- -->
PeriodicThread callbacks were invisible to Python's cyclic garbage collector and could accumulate when threads used bound methods as targets.<!-- -->
<!-- -->
AAP: This fix resolves an issue where the AppSec body-parsing hook consumed the websocket.connect ASGI message, causing ASGI/FastAPI WebSocket connect
websocket.connect ASGI message, causing ASGI/FastAPI WebSocket connections to fail with HTTP 500 when AppSec was enabled.<!-- -->
reasoning_content was missing from streamed chat completions in the OpenAI and LiteLLM integrations when an OpenAI-compatible reasoning provider (e.g. DeepSeek, Qwen) emitted delta.reasoning_content chunks. The aggregated message now captures reasoning text in the output message, matching non-streaming behavior.<!-- -->
DD_LLMOBS_AGENTLESS_ENABLED=1), APM traces are now exported agentlessly to Datadog's intake. This should not change user-facing behavior: both APM and LLMObs spans remain visible in the UI; LLMObs spans are simply no longer shipped separately for agentless users. Note that setting DD_APM_TRACING_ENABLED=false takes higher precedence and will result in LLMObs span events shipping separately as existing behavior.LLM Observability: when LLMObs is enabled in agentless mode (Datadog Agent not reachable or with DD_LLMOBS_AGENTLESS_ENABLED=1), APM traces are now ex
DD_LLMOBS_AGENTLESS_ENABLED=1), APM traces are now exported agentlessly to Datadog's intake. This should not change user-facing behavior: both APM and LLMObs spans remain visible in the UI; LLMObs spans are simply no longer shipped separately for agentless users. Note that setting DD_APM_TRACING_ENABLED=false takes higher precedence and will result in LLMObs span events shipping separately as existing behavior.CI Visibility: This fix resolves an issue where the Datadog UI showed No extra information available when viewing details about why a test was skipped
CI Visibility: This fix resolves an issue where the Datadog UI showed No extra information available when viewing details about why a test was skipped by Intelligent Test Runner, due to missing ITR correlation ID on test or suite events.
Fix crashes in uwsgi worker when exiting from SIGTERM.
IAST: This fix resolves an issue where IAST could report a false positive vulnerability against a request whose input did not actually contain tainted…
yaml/_yaml loaded during module cleanup, fixing an issue that broke PyYAML consumers such as Airflow.<!-- -->
<!-- -->
tracing: This fix resolves a rare issue that could cause an application to hang after forking a child process.
<!-- -->
git warnings when running tests with pytest-xdist. Simultaneous git fetch --update-shallow calls now retry with exponential back-off on .git/shallow.lock contention, skip the fetch entirely once a sibling worker has already unshallowed the repository, and run under the C locale so the contention check is robust against non-English git messages. git merge-base is deferred until after unshallowing so the required commits are available locally.<!-- -->
Your coding agent can read these notes before it upgrades. Set up the MCP server →