NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
PyPI, #486 most downloaded on PyPI
Datadog APM client library
Latest stable
4.17.2
Last release 3 days ago
19 Nov 2026
Ships on a steady schedule
a new release about every 2 weeks
Nearly every release is documented
notes for 60 of the last 60 stable releases
5 versions withdrawn
withdrawn after publishing
10 years old
871 releases · first in 2017
Drag across the ruler to compare, or tap a release to open it.
Fixes an issue where untrusted deserialization vulnerabilities are reported with an unrecognized vulnerability type and are not displayed correctly in…
Estimated end-of-life date, accurate to within three months: 05-2027
See the support level definitions for more information.
asyncio.gather. Also fixes asyncio tasks that could go missing from profiles after calling asyncio.gather.asyncio.create_task is sometimes reported without the stack of the Task that created it.runtime.python.gc.pause.time and runtime.python.gc.pause.max overcounting GC pause time when other threads held the GIL during a collection.Your coding agent can read these notes before it upgrades. Set up the MCP server →
Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 05-2027
See the support level definitions for more information.
DD_LLMOBS_ENABLED and Remote Configuration was enabled.faulthandler was enabled during the first seconds after startup.RLIMIT_AS). Under that pressure, some stack frames may be missing from profiles instead of crashing the process.Fixes missing vulnerability findings in response middleware around mounted web applications when nested request spans finish before the enclosing requ…
Estimated end-of-life date, accurate to within three months: 05-2027
See the support level definitions for more information.
DD_PYTEST_USE_NEW_PLUGIN and DD_PYTEST_USE_NEW_PLUGIN_BETA environment variables no longer have any effect; the pytest plugin is always used. Remove these environment variables from your configuration to silence the deprecation warning emitted while they are set.aiohttp client.ddtrace.contrib.aiohttp.trace_app. Request bodies sent with chunked transfer encoding or compression, or larger than the application's client_max_size, are not inspected.bytearray and memoryview queries, in SQL integrations and Django MySQL backends. Queries are inspected using the connection's encoding, with UTF-8 inspection for PostgreSQL SQL_ASCII connections.tags argument to AIGuardClient.evaluate(). Each key/value pair in the dictionary is set as a tag on the AI Guard span, so evaluations can be filtered by custom fields in queries, e.g. client.evaluate(messages, tags={"tenant": "acme"}). Keys starting with ai_guard. are reserved and ignored.ddtrace.aiguard.middleware, to control which requests AI Guard evaluates and to run custom logic around each evaluation. See the AI Guard documentation for how to use them.algoliasearch 3.x.DD_TRACE_ELEVENLABS_ENABLED=false to disable automatic instrumentation. For improved user-speaking colors, enable vad_score in the agent's Advanced settings under Client Events. Audio playback remains available without this optional event.LLMObs.agent(version=...), @agent(version=...), LLMObs.annotate(agent={"version": ...}) or LLMObs.annotation_context(agent={"version": ...}) is now also reported as an agent_version tag on child spans of that agent within the same process.http.client_ip and network.client.ip tags when AppSec or DD_TRACE_CLIENT_IP_ENABLED is active and the web framework instruments the request. No additional LLMObs configuration is required.ephemeral_30m_input_tokens and ephemeral_12h_input_tokens metrics for Responses API calls.client.decisions.create), capturing the input, questions, answers, and token usage.LLMObs.routing_context(), a context manager that sends LLM Observability spans and evaluation metrics created within it to other Datadog organizations instead of the configured one. Pass dd_api_key (and optionally dd_site) to route to one organization, or targets to send the same data to several. Routed data is sent directly to the LLM Observability intake; APM spans are not routed.web_search_count for OpenAI Responses API and Anthropic web searches, and storage_search_count for OpenAI file searches.system_one calls, capturing the request and response bodies and token usage. Images are recorded by reference, not by content.decision_provider (openai and typesafe respectively) to identify decision model spans.mcp_client_name, mcp_client_title, and mcp_client_version tags. Tool call spans are not tagged for servers on mcp<2 using stateless streamable HTTP, where the client's identity is unavailable.mcp 2.x, including clients and servers using the stateless 2026-07-28 MCP protocol. Previously, no APM or Agent Observability spans were created for MCP clients or servers when mcp 2.x was installed.DD_TRACE_HTTP_CLIENT_ERROR_STATUSES to configure which HTTP client response status codes mark a span as an error when DD_TRACE_OTEL_SEMANTICS_ENABLED=true. When it is not set, client status codes of 400 and above are errors.DD_TRACE_OTEL_SEMANTICS_ENABLED=true. When enabled, client spans use OpenTelemetry attribute names and low-cardinality resource names; the default remains the existing Datadog behavior.DD_TRACE_OTEL_SEMANTICS_ENABLED=true.DD_TRACE_OTEL_SEMANTICS_ENABLED. See the OpenTelemetry API docs for details.OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=grpc or OTEL_EXPORTER_OTLP_PROTOCOL=grpc to use gRPC; http/protobuf remains the default for traces. Applications that use the global gRPC setting but require HTTP trace export must set OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf. gRPC trace export currently supports plaintext http:// endpoints only.pytest.mark.dd_prevent_failure_suppression to override Datadog quarantine and disabling, and pytest.mark.dd_no_auto_retry to exclude Datadog automatic retries. Both markers support test functions, classes, modules, and individual parameter cases while preserving explicit pytest plugin retries.DD_CIVISIBILITY_GIT_UPLOAD_ENABLED (matching dd-trace-js and dd-trace-go). When false, the git metadata upload is skipped entirely; upload once per pipeline instead (e.g. datadog-ci git-metadata upload in a setup job). The pull request merge-base is still computed from local history when possible.DD_CIVISIBILITY_GIT_UNSHALLOW_ENABLED environment variable (matching dd-trace-js). When set to false, the plugin never runs git fetch --shallow-since on shallow clones; the /search_commits check and any pending packfile upload still run using the locally available history.TypeSafeClient.system_one and AsyncTypeSafeClient.system_one calls in the TypeSafe Python SDK (typesafe-sdk).-O or PYTHONOPTIMIZE) skipped the integrity check of the downloaded libddwaf library. The library is now checked against checksums pinned in ddtrace rather than ones downloaded alongside it.DD_AI_GUARD_TIMEOUT below 1000 resulted in a request timeout of 0, because the value is applied in whole seconds. Such values now log a warning at startup and fall back to the default of 10000.DD_AI_GUARD_ANALYZE_STREAM_RESPONSES_ENABLED is true. The stream is now read and evaluated before any chunk is returned, and a block returns no chunk.DD_AI_GUARD_ANALYZE_STREAM_RESPONSES_ENABLED set to true, a LangChain model that streams delivered tokens to callbacks (for example astream_events() or LangGraph stream_mode="messages") before AI Guard evaluated the response. Callbacks now receive the tokens after the evaluation, and none when it blocks.ValueError when an agent invocation finished in a different asyncio task, and where OpenAI and Anthropic calls in the task that started the invocation were then no longer evaluated.responses.create(stream=True)) were not evaluated when DD_AI_GUARD_ANALYZE_STREAM_RESPONSES_ENABLED was unset or false. Request inputs are now evaluated, matching Chat Completions and Anthropic streaming calls, so these calls can now be blocked when blocking is enabled. To keep evaluating without blocking, set DD_AI_GUARD_BLOCK=false; to disable AI Guard for OpenAI, set DD_AI_GUARD_OPENAI_ENABLED=false.DD_TEST_OPTIMIZATION_DISCOVERY_ENABLED enabled, collection errors now return exit code 2 while preserving partial discovery results, unless --continue-on-collection-errors is supplied.async_to_sync) could end with an internal coverage error, or silently lose or misattribute the coverage data of the tests that use them.error.stack or error.message exceed 5000 characters, the last 5000 characters are now kept.pytest commands in the same shallow checkout. Fetches are retried when the checked-out commit or shallow history changes.git fetch --shallow-since unshallow on shallow clones when the backend already knows all local commits and the pull request merge-base is not needed or user-supplied (DD_GIT_PULL_REQUEST_BASE_BRANCH_SHA); shallow-graph merge-bases are untrusted and recomputed.--lazy-apps --skip-atexit or a --master process with --py-call-uwsgi-fork-hooks.lazy-apps, py-call-uwsgi-fork-hooks or py-call-osafterfork option. Native frame attribution and exception profiling are now disabled in these workers. Enable lazy-apps or py-call-uwsgi-fork-hooks to keep them.sys.monitoring slot used by the profiler, by sharing a single monitoring tool with error tracking and falling back to another free slot when one is taken.RecursionError when pickling a profiler object has been fixed.failed to intern string: OutOfMemory when the process runs out of memory while the profiler exports a profile. The profile is now dropped instead.DD_ERROR_TRACKING_HANDLED_ERRORS_INCLUDE were not reported on Python 3.12 and later when the first handled exception occurred while the module was still being imported, causing all later handled exceptions from the same module to also be missed.DD_APPSEC_ENABLED was unset and AAP had not been activated through Remote Configuration, or because AAP failed to load at startup.DD_TRACE_SAMPLING_RULES rule and have no service-specific Agent rate are sampled at 100% instead of using the default rate provided by the Datadog Agent.config.<integration>.http.trace_headers(), were ignored when set before the integration was enabled./run request in AWS Lambda MicroVMs during tracer reconfiguration when trace delivery is slow, and prevents cleanup failures from interrupting reconfiguration or shutdown. Native exporter shutdown can still delay Python threads and send pending trace statistics after identity refresh.baggage in OTEL_PROPAGATORS is reported as unsupported and ignored.deployment.environment attribute instead of deployment.environment.name for DD_ENV.ZeroDivisionError raised from span.finish() when multiple threads write traces concurrently.dd= list-member of the W3C tracestate header could exceed 256 characters, causing proxies that enforce the W3C limit to drop the whole tracestate header.contextvars copy made inside a @tracer.wrap function kept the local variables of the finished call in memory.OTEL_RESOURCE_ATTRIBUTES can differ between traces, logs, and metrics. These unified service tags are now applied consistently across all three signals.trace_app failed with KeyError: 'disable_stream_timing_for_mem_leak' when the aiohttp integration was not patched, for example with DD_TRACE_ENABLED=false.http.url tag of aiohttp server spans lost the percent-encoding of the query string. The http.query.string tag now also keeps the percent-encoded query string.IndexError raised when collect_query_text is enabled and the search query is passed as a keyword argument.algoliasearch 4.0.0 or later was installed, raising TypeError: issubclass() arg 1 must be a class in Django applications.tracer.wrap() would surface as TypeError.multiprocessing.Process started from a thread pool).gevent is installed and remote configuration enables Exception Replay or Dynamic Instrumentation while the application is starting.TypeError, silently disabling that instrumentation for the call.sys.monitoring (such as Error Tracking handled-exception monitoring) could fail to enable with a ValueError on Python 3.12 and later when a garbage collection ran during startup.AttributeError that prevented the Symbol Database from loading on Python versions earlier than 3.9.8.falcon.asgi.App failed with a TypeError or CompatibilityError when the falcon integration was enabled, for example under ddtrace-run. ASGI applications now start normally but are not traced by the falcon integration; wrap them with ddtrace.contrib.asgi.TraceMiddleware to trace them.falcon.App failed when middleware was None, a single component, a tuple, or passed positionally, and where falcon.API applications on Falcon 3 produced duplicate falcon.request spans.resp.status is set to an integer, an http.HTTPStatus member, or bytes, or when resp.status_code is set (Falcon 4).google-adk 2.10.0 and later fail to start when tracing is enabled, or start normally but report no tool call or code execution spans. Tool calls and code execution are now traced on all supported google-adk versions.google-adk 2.12.0 fail to start with LLM Observability enabled, or start without tool call and code execution spans otherwise. Tool calls made through Runner.run_live are now also traced on google-adk 2.12.0.Action: followed by long content.DD_API_KEY in the application. Fetching other prompts still requires the appropriate API credentials.google_genai and vertexai LLM spans did not include tool_use_prompt_token_count in the input token count.LLMJudge with the anthropic provider raised TypeError: Object of type method is not JSON serializable when the model returned a thinking block before the text block (for example, Claude models with adaptive thinking). The judge now reads the first text block in the response.TracedStream/TracedAsyncStream finalization and the stream context managers.opentelemetry-exporter-otlp 1.45.0 or later.OTEL_LOGS_EXPORTER=otlp does not enable OTLP log export.ddtrace is imported in the master process and neither lazy-apps nor py-call-uwsgi-fork-hooks is set.AttributeError: 'EngineCoreOutput' object has no attribute 'num_cached_tokens'. This error stopped the vLLM output loop and caused all requests to fail. The integration now also handles tracing errors without interrupting vLLM.Producer.produce() was called with headers as a list of tuples or as None.tool resulted in an error message.logging.captureWarnings(True) had no effect when running under ddtrace-run with gevent installed: warnings were written directly to stderr in the default plain-text format instead of being routed through the py.warnings logger.Fixes missing vulnerability findings in response middleware around mounted web applications when nested request spans finish before the enclosing requ…
Estimated end-of-life date, accurate to within three months: 05-2027
See the support level definitions for more information.
DD_PYTEST_USE_NEW_PLUGIN and DD_PYTEST_USE_NEW_PLUGIN_BETA environment variables no longer have any effect; the pytest plugin is always used. Remove these environment variables from your configuration to silence the deprecation warning emitted while they are set.aiohttp client.ddtrace.contrib.aiohttp.trace_app. Request bodies sent with chunked transfer encoding or compression, or larger than the application's client_max_size, are not inspected.bytearray and memoryview queries, in SQL integrations and Django MySQL backends. Queries are inspected using the connection's encoding, with UTF-8 inspection for PostgreSQL SQL_ASCII connections.tags argument to AIGuardClient.evaluate(). Each key/value pair in the dictionary is set as a tag on the AI Guard span, so evaluations can be filtered by custom fields in queries, e.g. client.evaluate(messages, tags={"tenant": "acme"}). Keys starting with ai_guard. are reserved and ignored.ddtrace.aiguard.middleware, to control which requests AI Guard evaluates and to run custom logic around each evaluation. See the AI Guard documentation for how to use them.algoliasearch 3.x.DD_TRACE_ELEVENLABS_ENABLED=false to disable automatic instrumentation. For improved user-speaking colors, enable vad_score in the agent's Advanced settings under Client Events. Audio playback remains available without this optional event.LLMObs.agent(version=...), @agent(version=...), LLMObs.annotate(agent={"version": ...}) or LLMObs.annotation_context(agent={"version": ...}) is now also reported as an agent_version tag on child spans of that agent within the same process.http.client_ip and network.client.ip tags when AppSec or DD_TRACE_CLIENT_IP_ENABLED is active and the web framework instruments the request. No additional LLMObs configuration is required.ephemeral_30m_input_tokens and ephemeral_12h_input_tokens metrics for Responses API calls.client.decisions.create), capturing the input, questions, answers, and token usage.LLMObs.routing_context(), a context manager that sends LLM Observability spans and evaluation metrics created within it to other Datadog organizations instead of the configured one. Pass dd_api_key (and optionally dd_site) to route to one organization, or targets to send the same data to several. Routed data is sent directly to the LLM Observability intake; APM spans are not routed.web_search_count for OpenAI Responses API and Anthropic web searches, and storage_search_count for OpenAI file searches.system_one calls, capturing the request and response bodies and token usage. Images are recorded by reference, not by content.decision_provider (openai and typesafe respectively) to identify decision model spans.mcp_client_name, mcp_client_title, and mcp_client_version tags. Tool call spans are not tagged for servers on mcp<2 using stateless streamable HTTP, where the client's identity is unavailable.mcp 2.x, including clients and servers using the stateless 2026-07-28 MCP protocol. Previously, no APM or Agent Observability spans were created for MCP clients or servers when mcp 2.x was installed.DD_TRACE_HTTP_CLIENT_ERROR_STATUSES to configure which HTTP client response status codes mark a span as an error when DD_TRACE_OTEL_SEMANTICS_ENABLED=true. When it is not set, client status codes of 400 and above are errors.DD_TRACE_OTEL_SEMANTICS_ENABLED=true. When enabled, client spans use OpenTelemetry attribute names and low-cardinality resource names; the default remains the existing Datadog behavior.DD_TRACE_OTEL_SEMANTICS_ENABLED=true.DD_TRACE_OTEL_SEMANTICS_ENABLED. See the OpenTelemetry API docs for details.OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=grpc or OTEL_EXPORTER_OTLP_PROTOCOL=grpc to use gRPC; http/protobuf remains the default for traces. Applications that use the global gRPC setting but require HTTP trace export must set OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf. gRPC trace export currently supports plaintext http:// endpoints only.pytest.mark.dd_prevent_failure_suppression to override Datadog quarantine and disabling, and pytest.mark.dd_no_auto_retry to exclude Datadog automatic retries. Both markers support test functions, classes, modules, and individual parameter cases while preserving explicit pytest plugin retries.DD_CIVISIBILITY_GIT_UPLOAD_ENABLED (matching dd-trace-js and dd-trace-go). When false, the git metadata upload is skipped entirely; upload once per pipeline instead (e.g. datadog-ci git-metadata upload in a setup job). The pull request merge-base is still computed from local history when possible.DD_CIVISIBILITY_GIT_UNSHALLOW_ENABLED environment variable (matching dd-trace-js). When set to false, the plugin never runs git fetch --shallow-since on shallow clones; the /search_commits check and any pending packfile upload still run using the locally available history.TypeSafeClient.system_one and AsyncTypeSafeClient.system_one calls in the TypeSafe Python SDK (typesafe-sdk).-O or PYTHONOPTIMIZE) skipped the integrity check of the downloaded libddwaf library. The library is now checked against checksums pinned in ddtrace rather than ones downloaded alongside it.DD_AI_GUARD_TIMEOUT below 1000 resulted in a request timeout of 0, because the value is applied in whole seconds. Such values now log a warning at startup and fall back to the default of 10000.DD_AI_GUARD_ANALYZE_STREAM_RESPONSES_ENABLED is true. The stream is now read and evaluated before any chunk is returned, and a block returns no chunk.DD_AI_GUARD_ANALYZE_STREAM_RESPONSES_ENABLED set to true, a LangChain model that streams delivered tokens to callbacks (for example astream_events() or LangGraph stream_mode="messages") before AI Guard evaluated the response. Callbacks now receive the tokens after the evaluation, and none when it blocks.ValueError when an agent invocation finished in a different asyncio task, and where OpenAI and Anthropic calls in the task that started the invocation were then no longer evaluated.responses.create(stream=True)) were not evaluated when DD_AI_GUARD_ANALYZE_STREAM_RESPONSES_ENABLED was unset or false. Request inputs are now evaluated, matching Chat Completions and Anthropic streaming calls, so these calls can now be blocked when blocking is enabled. To keep evaluating without blocking, set DD_AI_GUARD_BLOCK=false; to disable AI Guard for OpenAI, set DD_AI_GUARD_OPENAI_ENABLED=false.DD_TEST_OPTIMIZATION_DISCOVERY_ENABLED enabled, collection errors now return exit code 2 while preserving partial discovery results, unless --continue-on-collection-errors is supplied.async_to_sync) could end with an internal coverage error, or silently lose or misattribute the coverage data of the tests that use them.error.stack or error.message exceed 5000 characters, the last 5000 characters are now kept.pytest commands in the same shallow checkout. Fetches are retried when the checked-out commit or shallow history changes.git fetch --shallow-since unshallow on shallow clones when the backend already knows all local commits and the pull request merge-base is not needed or user-supplied (DD_GIT_PULL_REQUEST_BASE_BRANCH_SHA); shallow-graph merge-bases are untrusted and recomputed.lazy-apps, py-call-uwsgi-fork-hooks or py-call-osafterfork option. Native frame attribution and exception profiling are now disabled in these workers. Enable lazy-apps or py-call-uwsgi-fork-hooks to keep them.sys.monitoring slot used by the profiler, by sharing a single monitoring tool with error tracking and falling back to another free slot when one is taken.RecursionError when pickling a profiler object has been fixed.DD_ERROR_TRACKING_HANDLED_ERRORS_INCLUDE were not reported on Python 3.12 and later when the first handled exception occurred while the module was still being imported, causing all later handled exceptions from the same module to also be missed.DD_APPSEC_ENABLED was unset and AAP had not been activated through Remote Configuration, or because AAP failed to load at startup.DD_TRACE_SAMPLING_RULES rule and have no service-specific Agent rate are sampled at 100% instead of using the default rate provided by the Datadog Agent.config.<integration>.http.trace_headers(), were ignored when set before the integration was enabled.baggage in OTEL_PROPAGATORS is reported as unsupported and ignored.deployment.environment attribute instead of deployment.environment.name for DD_ENV.ZeroDivisionError raised from span.finish() when multiple threads write traces concurrently.dd= list-member of the W3C tracestate header could exceed 256 characters, causing proxies that enforce the W3C limit to drop the whole tracestate header.contextvars copy made inside a @tracer.wrap function kept the local variables of the finished call in memory.OTEL_RESOURCE_ATTRIBUTES can differ between traces, logs, and metrics. These unified service tags are now applied consistently across all three signals.trace_app failed with KeyError: 'disable_stream_timing_for_mem_leak' when the aiohttp integration was not patched, for example with DD_TRACE_ENABLED=false.http.url tag of aiohttp server spans lost the percent-encoding of the query string. The http.query.string tag now also keeps the percent-encoded query string.IndexError raised when collect_query_text is enabled and the search query is passed as a keyword argument.algoliasearch 4.0.0 or later was installed, raising TypeError: issubclass() arg 1 must be a class in Django applications.tracer.wrap() would surface as TypeError.multiprocessing.Process started from a thread pool).gevent is installed and remote configuration enables Exception Replay or Dynamic Instrumentation while the application is starting.TypeError, silently disabling that instrumentation for the call.sys.monitoring (such as Error Tracking handled-exception monitoring) could fail to enable with a ValueError on Python 3.12 and later when a garbage collection ran during startup.AttributeError that prevented the Symbol Database from loading on Python versions earlier than 3.9.8.falcon.asgi.App failed with a TypeError or CompatibilityError when the falcon integration was enabled, for example under ddtrace-run. ASGI applications now start normally but are not traced by the falcon integration; wrap them with ddtrace.contrib.asgi.TraceMiddleware to trace them.falcon.App failed when middleware was None, a single component, a tuple, or passed positionally, and where falcon.API applications on Falcon 3 produced duplicate falcon.request spans.resp.status is set to an integer, an http.HTTPStatus member, or bytes, or when resp.status_code is set (Falcon 4).google-adk 2.10.0 and later fail to start when tracing is enabled, or start normally but report no tool call or code execution spans. Tool calls and code execution are now traced on all supported google-adk versions.google-adk 2.12.0 fail to start with LLM Observability enabled, or start without tool call and code execution spans otherwise. Tool calls made through Runner.run_live are now also traced on google-adk 2.12.0.Action: followed by long content.DD_API_KEY in the application. Fetching other prompts still requires the appropriate API credentials.google_genai and vertexai LLM spans did not include tool_use_prompt_token_count in the input token count.LLMJudge with the anthropic provider raised TypeError: Object of type method is not JSON serializable when the model returned a thinking block before the text block (for example, Claude models with adaptive thinking). The judge now reads the first text block in the response.TracedStream/TracedAsyncStream finalization and the stream context managers.opentelemetry-exporter-otlp 1.45.0 or later.OTEL_LOGS_EXPORTER=otlp does not enable OTLP log export.AttributeError: 'EngineCoreOutput' object has no attribute 'num_cached_tokens'. This error stopped the vLLM output loop and caused all requests to fail. The integration now also handles tracing errors without interrupting vLLM.Producer.produce() was called with headers as a list of tuples or as None.tool resulted in an error message.logging.captureWarnings(True) had no effect when running under ddtrace-run with gevent installed: warnings were written directly to stderr in the default plain-text format instead of being routed through the py.warnings logger.Fixes missing vulnerability findings in response middleware around mounted web applications when nested request spans finish before the enclosing requ…
Estimated end-of-life date, accurate to within three months: 05-2027
See the support level definitions for more information.
DD_PYTEST_USE_NEW_PLUGIN and DD_PYTEST_USE_NEW_PLUGIN_BETA environment variables no longer have any effect; the pytest plugin is always used. Remove these environment variables from your configuration to silence the deprecation warning emitted while they are set.aiohttp client.ddtrace.contrib.aiohttp.trace_app. Request bodies sent with chunked transfer encoding or compression, or larger than the application's client_max_size, are not inspected.bytearray and memoryview queries, in SQL integrations and Django MySQL backends. Queries are inspected using the connection's encoding, with UTF-8 inspection for PostgreSQL SQL_ASCII connections.tags argument to AIGuardClient.evaluate(). Each key/value pair in the dictionary is set as a tag on the AI Guard span, so evaluations can be filtered by custom fields in queries, e.g. client.evaluate(messages, tags={"tenant": "acme"}). Keys starting with ai_guard. are reserved and ignored.ddtrace.aiguard.middleware, to control which requests AI Guard evaluates and to run custom logic around each evaluation. See the AI Guard documentation for how to use them.algoliasearch 3.x.DD_TRACE_ELEVENLABS_ENABLED=false to disable automatic instrumentation. For improved user-speaking colors, enable vad_score in the agent's Advanced settings under Client Events. Audio playback remains available without this optional event.LLMObs.agent(version=...), @agent(version=...), LLMObs.annotate(agent={"version": ...}) or LLMObs.annotation_context(agent={"version": ...}) is now also reported as an agent_version tag on child spans of that agent within the same process.http.client_ip and network.client.ip tags when AppSec or DD_TRACE_CLIENT_IP_ENABLED is active and the web framework instruments the request. No additional LLMObs configuration is required.ephemeral_30m_input_tokens and ephemeral_12h_input_tokens metrics for Responses API calls.client.decisions.create), capturing the input, questions, answers, and token usage.LLMObs.routing_context(), a context manager that sends LLM Observability spans and evaluation metrics created within it to other Datadog organizations instead of the configured one. Pass dd_api_key (and optionally dd_site) to route to one organization, or targets to send the same data to several. Routed data is sent directly to the LLM Observability intake; APM spans are not routed.web_search_count for OpenAI Responses API and Anthropic web searches, and storage_search_count for OpenAI file searches.system_one calls, capturing the request and response bodies and token usage. Images are recorded by reference, not by content.decision_provider (openai and typesafe respectively) to identify decision model spans.mcp_client_name, mcp_client_title, and mcp_client_version tags. Tool call spans are not tagged for servers on mcp<2 using stateless streamable HTTP, where the client's identity is unavailable.mcp 2.x, including clients and servers using the stateless 2026-07-28 MCP protocol. Previously, no APM or Agent Observability spans were created for MCP clients or servers when mcp 2.x was installed.DD_TRACE_HTTP_CLIENT_ERROR_STATUSES to configure which HTTP client response status codes mark a span as an error when DD_TRACE_OTEL_SEMANTICS_ENABLED=true. When it is not set, client status codes of 400 and above are errors.DD_TRACE_OTEL_SEMANTICS_ENABLED=true. When enabled, client spans use OpenTelemetry attribute names and low-cardinality resource names; the default remains the existing Datadog behavior.DD_TRACE_OTEL_SEMANTICS_ENABLED=true.DD_TRACE_OTEL_SEMANTICS_ENABLED. See the OpenTelemetry API docs for details.OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=grpc or OTEL_EXPORTER_OTLP_PROTOCOL=grpc to use gRPC; http/protobuf remains the default for traces. Applications that use the global gRPC setting but require HTTP trace export must set OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf. gRPC trace export currently supports plaintext http:// endpoints only.pytest.mark.dd_prevent_failure_suppression to override Datadog quarantine and disabling, and pytest.mark.dd_no_auto_retry to exclude Datadog automatic retries. Both markers support test functions, classes, modules, and individual parameter cases while preserving explicit pytest plugin retries.TypeSafeClient.system_one and AsyncTypeSafeClient.system_one calls in the TypeSafe Python SDK (typesafe-sdk).-O or PYTHONOPTIMIZE) skipped the integrity check of the downloaded libddwaf library. The library is now checked against checksums pinned in ddtrace rather than ones downloaded alongside it.DD_AI_GUARD_TIMEOUT below 1000 resulted in a request timeout of 0, because the value is applied in whole seconds. Such values now log a warning at startup and fall back to the default of 10000.DD_AI_GUARD_ANALYZE_STREAM_RESPONSES_ENABLED is true. The stream is now read and evaluated before any chunk is returned, and a block returns no chunk.DD_AI_GUARD_ANALYZE_STREAM_RESPONSES_ENABLED set to true, a LangChain model that streams delivered tokens to callbacks (for example astream_events() or LangGraph stream_mode="messages") before AI Guard evaluated the response. Callbacks now receive the tokens after the evaluation, and none when it blocks.ValueError when an agent invocation finished in a different asyncio task, and where OpenAI and Anthropic calls in the task that started the invocation were then no longer evaluated.responses.create(stream=True)) were not evaluated when DD_AI_GUARD_ANALYZE_STREAM_RESPONSES_ENABLED was unset or false. Request inputs are now evaluated, matching Chat Completions and Anthropic streaming calls, so these calls can now be blocked when blocking is enabled. To keep evaluating without blocking, set DD_AI_GUARD_BLOCK=false; to disable AI Guard for OpenAI, set DD_AI_GUARD_OPENAI_ENABLED=false.DD_TEST_OPTIMIZATION_DISCOVERY_ENABLED enabled, collection errors now return exit code 2 while preserving partial discovery results, unless --continue-on-collection-errors is supplied.async_to_sync) could end with an internal coverage error, or silently lose or misattribute the coverage data of the tests that use them.error.stack or error.message exceed 5000 characters, the last 5000 characters are now kept.lazy-apps, py-call-uwsgi-fork-hooks or py-call-osafterfork option. Native frame attribution and exception profiling are now disabled in these workers. Enable lazy-apps or py-call-uwsgi-fork-hooks to keep them.sys.monitoring slot used by the profiler, by sharing a single monitoring tool with error tracking and falling back to another free slot when one is taken.RecursionError when pickling a profiler object has been fixed.DD_ERROR_TRACKING_HANDLED_ERRORS_INCLUDE were not reported on Python 3.12 and later when the first handled exception occurred while the module was still being imported, causing all later handled exceptions from the same module to also be missed.DD_APPSEC_ENABLED was unset and AAP had not been activated through Remote Configuration, or because AAP failed to load at startup.DD_TRACE_SAMPLING_RULES rule and have no service-specific Agent rate are sampled at 100% instead of using the default rate provided by the Datadog Agent.config.<integration>.http.trace_headers(), were ignored when set before the integration was enabled.baggage in OTEL_PROPAGATORS is reported as unsupported and ignored.deployment.environment attribute instead of deployment.environment.name for DD_ENV.ZeroDivisionError raised from span.finish() when multiple threads write traces concurrently.dd= list-member of the W3C tracestate header could exceed 256 characters, causing proxies that enforce the W3C limit to drop the whole tracestate header.contextvars copy made inside a @tracer.wrap function kept the local variables of the finished call in memory.OTEL_RESOURCE_ATTRIBUTES can differ between traces, logs, and metrics. These unified service tags are now applied consistently across all three signals.trace_app failed with KeyError: 'disable_stream_timing_for_mem_leak' when the aiohttp integration was not patched, for example with DD_TRACE_ENABLED=false.http.url tag of aiohttp server spans lost the percent-encoding of the query string. The http.query.string tag now also keeps the percent-encoded query string.IndexError raised when collect_query_text is enabled and the search query is passed as a keyword argument.algoliasearch 4.0.0 or later was installed, raising TypeError: issubclass() arg 1 must be a class in Django applications.tracer.wrap() would surface as TypeError.multiprocessing.Process started from a thread pool).gevent is installed and remote configuration enables Exception Replay or Dynamic Instrumentation while the application is starting.TypeError, silently disabling that instrumentation for the call.sys.monitoring (such as Error Tracking handled-exception monitoring) could fail to enable with a ValueError on Python 3.12 and later when a garbage collection ran during startup.AttributeError that prevented the Symbol Database from loading on Python versions earlier than 3.9.8.falcon.asgi.App failed with a TypeError or CompatibilityError when the falcon integration was enabled, for example under ddtrace-run. ASGI applications now start normally but are not traced by the falcon integration; wrap them with ddtrace.contrib.asgi.TraceMiddleware to trace them.falcon.App failed when middleware was None, a single component, a tuple, or passed positionally, and where falcon.API applications on Falcon 3 produced duplicate falcon.request spans.resp.status is set to an integer, an http.HTTPStatus member, or bytes, or when resp.status_code is set (Falcon 4).google-adk 2.10.0 and later fail to start when tracing is enabled, or start normally but report no tool call or code execution spans. Tool calls and code execution are now traced on all supported google-adk versions.google-adk 2.12.0 fail to start with LLM Observability enabled, or start without tool call and code execution spans otherwise. Tool calls made through Runner.run_live are now also traced on google-adk 2.12.0.Action: followed by long content.DD_API_KEY in the application. Fetching other prompts still requires the appropriate API credentials.google_genai and vertexai LLM spans did not include tool_use_prompt_token_count in the input token count.LLMJudge with the anthropic provider raised TypeError: Object of type method is not JSON serializable when the model returned a thinking block before the text block (for example, Claude models with adaptive thinking). The judge now reads the first text block in the response.TracedStream/TracedAsyncStream finalization and the stream context managers.opentelemetry-exporter-otlp 1.45.0 or later.OTEL_LOGS_EXPORTER=otlp does not enable OTLP log export.AttributeError: 'EngineCoreOutput' object has no attribute 'num_cached_tokens'. This error stopped the vLLM output loop and caused all requests to fail. The integration now also handles tracing errors without interrupting vLLM.Producer.produce() was called with headers as a list of tuples or as None.tool resulted in an error message.logging.captureWarnings(True) had no effect when running under ddtrace-run with gevent installed: warnings were written directly to stderr in the default plain-text format instead of being routed through the py.warnings logger.tracing: Fixes inconsistent handling of oversized or malformed baggage headers.
tracer.wrap() would surface as TypeError.contextvars copy made inside a @tracer.wrap functioEstimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 05-2027
See the support level definitions for more information.
--lazy-apps --skip-atexit or a --master process with --py-call-uwsgi-fork-hooks.failed to intern string: OutOfMemory when the process runs out of memory while the profiler exports a profile. The profile is now dropped instead.ddtrace is imported in the master process and neither lazy-apps nor py-call-uwsgi-fork-hooks is set.Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 05-2027
See the support level definitions for more information.
CI Visibility: Fixes repeated Git history fetches that slow down successive pytest commands in the same shallow checkout. Fetches are retried when the checked-out commit or shallow history changes.
CI Visibility: skip the git fetch --shallow-since unshallow on shallow clones when the backend already knows all local commits and the pull request merge-base is not needed or user-supplied (DD_GIT_PULL_REQUEST_BASE_BRANCH_SHA); shallow-graph merge-bases are untrusted and recomputed.
Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 05-2027
See the support level definitions for more information.
falcon.asgi.App failed with a TypeError or CompatibilityError when the falcon integration was enabled, for example under ddtrace-run. ASGI applications now start normally but are not traced by the falcon integration; wrap them with ddtrace.contrib.asgi.TraceMiddleware to trace them.falcon.App failed when middleware was None, a single component, a tuple, or passed positionally, and where falcon.API applications on Falcon 3 produced duplicate falcon.request spans.AttributeError that prevented the Symbol Database from loading on Python versions earlier than 3.9.8.Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 05-2027
See the support level definitions for more information.
internal: Fixes an issue where forked child processes hang and never exit, causing memory usage to grow and the parent process to stall, when several threads fork at the same time (for example, multiprocessing.Process started from a thread pool).
dynamic instrumentation: Fixes an issue where an application thread could hang indefinitely when probes, Exception Replay, or Code Origin produced data faster than it could be uploaded.
google-adk: Fixes an issue where applications using google-adk 2.12.0 fail to start with LLM Observability enabled, or start without tool call and code execution spans otherwise. Tool calls made through Runner.run_live are now also traced on google-adk 2.12.0.
otel: Fixes an issue where OTLP HTTP export requests are incorrectly traced when OpenTelemetry logs or metrics collection is enabled with opentelemetry-exporter-otlp 1.45.0 or later.
opentelemetry: Fixes an issue where OTEL_LOGS_EXPORTER=otlp does not enable OTLP log export.
tracing: Fixes an issue where baggage in OTEL_PROPAGATORS is reported as unsupported and ignored.
internal: Fixes an issue where wrapping an obfuscated function (for example PyArmor) on Python 3.15 proceeds into instrumentation setup instead of failing cleanly. Those functions still cannot be instrumented.
LLM Observability: This fix resolves an issue where google_genai and vertexai LLM spans did not include tool_use_prompt_token_count in the input token count.
Code Security (IAST): Fixes an issue where SQL values escaped by PyMySQL 1.2.1 and later are not recognized as sanitized.
tracing: Fixes a security vulnerability, as described in GHSA-3633-r77q-h3v3 .
google-adk 2.9.0 and later, tool call spans use the resource name FunctionTool._call_tool_async (one leading underscore) instead of FunctionTool.__call_tool_async, following a rename in google-adk itself. Update any monitor or dashboard that matches on the old resource name.DD_TRACE_STATS_CARDINALITY_LIMIT (default 7000), DD_TRACE_STATS_RESOURCE_CARDINALITY_LIMIT (default 1024), DD_TRACE_STATS_HTTP_ENDPOINT_CARDINALITY_LIMIT (default 512), DD_TRACE_STATS_PEER_TAGS_CARDINALITY_LIMIT (default 512) and DD_TRACE_STATS_ADDITIONAL_TAGS_CARDINALITY_LIMIT (default 100) to bound the number of distinct trace metrics aggregation keys tracked per time bucket when client-side stats computation is enabled. Values beyond a limit are aggregated together under a sentinel value.DD_TRACE_TRIO_ENABLED=false.agent argument of LLMObs.annotate() and LLMObs.annotation_context() now accepts name, instructions, model, model_settings and tools alongside version, and reports them as the agent's manifest on agent spans for manual annotation.DD_LLMOBS_SAMPLING_RULES environment variable, which mirrors the format of DD_TRACE_SAMPLING_RULES. See the documentation for details.ManagedPrompt.format() replaces with caller-provided text or tool message lists in their authored position.ManagedPrompt.config and accept it through the config argument when creating prompts or versions.AIGuardAbortError when the verdict is DENY or ABORT. A tool call decided by an instrumented agent is still evaluated only once. Streamed LangChain responses are not covered by this change.InvokeModelWithResponseStream request that ended in an error or was cancelled part-way produced a span with no output annotation. The text received before the failure is now tagged on the span.ConverseStream request that ended in an error or was cancelled part-way produced a span that was not marked as an error, making an interrupted request indistinguishable from a successful one.[paths] aliases from .coveragerc to SF: lines before uploading, fixing duplicate file entries for installed packages.-, ., @, or $.network.client.ip and ai_guard.network.client.ip contain an IP resolved from proxy headers instead of the peer address of the incoming connection. These tags are omitted when the peer address is unavailable.DD_AI_GUARD_ENDPOINT could be written to debug logs, exception messages and span error tags when an evaluation failed to reach the AI Guard service. Any user info in the endpoint is now removed before the request is sent, and the endpoint is redacted from failure messages. The underlying transport error is no longer chained onto the raised AIGuardClientError, which now reports that error's type and redacted message in its own message instead.google-adk 2.9.0 and later fail to start when tracing is enabled, or start normally but report no tool call or code execution spans. Tool calls and code execution are now traced on all supported google-adk versions.interceptors=None raises a TypeError when tracing is enabled.join on a bytes or bytearray separator with an element of the wrong type crashes the process instead of raising TypeError.% string formatting handled tainted text containing a lone surrogate character.pytest-timeout thread mode and terminate a pytest-xdist worker.DD_RUNTIME_METRICS_ENABLED is true.gen_ai.* attributes on APM spans could produce a duplicate LLM Observability span when LLM Observability was disabled.as, and where errors raised while closing the wrapped stream were missing from the span.DD_TRACE_OTEL_CTX_ENABLED to false.AttributeError: 'EngineCoreOutput' object has no attribute 'num_cached_tokens'. This error stopped the vLLM output loop and caused all requests to fail. The integration now also handles tracing errors without interrupting vLLM.google-adk 2.10.0 and later fail to start when tracing is enabled, or start normally but report no tool call or code execution spans. Tool calls and code execution are now traced on all supported google-adk versions.dd= list-member of the W3C tracestate header could exceed 256 characters, causing proxies that enforce the W3C limit to drop the whole tracestate header.LLMJudge with the anthropic provider raised TypeError: Object of type method is not JSON serializable when the model returned a thinking block before the text block (for example, Claude models with adaptive thinking). The judge now reads the first text block in the response.TracedStream/TracedAsyncStream finalization and the stream context managers.tracing: Fixes a security vulnerability, as described in GHSA-3633-r77q-h3v3 .
google-adk 2.9.0 and later, tool call spans use the resource name FunctionTool._call_tool_async (one leading underscore) instead of FunctionTool.__call_tool_async, following a rename in google-adk itself. Update any monitor or dashboard that matches on the old resource name.DD_TRACE_STATS_CARDINALITY_LIMIT (default 7000), DD_TRACE_STATS_RESOURCE_CARDINALITY_LIMIT (default 1024), DD_TRACE_STATS_HTTP_ENDPOINT_CARDINALITY_LIMIT (default 512), DD_TRACE_STATS_PEER_TAGS_CARDINALITY_LIMIT (default 512) and DD_TRACE_STATS_ADDITIONAL_TAGS_CARDINALITY_LIMIT (default 100) to bound the number of distinct trace metrics aggregation keys tracked per time bucket when client-side stats computation is enabled. Values beyond a limit are aggregated together under a sentinel value.DD_TRACE_TRIO_ENABLED=false.agent argument of LLMObs.annotate() and LLMObs.annotation_context() now accepts name, instructions, model, model_settings and tools alongside version, and reports them as the agent's manifest on agent spans for manual annotation.DD_LLMOBS_SAMPLING_RULES environment variable, which mirrors the format of DD_TRACE_SAMPLING_RULES. See the documentation for details.ManagedPrompt.format() replaces with caller-provided text or tool message lists in their authored position.ManagedPrompt.config and accept it through the config argument when creating prompts or versions.AIGuardAbortError when the verdict is DENY or ABORT. A tool call decided by an instrumented agent is still evaluated only once. Streamed LangChain responses are not covered by this change.InvokeModelWithResponseStream request that ended in an error or was cancelled part-way produced a span with no output annotation. The text received before the failure is now tagged on the span.ConverseStream request that ended in an error or was cancelled part-way produced a span that was not marked as an error, making an interrupted request indistinguishable from a successful one.[paths] aliases from .coveragerc to SF: lines before uploading, fixing duplicate file entries for installed packages.-, ., @, or $.network.client.ip and ai_guard.network.client.ip contain an IP resolved from proxy headers instead of the peer address of the incoming connection. These tags are omitted when the peer address is unavailable.DD_AI_GUARD_ENDPOINT could be written to debug logs, exception messages and span error tags when an evaluation failed to reach the AI Guard service. Any user info in the endpoint is now removed before the request is sent, and the endpoint is redacted from failure messages. The underlying transport error is no longer chained onto the raised AIGuardClientError, which now reports that error's type and redacted message in its own message instead.google-adk 2.9.0 and later fail to start when tracing is enabled, or start normally but report no tool call or code execution spans. Tool calls and code execution are now traced on all supported google-adk versions.interceptors=None raises a TypeError when tracing is enabled.join on a bytes or bytearray separator with an element of the wrong type crashes the process instead of raising TypeError.% string formatting handled tainted text containing a lone surrogate character.pytest-timeout thread mode and terminate a pytest-xdist worker.DD_RUNTIME_METRICS_ENABLED is true.gen_ai.* attributes on APM spans could produce a duplicate LLM Observability span when LLM Observability was disabled.as, and where errors raised while closing the wrapped stream were missing from the span.DD_TRACE_OTEL_CTX_ENABLED to false.Profiling: A crash that could occur in applications that use signal handlers to recover from crashes has been fixed.
google-adk: Fixes an issue where applications using google-adk 2.10.0 and later fail to start when tracing is enabled, or start normally but report no tool call or code execution spans. Tool calls and code execution are now traced on all supported google-adk versions.
dd= list-member of the W3C tracestate header could exceed 256 characters, causing proxies that enforce the W3C limit to drop the whole tracestate header.LLMJudge with the anthropic provider raised TypeError: Object of type method is not JSON serializable when the model returned a thinking block before the text block (for example, Claude models with adaptive thinking). The judge now reads the first text block in the response.TracedStream/TracedAsyncStream finalization and the stream context managers.tracing: Fixes a security vulnerability, as described in GHSA-3633-r77q-h3v3 .
google-adk: With google-adk 2.9.0 and later, tool call spans use the resource name FunctionTool._call_tool_async (one leading underscore) instead of FunctionTool.__call_tool_async, following a rename in google-adk itself. Update any monitor or dashboard that matches on the old resource name.
kafka: Adds support for Python 3.14.
azure_durable_functions: Adds tracing for orchestration function invocations and, when Durable Functions distributed tracing V2 is enabled, connects durable orchestration and activity spans to the originating trace without requiring OpenTelemetry configuration.
tracing: Adds DD_TRACE_STATS_CARDINALITY_LIMIT (default 7000), DD_TRACE_STATS_RESOURCE_CARDINALITY_LIMIT (default 1024), DD_TRACE_STATS_HTTP_ENDPOINT_CARDINALITY_LIMIT (default 512), DD_TRACE_STATS_PEER_TAGS_CARDINALITY_LIMIT (default 512) and DD_TRACE_STATS_ADDITIONAL_TAGS_CARDINALITY_LIMIT (default 100) to bound the number of distinct trace metrics aggregation keys tracked per time bucket when client-side stats computation is enabled. Values beyond a limit are aggregated together under a sentinel value.
trio: Adds native trace-context support for Trio tasks and worker threads on Linux. The integration is enabled by default and can be disabled with DD_TRACE_TRIO_ENABLED=false.
LLM Observability: The agent argument of LLMObs.annotate() and LLMObs.annotation_context() now accepts name, instructions, model, model_settings and tools alongside version, and reports them as the agent's manifest on agent spans for manual annotation.
LLM Observability: Introduces tag-based head sampling via the DD_LLMOBS_SAMPLING_RULES environment variable, which mirrors the format of DD_TRACE_SAMPLING_RULES. See the documentation for details.
LLM Observability: Managed chat prompts can now include named message placeholders that ManagedPrompt.format() replaces with caller-provided text or tool message lists in their authored position.
openfeature: adds safe direct Event Platform intake fallback for exposure and flag-evaluation events when the agentless provider cannot use the local Agent's EVP proxy route. Direct requests use API-key authentication and preserve at-most-once delivery for ambiguous local failures.
openfeature: By default, feature flag evaluation data sent to Datadog now uses hashed user identifiers and omits evaluation context attributes to limit sensitive data collection. Enabling full evaluation data for an environment includes the original identifiers and attributes to help investigate flag usage and unexpected behavior.
LLM Observability: Managed prompts now expose version-owned JSON configuration through ManagedPrompt.config and accept it through the config argument when creating prompts or versions.
AI Guard: Fixes an issue where model responses were never evaluated for LangChain chat model and LLM calls, leaving unsafe model output neither reported nor blocked. Such calls now run a second evaluation covering the model's answer, including any tool calls it requested, which raises AIGuardAbortError when the verdict is DENY or ABORT. A tool call decided by an instrumented agent is still evaluated only once. Streamed LangChain responses are not covered by this change.
LLM Observability: Fixes an issue where a LangChain model response blocked by AI Guard resulted in LLM Observability spans with empty output and no token usage.
LLM Observability: resolves an issue where a streamed Bedrock InvokeModelWithResponseStream request that ended in an error or was cancelled part-way produced a span with no output annotation. The text received before the failure is now tagged on the span.
LLM Observability: resolves an issue where a streamed Bedrock ConverseStream request that ended in an error or was cancelled part-way produced a span that was not marked as an error, making an interrupted request indistinguishable from a successful one.
CI Visibility: Code Coverage report uploads now apply [paths] aliases from .coveragerc to SF: lines before uploading, fixing duplicate file entries for installed packages.
dynamic instrumentation: Fixes an issue where sensitive values were not redacted when the name used -, ., @, or $.
LLM Observability: Fixes an issue where abandoned or disconnected streamed LLM requests can leave spans unfinished, causing memory to grow in long-running web workers.
AI Guard: Fixes an issue where network.client.ip and ai_guard.network.client.ip contain an IP resolved from proxy headers instead of the peer address of the incoming connection. These tags are omitted when the peer address is unavailable.
AI Guard: Fixes an issue where credentials embedded in DD_AI_GUARD_ENDPOINT could be written to debug logs, exception messages and span error tags when an evaluation failed to reach the AI Guard service. Any user info in the endpoint is now removed before the request is sent, and the endpoint is redacted from failure messages. The underlying transport error is no longer chained onto the raised AIGuardClientError, which now reports that error's type and redacted message in its own message instead.
CI Visibility: Fixes an issue where collecting per-test code coverage across threads could terminate a test run with an internal error or attribute coverage to the wrong test.
CI Visibility: Fixes an issue where code coverage path aliases from the active coverage configuration were not applied to LCOV source filenames when the mapped destination files did not exist on disk. This caused uploaded coverage reports to retain separate installed-package paths for the same repository file instead of deduplicating them to the canonical repository path.
internal: Fixes incorrect telemetry being collected when hitting cardinality limits during stats computation.
google-adk: Fixes an issue where applications using google-adk 2.9.0 and later fail to start when tracing is enabled, or start normally but report no tool call or code execution spans. Tool calls and code execution are now traced on all supported google-adk versions.
grpc: Fixes an issue where creating synchronous or asynchronous gRPC servers with interceptors=None raises a TypeError when tracing is enabled.
Code Security (IAST): Fixes an issue where calling join on a bytes or bytearray separator with an element of the wrong type crashes the process instead of raising TypeError.
AAP: Fixes false-positive and duplicate unvalidated redirect reports in Flask applications when a request parameter matches the response Location header.
Code Security (IAST): fixes an issue where taint tracking could abort the Python process when old-style % string formatting handled tainted text containing a lone surrogate character.
LLM Observability: Fixes an issue where formatting managed text or chat prompts consumes a closing JSON object brace immediately after a single-brace placeholder, producing malformed JSON. Braces surrounding placeholders are now preserved.
LLM Observability: Fixes an issue where agent attribution was propagated to downstream services from spans that had no agent ancestor. #19531
profiling: A bug where Lock profiling would not associate samples with spans and traces when PyTorch profiling was enabled has been fixed.
internal: Fixes an issue on macOS where starting a subprocess can deadlock while a hostname lookup is in progress.
tracing: Fixes an issue where traces can be dropped when flushing while the tracer is shutting down.
openfeature: Fixes an issue where flag evaluations return local defaults and the provider remains not ready when Remote Configuration completes its first poll during application startup.
profiling: Fixes an issue where an unexpected error while uploading a profile stopped the profiler from uploading any further profiles for the lifetime of the process.
profiling: An issue where unexpected errors while stopping the profiler prevented the profiler from stopping completely has been fixed.
CI Visibility: Fixes an issue where Auto Test Retries do not retry tests that use pytest-timeout thread mode and terminate a pytest-xdist worker.
tracing: Fixes an issue where the Python garbage collector stops and memory grows without limit when DD_RUNTIME_METRICS_ENABLED is true.
CI Visibility: Fixes an incompatibility between vcrpy and Test Optimization that prevents test traces from being uploaded to Datadog.
LLM Observability: An issue where where streamed LLM requests could leave spans unfinished and cause memory to grow in long-running processes has been fixed.
LLM Observability: Fixes an issue where gen_ai.* attributes on APM spans could produce a duplicate LLM Observability span when LLM Observability was disabled.
LLM Observability: Fixes an issue where a traced stream opened as a context manager through a stream manager could close before any chunks were read if the caller omitted as, and where errors raised while closing the wrapped stream were missing from the span.
langchain: Fixes an issue where discarding an unused traced stream could end an enclosing AI Guard evaluation early.
LLM Observability: Fixes an issue where APM spans annotated with gen_ai attributes could cause a duplicate LLM Observability span to be created.
LLM Observability: This fix resolves an issue where cancelled or failed OpenAI streamed requests produced no LLM Observability span at all. These requests now produce an LLM span carrying the input messages, whatever output arrived before the stream ended, and estimated token metrics.
tracing: Fixes a native memory leak on Linux when a thread exits while an OpenTelemetry thread context is still attached. Publication is enabled by default and can be disabled by setting DD_TRACE_OTEL_CTX_ENABLED to false.
Profiling: A crash that could occur in applications that use signal handlers to recover from crashes has been fixed.
code origin: Fixes an issue on Python 3.10 where an exception raised from an async generator decorated with tracer.wrap() would surface as TypeError .
tracer.wrap() would surface as TypeError.contextvars copy made inside a @tracer.wrap function kept the local variables of the finished call in memory.RLIMIT_AS). Under that pressure, some stack frames may be missing from profiles instead of crashing the process.google-adk: With google-adk 2.9.0 and later, tool call spans use the resource name FunctionTool._call_tool_async (one leading underscore) instead of F
google-adk 2.9.0 and later, tool call spans use the resource name FunctionTool._call_tool_async (one leading underscore) instead of FunctionTool.__call_tool_async, following a rename in google-adk itself. Update any monitor or dashboard that matches on the old resource name.google-adk 2.9.0 and later fail to start when tracing is enabled, or start normally but report no tool call or code execution spans. Tool calls and code execution are now traced on all supported google-adk versions.AttributeError: 'EngineCoreOutput' object has no attribute 'num_cached_tokens'. This error stopped the vLLM output loop and caused all requests tofail. The integration now also handles tracing errors without interrupting vLLM.LLMJudge with the anthropic provider raised TypeError: Object of type method is not JSON serializable when the model returned a thinking block before the text block (for example, Claude models with adaptive thinking). The judge now reads the first text block in the response.TracedStream/TracedAsyncStream finalization and the stream context managers.tracing: Fixes a security vulnerability, as described in GHSA-3633-r77q-h3v3 .
Code Security (IAST): fixes an issue where taint tracking could abort the Python process when old-style % string formatting handled tainted text conta
% string formatting handled tainted text containing a lone surrogate character.internal: Fixes an issue on macOS where starting a subprocess can deadlock while a hostname lookup is in progress.
DD_TRACE_OTEL_CTX_ENABLED is set to true and a thread exits while an OpenTelemetry thread context is still attached.Resolves an issue where a streamed Bedrock InvokeModelWithResponseStream request that ended in an error or was cancelled part-way produced a span with
InvokeModelWithResponseStream request that ended in an error or was cancelled part-way produced a span with no output annotation. The text received before the failure is now tagged on the span.ConverseStream request that ended in an error or was cancelled part-way produced a span that was not marked as an error, making an interrupted request indistinguishable from a successful one.DD_RUNTIME_METRICS_ENABLED is true.tracing: DD_TRACE_OTEL_CTX_ENABLED now defaults to false until a Linux-native memory leak caused by setting it to true is fixed.
pytest-timeout thread mode and terminate a pytest-xdist worker.Resolves an issue where gen_ai.* attributes were added to APM spans when LLM Observability was disabled. These attributes are now only emitted when LL
gen_ai.* attributes were added to APM spans when LLM Observability was disabled. These attributes are now only emitted when LLM Observability is enabled.…and reported by dependency scanners as CVE-2025-56005 . Existing redaction paths continue to work.
Estimated end-of-life date, accurate to within three months: 05-2027
See the support level definitions for more information.
pydantic_ai integration now reports agent configuration using the shared agent manifest schema. framework, name, model, model_settings, instructions, system_prompts and tools keep their names and meaning. New keys are metadata, extra_instructions, capabilities, data_contracts, memory_policies, guardrails and agent_settings.discard boolean field to DD_TRACE_SAMPLING_RULES rules to fully drop trace chunks which were rejected during sampling. See the DD_TRACE_SAMPLING_RULES configuration docs for details.ot.th and ot.rv) to spans exported through OTLP when OTEL_TRACES_EXPORTER=otlp is configured, allowing downstream collectors to preserve and extrapolate sampling decisions.DD_RUNTIME_METRICS_ENABLED is true.DD_TRACE_OTEL_CTX_ENABLED=false or DD_TRACE_ASYNCIO_ENABLED=false.OTEL_TRACES_SPAN_METRICS_ENABLED=true to enable it.DD_TRACE_STATS_ADDITIONAL_TAGS, which adds configured span tags to native v0.6 trace stats and OTLP trace metrics.Span.remove_tag(key) and Span.remove_metric(key) for removing a previously set tag or metric from a span. Both are no-ops if the key is not set.DD_AGENTLESS_ENABLED (default false), a single switch that submits all telemetry, traces, Remote Configuration, Dynamic Instrumentation, crash reports, Test Optimization and LLM Observability data directly to the Datadog intake instead of through a local Datadog Agent. It becomes the default for the per-product agentless settings, which can still be set individually to override it. DD_API_KEY is required when it is enabled. Note that tracer flares, runtime metrics and profiling are notably not supported in this mode.DD_TRACE_OTEL_CTX_ENABLED=false.DD_TRACE_OTEL_CTX_ENABLED is true (the default), propagated trace contexts are published to OpenTelemetry thread context records. This lets external readers retain trace and span correlation when no active span object is available.DD_TRACE_OTEL_CTX_ENABLED to false.DD_TRACE_PROPAGATION_AS_SPAN_LINKS environment variable, a comma-separated list of integration names (e.g. google_cloud_pubsub,kafka) for which propagated context is attached as span links instead of a parent-child relationship.InvokeModel spans for Anthropic models, matching the Converse API.openai, litellm, and anthropic integrations.gen_ai.* attributes, so model, provider, application, conversation, and token usage can be searched, faceted, and monitored directly in APM.llm span.image_parts on LLM span messages, so they can be rendered in LLM Observability. Note: images referenced by a URL or a file ID are not captured, and a very large inline image is replaced with a text placeholder.DD_APP_KEY on the application when a Datadog Agent is used.LLMObs.list_experiments() to query experiments by logical name, metadata containment (e.g. {"tags": ["git.commit.sha:abc123"]}). Returns run status and aggregate data (evals, token costs, error rates), so CI/CD workflows can compare a run against a baseline commit and gate on the result.version when starting an agent span, either with LLMObs.agent(name="my-agent", version="v3") or with the @agent(version="v3") decorator. For agents created by an auto-instrumented framework, use LLMObs.annotation_context(agent={"version": "v3"}). This sets the agent_version tag on agent spans. The agent argument accepts a plain dictionary or the new ddtrace.llmobs.Agent TypedDict class.image_parts on LLM span messages, for both Chat Completions and the Responses API, so image inputs render in LLM Observability. This also applies to the OpenAI Agents SDK and to the LiteLLM integration, which share the same message extraction. Images referenced by a remote URL or file_id are not fetched and keep their existing text reference. A single inline image whose base64 payload exceeds 4 MiB is left as an [image omitted: too large] marker instead. Note that this budget is per image, not per request: several inline images (or an image alongside inline audio) can still take a span event past the 5 MB per-event limit, at which point the event's input and output are replaced with a placeholder. To stop image bytes from being recorded, register a span processor with LLMObs.register_processor and remove the image_parts key from the messages on span.input.LLMObs.submit_feedback() for submitting end-user feedback associated with a span, trace, session, or customer-defined feedback join key.DD_API_KEY; set DD_FEATURE_FLAGS_CONFIGURATION_SOURCE=remote_config to keep loading configuration through the Datadog Agent's Remote Configuration.DD_CIVISIBILITY_DYNAMIC_ATR_ENABLED to enable dynamic Auto Test Retries budgets based on test duration, instead of the flat per-test retry limit. Optionally configure the five duration-based budgets with DD_CIVISIBILITY_DYNAMIC_ATR_BUCKETS (five comma-separated integers in [1, 20]), corresponding to the Early Flake Detection duration buckets.ci.pipeline.display_name tag for the Buildkite provider, populated from the BUILDKITE_PIPELINE_NAME environment variable.DD_TRACE_ANYIO_ENABLED=false.kafka in the DD_TRACE_PROPAGATION_AS_SPAN_LINKS environment variable. When enabled together with distributed tracing, the kafka.consume span starts a new trace and relates to every consumed message's producer via span links, instead of continuing a single producer's trace. This is disabled by default.DD_PROFILING_MEMORY_MEM_DOMAIN_ENABLED now defaults to true, so the heap profiler also tracks allocations in the Mem domain on Python 3.12+.DD_PROFILING_NATIVE_HEAP_ENABLED=true.DD_PROFILING_STACK_GC_ENABLED=true.build_py option --no-bundle-libddwaf, set through the [build_py] section of setup.cfg, which builds the package without a bundled libddwaf. The library is then loaded from the system at import time (libddwaf.so.2, then libddwaf.so, resolved by the dynamic linker, and required to be 2.x). Intended for Linux distribution packages; default builds bundle libddwaf as before.httpx2 client.httpx2>=2.0.0aiomysql, aiopg, psycopg async, and MySQL Connector/Python asyncio database drivers.tags argument to ddtrace.data_streams.set_produce_checkpoint and ddtrace.data_streams.set_consume_checkpoint. Pass a list of "key:value" strings to attach additional edge tags to a manual checkpoint alongside the automatically set type, topic, and direction tags, for example set_produce_checkpoint("eventbridge", detail_type, carrier_set, tags=["exchange:my-bus"]). Existing calls that do not pass tags are unaffected.http/protobuf protocol (the default) and http/json, selected via the OTEL_EXPORTER_OTLP_TRACES_PROTOCOL environment variable.urllib3 requests include an Exploit Prevention frame in their traceback, which caused ordinary application errors to be misattributed to Datadog. Enabling urllib3 APM tracing, which is off by default, still contributes a frame of its own.DD_APPSEC_RULES to an empty value prevents AppSec rules from being updated through remote configuration.subprocess commands are passed as tuples.DD_AGENTLESS_ENABLED is set. The errors-intake upload was incorrectly reusing the same endpoint configured for the telemetry crash-report path, sending it to the telemetry intake host instead of the dedicated errors-intake host.InvokeModel integration only captured the first content block of an Anthropic response, so output messages were empty or truncated when a response began with a tool_use or thinking block.DD_APM_TRACING_ENABLED=false discarded every trace, so AI Guard, App and API Protection, Code Security and Software Composition Analysis reported no spans while running in standalone mode. These traces are now sent, and APM billing stays opted out.google_genai responses was merged into the output message instead of being captured as a separate reasoning message._DD_TRACE_WRITER_ADDITIONAL_HEADERS header values containing a whitespace (for example Bearer <token>) were incorrectly parsed, resulting in a truncated header value that caused LLM Observability spans to be dropped.claude-agent-sdk versions prior to 0.1.49, which do not report usage on the assistant message. Note that explicitly setting include_partial_messages=False on ClaudeAgentOptions opts out of this behavior and will result in inaccurate per-turn output token counts.agent_attribution missing or pointing at the wrong ancestor.TypeError raised during streamed tool-call reconstruction when an OpenAI-compatible backend (e.g. DashScope/Qwen) emits a tool-call delta with function.arguments or custom.input set to None instead of "".DD_FEATURE_FLAGS_CONFIGURATION_SOURCE is set to agentless or remote_config.LLMObs.annotate(agent=...) are also now coerced to strings.[image]; remote URL and file_id references are unchanged.[image] marker, or as the screenshot's URL when it is a remote reference.pytest plugin entered a code coverage collection context around every test even when code coverage was disabled, which reported coverage as active while no coverage was being collected and, on Python 3.12 and later, restarted sys.monitoring events once per test.TypeError.ValueError: I/O operation on closed file logging tracebacks at the end of pytest sessions when standard stream handlers point to streams closed by pytest. Tracer logs continue to reach healthy root handlers, including file output and log forwarding.site-packages or dist-packages directories.pass instead of fail when pytest exits with a non-OK, non-NO_TESTS_COLLECTED exit code (e.g. collection errors, interruptions, internal errors).PutEvents now propagates Data Streams Monitoring (DSM) context when DD_DATA_STREAMS_ENABLED is enabled.invoke_model_with_response_stream calls reported all token usage metrics as 0. Token counts are now extracted from the streamed response and set on the span.client_error, making them difficult to diagnose.webbrowser.open, or for calls to urllib.request.urlopen that pass the URL positionally or go through a custom opener installed with urllib.request.install_opener.urllib.request.urlopen and http.client requests include an internal Datadog frame in their traceback when Exploit Prevention is enabled, which caused ordinary application errors to be misattributed to the tracer.open() or pathlib.Path.open() are reported with a corrupted traceback when Exploit Prevention is enabled, showing a duplicated caller frame and omitting the frame that actually raised.OTEL_* environment variables override ddtrace defaults instead of being treated as unset.http.client.request span unfinished and active, so it is never sent and every later span on that thread is parented to it.@tracer.wrap on functions created at run time has been fixed.mysql.connector.Connect().rq.worker.perform_job span was not created when jobs were processed by rq.SimpleWorker on rq >= 2.7.WebDriver.get, WebDriver.quit and WebDriver.close calls made.tracer.configure(apm_tracing_disabled=True) did not apply the standalone sampling settings, so traces were sent without the one trace per minute limit and without the _dd.apm.enabled=0 metric that opts them out of APM billing.datadog.runtime_id resource attribute on OTLP trace and trace-metrics exports was a randomly generated value instead of the tracer's runtime id, preventing those exports from being correlated with the other signals emitted by the same process.aiohttp, httpx, requests, urllib3) where a strong reference to each response object was retained, preventing garbage collection of response bodies and causing linear RSS growth on sequential requests.DD_TRACE_OTEL_SEMANTICS_ENABLED=true, HTTP client spans now use the HTTP method as their resource name. This keeps the resource name distinct from the operation name and adds method-level information that better aligns with OpenTelemetry semantics; default resource names are unchanged.DD_TRACE_OTEL_CTX_ENABLED environment variable to false.runtime.python.* metrics stopped carrying the dd.internal.entity_id tag after the first flush, even when DD_ENTITY_ID was set. This prevented the Datadog Agent from attaching pod_name and kube_namespace to runtime metrics in Kubernetes.v0.5 trace API (DD_TRACE_API_VERSION=v0.5) drops traces when a single payload contains between 65,536 and 131,071 unique tag keys and values.deployment.environment.name attribute in OTEL_RESOURCE_ATTRIBUTES.mcp is importable but is not the MCP SDK. Patching is now skipped instead of raising.--lazy-apps and --skip-atexit can crash while workers shut down on Python 3.12 or later.get_final_message() or get_final_text().django.setup().ddtrace-run (or single-step instrumentation) could raise a PydanticSchemaGenerationError at import time for models that use forward references, when gevent is installed and the application runs on CPython 3.14 or later. The annotationlib module, which now owns the ForwardRef class on CPython 3.14, is preserved during module cloning so that pydantic resolves forward references correctly.AttributeError on google-adk >= 2.7.0 caused by the removal of __call_tool_live. Applications with LLM Observability enabled failed to start.google-adk < 2.7.0 where streaming tool call spans recorded no output.SystemError raised inside an instrumented request when an operation handled a container holding an object that cannot be hashed, such as the cookie objects Django builds in HttpResponse.set_cookie, and fixes the same error when a regular expression matched against a bytearray.NameError raised from application code that calls eval() when the wrapt C extension is unavailable and its pure-Python implementation is used, which caused Code Security to resolve the wrong caller scope.gunicorn or uWSGI) with no configuration, so every subsequent flag evaluation returned the caller-provided default with the PROVIDER_NOT_READY error code.ddtrace.openfeature, preventing the application from starting. Such values are now logged and the documented default is used instead.opentelemetry-exporter-otlp-proto-grpc 1.34.0 or later.DD_LOGS_OTEL_ENABLED=true) could cause the tracer to export its own internal log records, producing a growing volume of exported logs. The tracer's own log records and the OpenTelemetry exporter's log records are now excluded from OpenTelemetry logs export, while application logs continue to be collected and exported as before.pytorch.rank span could end too soon.requests calls pointed at the wrong span when the urllib3 integration was also enabled (for example via DD_TRACE_URLLIB3_ENABLED=true), causing those requests to be missing or show incorrect latency in downstream traces.command is passed as a keyword argument to execute or executemany.ModuleNotFoundError: No module named 'vllm.v1.engine.processor', silently disabling all vLLM tracing and metrics.pytest plugin performs once per test, which is most noticeable on large suites of fast tests.python-jsonpath in place of the vendored jsonpath-ng and ply, so ddtrace no longer ships ply, which is unmaintained and reported by dependency scanners as CVE-2025-56005. Existing redaction paths continue to work.tracer parameter of ddtrace.runtime.RuntimeMetrics.enable is deprecated and will be removed in a future version. It has no effect.DD_GOOGLE_CLOUD_PUBSUB_PROPAGATION_AS_SPAN_LINKS environment variable is deprecated and will be removed in a future release. Add google_cloud_pubsub to the DD_TRACE_PROPAGATION_AS_SPAN_LINKS list instead.Estimated end-of-life date, accurate to within three months: 05-2027
Estimated end-of-life date, accurate to within three months: 05-2027
See the support level definitions for more information.
discard boolean field to DD_TRACE_SAMPLING_RULES rules to fully drop trace chunks which were rejected during sampling. See the DD_TRACE_SAMPLING_RULES configuration docs for details.ot.th and ot.rv) to spans exported through OTLP when OTEL_TRACES_EXPORTER=otlp is configured, allowing downstream collectors to preserve and extrapolate sampling decisions.DD_RUNTIME_METRICS_ENABLED is true.DD_TRACE_OTEL_CTX_ENABLED=false or DD_TRACE_ASYNCIO_ENABLED=false.InvokeModel spans for Anthropic models, matching the Converse API.openai, litellm, and anthropic integrations.gen_ai.* attributes, so model, provider, application, conversation, and token usage can be searched, faceted, and monitored directly in APM.llm span.DD_CIVISIBILITY_DYNAMIC_ATR_ENABLED to enable dynamic Auto Test Retries budgets based on test duration, instead of the flat per-test retry limit. Optionally configure the five duration-based budgets with DD_CIVISIBILITY_DYNAMIC_ATR_BUCKETS (five comma-separated integers in [1, 20]), corresponding to the Early Flake Detection duration buckets.DD_TRACE_ANYIO_ENABLED=false.kafka in the DD_TRACE_PROPAGATION_AS_SPAN_LINKS environment variable. When enabled together with distributed tracing, the kafka.consume span starts a new trace and relates to every consumed message's producer via span links, instead of continuing a single producer's trace. This is disabled by default.DD_PROFILING_MEMORY_MEM_DOMAIN_ENABLED now defaults to true, so the heap profiler also tracks allocations in the Mem domain on Python 3.12+.build_py option --no-bundle-libddwaf, set through the [build_py] section of setup.cfg, which builds the package without a bundled libddwaf. The library is then loaded from the system at import time (libddwaf.so.2, then libddwaf.so, resolved by the dynamic linker, and required to be 2.x). Intended for Linux distribution packages; default builds bundle libddwaf as before.urllib3 requests include an Exploit Prevention frame in their traceback, which caused ordinary application errors to be misattributed to Datadog. Enabling urllib3 APM tracing, which is off by default, still contributes a frame of its own.DD_AGENTLESS_ENABLED is set. The errors-intake upload was incorrectly reusing the same endpoint configured for the telemetry crash-report path, sending it to the telemetry intake host instead of the dedicated errors-intake host.InvokeModel integration only captured the first content block of an Anthropic response, so output messages were empty or truncated when a response began with a tool_use or thinking block.DD_APM_TRACING_ENABLED=false discarded every trace, so AI Guard, App and API Protection, Code Security and Software Composition Analysis reported no spans while running in standalone mode. These traces are now sent, and APM billing stays opted out.google_genai responses was merged into the output message instead of being captured as a separate reasoning message.pytest plugin entered a code coverage collection context around every test even when code coverage was disabled, which reported coverage as active while no coverage was being collected and, on Python 3.12 and later, restarted sys.monitoring events once per test.TypeError.ValueError: I/O operation on closed file logging tracebacks at the end of pytest sessions when standard stream handlers point to streams closed by pytest. Tracer logs continue to reach healthy root handlers, including file output and log forwarding.PutEvents now propagates Data Streams Monitoring (DSM) context when DD_DATA_STREAMS_ENABLED is enabled.client_error, making them difficult to diagnose.webbrowser.open, or for calls to urllib.request.urlopen that pass the URL positionally or go through a custom opener installed with urllib.request.install_opener.urllib.request.urlopen and http.client requests include an internal Datadog frame in their traceback when Exploit Prevention is enabled, which caused ordinary application errors to be misattributed to the tracer.OTEL_* environment variables override ddtrace defaults instead of being treated as unset.http.client.request span unfinished and active, so it is never sent and every later span on that thread is parented to it.mysql.connector.Connect().rq.worker.perform_job span was not created when jobs were processed by rq.SimpleWorker on rq >= 2.7.WebDriver.get, WebDriver.quit and WebDriver.close calls made.tracer.configure(apm_tracing_disabled=True) did not apply the standalone sampling settings, so traces were sent without the one trace per minute limit and without the _dd.apm.enabled=0 metric that opts them out of APM billing.datadog.runtime_id resource attribute on OTLP trace and trace-metrics exports was a randomly generated value instead of the tracer's runtime id, preventing those exports from being correlated with the other signals emitted by the same process.mcp is importable but is not the MCP SDK. Patching is now skipped instead of raising.pytest plugin performs once per test, which is most noticeable on large suites of fast tests.The tracer parameter of ddtrace.runtime.RuntimeMetrics.enable is deprecated and will be removed in a future version. It has no effect.
Estimated end-of-life date, accurate to within three months: 05-2027
See the support level definitions for more information.
pydantic_ai integration now reports agent configuration using the shared agent manifest schema. framework, name, model, model_settings, instructions, system_prompts and tools keep their names and meaning. New keys are metadata, extra_instructions, capabilities, data_contracts, memory_policies, guardrails and agent_settings.tracer parameter of ddtrace.runtime.RuntimeMetrics.enable is deprecated and will be removed in a future version. It has no effect.DD_GOOGLE_CLOUD_PUBSUB_PROPAGATION_AS_SPAN_LINKS environment variable is deprecated and will be removed in a future release. Add google_cloud_pubsub to the DD_TRACE_PROPAGATION_AS_SPAN_LINKS list instead.httpx2>=2.0.0OTEL_TRACES_SPAN_METRICS_ENABLED=true to enable it.DD_TRACE_STATS_ADDITIONAL_TAGS, which adds configured span tags to native v0.6 trace stats and OTLP trace metrics.DD_TRACE_OTEL_CTX_ENABLED=false.DD_TRACE_OTEL_CTX_ENABLED is true (the default), propagated trace contexts are published to OpenTelemetry thread context records. This lets external readers retain trace and span correlation when no active span object is available.DD_TRACE_OTEL_CTX_ENABLED to false.DD_TRACE_PROPAGATION_AS_SPAN_LINKS environment variable, a comma-separated list of integration names (e.g. google_cloud_pubsub,kafka) for which propagated context is attached as span links instead of a parent-child relationship.DD_PROFILING_NATIVE_HEAP_ENABLED=true.DD_PROFILING_STACK_GC_ENABLED=true.httpx2 client.aiomysql, aiopg, psycopg async, and MySQL Connector/Python asyncio database drivers.image_parts on LLM span messages, so they can be rendered in LLM Observability. Note: images referenced by a URL or a file ID are not captured, and a very large inline image is replaced with a text placeholder.DD_APP_KEY on the application when a Datadog Agent is used.image_parts on LLM span messages, for both Chat Completions and the Responses API, so image inputs render in LLM Observability. This also applies to the OpenAI Agents SDK and to the LiteLLM integration, which share the same message extraction. Images referenced by a remote URL or file_id are not fetched and keep their existing text reference. A single inline image whose base64 payload exceeds 4 MiB is left as an [image omitted: too large] marker instead. Note that this budget is per image, not per request: several inline images (or an image alongside inline audio) can still take a span event past the 5 MB per-event limit, at which point the event's input and output are replaced with a placeholder. To stop image bytes from being recorded, register a span processor with LLMObs.register_processor and remove the image_parts key from the messages on span.input.aiohttp, httpx, requests, urllib3) where a strong reference to each response object was retained, preventing garbage collection of response bodies and causing linear RSS growth on sequential requests.DD_TRACE_OTEL_SEMANTICS_ENABLED=true, HTTP client spans now use the HTTP method as their resource name. This keeps the resource name distinct from the operation name and adds method-level information that better aligns with OpenTelemetry semantics; default resource names are unchanged.runtime.python.* metrics stopped carrying the dd.internal.entity_id tag after the first flush, even when DD_ENTITY_ID was set. This prevented the Datadog Agent from attaching pod_name and kube_namespace to runtime metrics in Kubernetes.deployment.environment.name attribute in OTEL_RESOURCE_ATTRIBUTES._DD_TRACE_WRITER_ADDITIONAL_HEADERS header values containing a whitespace (for example Bearer <token>) were incorrectly parsed, resulting in a truncated header value that caused LLM Observability spans to be dropped.claude-agent-sdk versions prior to 0.1.49, which do not report usage on the assistant message. Note that explicitly setting include_partial_messages=False on ClaudeAgentOptions opts out of this behavior and will result in inaccurate per-turn output token counts.agent_attribution missing or pointing at the wrong ancestor.DD_FEATURE_FLAGS_CONFIGURATION_SOURCE is set to agentless or remote_config.LLMObs.annotate(agent=...) are also now coerced to strings.[image]; remote URL and file_id references are unchanged.[image] marker, or as the screenshot's URL when it is a remote reference.DD_APPSEC_RULES to an empty value prevents AppSec rules from being updated through remote configuration.subprocess commands are passed as tuples.open() or pathlib.Path.open() are reported with a corrupted traceback when Exploit Prevention is enabled, showing a duplicated caller frame and omitting the frame that actually raised.AttributeError on google-adk >= 2.7.0 caused by the removal of __call_tool_live. Applications with LLM Observability enabled failed to start.google-adk < 2.7.0 where streaming tool call spans recorded no output.SystemError raised inside an instrumented request when an operation handled a container holding an object that cannot be hashed, such as the cookie objects Django builds in HttpResponse.set_cookie, and fixes the same error when a regular expression matched against a bytearray.NameError raised from application code that calls eval() when the wrapt C extension is unavailable and its pure-Python implementation is used, which caused Code Security to resolve the wrong caller scope.gunicorn or uWSGI) with no configuration, so every subsequent flag evaluation returned the caller-provided default with the PROVIDER_NOT_READY error code.ddtrace.openfeature, preventing the application from starting. Such values are now logged and the documented default is used instead.opentelemetry-exporter-otlp-proto-grpc 1.34.0 or later.command is passed as a keyword argument to execute or executemany.ModuleNotFoundError: No module named 'vllm.v1.engine.processor', silently disabling all vLLM tracing and metrics.tracing: DD_TRACE_OTEL_CTX_ENABLED now defaults to false until a Linux-native memory leak caused by setting it to true is fixed.
pytest-timeout thread mode and terminate a pytest-xdist worker.DD_APM_TRACING_ENABLED=false discarded every trace, so AI Guard, App and API Protection, Code Security and Software Composition Analysis reported no spans while running in standalone mode. These traces are now sent, and APM billing stays opted out.tracer.configure(apm_tracing_disabled=True) did not apply the standalone sampling settings, so traces were sent without the one trace per minute limit and without the _dd.apm.enabled=0 metric that opts them out of APM billing.Estimated end-of-life date, accurate to within three months: 05-2027
Estimated end-of-life date, accurate to within three months: 05-2027
See the support level definitions for more information.
google-adk: Fixes an AttributeError on google-adk >= 2.7.0 caused by the removal of __call_tool_live. Applications with LLM Observability enabled failed to start.
google-adk: Fixes an issue on google-adk < 2.7.0 where streaming tool call spans recorded no output.
vllm: Fixes an issue where the integration failed to instrument vLLM >= 0.14.0 with ModuleNotFoundError: No module named 'vllm.v1.engine.processor', silently disabling all vLLM tracing and metrics.
CI Visibility: Fixes an issue where test sessions running with pytest-xdist on shallow Git checkouts can spend excessive time unshallowing the repository and may fail or time out during collection.
LLM Observability: Fixes an issue where reasoning content from streamed google_genai responses was merged into the output message instead of being captured as a separate reasoning message.
LLM Observability: Fixes an issue where the agentless exporter dropped traces due to non-JSON-serializable objects being stored on a span. All LLM Observability span data is now made JSON-serializable before submission. Non-string span tag keys, span metric keys, and agent versions supplied through LLMObs.annotate(agent=...) are also coerced to strings.
MCP: Resolves an error logged when a module named mcp is importable but is not the MCP SDK. Patching is now skipped instead of raising an error.
Profiling: Fast memory copy is now automatically disabled when Python is running as an embedded interpreter.
As a result, ddtrace no longer ships ply , which is unmaintained and reported by dependency scanners as CVE-2025-56005 . Existing redaction paths cont…
Estimated end-of-life date, accurate to within three months: 05-2027
See the support level definitions for more information.
AI Guard:
CI Visibility:
ci.pipeline.display_name tag for the Buildkite provider, populated from the BUILDKITE_PIPELINE_NAME environment variable.Data Streams Monitoring:
tags argument to ddtrace.data_streams.set_produce_checkpoint and ddtrace.data_streams.set_consume_checkpoint. Pass a list of "key:value" strings to attach additional edge tags to a manual checkpoint alongside the automatically set type, topic, and direction tags. For example: set_produce_checkpoint("eventbridge", detail_type, carrier_set, tags=["exchange:my-bus"]). Existing calls that do not pass tags are unaffected.LLM Observability:
LLMObs.list_experiments() to query experiments by logical name and metadata containment, such as {"tags": ["git.commit.sha:abc123"]}. It returns run status and aggregate data—including evaluations, token costs, and error rates—so CI/CD workflows can compare a run against a baseline commit and gate on the result.version when starting an agent span, either with LLMObs.agent(name="my-agent", version="v3") or with the @agent(version="v3") decorator. For agents created by an auto-instrumented framework, use LLMObs.annotation_context(agent={"version": "v3"}). This sets the agent_version tag on agent spans. The agent argument accepts a plain dictionary or the new ddtrace.llmobs.Agent TypedDict class.LLMObs.submit_feedback() for submitting end-user feedback associated with a span, trace, session, or customer-defined feedback join key.OpenFeature:
DD_API_KEY. Set DD_FEATURE_FLAGS_CONFIGURATION_SOURCE=remote_config to continue loading configuration through the Datadog Agent's Remote Configuration.OpenTelemetry:
http/protobuf protocol—the default—and http/json, selected through the OTEL_EXPORTER_OTLP_TRACES_PROTOCOL environment variable.Tracing:
Span.remove_tag(key) and Span.remove_metric(key) for removing a previously set tag or metric from a span. Both methods are no-ops if the key is not set.AAP:
Botocore:
invoke_model_with_response_stream calls reported all token usage metrics as 0. Token counts are now extracted from the streamed response and set on the span.CI Visibility:
site-packages or dist-packages directories.pass instead of fail when pytest exited with a non-OK, non-NO_TESTS_COLLECTED exit code, such as for collection errors, interruptions, or internal errors.Code Origin for Spans:
Code Security:
Django:
django.setup().Gevent:
ddtrace-run or single-step instrumentation could raise a PydanticSchemaGenerationError at import time for models that use forward references when gevent is installed and the application runs on CPython 3.14 or later. The annotationlib module, which now owns the ForwardRef class on CPython 3.14, is preserved during module cloning so that Pydantic resolves forward references correctly.Kombu:
OpenTelemetry Logs:
DD_LOGS_OTEL_ENABLED=true could cause the tracer to export its own internal log records, producing a growing volume of exported logs. The tracer's own log records and the OpenTelemetry exporter's log records are now excluded, while application logs continue to be collected and exported.Profiling:
--lazy-apps and --skip-atexit could crash while workers shut down on Python 3.12 or later.PyTorch:
pytorch.rank span could end too soon.Requests:
requests calls pointed to the wrong span when the urllib3 integration was also enabled—for example, through DD_TRACE_URLLIB3_ENABLED=true—causing those requests to be missing or to show incorrect latency in downstream traces.Snowflake:
command was passed as a keyword argument to execute or executemany.Tracing:
DD_TRACE_OTEL_CTX_ENABLED environment variable to false._dd.svc_src was incorrectly set to either m or the service name.Botocore:
python-jsonpath instead of the vendored jsonpath-ng and ply. As a result, ddtrace no longer ships ply, which is unmaintained and reported by dependency scanners as CVE-2025-56005. Existing redaction paths continue to work.code origin for spans: prevent view or traced functions from raising an exception in situations where their number is large.
tracing: avoid reentrant SpanData borrow panic.
tracing: avoid reentrant SpanData borrow panic.
snowflake: Fixes an issue where traced cursor calls fail when command is passed as a keyword argument to execute or executemany.
command is passed as a keyword argument to execute or executemany.<!-- -->
…and flagged by dependency scanners for CVE-2025-56005 . Existing redaction paths continue to work.
ci.pipeline.display_name tag for the Buildkite provider, populated from the BUILDKITE_PIPELINE_NAME environment variable.tags argument to ddtrace.data_streams.set_produce_checkpoint and ddtrace.data_streams.set_consume_checkpoint. Pass a list of "key:value" strings to attach additional edge tags to a manual checkpoint alongside the automatically set type, topic, and direction tags—for example, set_produce_checkpoint("eventbridge", detail_type, carrier_set, tags=["exchange:my-bus"]). Existing calls that do not pass tags are unaffected.LLMObs.list_experiments() to query experiments by logical name and metadata containment (for example, {"tags": ["git.commit.sha:abc123"]}). It returns run status and aggregate data (evaluations, token costs, and error rates), allowing CI/CD workflows to compare a run against a baseline commit and gate on the result.version when starting an agent span, either with LLMObs.agent(name="my-agent", version="v3") or with the @agent(version="v3") decorator. For agents created by an auto-instrumented framework, use LLMObs.annotation_context(agent={"version": "v3"}). This sets the agent_version tag on agent spans. The agent argument accepts either a plain dictionary or the new ddtrace.llmobs.Agent TypedDict class.LLMObs.submit_feedback() for submitting end-user feedback associated with a span, trace, session, or customer-defined feedback join key.DD_API_KEY; set DD_FEATURE_FLAGS_CONFIGURATION_SOURCE=remote_config to continue loading configuration through the Datadog Agent's Remote Configuration.http/protobuf protocol (the default) and http/json, selected via the OTEL_EXPORTER_OTLP_TRACES_PROTOCOL environment variable.Span.remove_tag(key) and Span.remove_metric(key) for removing a previously set tag or metric from a span. Both are no-ops if the key is not set.invoke_model_with_response_stream calls reported all token usage metrics as 0. Token counts are now extracted from the streamed response and set on the span.site-packages or dist-packages directories.pass instead of fail when pytest exited with a non-OK, non-NO_TESTS_COLLECTED exit code (for example, because of collection errors, interruptions, or internal errors).django.setup().ddtrace-run (or single-step instrumentation) could raise a PydanticSchemaGenerationError at import time for models that use forward references when gevent is installed and the application runs on CPython 3.14 or later. The annotationlib module, which now owns the ForwardRef class on CPython 3.14, is preserved during module cloning so that pydantic resolves forward references correctly.DD_LOGS_OTEL_ENABLED=true) could cause the tracer to export its own internal log records, producing a growing volume of exported logs. The tracer's own log records and the OpenTelemetry exporter's log records are now excluded from OpenTelemetry logs export, while application logs continue to be collected and exported as before.--lazy-apps and --skip-atexit could crash while workers shut down on Python 3.12 or later.pytorch.rank span could end too soon.requests calls pointed to the wrong span when the urllib3 integration was also enabled (for example, via DD_TRACE_URLLIB3_ENABLED=true), causing those requests to be missing or to show incorrect latency in downstream traces.DD_TRACE_OTEL_CTX_ENABLED environment variable to false._dd.svc_src was incorrectly set to either "m" or the service name.python-jsonpath in place of the vendored jsonpath-ng and ply, so ddtrace no longer ships ply, which is unmaintained and flagged by dependency scanners for CVE-2025-56005. Existing redaction paths continue to work.tracing: DD_TRACE_OTEL_CTX_ENABLED now defaults to false until a Linux-native memory leak caused by setting it to true is fixed.
CI visibility: Fixes an issue where test sessions running with pytest-xdist on shallow Git checkouts can spend excessive time unshallowing the repository and may fail or time out during collection.
OpenTelemetry logs: Fixes an issue where enabling OpenTelemetry logs (DD_LOGS_OTEL_ENABLED=true, or OTEL_SDK_DISABLED=false) could make the process stop responding and export an ever-growing volume of its own log records. Application logs are still collected and exported.
code origin for spans: prevent view or traced functions from raising an exception in situations where their number is large.
code origin for spans: prevent view or traced functions from raising an exception in situations where their number is large.
Profiling: Fast memory copy is now automatically disabled when Python is running as an embedded interpreter.
snowflake: Fixes an issue where traced cursor calls fail when command is passed as a keyword argument to execute or executemany.
command is passed as a keyword argument to execute or executemany.Deprecates the ml_app argument and DD_LLMOBS_ML_APP environment variable, which will be removed in the next major version of ddtrace. Use agent_servic…
ml_app argument and DD_LLMOBS_ML_APP environment variable, which will be removed in the next major version of ddtrace. Use agent_service and DD_SERVICE instead to group traces belonging to a root agentic service (i.e. previously known as ML application).ddtrace.appsec.ai_guard package to the top-level ddtrace.aiguard package. Importing from ddtrace.appsec.ai_guard is deprecated, now emits a ddtrace.DDTraceDeprecationWarning, and will be removed in 5.0.0. As an alternative, update imports such as from ddtrace.appsec.ai_guard import AIGuardClient to from ddtrace.aiguard import AIGuardClient. The Strands Agents classes are the exception: import them from ddtrace.aiguard.integrations.strands (AIGuardStrandsPlugin and AIGuardStrandsHookProvider are not re-exported from the top-level package).ddtrace/contrib/internal/pytest) is deprecated and will be removed in ddtrace 5.0.0. This plugin is active only when DD_PYTEST_USE_NEW_PLUGIN is explicitly set to false or 0. The new pytest plugin (ddtrace/testing/internal/pytest) is already the default. To migrate, remove the DD_PYTEST_USE_NEW_PLUGIN environment variable from your configuration.label parameter of LLMObs.get_prompt() and LLMObs.refresh_prompt() is deprecated. Set DD_ENV instead; the prompt version is resolved for that environment.ManagedPrompt.label property is deprecated. #19399DD_TRACE_OTEL_SEMANTICS_ENABLED to opt in to OpenTelemetry trace semantics for spans created through OpenTelemetry APIs and for OpenTelemetry Protocol (OTLP) trace export.DD_AI_GUARD_OPENAI_ENABLED switch to the Anthropic and LangChain auto-instrumentations: DD_AI_GUARD_ANTHROPIC_ENABLED and DD_AI_GUARD_LANGCHAIN_ENABLED (both true by default). Set either to false to disable AI Guard for that specific provider/framework without affecting other integrations or requiring a tracer version rollback.DD_APPSEC_AGENTIC_ONBOARDING environment variable. Its value is reported verbatim in configuration telemetry, letting Datadog record that a service was onboarded through the agentic onboarding solution.DD_BOTOCORE_BEDROCK_RESOLVE_INFERENCE_PROFILE (default false). When enabled, Bedrock calls using an application-inference-profile ARN as modelId resolve the underlying foundation model, so the span reports the real model and cost populates. Requires DD_LLMOBS_ENABLED=true and bedrock:GetInferenceProfile IAM permission.Session object on LLM Observability spans.agent_service as a preferred alias for the ml_app argument across SDK methods (LLMObs.enable(...), span constructors, decorators, submit_evaluation(...), get_spans(...), and publish_evaluator(...)). When provided, agent_service takes precedence over the deprecated ml_app argument and over service. When neither agent_service nor ml_app is set, application identity continues to resolve from a propagated parent value, then DD_LLMOBS_ML_APP, then DD_SERVICE.LLMObs.get_prompt() and rendered with .format() to the resulting LLM span.LLMObs.get_prompt() now resolves environment-scoped prompt versions and supports targeting for A/B testing. Set DD_ENV to select the version for that environment. The signature is now keyword-only after prompt_id and accepts targeting_key (sticky bucketing) and **attributes (targeting rules). #18127version argument to LLMObs.get_prompt() for retrieving an exact prompt version. #19050LLMObs.annotate now accept an optional image_parts field, a list of images each with a mime_type and either inline base64 content or an attachment_key. This lets images be collected and rendered in LLM Observability.client.realtime.connect(...), both sync and async). Each conversation turn is captured as an llm span with the user and assistant transcripts, audio, token usage, and any tool calls (function and MCP), grouped per connection so the whole conversation is traced end to end. Realtime instrumentation can be disabled on its own, without affecting the rest of the OpenAI integration, by setting DD_OPENAI_REALTIME_ENABLED=false.LLMObs for creating, updating, deleting, and listing prompts and prompt versions in the prompt registry. #18186env_ids argument to prompt creation and version update methods, allowing prompt versions to be deployed to feature-flag environments.session_id now propagates to following spans in a trace. If session_id is set explicitly on a span, this will take precedence over the inherited or propagated value.mistralai client.mistralai client.DD_CODE_COVERAGE_FLAGS to group and filter uploaded code coverage reports.DD_EXPERIMENTAL_FLAGGING_PROVIDER_SPAN_ENRICHMENT_ENABLED=true, the provider attaches feature-flag evaluation metadata (ffe_flags_enc, ffe_subjects_enc, and ffe_runtime_defaults) to the root APM span so traces and errors can be filtered by active flag variant. This is off by default and, when disabled, adds no per-span overhead.flagevaluation track), lighting up server-side flag-evaluation observability. The existing feature_flag.evaluations OpenTelemetry metric behavior is unchanged. The new emission is gated by DD_FLAGGING_EVALUATION_COUNTS_ENABLED (default enabled).get_final_message() or get_final_text().yaml/_yaml loaded during module cleanup, fixing an issue that broke PyYAML consumers such as Airflow.DD_PROFILING_MAX_FRAMES by clamping it before stack collection so samples stay within the backend's 600-location limit.SIGSEGV/SIGBUS handler that the profiler cannot safely chain with (e.g. CUDA, PyTorch, abseil via vLLM/gRPC; see sampler.cpp). The sampler upgrades to the faster fault-recovery copy if it still owns both fault handlers afterwards. Otherwise, it permanently falls back to the syscall-based copy.v0.5 trace API (DD_TRACE_API_VERSION=v0.5) drops traces when a single payload contains between 65,536 and 131,071 unique tag keys and values.@tracer.wrap on functions created at run time has been fixed.DD_TRACE_OTEL_CTX_ENABLED environment variable to false.TypeError raised during streamed tool-call reconstruction when an OpenAI-compatible backend (e.g. DashScope/Qwen) emits a tool-call delta with function.arguments or custom.input set to None instead of "".AssistantMessage chunks that share a message_id, the integration emitted one llm span per chunk and counted the turn's token usage multiple times. Chunks sharing a message_id are now merged into a single llm span with the usage counted once. This applies to claude-agent-sdk 0.1.51 and later, which expose message_id; earlier versions retain the previous behavior.google_adk integration dropped an agent span (logging missing span kind in span context) and orphaned its child spans when a Gemini response part could not be parsed. The span kind is now set before input/output extraction, so a parsing failure degrades to empty input/output instead of dropping the span.google_adk integrations rendered unhandled Gemini response parts (such as inline_data, file_data, empty parts, or thought-signature parts) as a confusing Unsupported file type placeholder. These parts now produce a concise summary or empty content. - IAST: A crash occurring when using uvloop has been fixed.DD_LLMOBS_OVERRIDE_ORIGIN had no effect when the resolved export mode routed events alongside the APM trace (APM_AGENTLESS or APM_AGENT). Setting this variable now always routes LLM Observability events directly through the LLM Observability writer so the override is respected.model_settings contained non-JSON-serializable provider sentinel values.TypeError: 'tuple' object does not support item assignment raised when passing headers as a sequence of (key, value) tuples (aiohttp's LooseHeaders type) to request methods such as get or post.aws.durable.replayed value and an aws.durable.operation_attempt one higher than its original execution.No extra information available when viewing details about why a test was skipped by Intelligent Test Runner, due to missing ITR correlation ID on test or suite events.DD_CIVISIBILITY_CODE_COVERAGE_REPORT_UPLOAD_ENABLED=true) alongside Test Impact Analysis caused per-test coverage data to be silently empty. The two features can now be enabled together.resource remaining as the raw URL path when a worker is killed before start_response runs (e.g. gunicorn --timeout). The route pattern is now populated from request.url_rule during preprocess_request.% string formatting handled tainted text containing literal IAST evidence marker delimiters.copy_ranges_from_strings and copy_and_shift_ranges_from_strings to the active request slot, matching the scoped taint read path. Previously these copy helpers resolved the taint map by scanning all request slots, so a concurrent or still-open request could capture the derived taint and the current request would miss the transformed tainted input.llm-kind LLM Observability spans for a single call when the langchain and google_genai integrations were both enabled, causing downstream cost and token metrics to be double-counted.openfeature-sdk 0.10.0 or later.break-ing out of astream() or stream() when an __interrupt__ is received in a human-in-the-loop workflow) left the graph's root span unfinished, so the trace failed to assemble and did not appear in Datadog. The root span is now finished when the stream is torn down, so interrupted runs are traced. - LLM Observability: This fix resolves an issue where annotating a span with a metric key containing a dot (e.g. anomaly.query_count) prevented the span from being ingested. Dots in metric keys are now replaced with underscores.openai-agents >= 0.8.0. The integration now instruments those versions, restoring agent manifest capture.DD_TRACE_CLOUD_REQUEST_PAYLOAD_TAGGING or DD_TRACE_CLOUD_RESPONSE_PAYLOAD_TAGGING is enabled.LLM Observability: Resolves a TypeError raised during streamed tool-call reconstruction when an OpenAI-compatible backend (e.g. DashScope/Qwen) e mits
TypeError raised during streamed tool-call reconstruction when an OpenAI-compatible backend (e.g. DashScope/Qwen) e
mits a tool-call delta with function.arguments or custom.input set to None instead of "".<!-- -->
Nothing published for this version
Deprecates the ml_app argument and DD_LLMOBS_ML_APP environment variable, which will be removed in the next major version of ddtrace. Use agent_servic…
ml_app argument and DD_LLMOBS_ML_APP environment variable, which will be removed in the next major version of ddtrace. Use agent_service and DD_SERVICE instead to group traces belonging to a root agentic service (i.e. previously known as ML application).AI Guard:
ddtrace.appsec.ai_guard package to the top-level ddtrace.aiguard package. Importing from ddtrace.appsec.ai_guard is deprecated, now emits a ddtrace.DDTraceDeprecationWarning, and will be removed in 5.0.0. As an alternative, update imports such as from ddtrace.appsec.ai_guard import AIGuardClient to from ddtrace.aiguard import AIGuardClient. The Strands Agents classes are the exception: import them from ddtrace.aiguard.integrations.strands (AIGuardStrandsPlugin and AIGuardStrandsHookProvider are not re-exported from the top-level package).pytest:
ddtrace/contrib/internal/pytest) is deprecated and will be removed in ddtrace 5.0.0. This plugin is active only when DD_PYTEST_USE_NEW_PLUGIN is explicitly set to false or 0. The new pytest plugin (ddtrace/testing/internal/pytest) is already the default. To migrate, remove the DD_PYTEST_USE_NEW_PLUGIN environment variable from your configuration.LLM Observability:
label parameter of LLMObs.get_prompt() and LLMObs.refresh_prompt() is deprecated. Set DD_ENV instead; the prompt version is resolved for that environment.Tracing:
DD_TRACE_OTEL_SEMANTICS_ENABLED to opt in to OpenTelemetry trace semantics for spans created through OpenTelemetry APIs and for OpenTelemetry Protocol (OTLP) trace export.AI Guard:
DD_AI_GUARD_OPENAI_ENABLED switch to the Anthropic and LangChain auto-instrumentations: DD_AI_GUARD_ANTHROPIC_ENABLED and DD_AI_GUARD_LANGCHAIN_ENABLED (both true by default). Set either to false to disable AI Guard for that specific provider/framework without affecting other integrations or requiring a tracer version rollback.AAP:
DD_APPSEC_AGENTIC_ONBOARDING environment variable. Its value is reported verbatim in configuration telemetry, letting Datadog record that a service was onboarded through the agentic onboarding solution.LLM Observability:
DD_BOTOCORE_BEDROCK_RESOLVE_INFERENCE_PROFILE (default false). When enabled, Bedrock calls using an application-inference-profile ARN as modelId resolve the underlying foundation model, so the span reports the real model and cost populates. Requires DD_LLMOBS_ENABLED=true and bedrock:GetInferenceProfile IAM permission.Session object on LLM Observability spans.agent_service as a preferred alias for the ml_app argument across SDK methods (LLMObs.enable(...), span constructors, decorators, submit_evaluation(...), get_spans(...), and publish_evaluator(...)). When provided, agent_service takes precedence over the deprecated ml_app argument and over service. When neither agent_service nor ml_app is set, application identity continues to resolve from a propagated parent value, then DD_LLMOBS_ML_APP, then DD_SERVICE.LLMObs.get_prompt() and rendered with .format() to the resulting LLM span.LLMObs.get_prompt() now resolves environment-scoped prompt versions and supports targeting for A/B testing. Set DD_ENV to select the version for that environment. The signature is now keyword-only after prompt_id and accepts targeting_key (sticky bucketing) and **attributes (targeting rules). #18127version argument to LLMObs.get_prompt() for retrieving an exact prompt version. #19050LLMObs.annotate now accept an optional image_parts field, a list of images each with a mime_type and either inline base64 content or an attachment_key. This lets images be collected and rendered in LLM Observability.client.realtime.connect(...), both sync and async). Each conversation turn is captured as an llm span with the user and assistant transcripts, audio, token usage, and any tool calls (function and MCP), grouped per connection so the whole conversation is traced end to end. Realtime instrumentation can be disabled on its own, without affecting the rest of the OpenAI integration, by setting DD_OPENAI_REALTIME_ENABLED=false.LLMObs for creating, updating, deleting, and listing prompts and prompt versions in the prompt registry. #18186env_ids argument to prompt creation and version update methods, allowing prompt versions to be deployed to feature-flag environments.session_id now propagates to following spans in a trace. If session_id is set explicitly on a span, this will take precedence over the inherited or propagated value.mistralai client.CI visibility:
DD_CODE_COVERAGE_FLAGS to group and filter uploaded code coverage reports.Crashtracking:
Stats computation:
Profiling:
openfeature:
DD_EXPERIMENTAL_FLAGGING_PROVIDER_SPAN_ENRICHMENT_ENABLED=true, the provider attaches feature-flag evaluation metadata (ffe_flags_enc, ffe_subjects_enc, and ffe_runtime_defaults) to the root APM span so traces and errors can be filtered by active flag variant. This is off by default and, when disabled, adds no per-span overhead.flagevaluation track), lighting up server-side flag-evaluation observability. The existing feature_flag.evaluations OpenTelemetry metric behavior is unchanged. The new emission is gated by DD_FLAGGING_EVALUATION_COUNTS_ENABLED (default enabled).mistralai:
mistralai client.Profiling:
aiohttp:
TypeError: 'tuple' object does not support item assignment raised when passing headers as a sequence of (key, value) tuples (aiohttp's LooseHeaders type) to request methods such as get or post.aws_durable_execution_sdk_python:
aws.durable.replayed value and an aws.durable.operation_attempt one higher than its original execution.CI Visibility:
No extra information available when viewing details about why a test was skipped by Intelligent Test Runner, due to missing ITR correlation ID on test or suite events.DD_CIVISIBILITY_CODE_COVERAGE_REPORT_UPLOAD_ENABLED=true) alongside Test Impact Analysis caused per-test coverage data to be silently empty. The two features can now be enabled together.LLM Observability:
AssistantMessage chunks that share a message_id, the integration emitted one llm span per chunk and counted the turn's token usage multiple times. Chunks sharing a message_id are now merged into a single llm span with the usage counted once. This applies to claude-agent-sdk 0.1.51 and later, which expose message_id; earlier versions retain the previous behavior.google_adk integration dropped an agent span (logging missing span kind in span context) and orphaned its child spans when a Gemini response part could not be parsed. The span kind is now set before input/output extraction, so a parsing failure degrades to empty input/output instead of dropping the span.google_adk integrations rendered unhandled Gemini response parts (such as inline_data, file_data, empty parts, or thought-signature parts) as a confusing Unsupported file type placeholder. These parts now produce a concise summary or empty content.LLMObs.submit_evaluation() incorrectly tagging evaluations with source:otel based on the global DD_TRACE_OTEL_ENABLED flag. The tag is now controlled by the is_otel field on the ExportedLLMObsSpan dictionary: True for OTel gen.ai instrumented spans, False for Datadog LLMObs spans. LLMObs.export_span() sets this field automatically.model_settings contained non-JSON-serializable provider sentinel values.Tracing:
flask:
resource remaining as the raw URL path when a worker is killed before start_response runs (e.g. gunicorn --timeout). The route pattern is now populated from request.url_rule during preprocess_request.Code Security (IAST):
% string formatting handled tainted text containing literal IAST evidence marker delimiters.copy_ranges_from_strings and copy_and_shift_ranges_from_strings to the active request slot, matching the scoped taint read path. Previously these copy helpers resolved the taint map by scanning all request slots, so a concurrent or still-open request could capture the derived taint and the current request would miss the transformed tainted input.uvloop has been fixed.langchain:
llm-kind LLM Observability spans for a single call when the langchain and google_genai integrations were both enabled, causing downstream cost and token metrics to be double-counted.openfeature:
openfeature-sdk 0.10.0 or later.AAP:
Fixed an issue that could have caused the internal symbol DB component to restart in nested fork child processes, potentially leading to high memory usage.
Fix crashes in uwsgi worker when exiting from SIGTERM.
openai_agents:
openai-agents >= 0.8.0. The integration now instruments those versions, restoring agent manifest capture.profiling:
DD_PROFILING_MAX_FRAMES by clamping it before stack collection so samples stay within the backend's 600-location limit.DD_TRACE_CLOUD_REQUEST_PAYLOAD_TAGGING or DD_TRACE_CLOUD_RESPONSE_PAYLOAD_TAGGING is enabled.anthropic: Fixes an issue where spans were not exported when streamed responses were consumed using get_final_message() or get_final_text().
get_final_message() or get_final_text().command is passed as a keyword argument to execute or executemany.v0.5 trace API (DD_TRACE_API_VERSION=v0.5) drops traces when a single payload contains between 65,536 and 131,071 unique tag keys and values.@tracer.wrap on functions created at run time has been fixed.aws_durable_execution_sdk_python: This fix resolves an issue where a replayed durable operation that had failed permanently reported an incorrect aws.
aws.durable.replayed value and an aws.durable.operation_attempt one higher than its original execution.CI Visibility: This fix resolves an issue where the Datadog UI showed No extra information available when viewing details about why a test was skipped
CI Visibility: This fix resolves an issue where the Datadog UI showed No extra information available when viewing details about why a test was
skipped by Intelligent Test Runner, due to missing ITR correlation ID on test or suite events.
Code Security (IAST): fixes an issue where taint tracking could abort the Python process when old-style % string formatting handled tainted
text containing literal IAST evidence marker delimiters.
Code Security (IAST): This fix scopes copy_ranges_from_strings and copy_and_shift_ranges_from_strings to the active request slot, matching
the scoped taint read path. Previously these copy helpers resolved the taint map by scanning all request slots, so a concurrent or still-open
request could capture the derived taint and the current request would miss the transformed tainted input.
LLM Observability: Resolves a TypeError raised during streamed tool-call reconstruction when an OpenAI-compatible backend (e.g.
DashScope/Qwen) emits a tool-call delta with function.arguments or custom.input set to None instead of "".
Fix crashes in uwsgi worker when exiting from SIGTERM.
IAST: A crash occurring when using uvloop has been fixed.
LLM Observability: Resolves an issue where the OpenAI and LiteLLM integrations reported zero cost for OpenRouter requests using a
bring-your-own-key setup.
Fixed an issue where sensitive content could be left unredacted in vulnerability reports when a sensitive value overlapped a tainted range.
v0.4 trace API version (v0.5 cannot carry the LLMObs span data). Setting DD_TRACE_API_VERSION=v0.5 with LLMObs enabled logs a warning and downgrades to v0.4. No user action is required.<!-- -->
AI Guard:
create_agent.DD_AI_GUARD_OPENAI_ENABLED environment variable (true by default) as a per-provider kill switch. Set it to false to disable AI Guard auto-instrumentation of the OpenAI SDK without affecting other providers or requiring a tracer version rollback.DD_AI_GUARD_ANALYZE_STREAM_RESPONSES_ENABLED (default off); streams are buffered and evaluated before any chunk is delivered, raising AIGuardAbortError on a block.DD_APM_TRACING_ENABLED=false), traces produced by AI Guard are still sent to Datadog, kept with USER_KEEP sampling priority and the AI Guard decision maker so that they can be attributed to AI Guard, while APM host billing is not triggered.DD_AI_GUARD_ANALYZE_STREAM_RESPONSES_ENABLED=true (default false), all streamed chunks are buffered, the full assistant response is evaluated by AI Guard, and chunks are replayed to the caller only after an ALLOW verdict. On DENY/ABORT, AIGuardAbortError is raised and zero chunks are delivered. When the flag is off (the default), streaming behavior is unchanged.AAP:
_dd.appsec.normalized_route) for Tornado, following RFC-1103 and the existing FastAPI, Starlette, Django, and Flask implementations. The tag is emitted on every request span that already carries http.route when the API Security feature is active. Tornado routes using named capturing groups ((?P<name>...)) produce named parameters (e.g. /users/{id}); positional groups produce auto-numbered placeholders (e.g. {param1}). Optional trailing-slash patterns (/?) are treated as not declaring a trailing slash per RFC-1103 rule 1. #18398AppSec:
chat.completions, completions, or responses call is made during a web request, the WAF address server.business_logic.llm.event is emitted with the provider and model name. This enables WAF rules to tag the trace with appsec.events.llm.call.provider and appsec.events.llm.call.model, surfacing LLM-backed endpoints in the API Endpoint Catalog. Supports both sync and async OpenAI clients, Azure OpenAI (engine kwarg), and streaming. Only the first LLM call per request is recorded (endpoint-level detection, not per-invocation).aws_durable_execution_sdk_python:
aws.durable.operation_attempt tag to aws.durable.step and aws.durable.wait_for_condition spans. The tag is 0 for the original attempt and 1, 2, etc. for each subsequent retry.ray_serve:
ray[serve]>=2.47.1. When Ray is instrumented, ddtrace now traces Serve HTTP and gRPC requests, deployment routing, deployment method execution, and calls made with DeploymentHandle.remote(), with distributed trace context propagated across Serve deployments. See the Ray documentation for more information. #18076LLM Observability:
LLMObs.annotate now accept an optional audio_parts field, a list of audio segments each with a mime_type and either inline base64 content or an attachment_key. This lets audio (for example, speech-to-text and text-to-speech payloads) be collected and rendered in LLM Observability.sample_rate argument to LLMObs.enable() to configure the proportion of LLMObs traces to sample (between 0.0 and 1.0) in code. This takes precedence over the DD_LLMOBS_SAMPLE_RATE environment variable.SyncExperiment.rerun_evaluators() to re-run evaluators on the stored task outputs from a previous experiment run without re-executing the task function. Reads from experiment.result (set by run() or pull()) and returns a new SyncExperiment with fresh evaluations while preserving original span IDs, trace IDs, inputs, and outputs via replay span copies linked back to the originals via parent_experiment_span_id. The missing_task_strategy parameter controls behavior when prior rows contain errors: "raise" (default) raises immediately, "skip" omits the row, and "retry" re-executes the task for that row.gpt-audio, gpt-audio-mini, gpt-4o-audio-preview) as audio_parts on LLM span messages. Note: audio in streamed responses (stream=True) is not captured yet.DD_LLMOBS_SAMPLE_RATE, which controls the proportion of LLM Observability spans (between 0.0 and 1.0, defaults to 1.0) that are retained. This does not affect APM span retention or the accuracy of LLM Observability metrics such as token usage, and cost.metadata to the data exposed to the span processor registered with LLMObs.enable(span_processor=...) or LLMObs.register_processor(...). Previously only input and output were exposed; the processor can now read and modify metadata to redact or remove sensitive values that integrations record there.mistralai client chat and embedding calls.profiling:
mistralai:
mistralai client chat and embedding calls.pytorch:
pytorch.rank lifetime span for PyTorch distributed training. The span opens at init_process_group and closes at destroy_process_group or process exit. Tags include rank, world_size, framework (DDP / FSDP / DeepSpeed), launcher, torch.distributed.backend, and training_job_id (resolved from launcher environment variables). When running under Ray Train, ray.train.run_name, ray.submission_id, and ray.metadata.* are also applied. Enable with DD_PATCH_MODULES=pytorch:true.CI Visibility:
DD_TEST_MANAGEMENT_ATTEMPT_TO_FIX_RETRIES environment variable is now respected by the ddtrace.testing internal pytest plugin. When set to a valid integer, it overrides the retry count returned by the Datadog API for "attempt to fix" tests.Library Injection:
DD_INJECT_EXPERIMENTAL_OVERRIDE_USER_DDTRACE=true can be added to the environment.LLM Observability:
us3, us5, ap1, and ap2 Datadog sites. This affected customers on these sites when no Datadog Agent was running or agentless export was explicitly enabled (DD_LLMOBS_AGENTLESS_ENABLED=1).openai integration where streamed chat completion spans under-reported output_tokens and total_tokens for OpenAI-compatible providers that emit a cumulative usage object on every streamed chunk._DD_LLMOBS_WRITER_TIMEOUT, default 5 seconds) and instead used the 2 second connection default, causing intermittent TimeoutError and dropped span events on high-latency connections to the agent or intake.model_name with the request deployment name (e.g. my-deployment) instead of the canonical model returned by the API (e.g. gpt-4o-2024-08-06). The integration now prefers the response model for the azure and azure_text providers.metadata containing non-string keys (e.g. int/float/bool) could be dropped during ingestion. Metadata keys are now stringified before encoding.Omit/NotGiven sentinel values) as noise in LLM span metadata.anomaly.query_count) prevented the span from being ingested. Dots in metric keys are now replaced with underscores.DD_LLMOBS_OVERRIDE_ORIGIN had no effect when the resolved export mode routed events alongside the APM trace (APM_AGENTLESS or APM_AGENT). Setting this variable now always routes LLM Observability events directly through the LLM Observability writer so the override is respected.LLMObs.annotate(tags=...) could cause spans to be dropped during ingestion.AI Guard:
document content blocks, which could cause evaluation to be skipped for prompts whose only content was a document. Document text is now scanned and binary documents leave a placeholder so evaluation still runs.block dynamic parameter, supplied in the incoming request body, to disable the operator-configured blocking policy and let a denied request proceed in monitor mode. Request-supplied parameters can now only strengthen blocking (turning monitor mode into blocking); they can no longer weaken an enabled blocking policy.langchain 1.x:
aiokafka:
kafka.partition and kafka.message_offset span tags being incorrectly set to -1. Consumer spans no longer misreport 0 (e.g. single-partition topics or the first message in a partition) due to falsy evaluation. Producer spans and any span where the value is genuinely unknown now omit these tags entirely rather than emitting a misleading -1.anthropic:
model_provider as unknown for the AnthropicVertex and AnthropicBedrock clients. The provider is now resolved to google and amazon respectively.AAP:
Content-Type header for the Django, Tornado, Flask, and FastAPI integrations.urllib3 was not inspected for SSRF and API Security (API10).ASM:
ddtrace.contrib.trace_utils.set_user if neither ddtrace.appsec.trace_utils nor the user tracking SDK had been imported. The user blocking decision is now always enforced when AppSec is enabled.bytes commands passed to os.system, os.spawn*, and subprocess (for example os.system(b"...") or subprocess.run(b"...", shell=True)) bypassed the Exploit Prevention (RASP) shell and command injection checks. bytes commands are valid on POSIX systems and are now passed to the security checks like str commands.anonymization mode. It is now hashed, consistently with the user id and login.aws_durable_execution_sdk_python:
aws.durable.operation_attempt span tag was reported off by one for replayed durable operations, causing a resumed (replayed) execution to record a different attempt number than the original execution for the same operation. Replayed operations now report the same 0-indexed attempt number as a fresh execution.CI visibility:
ValueError: tool 1 is already in use error that occurred on Python 3.12+ when running pytest with coverage.py or pytest-cov alongside ddtrace. Both tools previously competed for sys.monitoring.COVERAGE_ID (slot 1). ddtrace now tries tool slots in priority order (4, 3, 1), falling back gracefully if all candidate slots are occupied.ResourceWarning: unclosed socket that occurred when using the pytest plugin. The background writer thread's thread-local HTTP connection was not closed on thread exit.database monitoring:
DD_DBM_PROPAGATION_MODE was set to full, service, or dynamic_service.elasticsearch:
coro.close() immediately on the error path. Affects Elasticsearch and OpenSearch integrations.Tracing:
TypeError: object NoneType can't be used in 'await' expression error. This occurs when the generator body awaits a coroutine that suspends to the event loop before its first yield. Python 3.9 and 3.10 are not affected.os.fork() could be sent twice, once by the parent and once by the child./v0.7/config payloads in multi-threaded applications (e.g. uWSGI).@tracer.wrap() to an async generator now forwards sent values, thrown exceptions, and close requests to the underlying generator, so it behaves like the unwrapped generator in all cases. Previously the wrapper only relayed values during forward iteration, so sent values were dropped and try/finally cleanup was skipped whenever the generator was closed early or received a thrown exception.TypeError: cannot pickle '_thread.lock' raised when serializing ddtrace-instrumented objects with cloudpickle. Fork-safe locks and events (ddtrace.internal.forksafe.Lock / Event) are now pickle-safe, and unpickling reconstructs a fresh instance.network.client.ip to contain the actual TCP peer IP address (i.e. REMOTE_ADDR) rather than the resolved HTTP client IP (which could originate from forwarding headers such as X-Forwarded-For). This affects the Flask, Django, ASGI (FastAPI/Starlette), and Tornado integrations. The resolved HTTP client IP continues to be reported under http.client_ip, which is the value used by App and API Protection (AAP) for IP-based threat detection and blocking.botocore:
_datadog message attribute is now read from the top-level SQS MessageAttributes first, and the message body is only parsed for SNS-to-SQS notifications.PutRecords now propagates trace context to each record in the batch. This enables trace context propagation to correctly map to each individual record, as well as enabling the correct tracking of messages through Data Streams Monitoring. Previously, only the first record would ever have been tracked.botocore, aiobotocore:
The request signature we calculated does not match the signature you provided when distributed tracing was enabled.celery:
DD_TRACE_REMOVE_INTEGRATION_SERVICE_NAMES_ENABLED, causing spans to always use celery-worker/celery-producer instead of the configured global service name.bootstrap:
yaml/_yaml loaded during module cleanup, fixing an issue that broke PyYAML consumers such as Airflow.Django:
process_exception, were not preserved, causing Django to skip exception handlers under ASGI.fastapi:
grpc:
grpc.aio client spans intermittently reporting error.message as "Internal error from Core" instead of the application-set abort details when a server aborts the RPC. #18015iast:
litellm:
AttributeError when metadata=None is passed to traced litellm.completion() calls. #18033MCP:
code origin:
sqlalchemy:
trace_engine() is called repeatedly for the same engine.core:
google_genai:
TypeError when Google GenAI returns None for optional response fields (token counts or candidates) introduced in google-genai>=2.6.0.Code Security (IAST):
re.finditer eagerly consumed the entire match iterator to taint every match, which removed the laziness of re.finditer and allowed a request-controlled input with many matches to allocate excess memory. Matches are now tainted lazily as the application iterates over them.kafka:
langgraph:
break-ing out of astream() or stream() when an __interrupt__ is received in a human-in-the-loop workflow) left the graph's root span unfinished, so the trace failed to assemble and did not appear in Datadog. The root span is now finished when the stream is torn down, so interrupted runs are traced.otel:
profiling:
SIGSEGV/SIGBUS handler that the profiler cannot wrap (e.g. CUDA, PyTorch, etc.). The sampler upgrades to the faster fault-recovery copy if it still owns both fault handlers afterwards. Otherwise, it permanently falls back to the syscall-based copy.protobuf:
redis:
out.host and server.address tags on spans produced by redis.cluster.RedisCluster and redis.asyncio.cluster.RedisCluster (redis-py >=4.1). These clients do not expose a connection_pool attribute, so connection metadata was never extracted and the Datadog inferred-entity feature fell back to the generic peer.db.system: redis identifier instead of resolving to the specific cluster hostname. Spans now include out.host and server.address derived from the cluster's startup node, enabling the service map to display a host-specific Redis entity.remoteconfig:
acknowledged before the product received it. The state is now set to acknowledged only after the configuration has been delivered to the product, and to error only when the payload is malformed and cannot be deserialized.starlette:
CORSMiddleware handled an OPTIONS request, the span resource was set to the raw URL path (e.g. OPTIONS /users/123) instead of the route template (e.g. OPTIONS /users/{user_id}). The route pattern is now resolved via the app's route tree when CORSMiddleware short-circuits before the Starlette router runs. Fixes #14321.Fixed an issue where sensitive content could be left unredacted in vulnerability reports when a sensitive value overlapped a tainted range.
Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
v0.4 trace API version (v0.5 cannot carry the LLMObs span data). Setting DD_TRACE_API_VERSION=v0.5 with LLMObs enabled logs a warning and downgrades to v0.4. No user action is required.DD_AI_GUARD_OPENAI_ENABLED environment variable (true by default) as a per-provider kill switch. Set it to false to disable AI Guard auto-instrumentation of the OpenAI SDK without affecting other providers or requiring a tracer version rollback.create_agent.DD_AI_GUARD_ANALYZE_STREAM_RESPONSES_ENABLED (default off); streams are buffered and evaluated before any chunk is delivered, raising AIGuardAbortError on a block.DD_APM_TRACING_ENABLED=false), traces produced by AI Guard are still sent to Datadog, kept with USER_KEEP sampling priority and the AI Guard decision maker so that they can be attributed to AI Guard, while APM host billing is not triggered.DD_AI_GUARD_ANALYZE_STREAM_RESPONSES_ENABLED=true (default false), all streamed chunks are buffered, the full assistant response is evaluated by AI Guard, and chunks are replayed to the caller only after an ALLOW verdict. On DENY/ABORT, AIGuardAbortError is raised and zero chunks are delivered. When the flag is off (the default), streaming behavior is unchanged._dd.appsec.normalized_route) for Tornado, following RFC-1103 and the existing FastAPI, Starlette, Django, and Flask implementations. The tag is emitted on every request span that already carries http.route when the API Security feature is active. Tornado routes using named capturing groups ((?P<name>...)) produce named parameters (e.g. /users/{id}); positional groups produce auto-numbered placeholders (e.g. {param1}). Optional trailing-slash patterns (/?) are treated as not declaring a trailing slash per RFC-1103 rule 1. #18398chat.completions, completions, or responses call is made during a web request, the WAF address server.business_logic.llm.event is emitted with the provider and model name. This enables WAF rules to tag the trace with appsec.events.llm.call.provider and appsec.events.llm.call.model, surfacing LLM-backed endpoints in the API Endpoint Catalog. Supports both sync and async OpenAI clients, Azure OpenAI (engine kwarg), and streaming. Only the first LLM call per request is recorded (endpoint-level detection, not per-invocation).aws.durable.operation_attempt tag to aws.durable.step and aws.durable.wait_for_condition spans. The tag is 0 for the original attempt and 1, 2, etc. for each subsequent retry.ray[serve]>=2.47.1. When Ray is instrumented, ddtrace now traces Serve HTTP and gRPC requests, deployment routing, deployment method execution, and calls made with DeploymentHandle.remote(), with distributed trace context propagated across Serve deployments. See the Ray documentation for more information. #18076LLMObs.annotate now accept an optional audio_parts field, a list of audio segments each with a mime_type and either inline base64 content or an attachment_key. This lets audio (for example, speech-to-text and text-to-speech payloads) be collected and rendered in LLM Observability.sample_rate argument to LLMObs.enable() to configure the proportion of LLMObs traces to sample (between 0.0 and 1.0) in code. This takes precedence over the DD_LLMOBS_SAMPLE_RATE environment variable.SyncExperiment.rerun_evaluators() to re-run evaluators on the stored task outputs from a previous experiment run without re-executing the task function. Reads from experiment.result (set by run() or pull()) and returns a new SyncExperiment with fresh evaluations while preserving original span IDs, trace IDs, inputs, and outputs via replay span copies linked back to the originals via parent_experiment_span_id. The missing_task_strategy parameter controls behavior when prior rows contain errors: "raise" (default) raises immediately, "skip" omits the row, and "retry" re-executes the task for that row.gpt-audio, gpt-audio-mini, gpt-4o-audio-preview) as audio_parts on LLM span messages. Note: audio in streamed responses (stream=True) is not captured yet.DD_LLMOBS_SAMPLE_RATE, which controls the proportion of LLM Observability spans (between 0.0 and 1.0, defaults to 1.0) that are retained. This does not affect APM span retention or the accuracy of LLM Observability metrics such as token usage, and cost.metadata to the data exposed to the span processor registered with LLMObs.enable(span_processor=...) or LLMObs.register_processor(...). Previously only input and output were exposed; the processor can now read and modify metadata to redact or remove sensitive values that integrations record there.mistralai client chat and embedding calls.mistralai client chat and embedding calls.pytorch.rank lifetime span for PyTorch distributed training. The span opens at init_process_group and closes at destroy_process_group or process exit. Tags include rank, world_size, framework (DDP / FSDP / DeepSpeed), launcher, torch.distributed.backend, and training_job_id (resolved from launcher environment variables). When running under Ray Train, ray.train.run_name, ray.submission_id, and ray.metadata.* are also applied. Enable with DD_PATCH_MODULES=pytorch:true.DD_INJECT_EXPERIMENTAL_OVERRIDE_USER_DDTRACE=true can be added to the environment.DD_TEST_MANAGEMENT_ATTEMPT_TO_FIX_RETRIES environment variable is now respected by the ddtrace.testing internal pytest plugin. When set to a valid integer, it overrides the retry count returned by the Datadog API for "attempt to fix" tests.us3, us5, ap1, and ap2 Datadog sites. This affected customers on these sites when no Datadog Agent was running or agentless export was explicitly enabled (DD_LLMOBS_AGENTLESS_ENABLED=1).openai integration where streamed chat completion spans under-reported output_tokens and total_tokens for OpenAI-compatible providers that emit a cumulative usage object on every streamed chunk._DD_LLMOBS_WRITER_TIMEOUT, default 5 seconds) and instead used the 2 second connection default, causing intermittent TimeoutError and dropped span events on high-latency connections to the agent or intake.model_name with the request deployment name (e.g. my-deployment) instead of the canonical model returned by the API (e.g. gpt-4o-2024-08-06). The integration now prefers the response model for the azure and azure_text providers.metadata containing non-string keys (e.g. int/float/bool) could be dropped during ingestion. Metadata keys are now stringified before encoding.Omit/NotGiven sentinel values) as noise in LLM span metadata.LLMObs.annotate(tags=...) could cause spans to be dropped during ingestion.document content blocks, which could cause evaluation to be skipped for prompts whose only content was a document. Document text is now scanned and binary documents leave a placeholder so evaluation still runs.block dynamic parameter, supplied in the incoming request body, to disable the operator-configured blocking policy and let a denied request proceed in monitor mode. Request-supplied parameters can now only strengthen blocking (turning monitor mode into blocking); they can no longer weaken an enabled blocking policy.kafka.partition and kafka.message_offset span tags being incorrectly set to -1. Consumer spans no longer misreport 0 (e.g. single-partition topics or the first message in a partition) due to falsy evaluation. Producer spans and any span where the value is genuinely unknown now omit these tags entirely rather than emitting a misleading -1.model_provider as unknown for the AnthropicVertex and AnthropicBedrock clients. The provider is now resolved to google and amazon respectively.Content-Type header for the Django, Tornado, Flask, and FastAPI integrations.urllib3 was not inspected for SSRF and API Security (API10).ddtrace.contrib.trace_utils.set_user if neither ddtrace.appsec.trace_utils nor the user tracking SDK had been imported. The user blocking decision is now always enforced when AppSec is enabled.bytes commands passed to os.system, os.spawn*, and subprocess (for example os.system(b"...") or subprocess.run(b"...", shell=True)) bypassed the Exploit Prevention (RASP) shell and command injection checks. bytes commands are valid on POSIX systems and are now passed to the security checks like str commands.anonymization mode. It is now hashed, consistently with the user id and login.aws.durable.operation_attempt span tag was reported off by one for replayed durable operations, causing a resumed (replayed) execution to record a different attempt number than the original execution for the same operation. Replayed operations now report the same 0-indexed attempt number as a fresh execution.ValueError: tool 1 is already in use error that occurred on Python 3.12+ when running pytest with coverage.py or pytest-cov alongside ddtrace. Both tools previously competed for sys.monitoring.COVERAGE_ID (slot 1). ddtrace now tries tool slots in priority order (4, 3, 1), falling back gracefully if all candidate slots are occupied.ResourceWarning: unclosed socket that occurred when using the pytest plugin. The background writer thread's thread-local HTTP connection was not closed on thread exit.DD_DBM_PROPAGATION_MODE was set to full, service, or dynamic_service.coro.close() immediately on the error path. Affects Elasticsearch and OpenSearch integrations.TypeError: object NoneType can't be used in 'await' expression error. This occurs when the generator body awaits a coroutine that suspends to the event loop before its first yield. Python 3.9 and 3.10 are not affected.os.fork() could be sent twice, once by the parent and once by the child./v0.7/config payloads in multi-threaded applications (e.g. uWSGI).@tracer.wrap() to an async generator now forwards sent values, thrown exceptions, and close requests to the underlying generator, so it behaves like the unwrapped generator in all cases. Previously the wrapper only relayed values during forward iteration, so sent values were dropped and try/finally cleanup was skipped whenever the generator was closed early or received a thrown exception.TypeError: cannot pickle '_thread.lock' raised when serializing ddtrace-instrumented objects with cloudpickle. Fork-safe locks and events (ddtrace.internal.forksafe.Lock / Event) are now pickle-safe, and unpickling reconstructs a fresh instance.network.client.ip to contain the actual TCP peer IP address (i.e. REMOTE_ADDR) rather than the resolved HTTP client IP (which could originate from forwarding headers such as X-Forwarded-For). This affects the Flask, Django, ASGI (FastAPI/Starlette), and Tornado integrations. The resolved HTTP client IP continues to be reported under http.client_ip, which is the value used by App and API Protection (AAP) for IP-based threat detection and blocking._datadog message attribute is now read from the top-level SQS MessageAttributes first, and the message body is only parsed for SNS-to-SQS notifications.PutRecords now propagates trace context to each record in the batch. This enables trace context propagation to correctly map to each individual record, as well as enabling the correct tracking of messages through Data Streams Monitoring. Previously, only the first record would ever have been tracked.The request signature we calculated does not match the signature you provided when distributed tracing was enabled.DD_TRACE_REMOVE_INTEGRATION_SERVICE_NAMES_ENABLED, causing spans to always use celery-worker/celery-producer instead of the configured global service name.process_exception, were not preserved, causing Django to skip exception handlers under ASGI.grpc.aio client spans intermittently reporting error.message as "Internal error from Core" instead of the application-set abort details when a server aborts the RPC. #18015re.finditer eagerly consumed the entire match iterator to taint every match, which removed the laziness of re.finditer and allowed a request-controlled input with many matches to allocate excess memory. Matches are now tainted lazily as the application iterates over them.AttributeError when metadata=None is passed to traced litellm.completion() calls. #18033trace_engine() is called repeatedly for the same engine.TypeError when Google GenAI returns None for optional response fields (token counts or candidates) introduced in google-genai>=2.6.0.out.host and server.address tags on spans produced by redis.cluster.RedisCluster and redis.asyncio.cluster.RedisCluster (redis-py >=4.1). These clients do not expose a connection_pool attribute, so connection metadata was never extracted and the Datadog inferred-entity feature fell back to the generic peer.db.system: redis identifier instead of resolving to the specific cluster hostname. Spans now include out.host and server.address derived from the cluster's startup node, enabling the service map to display a host-specific Redis entity.acknowledged before the product received it. The state is now set to acknowledged only after the configuration has been delivered to the product, and to error only when the payload is malformed and cannot be deserialized.CORSMiddleware handled an OPTIONS request, the span resource was set to the raw URL path (e.g. OPTIONS /users/123) instead of the route template (e.g. OPTIONS /users/{user_id}). The route pattern is now resolved via the app's route tree when CORSMiddleware short-circuits before the Starlette router runs. Fixes #14321.Fixed an issue where sensitive content could be left unredacted in vulnerability reports when a sensitive value overlapped a tainted range.
Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
v0.4 trace API version (v0.5 cannot carry the LLMObs span data). Setting DD_TRACE_API_VERSION=v0.5 with LLMObs enabled logs a warning and downgrades to v0.4. No user action is required.DD_AI_GUARD_OPENAI_ENABLED environment variable (true by default) as a per-provider kill switch. Set it to false to disable AI Guard auto-instrumentation of the OpenAI SDK without affecting other providers or requiring a tracer version rollback.create_agent.DD_AI_GUARD_ANALYZE_STREAM_RESPONSES_ENABLED (default off); streams are buffered and evaluated before any chunk is delivered, raising AIGuardAbortError on a block.DD_APM_TRACING_ENABLED=false), traces produced by AI Guard are still sent to Datadog, kept with USER_KEEP sampling priority and the AI Guard decision maker so that they can be attributed to AI Guard, while APM host billing is not triggered.DD_AI_GUARD_ANALYZE_STREAM_RESPONSES_ENABLED=true (default false), all streamed chunks are buffered, the full assistant response is evaluated by AI Guard, and chunks are replayed to the caller only after an ALLOW verdict. On DENY/ABORT, AIGuardAbortError is raised and zero chunks are delivered. When the flag is off (the default), streaming behavior is unchanged._dd.appsec.normalized_route) for Tornado, following RFC-1103 and the existing FastAPI, Starlette, Django, and Flask implementations. The tag is emitted on every request span that already carries http.route when the API Security feature is active. Tornado routes using named capturing groups ((?P<name>...)) produce named parameters (e.g. /users/{id}); positional groups produce auto-numbered placeholders (e.g. {param1}). Optional trailing-slash patterns (/?) are treated as not declaring a trailing slash per RFC-1103 rule 1. #18398chat.completions, completions, or responses call is made during a web request, the WAF address server.business_logic.llm.event is emitted with the provider and model name. This enables WAF rules to tag the trace with appsec.events.llm.call.provider and appsec.events.llm.call.model, surfacing LLM-backed endpoints in the API Endpoint Catalog. Supports both sync and async OpenAI clients, Azure OpenAI (engine kwarg), and streaming. Only the first LLM call per request is recorded (endpoint-level detection, not per-invocation).aws.durable.operation_attempt tag to aws.durable.step and aws.durable.wait_for_condition spans. The tag is 0 for the original attempt and 1, 2, etc. for each subsequent retry.ray[serve]>=2.47.1. When Ray is instrumented, ddtrace now traces Serve HTTP and gRPC requests, deployment routing, deployment method execution, and calls made with DeploymentHandle.remote(), with distributed trace context propagated across Serve deployments. See the Ray documentation for more information. #18076LLMObs.annotate now accept an optional audio_parts field, a list of audio segments each with a mime_type and either inline base64 content or an attachment_key. This lets audio (for example, speech-to-text and text-to-speech payloads) be collected and rendered in LLM Observability.sample_rate argument to LLMObs.enable() to configure the proportion of LLMObs traces to sample (between 0.0 and 1.0) in code. This takes precedence over the DD_LLMOBS_SAMPLE_RATE environment variable.SyncExperiment.rerun_evaluators() to re-run evaluators on the stored task outputs from a previous experiment run without re-executing the task function. Reads from experiment.result (set by run() or pull()) and returns a new SyncExperiment with fresh evaluations while preserving original span IDs, trace IDs, inputs, and outputs via replay span copies linked back to the originals via parent_experiment_span_id. The missing_task_strategy parameter controls behavior when prior rows contain errors: "raise" (default) raises immediately, "skip" omits the row, and "retry" re-executes the task for that row.gpt-audio, gpt-audio-mini, gpt-4o-audio-preview) as audio_parts on LLM span messages. Note: audio in streamed responses (stream=True) is not captured yet.DD_LLMOBS_SAMPLE_RATE, which controls the proportion of LLM Observability spans (between 0.0 and 1.0, defaults to 1.0) that are retained. This does not affect APM span retention or the accuracy of LLM Observability metrics such as token usage, and cost.metadata to the data exposed to the span processor registered with LLMObs.enable(span_processor=...) or LLMObs.register_processor(...). Previously only input and output were exposed; the processor can now read and modify metadata to redact or remove sensitive values that integrations record there.mistralai client chat and embedding calls.mistralai client chat and embedding calls.pytorch.rank lifetime span for PyTorch distributed training. The span opens at init_process_group and closes at destroy_process_group or process exit. Tags include rank, world_size, framework (DDP / FSDP / DeepSpeed), launcher, torch.distributed.backend, and training_job_id (resolved from launcher environment variables). When running under Ray Train, ray.train.run_name, ray.submission_id, and ray.metadata.* are also applied. Enable with DD_PATCH_MODULES=pytorch:true.DD_INJECT_EXPERIMENTAL_OVERRIDE_USER_DDTRACE=true can be added to the environment.DD_TEST_MANAGEMENT_ATTEMPT_TO_FIX_RETRIES environment variable is now respected by the ddtrace.testing internal pytest plugin. When set to a valid integer, it overrides the retry count returned by the Datadog API for "attempt to fix" tests.us3, us5, ap1, and ap2 Datadog sites. This affected customers on these sites when no Datadog Agent was running or agentless export was explicitly enabled (DD_LLMOBS_AGENTLESS_ENABLED=1).openai integration where streamed chat completion spans under-reported output_tokens and total_tokens for OpenAI-compatible providers that emit a cumulative usage object on every streamed chunk._DD_LLMOBS_WRITER_TIMEOUT, default 5 seconds) and instead used the 2 second connection default, causing intermittent TimeoutError and dropped span events on high-latency connections to the agent or intake.model_name with the request deployment name (e.g. my-deployment) instead of the canonical model returned by the API (e.g. gpt-4o-2024-08-06). The integration now prefers the response model for the azure and azure_text providers.metadata containing non-string keys (e.g. int/float/bool) could be dropped during ingestion. Metadata keys are now stringified before encoding.Omit/NotGiven sentinel values) as noise in LLM span metadata.LLMObs.annotate(tags=...) could cause spans to be dropped during ingestion.document content blocks, which could cause evaluation to be skipped for prompts whose only content was a document. Document text is now scanned and binary documents leave a placeholder so evaluation still runs.block dynamic parameter, supplied in the incoming request body, to disable the operator-configured blocking policy and let a denied request proceed in monitor mode. Request-supplied parameters can now only strengthen blocking (turning monitor mode into blocking); they can no longer weaken an enabled blocking policy.kafka.partition and kafka.message_offset span tags being incorrectly set to -1. Consumer spans no longer misreport 0 (e.g. single-partition topics or the first message in a partition) due to falsy evaluation. Producer spans and any span where the value is genuinely unknown now omit these tags entirely rather than emitting a misleading -1.model_provider as unknown for the AnthropicVertex and AnthropicBedrock clients. The provider is now resolved to google and amazon respectively.Content-Type header for the Django, Tornado, Flask, and FastAPI integrations.urllib3 was not inspected for SSRF and API Security (API10).ddtrace.contrib.trace_utils.set_user if neither ddtrace.appsec.trace_utils nor the user tracking SDK had been imported. The user blocking decision is now always enforced when AppSec is enabled.bytes commands passed to os.system, os.spawn*, and subprocess (for example os.system(b"...") or subprocess.run(b"...", shell=True)) bypassed the Exploit Prevention (RASP) shell and command injection checks. bytes commands are valid on POSIX systems and are now passed to the security checks like str commands.anonymization mode. It is now hashed, consistently with the user id and login.aws.durable.operation_attempt span tag was reported off by one for replayed durable operations, causing a resumed (replayed) execution to record a different attempt number than the original execution for the same operation. Replayed operations now report the same 0-indexed attempt number as a fresh execution.ValueError: tool 1 is already in use error that occurred on Python 3.12+ when running pytest with coverage.py or pytest-cov alongside ddtrace. Both tools previously competed for sys.monitoring.COVERAGE_ID (slot 1). ddtrace now tries tool slots in priority order (4, 3, 1), falling back gracefully if all candidate slots are occupied.ResourceWarning: unclosed socket that occurred when using the pytest plugin. The background writer thread's thread-local HTTP connection was not closed on thread exit.DD_DBM_PROPAGATION_MODE was set to full, service, or dynamic_service.coro.close() immediately on the error path. Affects Elasticsearch and OpenSearch integrations.TypeError: object NoneType can't be used in 'await' expression error. This occurs when the generator body awaits a coroutine that suspends to the event loop before its first yield. Python 3.9 and 3.10 are not affected.os.fork() could be sent twice, once by the parent and once by the child./v0.7/config payloads in multi-threaded applications (e.g. uWSGI).@tracer.wrap() to an async generator now forwards sent values, thrown exceptions, and close requests to the underlying generator, so it behaves like the unwrapped generator in all cases. Previously the wrapper only relayed values during forward iteration, so sent values were dropped and try/finally cleanup was skipped whenever the generator was closed early or received a thrown exception.TypeError: cannot pickle '_thread.lock' raised when serializing ddtrace-instrumented objects with cloudpickle. Fork-safe locks and events (ddtrace.internal.forksafe.Lock / Event) are now pickle-safe, and unpickling reconstructs a fresh instance.network.client.ip to contain the actual TCP peer IP address (i.e. REMOTE_ADDR) rather than the resolved HTTP client IP (which could originate from forwarding headers such as X-Forwarded-For). This affects the Flask, Django, ASGI (FastAPI/Starlette), and Tornado integrations. The resolved HTTP client IP continues to be reported under http.client_ip, which is the value used by App and API Protection (AAP) for IP-based threat detection and blocking._datadog message attribute is now read from the top-level SQS MessageAttributes first, and the message body is only parsed for SNS-to-SQS notifications.PutRecords now propagates trace context to each record in the batch. This enables trace context propagation to correctly map to each individual record, as well as enabling the correct tracking of messages through Data Streams Monitoring. Previously, only the first record would ever have been tracked.The request signature we calculated does not match the signature you provided when distributed tracing was enabled.DD_TRACE_REMOVE_INTEGRATION_SERVICE_NAMES_ENABLED, causing spans to always use celery-worker/celery-producer instead of the configured global service name.process_exception, were not preserved, causing Django to skip exception handlers under ASGI.grpc.aio client spans intermittently reporting error.message as "Internal error from Core" instead of the application-set abort details when a server aborts the RPC. #18015re.finditer eagerly consumed the entire match iterator to taint every match, which removed the laziness of re.finditer and allowed a request-controlled input with many matches to allocate excess memory. Matches are now tainted lazily as the application iterates over them.AttributeError when metadata=None is passed to traced litellm.completion() calls. #18033trace_engine() is called repeatedly for the same engine.TypeError when Google GenAI returns None for optional response fields (token counts or candidates) introduced in google-genai>=2.6.0.out.host and server.address tags on spans produced by redis.cluster.RedisCluster and redis.asyncio.cluster.RedisCluster (redis-py >=4.1). These clients do not expose a connection_pool attribute, so connection metadata was never extracted and the Datadog inferred-entity feature fell back to the generic peer.db.system: redis identifier instead of resolving to the specific cluster hostname. Spans now include out.host and server.address derived from the cluster's startup node, enabling the service map to display a host-specific Redis entity.acknowledged before the product received it. The state is now set to acknowledged only after the configuration has been delivered to the product, and to error only when the payload is malformed and cannot be deserialized.CORSMiddleware handled an OPTIONS request, the span resource was set to the raw URL path (e.g. OPTIONS /users/123) instead of the route template (e.g. OPTIONS /users/{user_id}). The route pattern is now resolved via the app's route tree when CORSMiddleware short-circuits before the Starlette router runs. Fixes #14321.CI visibility: Fixes an issue where test sessions running with pytest-xdist on shallow Git checkouts can spend excessive time unshallowing the reposit
code origin for spans: prevent view or traced functions from raising an exception in situations where their number is large.
profiling: Fixes potential process memory growth when a stack profiling sampling cycle exits before completing a sample.
openfeature: This fix resolves an issue where the feature flagging provider failed to load with openfeature-sdk 0.10.0 or later.
openfeature-sdk 0.10.0 or later.<!-- -->
<!-- -->
@tracer.wrap on functions created at run time has been fixed.Fix crashes in uwsgi worker when exiting from SIGTERM.
Fix a potential memory leak triggered by having code origin for span explicitly disabled via environment variables.
<!-- -->
This fix resolves an issue where the Datadog UI showed No extra information available when viewing details about why a test was skipped by Intelligent
CI Visibility:
No extra information available when viewing details about why a test was skipped by Intelligent Test Runner, due to missing ITR correlation ID on test or suite events.Code Security (IAST):
copy_ranges_from_strings and copy_and_shift_ranges_from_strings to the active request slot, matching the scoped taint read path. Previously these copy helpers resolved the taint map by scanning all request slots, so a concurrent or still-open request could capture the derived taint and the current request would miss the transformed tainted input.MCP:
code origin:
IAST:
uvloop has been fixed.LLM Observability:
bootstrap: keep yaml/_yaml loaded during module cleanup, fixing an issue that broke PyYAML consumers such as Airflow.
yaml/_yaml loaded during module cleanup, fixing an issue that broke PyYAML consumers such as Airflow.<!-- -->
% string formatting handled tainted text containing literal IAST evidence marker delimiters.<!-- -->
<!-- -->
anomaly.query_count) prevented the span from being ingested. Dots in metric keys are now replaced with underscores.<!-- -->
IAST: This fix resolves an issue where IAST could report a false positive vulnerability against a request whose input did not actually contain tainted…
SCRIPT_NAME now expose the client-hit resource on a new flask.resource.full tag (e.g. GET /api/v2/users). The span resource and flask.url_rule tag are unchanged. The tag is only set when its value would differ from span.resource.<!-- -->
HEAD for any GET route and Flask's auto-handled OPTIONS for every route — not just the methods listed in methods=[...].<!-- -->
v0.4 trace API version (v0.5 cannot carry the LLMObs span data). Setting DD_TRACE_API_VERSION=v0.5 with LLMObs enabled logs a warning and downgrades to v0.4. No user action is required.x-datadog-endpoint-scan and x-datadog-security-test HTTP request headers on service entry spans unconditionally as http.request.headers.x-datadog-endpoint-scan and http.request.headers.x-datadog-security-test tags. These markers identify Datadog-originated endpoint scans and security tests so the API inventory pipeline can distinguish scan/test traffic from real user traffic. The headers are tagged regardless of DD_TRACE_HEADER_TAGS configuration or AppSec enablement, and are not propagated to downstream services.<!-- -->
dynamic_service as a new DD_DBM_PROPAGATION_MODE value. Set DD_DBM_PROPAGATION_MODE=dynamic_service to inject DBM service metadata and the SQL base hash without injecting trace context.<!-- -->
Messages.create / Messages.stream and their async and Beta variants), and evaluation is automatically skipped when a framework integration (LangChain) is already evaluating the same call.<!-- -->
kafka.partition and kafka.message_offset tags to the producer span once the broker acknowledges the send. The partition reflects the partition the broker actually assigned to the message, which may differ from the partition the caller requested. Mirrors the behavior added to the Java tracer in DataDog/dd-trace-java#11107.<!-- -->
_dd.appsec.normalized_route span tag for FastAPI and Starlette request spans when API Security is enabled. The tag follows RFC-1103 and provides a per-request, framework-agnostic representation of the matched route — converter types are stripped, multi-parameter URL segments are combined with +, path catch-all parameters are emitted as a single tail element, and trailing slashes are preserved as declared. Mount-prefixed sub-application routes are reported with their full assembled path.<!-- -->
_dd.appsec.normalized_route span tag support to Flask request spans when API Security is enabled. Flask / Werkzeug <converter:name> route syntax is normalized following RFC-1103: converter types are stripped, multi-parameter URL segments (e.g. /<first>.<last>/) are combined with +, <path:name> catch-all parameters are emitted as a single tail element, and trailing slashes are preserved as declared. Routes served through DispatcherMiddleware sub-apps are reported with their full assembled path (mount prefix included).<!-- -->
claude_agent_sdk integration now emits span links between step, LLM, and tool spans so multi-step traces render the sequencing between LLM calls and tool calls.<!-- -->
aws_durable_execution_sdk_python for details and opt-out configuration.<!-- -->
werkzeug.middleware.dispatcher.DispatcherMiddleware. Sub-app routes are reported with their full mounted path.<!-- -->
DD_DATA_STREAMS_ENABLED=true.<!-- -->
git.commit.sha and git.repository_url. Values come from DD_GIT_COMMIT_SHA / DD_GIT_REPOSITORY_URL or the main package's Project-URL metadata, falling back to running git against the current working directory. Honors DD_TRACE_GIT_METADATA_ENABLED and user-supplied tags with the same keys.<!-- -->
<!-- -->
MultiEvaluatorResult to emit multiple named evaluation metrics from a single evaluator call. Each sub-value may itself be an EvaluatorResult carrying its own reasoning, assessment, metadata, and tags. By default emitted metric labels are prefixed with the evaluator's name ("<evaluator_name>-<key>"); pass prefix=False to emit raw keys.<!-- -->
LLMObs.pull_experiment(experiment_id) to fetch a previously-run experiment from the Datadog backend by UUID. The returned SyncExperiment has its .result populated and is ready for downstream inspection (e.g. .as_dataframe()) without re-executing the original task.<!-- -->
task and dataset are now optional when creating an experiment via LLMObs.experiment() / LLMObs.async_experiment(), supporting pull-based workflows. Calling run() without providing task and dataset raises a ValueError.<!-- -->
DD_LLMOBS_SAMPLE_RATE, which controls the proportion of LLM Observability spans (between 0.0 and 1.0, defaults to 1.0) that are retained. This does not affect APM span retention or the accuracy of LLM Observability metrics such as token usage, and cost.<!-- -->
version field is set on a parent LLM span via tool_definitions, the resolved tool version is now also written onto manually-started child tool spans (created via LLMObs.tool()) as meta.tool.version. Previously, this propagation only occurred for tool spans produced by auto-instrumented integrations (OpenAI, LangChain, OpenAI Agents, ReAct) through the LinkTracker dispatch mechanism.<!-- -->
version field is set on a parent LLM span via tool_definitions, the resolved tool version is now also written onto the corresponding child tool span as meta.tool.version. This allows the LLM Observability UI to aggregate tool version counts and filter tool spans by version without correlating to the parent LLM span.<!-- -->
<!-- -->
PYMEM_DOMAIN_MEM (PyMem_Malloc/Calloc/Realloc) in addition to the existing PYMEM_DOMAIN_OBJ coverage when DD_PROFILING_MEMORY_MEM_DOMAIN_ENABLED=true is set. This support is effective on Python 3.12 and newer, making some previously invisible allocations visible in heap profiles in those environments.<!-- -->
DD_ML_JOB_ENV — a single opt-in env var for forwarding configuration into Ray job workers. Set it on the dashboard process as a semicolon-separated list of KEY:VALUE pairs, e.g. DD_SERVICE:my-svc;DD_AGENT_HOST:10.0.0.1, and workers receive them verbatim as DD_SERVICE=my-svc, DD_AGENT_HOST=10.0.0.1, etc.<!-- -->
DD_INJECT_EXPERIMENTAL_OVERRIDE_USER_DDTRACE=true can be added to the environment.<!-- -->
CI Visibility: Adds ddtrace.testing.logs.DDTestLogsHandler, a public logging.Handler for shipping log records to the Datadog logs intake correlated with CI Visibility test traces, intended for use in subprocesses that execute tests outside of the main pytest process. Also adds ddtrace.testing.logs.CorrelationFilter, a base class for stamping dd.trace_id / dd.span_id onto log records, and a ready-made ddtrace.testing.logs.ThreadLocalCorrelationFilter for the common thread-local case. Subclass CorrelationFilter to plug in other concurrency models (asyncio ContextVar, global state, etc.).
Example:
from pydantic_evals.evaluators import EqualsExpected import logging
from ddtrace.testing.logs import DDTestLogsHandler from ddtrace.testing.logs import ThreadLocalCorrelationFilter
handler = DDTestLogsHandler(service="my-service") correlation = ThreadLocalCorrelationFilter() handler.addFilter(correlation) logging.getLogger().addHandler(handler)
while True:
job = queue.get() correlation.set_context(trace_id=job.trace_id, span_id=job.span_id) run_test(job.item)handler.close()
websocket.connect ASGI message, causing ASGI/FastAPI WebSocket connections to fail with HTTP 500 when AppSec was enabled.<!-- -->
us3, us5, ap1, and ap2 Datadog sites. This affected customers on these sites when no Datadog Agent was running or agentless export was explicitly enabled (DD_LLMOBS_AGENTLESS_ENABLED=1).<!-- -->
confluent_kafka integration.<!-- -->
waf.duration, waf.duration_ext, rasp.duration, and rasp.duration_ext.<!-- -->
<!-- -->
cosmosdb.query span resource names included per-item identifiers (such as document, user, and permission ids) from the request URI, which caused unbounded resource cardinality. The integration now redacts those ids by replacing them with ? while keeping the database (dbs) and collection (colls) names intact. For example, Read /dbs/myDb/colls/myColl/docs/item1 is now reported as Read /dbs/myDb/colls/myColl/docs/?.<!-- -->
DD_CIVISIBILITY_BACKEND_API_TIMEOUT_MILLIS environment variable (previously missing) to override it. The value is expressed in milliseconds (e.g. 60000 for 60 seconds), consistent with the Java tracer. The same timeout now applies uniformly to all backend requests, including skippable test fetches.<!-- -->
DD_DBM_PROPAGATION_MODE was set to full, service, or dynamic_service.<!-- -->
wrapt==2.2.0 from the supported dependency range to avoid a regression that breaks wrapped C descriptors.<!-- -->
DD_KOMBU_SERVICE, DD_JINJA2_SERVICE, DD_DJANGO_DATABASE_SERVICE, DD_CELERY_PRODUCER_SERVICE, DD_CELERY_WORKER_SERVICE, and DD_DJANGO_CACHE_SERVICE now override the service tag on their respective spans. Previously they were ignored in favor of the DD_<INTEGRATION>_SERVICE_NAME form.<!-- -->
config.kombu["service"] before patching was applied to producer spans but not consumer spans. Both now read from the same service config source.<!-- -->
None value for req.uri_template caused a TypeError when constructing the route in the Falcon tracing middleware.<!-- -->
Content-Type header carried a parameter (most commonly charset=utf-8).<!-- -->
TypeError: object NoneType can't be used in 'await' expression error. This occurs when the generator body awaits a coroutine that suspends to the event loop before its first yield. Python 3.9 and 3.10 are not affected.<!-- -->
_datadog message attribute is now read from the top-level SQS MessageAttributes first, and the message body is only parsed for SNS-to-SQS notifications.<!-- -->
<!-- -->
DD_TRACE_REMOVE_INTEGRATION_SERVICE_NAMES_ENABLED, causing spans to always use celery-worker/celery-producer instead of the configured global service name.<!-- -->
<!-- -->
unknown error category from the upstream Claude Agent SDK instead of a descriptive API error. The integration now surfaces the detailed error message from the assistant message content.<!-- -->
ClaudeSDKClient.query() calls produced nested agent spans instead of separate root traces. Agent spans are now finalized eagerly when a ResultMessage is observed, rather than relying on the wrapping async generator's finally block to execute.<!-- -->
*args / **kwargs for any function that declares keyword-only parameters alongside variadic arguments.<!-- -->
process_exception, were not preserved, causing Django to skip exception handlers under ASGI.<!-- -->
os.fork() could be sent twice, once by the parent and once by the child.<!-- -->
/v0.7/config payloads in multi-threaded applications (e.g. uWSGI).<!-- -->
<!-- -->
<!-- -->
langchain_aws.ChatBedrockConverse spans reported an opaque inference-profile ARN identifier as the model name when an inference profile was used. base_model_id which represents the underlying foundation model is now checked first when extracting model names, and the botocore Bedrock integration reads the resolved base model from a shared in-process cache populated by langchain so the same resolution applies to the underlying bedrock-runtime span.<!-- -->
openai integration where streamed chat completion spans under-reported output_tokens and total_tokens for OpenAI-compatible providers that emit a cumulative usage object on every streamed chunk.<!-- -->
reasoning_content was missing from streamed chat completions in the OpenAI and LiteLLM integrations when an OpenAI-compatible reasoning provider (e.g. DeepSeek, Qwen) emitted delta.reasoning_content chunks. The aggregated message now captures reasoning text in the output message, matching non-streaming behavior.<!-- -->
_DD_LLMOBS_WRITER_TIMEOUT, default 5 seconds) and instead used the 2 second connection default, causing intermittent TimeoutError and dropped span events on high-latency connections to the agent or intake.<!-- -->
gevent monkey-patching threading lock primitives in place after the profiler had already patched them.<!-- -->
Content-Type header carried parameters being dropped.<!-- -->
<!-- -->
awake() on a PeriodicThread after it had been stopped would block forever. awake() now returns immediately in that case.<!-- -->
PeriodicThread callbacks were invisible to Python's cyclic garbage collector and could accumulate when threads used bound methods as targets.<!-- -->
<!-- -->
pydantic_ai.tool / pydantic_ai.agent) and the resource name is the specific tool or agent name, matching Datadog APM convention. This restores per-tool and per-agent grouping on APM service and resource pages. LLM Observability views are unaffected.<!-- -->
pod_name) on hosts using cgroup v2.<!-- -->
<!-- -->
= in the value. tracing: The subprocess integration now correctly scrubs env-var values for env-var names containing digits.<!-- -->
<!-- -->
@tracer.wrap() to an async generator now forwards sent values, thrown exceptions, and close requests to the underlying generator, so it behaves like the unwrapped generator in all cases. Previously the wrapper only relayed values during forward iteration, so sent values were dropped and try/finally cleanup was skipped whenever the generator was closed early or received a thrown exception.<!-- -->
<!-- -->
git warnings when running tests with pytest-xdist. Simultaneous git fetch --update-shallow calls now retry with exponential back-off on .git/shallow.lock contention, skip the fetch entirely once a sibling worker has already unshallowed the repository, and run under the C locale so the contention check is robust against non-English git messages. git merge-base is deferred until after unshallowing so the required commits are available locally.<!-- -->
<!-- -->
<!-- -->
<!-- -->
<!-- -->
<!-- -->
metadata containing non-string keys (e.g. int/float/bool) could be dropped during ingestion. Metadata keys are now stringified before encoding.<!-- -->
Omit/NotGiven sentinel values) as noise in LLM span metadata.<!-- -->
_current_trace_context did not propagate _dd.p.llmobs_parent_id into the trace context metadata. Without this value, downstream consumers (e.g. dd-trace-go) could not locate the active LLMObs parent span and would create orphaned child traces instead of correctly continuing the parent trace.<!-- -->
LLMObsSpan now exposes input_documents and output_documents for mutation or removal before export.<!-- -->
openai spans when an OpenAI (or AsyncOpenAI) client and an AzureOpenAI (or AsyncAzureOpenAI) client are instantiated at the same time. Provider is now determined per-call rather than from the most recently constructed client.<!-- -->
<!-- -->
<!-- -->
<!-- -->
<!-- -->
SIGINT and SIGTERM signals.<!-- -->
<!-- -->
<!-- -->
ddtrace-run caused a traceback on keyboard interrupt.<!-- -->
confluent-kafka integration where internal list_topics calls triggered librdkafka background metadata-refresh tasks.OTEL_METRIC_EXPORT_TIMEOUT and OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE to the allowlist of recognized OpenTelemetry environment variables. Previously, setting any of these variables produced an inaccurate OpenTelemetry configuration ... is not supported by Datadog warning at startup, even though the variables are valid OpenTelemetry SDK settings.<!-- -->
DD_LLMOBS_AGENTLESS_ENABLED=1), APM traces are now exported agentlessly to Datadog's intake. This should not change user-facing behavior: both APM and LLMObs spans remain visible in the UI; LLMObs spans are simply no longer shipped separately for agentless users. Note that setting DD_APM_TRACING_ENABLED=false takes higher precedence and will result in LLMObs span events shipping separately as existing behavior.<!-- -->
info.profiler.settings on the upload event, so profiles can be filtered by individual settings.<!-- -->
socket.gethostname(), ray.get_runtime_context().get_*()) are amortized across the thread lifetime, reducing overhead for high-frequency Ray task and actor-method instrumentation.AAP: Fix an issue which could make the Threat Protection panel wrongly report that a library update was required.
<!-- -->
DD_DBM_PROPAGATION_MODE was set to full, service, or dynamic_service.<!-- -->
_datadog message attribute is now read from the top-level SQS MessageAttributes first, and the message body is only parsed for SNS-to-SQS notifications.<!-- -->
process_exception, were not preserved, causing Django to skip exception handlers under ASGI.<!-- -->
<!-- -->
Nothing published for this version
tracing: Resolves an issue where wrapping an async generator on Python 3.11 through 3.14 raises a TypeError: object NoneType can't be used in 'await'
tracing: Resolves an issue where wrapping an async generator on Python 3.11 through 3.14 raises a TypeError: object NoneType can't be used in 'await' expression error. This occurs when the generator body awaits a coroutine that suspends to the event loop before its first yield. Python 3.9 and 3.10 are not affected.
LLM Observability: Fixes an issue where the span writer ignored its configured request timeout (_DD_LLMOBS_WRITER_TIMEOUT, default 5 seconds) and instead used the 2 second connection default, causing intermittent TimeoutError and dropped span events on high-latency connections to the agent or intake.
celery: Fixes Celery worker and producer spans ignoring DD_TRACE_REMOVE_INTEGRATION_SERVICE_NAMES_ENABLED, causing spans to always use celery-worker/c
celery: Fixes Celery worker and producer spans ignoring DD_TRACE_REMOVE_INTEGRATION_SERVICE_NAMES_ENABLED, causing spans to always use celery-worker/celery-producer instead of the configured global service name.
tracing: This fix resolves an issue where traces buffered before os.fork() could be sent twice, once by the parent and once by the child.
tracing: Applying @tracer.wrap() to an async generator now forwards sent values, thrown exceptions, and close requests to the underlying generator, so it behaves like the unwrapped generator in all cases. Previously the wrapper only relayed values during forward iteration, so sent values were dropped and try/finally cleanup was skipped whenever the generator was closed early or received a thrown exception.
tracing: A rare crash happening on versions of CPython prior to 3.12 has been fixed.
LLM Observability: Resolves an issue where spans annotated with metadata containing non-string keys (e.g. int/float/bool) could be dropped during ingestion. Metadata keys are now stringified before encoding.
LLM Observability: Resolves an issue where the OpenAI integration recorded unset request parameters (OpenAI SDK's Omit/NotGiven sentinel values) as noise in LLM span metadata.
profiling: Rare crashes that could happen post-fork in fork-based applications have been fixed.
LLM Observability: Adds support for DD_LLMOBS_SAMPLE_RATE, which controls the proportion of LLM Observability spans (between 0.0 and 1.0, defaults to
DD_LLMOBS_SAMPLE_RATE, which controls the proportion of LLM Observability spans (between 0.0 and 1.0, defaults to 1.0) that are retained. This does not affect APM span retention or the accuracy of LLM Observability metrics such as token usage, and cost.LLM Observability: when APM and LLMObs are both enabled, the APM trace writer now uses the v0.4 trace API version (v0.5 cannot carry the LLMObs span d
v0.4 trace API version (v0.5 cannot carry the LLMObs span data). Setting DD_TRACE_API_VERSION=v0.5 with LLMObs enabled logs a warning and downgrades to v0.4. No user action is required.<!-- -->
DD_INJECT_EXPERIMENTAL_OVERRIDE_USER_DDTRACE=true can be added to the environment.us3, us5, ap1, and ap2 Datadog sites. This affected customers on these sites when no Datadog Agent was running or agentless export was explicitly enabled (DD_LLMOBS_AGENTLESS_ENABLED=1).<!-- -->
/v0.7/config payloads in multi-threaded applications (e.g. uWSGI).<!-- -->
<!-- -->
<!-- -->
<!-- -->
IAST: This fix resolves an issue where IAST could report a false positive vulnerability against a request whose input did not actually contain tainted…
SCRIPT_NAME now expose the client-hit resource on a new flask.resource.full tag (e.g. GET /api/v2/users). The span resource and flask.url_rule tag are unchanged. The tag is only set when its value would differ from span.resource.<!-- -->
HEAD for any GET route and Flask's auto-handled OPTIONS for every route — not just the methods listed in methods=[...].x-datadog-endpoint-scan and x-datadog-security-test HTTP request headers on service entry spans unconditionally as http.request.headers.x-datadog-endpoint-scan and http.request.headers.x-datadog-security-test tags. These markers identify Datadog-originated endpoint scans and security tests so the API inventory pipeline can distinguish scan/test traffic from real user traffic. The headers are tagged regardless of DD_TRACE_HEADER_TAGS configuration or AppSec enablement, and are not propagated to downstream services.<!-- -->
dynamic_service as a new DD_DBM_PROPAGATION_MODE value. Set DD_DBM_PROPAGATION_MODE=dynamic_service to inject DBM service metadata and the SQL base hash without injecting trace context.<!-- -->
Messages.create / Messages.stream and their async and Beta variants), and evaluation is automatically skipped when a framework integration (LangChain) is already evaluating the same call.<!-- -->
kafka.partition and kafka.message_offset tags to the producer span once the broker acknowledges the send. The partition reflects the partition the broker actually assigned to the message, which may differ from the partition the caller requested. Mirrors the behavior added to the Java tracer in DataDog/dd-trace-java#11107.<!-- -->
_dd.appsec.normalized_route span tag for FastAPI and Starlette request spans when API Security is enabled. The tag follows RFC-1103 and provides a per-request, framework-agnostic representation of the matched route — converter types are stripped, multi-parameter URL segments are combined with +, path catch-all parameters are emitted as a single tail element, and trailing slashes are preserved as declared. Mount-prefixed sub-application routes are reported with their full assembled path.<!-- -->
_dd.appsec.normalized_route span tag support to Flask request spans when API Security is enabled. Flask / Werkzeug <converter:name> route syntax is normalized following RFC-1103: converter types are stripped, multi-parameter URL segments (e.g. /<first>.<last>/) are combined with +, <path:name> catch-all parameters are emitted as a single tail element, and trailing slashes are preserved as declared. Routes served through DispatcherMiddleware sub-apps are reported with their full assembled path (mount prefix included).<!-- -->
claude_agent_sdk integration now emits span links between step, LLM, and tool spans so multi-step traces render the sequencing between LLM calls and tool calls.<!-- -->
aws_durable_execution_sdk_python for details and opt-out configuration.<!-- -->
werkzeug.middleware.dispatcher.DispatcherMiddleware. Sub-app routes are reported with their full mounted path.<!-- -->
DD_DATA_STREAMS_ENABLED=true.<!-- -->
git.commit.sha and git.repository_url. Values come from DD_GIT_COMMIT_SHA / DD_GIT_REPOSITORY_URL or the main package's Project-URL metadata, falling back to running git against the current working directory. Honors DD_TRACE_GIT_METADATA_ENABLED and user-supplied tags with the same keys.<!-- -->
MultiEvaluatorResult to emit multiple named evaluation metrics from a single evaluator call. Each sub-value may itself be an EvaluatorResult carrying its own reasoning, assessment, metadata, and tags. By default emitted metric labels are prefixed with the evaluator's name ("<evaluator_name>-<key>"); pass prefix=False to emit raw keys.<!-- -->
LLMObs.pull_experiment(experiment_id) to fetch a previously-run experiment from the Datadog backend by UUID. The returned SyncExperiment has its .result populated and is ready for downstream inspection (e.g. .as_dataframe()) without re-executing the original task.<!-- -->
task and dataset are now optional when creating an experiment via LLMObs.experiment() / LLMObs.async_experiment(), supporting pull-based workflows. Calling run() without providing task and dataset raises a ValueError.<!-- -->
version field is set on a parent LLM span via tool_definitions, the resolved tool version is now also written onto manually-started child tool spans (created via LLMObs.tool()) as meta.tool.version. Previously, this propagation only occurred for tool spans produced by auto-instrumented integrations (OpenAI, LangChain, OpenAI Agents, ReAct) through the LinkTracker dispatch mechanism.<!-- -->
version field is set on a parent LLM span via tool_definitions, the resolved tool version is now also written onto the corresponding child tool span as meta.tool.version. This allows the LLM Observability UI to aggregate tool version counts and filter tool spans by version without correlating to the parent LLM span.<!-- -->
<!-- -->
PYMEM_DOMAIN_MEM (PyMem_Malloc/Calloc/Realloc) in addition to the existing PYMEM_DOMAIN_OBJ coverage when DD_PROFILING_MEMORY_MEM_DOMAIN_ENABLED=true is set. This support is effective on Python 3.12 and newer, making some previously invisible allocations visible in heap profiles in those environments.<!-- -->
DD_ML_JOB_ENV — a single opt-in env var for forwarding configuration into Ray job workers. Set it on the dashboard process as a semicolon-separated list of KEY:VALUE pairs, e.g. DD_SERVICE:my-svc;DD_AGENT_HOST:10.0.0.1, and workers receive them verbatim as DD_SERVICE=my-svc, DD_AGENT_HOST=10.0.0.1, etc.<!-- -->
CI Visibility: Adds ddtrace.testing.logs.DDTestLogsHandler, a public logging.Handler for shipping log records to the Datadog logs intake correlated with CI Visibility test traces, intended for use in subprocesses that execute tests outside of the main pytest process. Also adds ddtrace.testing.logs.CorrelationFilter, a base class for stamping dd.trace_id / dd.span_id onto log records, and a ready-made ddtrace.testing.logs.ThreadLocalCorrelationFilter for the common thread-local case. Subclass CorrelationFilter to plug in other concurrency models (asyncio ContextVar, global state, etc.).
Example:
from pydantic_evals.evaluators import EqualsExpected import logging
from ddtrace.testing.logs import DDTestLogsHandler from ddtrace.testing.logs import ThreadLocalCorrelationFilter
handler = DDTestLogsHandler(service="my-service") correlation = ThreadLocalCorrelationFilter() handler.addFilter(correlation) logging.getLogger().addHandler(handler)
while True:
job = queue.get() correlation.set_context(trace_id=job.trace_id, span_id=job.span_id) run_test(job.item)handler.close()
websocket.connect ASGI message, causing ASGI/FastAPI WebSocket connections to fail with HTTP 500 when AppSec was enabled.<!-- -->
confluent_kafka integration.<!-- -->
waf.duration, waf.duration_ext, rasp.duration, and rasp.duration_ext.<!-- -->
cosmosdb.query span resource names included per-item identifiers (such as document, user, and permission ids) from the request URI, which caused unbounded resource cardinality. The integration now redacts those ids by replacing them with ? while keeping the database (dbs) and collection (colls) names intact. For example, Read /dbs/myDb/colls/myColl/docs/item1 is now reported as Read /dbs/myDb/colls/myColl/docs/?.<!-- -->
DD_CIVISIBILITY_BACKEND_API_TIMEOUT_MILLIS environment variable (previously missing) to override it. The value is expressed in milliseconds (e.g. 60000 for 60 seconds), consistent with the Java tracer. The same timeout now applies uniformly to all backend requests, including skippable test fetches.<!-- -->
wrapt==2.2.0 from the supported dependency range to avoid a regression that breaks wrapped C descriptors.<!-- -->
DD_KOMBU_SERVICE, DD_JINJA2_SERVICE, DD_DJANGO_DATABASE_SERVICE, DD_CELERY_PRODUCER_SERVICE, DD_CELERY_WORKER_SERVICE, and DD_DJANGO_CACHE_SERVICE now override the service tag on their respective spans. Previously they were ignored in favor of the DD_<INTEGRATION>_SERVICE_NAME form.<!-- -->
config.kombu["service"] before patching was applied to producer spans but not consumer spans. Both now read from the same service config source.<!-- -->
None value for req.uri_template caused a TypeError when constructing the route in the Falcon tracing middleware.<!-- -->
Content-Type header carried a parameter (most commonly charset=utf-8).<!-- -->
<!-- -->
<!-- -->
unknown error category from the upstream Claude Agent SDK instead of a descriptive API error. The integration now surfaces the detailed error message from the assistant message content.<!-- -->
ClaudeSDKClient.query() calls produced nested agent spans instead of separate root traces. Agent spans are now finalized eagerly when a ResultMessage is observed, rather than relying on the wrapping async generator's finally block to execute.<!-- -->
*args / **kwargs for any function that declares keyword-only parameters alongside variadic arguments.<!-- -->
<!-- -->
langchain_aws.ChatBedrockConverse spans reported an opaque inference-profile ARN identifier as the model name when an inference profile was used. base_model_id which represents the underlying foundation model is now checked first when extracting model names, and the botocore Bedrock integration reads the resolved base model from a shared in-process cache populated by langchain so the same resolution applies to the underlying bedrock-runtime span.<!-- -->
reasoning_content was missing from streamed chat completions in the OpenAI and LiteLLM integrations when an OpenAI-compatible reasoning provider (e.g. DeepSeek, Qwen) emitted delta.reasoning_content chunks. The aggregated message now captures reasoning text in the output message, matching non-streaming behavior.<!-- -->
gevent monkey-patching threading lock primitives in place after the profiler had already patched them.<!-- -->
Content-Type header carried parameters being dropped.<!-- -->
<!-- -->
awake() on a PeriodicThread after it had been stopped would block forever. awake() now returns immediately in that case.<!-- -->
PeriodicThread callbacks were invisible to Python's cyclic garbage collector and could accumulate when threads used bound methods as targets.<!-- -->
<!-- -->
pydantic_ai.tool / pydantic_ai.agent) and the resource name is the specific tool or agent name, matching Datadog APM convention. This restores per-tool and per-agent grouping on APM service and resource pages. LLM Observability views are unaffected.<!-- -->
<!-- -->
= in the value. tracing: The subprocess integration now correctly scrubs env-var values for env-var names containing digits.<!-- -->
<!-- -->
git warnings when running tests with pytest-xdist. Simultaneous git fetch --update-shallow calls now retry with exponential back-off on .git/shallow.lock contention, skip the fetch entirely once a sibling worker has already unshallowed the repository, and run under the C locale so the contention check is robust against non-English git messages. git merge-base is deferred until after unshallowing so the required commits are available locally.<!-- -->
<!-- -->
_current_trace_context did not propagate _dd.p.llmobs_parent_id into the trace context metadata. Without this value, downstream consumers (e.g. dd-trace-go) could not locate the active LLMObs parent span and would create orphaned child traces instead of correctly continuing the parent trace.<!-- -->
LLMObsSpan now exposes input_documents and output_documents for mutation or removal before export.<!-- -->
openai spans when an OpenAI (or AsyncOpenAI) client and an AzureOpenAI (or AsyncAzureOpenAI) client are instantiated at the same time. Provider is now determined per-call rather than from the most recently constructed client.<!-- -->
<!-- -->
<!-- -->
<!-- -->
<!-- -->
SIGINT and SIGTERM signals.<!-- -->
ddtrace-run caused a traceback on keyboard interrupt.<!-- -->
confluent-kafka integration where internal list_topics calls triggered librdkafka background metadata-refresh tasks.OTEL_METRIC_EXPORT_TIMEOUT and OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE to the allowlist of recognized OpenTelemetry environment variables. Previously, setting any of these variables produced an inaccurate OpenTelemetry configuration ... is not supported by Datadog warning at startup, even though the variables are valid OpenTelemetry SDK settings.<!-- -->
DD_LLMOBS_AGENTLESS_ENABLED=1), APM traces are now exported agentlessly to Datadog's intake. This should not change user-facing behavior: both APM and LLMObs spans remain visible in the UI; LLMObs spans are simply no longer shipped separately for agentless users. Note that setting DD_APM_TRACING_ENABLED=false takes higher precedence and will result in LLMObs span events shipping separately as existing behavior.<!-- -->
info.profiler.settings on the upload event, so profiles can be filtered by individual settings.<!-- -->
socket.gethostname(), ray.get_runtime_context().get_*()) are amortized across the thread lifetime, reducing overhead for high-frequency Ray task and actor-method instrumentation.One column per quarter.