NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
PyPI, #486 most downloaded on PyPI
Datadog APM client library
Latest stable
4.17.2
Last release today
19 Nov 2026
Ships on a steady schedule
a new release about every 2 weeks
Nearly every release is documented
notes for 60 of the last 60 stable releases
5 versions withdrawn
withdrawn after publishing
10 years old
871 releases · first in 2017
Drag across the ruler to compare, or tap a release to open it.
Span.set_struct_tag is deprecated and will be removed in v4.0.0 with no direct replacement.
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
tiktoken library, and insteadstream_options={"include_usage": True} in the OpenAI request.Span.set_struct_tag is deprecated and will be removed in v4.0.0 with no direct replacement.Span.get_struct_tag is deprecated and will be removed in v4.0.0 with no direct replacement.Span.set_tag_str is deprecated and will be removed in version 4.0.0.
As an alternative to Span.set_tag_str, you can use Span.set_tag instead.DD_PROFILING_STACK_V2_ENABLED=false will no longer have an effect starting in 4.0.urllib3 and requests. It does not require enabling APM instrumentation for urllib3 anymore.threading.RLock (reentrant lock) profiling. The Lock profiler now tracks both threading.Lock and threading.RLock usage, providing comprehensive lock contention visibility for Python applications.version argument to LLMObs.pull_datasetversion and latest_version to provide information on the version of the dataset that is being worked with and the latest global version of the dataset, respectivelyversion field. The version field is now omitted unless explicitly set by the user.IndexError.assessment argument in submit_evaluation().
assessment now refers to whether the evaluation itself passes or fails according to your application, rather than the validity of the evaluation result.langchain integration would incorrectly mark Azure OpenAI calls as duplicate llm operations even if the openai integration was enabled.
The langchain integration will trace Azure OpenAI spans as workflow spans if there is an equivalent llm span from the openai integration.async for model in client.models.list()) caused a TypeError: 'async for' requires an object with __aiter__ method, got coroutine. See issue #14574.KeyError exceptions in test runs when gevent is detected within the environment.ray.init().ray.data._internal to the module denylist._acquire method of the Lock profiler (note: this only occurs when assertions are enabled.)DD_PROFILING_API_TIMEOUT doesn't have any effect, and is marked to be removed in upcoming 4.0 release. New environment variable DD_PROFILING_API_TIMEOUT_MS is introduced to configure timeout for uploading profiles to the backend. The default value is 10000 ms (10 seconds)ValueError: Formatting field not found in record: 'dd.service'.Your coding agent can read these notes before it upgrades. Set up the MCP server →
Span.set_struct_tag is deprecated and will be removed in v4.0.0 with no direct replacement.
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
tiktoken library, and insteadstream_options={"include_usage": True} in the OpenAI request.Span.set_struct_tag is deprecated and will be removed in v4.0.0 with no direct replacement.Span.get_struct_tag is deprecated and will be removed in v4.0.0 with no direct replacement.Span.set_tag_str is deprecated and will be removed in version 4.0.0.
As an alternative to Span.set_tag_str, you can use Span.set_tag instead.DD_PROFILING_STACK_V2_ENABLED=false will no longer have an effect starting in 4.0.urllib3 and requests. It does not require enabling APM instrumentation for urllib3 anymore.threading.RLock (reentrant lock) profiling. The Lock profiler now tracks both threading.Lock and threading.RLock usage, providing comprehensive lock contention visibility for Python applications.version argument to LLMObs.pull_datasetversion and latest_version to provide information on the version of the dataset that is being worked with and the latest global version of the dataset, respectivelyversion field. The version field is now omitted unless explicitly set by the user.IndexError.assessment argument in submit_evaluation().
assessment now refers to whether the evaluation itself passes or fails according to your application, rather than the validity of the evaluation result.langchain integration would incorrectly mark Azure OpenAI calls as duplicate llm operations even if the openai integration was enabled.
The langchain integration will trace Azure OpenAI spans as workflow spans if there is an equivalent llm span from the openai integration.async for model in client.models.list()) caused a TypeError: 'async for' requires an object with __aiter__ method, got coroutine. See issue #14574.KeyError exceptions in test runs when gevent is detected within the environment.ray.init().ray.data._internal to the module denylist._acquire method of the Lock profiler (note: this only occurs when assertions are enabled.)DD_PROFILING_API_TIMEOUT doesn't have any effect, and is marked to be removed in upcoming 4.0 release. New environment variable DD_PROFILING_API_TIMEOUT_MS is introduced to configure timeout for uploading profiles to the backend. The default value is 10000 ms (10 seconds)ValueError: Formatting field not found in record: 'dd.service'.Span.set_struct_tag is deprecated and will be removed in v4.0.0 with no direct replacement.
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
Span.set_struct_tag is deprecated and will be removed in v4.0.0 with no direct replacement.Span.get_struct_tag is deprecated and will be removed in v4.0.0 with no direct replacement.Span.set_tag_str is deprecated and will be removed in version 4.0.0.
As an alternative to Span.set_tag_str, you can use Span.set_tag instead.DD_PROFILING_STACK_V2_ENABLED=false will no longer have an effect starting in 4.0.urllib3 and requests. It does not require enabling APM instrumentation for urllib3 anymore.threading.RLock (reentrant lock) profiling. The Lock profiler now tracks both threading.Lock and threading.RLock usage, providing comprehensive lock contention visibility for Python applications.version argument to LLMObs.pull_datasetversion and latest_version to provide information on the version of the dataset that is being worked with and the latest global version of the dataset, respectivelyversion field. The version field is now omitted unless explicitly set by the user.IndexError.assessment argument in submit_evaluation().
assessment now refers to whether the evaluation itself passes or fails according to your application, rather than the validity of the evaluation result.langchain integration would incorrectly mark Azure OpenAI calls as duplicate llm operations even if the openai integration was enabled.
The langchain integration will trace Azure OpenAI spans as workflow spans if there is an equivalent llm span from the openai integration.async for model in client.models.list()) caused a TypeError: 'async for' requires an object with __aiter__ method, got coroutine. See issue #14574.KeyError exceptions in test runs when gevent is detected within the environment.ray.init().ray.data._internal to the module denylist.DD_PROFILING_API_TIMEOUT doesn't have any effect, and is marked to be removed in upcoming 4.0 release. New environment variable DD_PROFILING_API_TIMEOUT_MS is introduced to configure timeout for uploading profiles to the backend. The default value is 10000 ms (10 seconds)ValueError: Formatting field not found in record: 'dd.service'.Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
CI Visibility: This fix addresses a performance issue where repository tags were fetched during the unshallow process to extract commit metadata, causing slowdowns in repositories with many tags.
LLM Observability: Resolves an issue in the bedrock integration where invoking cohere rerank models would result in missing spans due to output formatting index errors.
opentelemetry:
ray: This fix resolves an issue where the tracer raised an error when submitting Ray tasks without explicitly calling ray.init().
tracer: This fix resolves an issue where an application instrumented by ddtrace could crash at start. Fix compatibility with zope.event==6.0
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
version field. The version field is now omitted unless explicitly set by the user.<!-- -->
async for model in client.models.list()) caused a TypeError: 'async for' requires an object with __aiter__ method, got coroutine. See issue #14574.<!-- -->
IndexError.<!-- -->
<!-- -->
<!-- -->
ValueError: Formatting field not found in record: 'dd.service'.Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
CI Visibility: This fix resolves performance issue affecting coverage collection for Python 3.12+
LLM Observability
assessment argument in submit_evaluation().
assessment now refers to whether the evaluation itself passes or fails according to your application, rather than the validity of the evaluation result.langchain integration would incorrectly mark Azure OpenAI calls as duplicate llm operations even if the openai integration was enabled. The langchain integration will trace Azure OpenAI spans as workflow spans if there is an equivalent llm span from the openai integration.<!-- -->
<!-- -->
Pin to wrapt<2 until we can ensure full compatibility with the breaking changes.
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
LLMObs.enableLLMObs.submit_evaluation_for() has been deprecated and will be removed in a future version. It will be replaced with LLMObs.submit_evaluation() which will take the signature of the original LLMObs.submit_evaluation_for() method in ddtrace version 4.0. Please use LLMObs.submit_evaluation() for submitting evaluations moving forward.
To migrate:
LLMObs.submit_evaluation_for(...) users: rename to LLMObs.submit_evaluation(...)LLMObs.submit_evaluation_for(...) users: rename the span_context argument to span, i.e. LLMObs.submit_evaluation(span_context={"span_id": ..., "trace_id": ...}, ...) to LLMObs.submit_evaluation(span={"span_id": ..., "trace_id": ...}, ...)Tracer.on_start_span and Tracer.deregister_on_start_span are deprecated and will be removed in v4.0.0 with no planned replacement.asyncio.create_task().asyncio and futures integrations are now enabled by default on LLMObs.enable(), which enables asynchronous context propagation for those libraries.LLMObs.submit_evaluation() and LLMObs.submit_evaluation_for() methods now accept a reasoning argument to denote an explanation of the evaluation results.parse() methods used for structured outputs.LLMObs.submit_evaluation_for() method now accepts a assessment argument to denote
whether or not the evaluation is valid or correct. Accepted values are either "pass" or "fail".parse() methods for structured outputs on chat.completions and responses endpoints (available in OpenAI SDK >= 1.92.0).track_user_id in the ATO SDK, which is equivalent to track_user but does not require the login, only the user id.requests with urllib3\<2.wrapt<2 until we can ensure full compatibility with the breaking changes.sumy package installs files under tests/* in site-packages, and this would cause any modules under tests.* to be considered third-party.AttributeError crash in certain configurations./v1/logs path is correctly added to prevent log payloads from being dropped by the Agent when using OTEL_EXPORTER_OTLP_ENDPOINT configuration. Metrics and traces are unaffected.Pin to wrapt<2 until we can ensure full compatibility with the breaking changes.
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
LLMObs.enableLLMObs.submit_evaluation_for() has been deprecated and will be removed in a future version. It will be replaced with LLMObs.submit_evaluation() which will take the signature of the original LLMObs.submit_evaluation_for() method in ddtrace version 4.0. Please use LLMObs.submit_evaluation() for submitting evaluations moving forward.
To migrate:
LLMObs.submit_evaluation_for(...) users: rename to LLMObs.submit_evaluation(...)LLMObs.submit_evaluation_for(...) users: rename the span_context argument to span, i.e. LLMObs.submit_evaluation(span_context={"span_id": ..., "trace_id": ...}, ...) to LLMObs.submit_evaluation(span={"span_id": ..., "trace_id": ...}, ...)Tracer.on_start_span and Tracer.deregister_on_start_span are deprecated and will be removed in v4.0.0 with no planned replacement.asyncio.create_task().asyncio and futures integrations are now enabled by default on LLMObs.enable(), which enables asynchronous context propagation for those libraries.LLMObs.submit_evaluation() and LLMObs.submit_evaluation_for() methods now accept a reasoning argument to denote an explanation of the evaluation results.parse() methods used for structured outputs.LLMObs.submit_evaluation_for() method now accepts a assessment argument to denote
whether or not the evaluation is valid or correct. Accepted values are either "pass" or "fail".parse() methods for structured outputs on chat.completions and responses endpoints (available in OpenAI SDK >= 1.92.0).track_user_id in the ATO SDK, which is equivalent to track_user but does not require the login, only the user id.requests with urllib3\<2.wrapt<2 until we can ensure full compatibility with the breaking changes.sumy package installs files under tests/* in site-packages, and this would cause any modules under tests.* to be considered third-party.AttributeError crash in certain configurations./v1/logs path is correctly added to prevent log payloads from being dropped by the Agent when using OTEL_EXPORTER_OTLP_ENDPOINT configuration. Metrics and traces are unaffected.Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
version field. The version field is now omitted unless explicitly set by the user.<!-- -->
<!-- -->
<!-- -->
ValueError: Formatting field not found in record: 'dd.service'.Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
<!-- -->
langchain integration would incorrectly mark Azure OpenAI calls as duplicate llm operations even if the openai integration was enabled. The langchain integration will trace Azure OpenAI spans as workflow spans if there is an equivalent llm span from the openai integration.<!-- -->
Pin to wrapt<2 until we can ensure full compatibility with the breaking changes
wrapt<2 until we can ensure full compatibility with the breaking changesEstimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
sumy package installs files under tests/* in site-packages, and this would cause any modules under tests.* to be considered third-party.<!-- -->
<!-- -->
AttributeError crash in certain configurations.<!-- -->
<!-- -->
<!-- -->
<!-- -->
vertica: The vertica integration is deprecated and will be removed in a future version, around the same time that ddtrace drops support for Python 3.9…
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
opentelemetry: Adds default configurations for the OpenTelemetry Metrics API implementation to improve the Datadog user experience. This includes the following configurations:
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT is set to the default Datadog Agent endpoint, or localhost if not foundOTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE is set to deltaOTEL_METRIC_EXPORT_INTERVAL is set to 10000OTEL_METRIC_EXPORT_TIMEOUT is set to 7500LLM Observability: MCP integration also traces ClientSession contexts, ClientSession.initialize, and ClientSession.list_tools.
ray: This introduces a Ray core integration that traces Ray jobs, remote tasks, and actor method calls. Supported for Ray >= 2.46.0.
To enable tracing, start the Ray head with --tracing-startup-hook=ddtrace.contrib.ray:setup_tracing then submit jobs as usual.
AttributeError that could occur when tracing Google ADK agent runs, due to the agent model attribute not being defined for SequentialAgent class.ImportError for when using langchain_core>=0.3.76.DD_APM_TRACING_ENABLED=0 when using LLM Observability.vertica: The vertica integration is deprecated and will be removed in a future version, around the same time that ddtrace drops support for Python 3.9…
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
opentelemetry: Adds default configurations for the OpenTelemetry Metrics API implementation to improve the Datadog user experience. This includes the following configurations:
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT is set to the default Datadog Agent endpoint, or localhost if not foundOTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE is set to deltaOTEL_METRIC_EXPORT_INTERVAL is set to 10000OTEL_METRIC_EXPORT_TIMEOUT is set to 7500LLM Observability: MCP integration also traces ClientSession contexts, ClientSession.initialize, and ClientSession.list_tools.
ray: This introduces a Ray core integration that traces Ray jobs, remote tasks, and actor method calls. Supported for Ray >= 2.46.0.
To enable tracing, start the Ray head with --tracing-startup-hook=ddtrace.contrib.ray:setup_tracing then submit jobs as usual.
AttributeError that could occur when tracing Google ADK agent runs, due to the agent model attribute not being defined for SequentialAgent class.ImportError for when using langchain_core>=0.3.76.vertica: The vertica integration is deprecated and will be removed in a future version, around the same time that ddtrace drops support for Python 3.9…
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
opentelemetry: Adds default configurations for the OpenTelemetry Metrics API implementation to improve the Datadog user experience. This includes the following configurations:
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT is set to the default Datadog Agent endpoint, or localhost if not foundOTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE is set to deltaOTEL_METRIC_EXPORT_INTERVAL is set to 10000OTEL_METRIC_EXPORT_TIMEOUT is set to 7500LLM Observability: MCP integration also traces ClientSession contexts, ClientSession.initialize, and ClientSession.list_tools.
ray: This introduces a Ray core integration that traces Ray jobs, remote tasks, and actor method calls. Supported for Ray >= 2.46.0.
To enable tracing, start the Ray head with --tracing-startup-hook=ddtrace.contrib.ray:setup_tracing then submit jobs as usual.
AttributeError that could occur when tracing Google ADK agent runs, due to the agent model attribute not being defined for SequentialAgent class.ImportError for when using langchain_core>=0.3.76.Pin to wrapt<2 until we can ensure full compatibility with the breaking changes.
Pin to wrapt<2 until we can ensure full compatibility with the breaking changes.
CI Visibility: This fix resolves an issue where tests would be incorrectly detected as third-party code if a third-party package containing a folder with the same name as the tests folder was installed. For instance, the sumy package installs files under tests/* in site-packages, and this would cause any modules under tests.* to be considered third-party.
langchain: Resolves an issue where langchain patching would throw an ImportError for when using langchain_core>=0.3.76.
Untrusted Serialization detection, which will be displayed on your DataDog Vulnerability Explorer dashboard. See the Application Vulnerability Managem…
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
DD_DJANGO_TRACING_MINIMAL environment variable for performance-sensitive applications. When enabled, this disables Django ORM, cache, and template instrumentation while keeping middleware instrumentation enabled. This significantly reduces overhead by removing Django-specific spans while preserving visibility into the underlying database drivers, cache clients, and other integrations. For example, with this enabled, Django ORM query spans are disabled but database driver spans (e.g., psycopg, MySQLdb) will still be created. To enable minimal tracing, set DD_DJANGO_TRACING_MINIMAL=true.aws.partition tag onto AWS traces based on the region for the boto, botocore, and aiobotocore integrations.DD_CIVISIBILITY_ENABLED (with default value True) so it can be disabled to avoid sending traces to the Test Visibility product from the test runners.DD_TRACE_RESOURCE_RENAMING_ENABLED="true"usedforsecurity=False.tags and chat_template, and a new Prompt TypedDict class that would be used in annotation and annotation_context.DatadogSampler from getting recreated whenever the SpanAggregator is reset, and instead updates the rate limiter that the sampler uses.PyErr_Format.-m entries in the denylist.asyncio.Tasks.PyObject_Realloc call, which can lead to accessing freed memory.--skip-atexit is not set when --lazy or --lazy-apps is set on uWSGI<2.0.30. This is to prevent crashes from profiling native extension modules. See https://github.com/unbit/uwsgi/pull/2726 for details.c_char type raised a TypeError, this now uses the 'c' typecode for better compatibility across versions.DD_GIT_COMMIT_SHA and DD_GIT_REPOSITORY_URL are defined before using the git command.Untrusted Serialization detection, which will be displayed on your DataDog Vulnerability Explorer dashboard. See the Application Vulnerability Managem…
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
DD_DJANGO_TRACING_MINIMAL environment variable for performance-sensitive applications. When enabled, this disables Django ORM, cache, and template instrumentation while keeping middleware instrumentation enabled. This significantly reduces overhead by removing Django-specific spans while preserving visibility into the underlying database drivers, cache clients, and other integrations. For example, with this enabled, Django ORM query spans are disabled but database driver spans (e.g., psycopg, MySQLdb) will still be created. To enable minimal tracing, set DD_DJANGO_TRACING_MINIMAL=true.aws.partition tag onto AWS traces based on the region for the boto, botocore, and aiobotocore integrations.DD_CIVISIBILITY_ENABLED (with default value True) so it can be disabled to avoid sending traces to the Test Visibility product from the test runners.DD_TRACE_RESOURCE_RENAMING_ENABLED="true"usedforsecurity=False.tags and chat_template, and a new Prompt TypedDict class that would be used in annotation and annotation_context.DatadogSampler from getting recreated whenever the SpanAggregator is reset, and instead updates the rate limiter that the sampler uses.PyErr_Format.-m entries in the denylist.asyncio.Tasks.PyObject_Realloc call, which can lead to accessing freed memory.--skip-atexit is not set when --lazy or --lazy-apps is set on uWSGI<2.0.30. This is to prevent crashes from profiling native extension modules. See https://github.com/unbit/uwsgi/pull/2726 for details.c_char type raised a TypeError, this now uses the 'c' typecode for better compatibility across versions.DD_GIT_COMMIT_SHA and DD_GIT_REPOSITORY_URL are defined before using the git command.Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
AAP:
CI Visibility: This fix solves an issue where the ITR skip count metric was aggregating skipped tests even when skipping level was set to suite. It will now count appropriately (skipped suites or skipped tests) depending on ITR skip level.
Estimated end-of-life date, accurate to within three months: 09-2026 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 09-2026 See the support level definitions for more information.
<!-- -->
<!-- -->
python -m and deny py_compile.Estimated end-of-life date, accurate to within three months: 09-2026 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 09-2026 See the support level definitions for more information.
<!-- -->
<!-- -->
DD_GIT_COMMIT_SHA and DD_GIT_REPOSITORY_URL are defined before using the git command.Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
ml_app is now optional, defaulting to service. while it is still recommended to set ml_app, enabling LLM Observability will no longer throw if one is not provided or propagated from an upstream service.tool_definitions parameter to the LLMObs.annotate() method for tool calling scenarios. Users can now pass a list of tool definition dictionaries directly to annotate LLM spans with available tools. Each tool definition must include a name (string) field, with optional description (string) and schema (JSON-serializable dictionary) fields.create_dataset, create_dataset_from_csv and pull_dataset) argument name is changed to dataset_name.ValueError: coroutine already executing on Python 3.13+ with django.utils.decorators.async_only_middleware."Error: expected pool connect callback to return an instance of 'asyncpg.connection.Connection', got 'ddtrace.contrib.internal.asyncpg.patch._TracedConnection'" due to using the custom connect option. With this fix, postgres.connect spans will be created when this option is used.pkg_resources, either directly or indirectly.Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
ml_app is now optional, defaulting to service. while it is still recommended to set ml_app, enabling LLM Observability will no longer throw if one is not provided or propagated from an upstream service.tool_definitions parameter to the LLMObs.annotate() method for tool calling scenarios. Users can now pass a list of tool definition dictionaries directly to annotate LLM spans with available tools. Each tool definition must include a name (string) field, with optional description (string) and schema (JSON-serializable dictionary) fields.create_dataset, create_dataset_from_csv and pull_dataset) argument name is changed to dataset_name.ValueError: coroutine already executing on Python 3.13+ with django.utils.decorators.async_only_middleware."Error: expected pool connect callback to return an instance of 'asyncpg.connection.Connection', got 'ddtrace.contrib.internal.asyncpg.patch._TracedConnection'" due to using the custom connect option. With this fix, postgres.connect spans will be created when this option is used.pkg_resources, either directly or indirectly.Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
ml_app is now optional, defaulting to service. while it is still recommended to set ml_app, enabling LLM Observability will no longer throw if one is not provided or propagated from an upstream service.tool_definitions parameter to the LLMObs.annotate() method for tool calling scenarios. Users can now pass a list of tool definition dictionaries directly to annotate LLM spans with available tools. Each tool definition must include a name (string) field, with optional description (string) and schema (JSON-serializable dictionary) fields."Error: expected pool connect callback to return an instance of 'asyncpg.connection.Connection', got 'ddtrace.contrib.internal.asyncpg.patch._TracedConnection'" due to using the custom connect option. With this fix, postgres.connect spans will be created when this option is used.pkg_resources, either directly or indirectly.Estimated end-of-life date, accurate to within three months: 09-2026 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 09-2026 See the support level definitions for more information.
tracing: Fixes encoding bytes objects as span attributes by truncating byte string, rather than throwing PyErr_Format.
libinjection: allow deny listing python modules executed with python -m and deny py_compile.
Estimated end-of-life date, accurate to within three months: 09-2026 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 09-2026 See the support level definitions for more information.
sampling: This change prevents the DatadogSampler from getting recreated whenever the SpanAggregator is reset, and instead updates the rate limiter that the sampler uses.
source code integration: check that DD_GIT_COMMIT_SHA and DD_GIT_REPOSITORY_URL are defined before using the git command.
CI Visibility: This fix resolves an issue where coverage from sessions with pytest-xdist were not submitted with the proper session id, preventing Test Impact Analysis feature from working properly.
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
__repr__ on SpanPointer objected raised an AttributeError. This caused aws_lambdas to crash when debug logging was enabled.Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
ValueError: coroutine already executing on Python 3.13+ with django.utils.decorators.async_only_middleware.tracing: ddtrace.tracer.Pin is deprecated and will be removed in version 4.0.0. To manage configuration of the tracer or integrations please use envir…
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
ddtrace.tracer.Pin is deprecated and will be removed in version 4.0.0. To manage configuration of the tracer or integrations please use environment variables.opentelemetry
opentelemetry-otlp-exporter and set DD_LOGS_OTEL_ENABLED=true.LLM Observability
LLMObs.annotate() method now accept input and output data with optional to ol_calls and tool_results fields for function calling scenarios.Django: Added the DD_DJANGO_ALWAYS_CREATE_DATABASE_SPANS config option (default: true).
When enabled, the Django integration always generates a database span for every operation, even if the underlying database engine is already instrumented. This ensures complete coverage but may produce duplicat e spans and extra overhead.
When disabled, spans are only created if the database engine is not instrumented.
To avoid multiple spans per database call, we recommend disabling this option. DD_DJANGO_ALWAYS_CREATE_DATABASE_SPANS=false
AAP
AAP: Fixes an issue where security signals would be incorrectly reported on an inferred proxy service instead of the current service.
CI Visibility: This fix resolves an issue where the pytest plugin would hold a reference to test exceptions beyond the end of the test, preventing them from being garbage-collected and increasing memory usage.
psycopg: This fix resolves a potential circular import with the psycopg3 contrib.
internal: This fix resolves an issue where the tracer flare was not sent when DD_TRACE_AGENT_URL was not set, as the default URL was not used.
tracing
websocket.receive span not closing exactly when another websocket.receive span was opened.websocket.close parent should be the handshake span when configuration is disabled.ddtrace.trace.tracer.configure(...) resets the trace writer buffer, causing spans to be dropped.Code Security: Fixed a crash in the taint-aware modulo aspect when formatting SQLAlchemy objects whose __repr__ can raise (e.g., inside complex CASE expressions).
LLM Observability: Properly parse DD_TAGS onto LLM Observability span events' tags.
sampling
_dd.p.dm=-3) during distributed tracing header extraction.ssi, crashtracker: This fix resolves an issue where crashtracker receiver binary was not available in an injected environment.
tracing: ddtrace.tracer.Pin is deprecated and will be removed in version 4.0.0. To manage configuration of the tracer or integrations please use envir…
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
ddtrace.tracer.Pin is deprecated and will be removed in version 4.0.0. To manage configuration of the tracer or integrations please use environment variables.opentelemetry
opentelemetry-otlp-exporter and set DD_LOGS_OTEL_ENABLED=true.LLM Observability
LLMObs.annotate() method now accept input and output data with optional to ol_calls and tool_results fields for function calling scenarios.Django: Added the DD_DJANGO_ALWAYS_CREATE_DATABASE_SPANS config option (default: true).
When enabled, the Django integration always generates a database span for every operation, even if the underlying database engine is already instrumented. This ensures complete coverage but may produce duplicat e spans and extra overhead.
When disabled, spans are only created if the database engine is not instrumented.
To avoid multiple spans per database call, we recommend disabling this option. DD_DJANGO_ALWAYS_CREATE_DATABASE_SPANS=false
AAP
AAP: Fixes an issue where security signals would be incorrectly reported on an inferred proxy service instead of the current service.
CI Visibility: This fix resolves an issue where the pytest plugin would hold a reference to test exceptions beyond the end of the test, preventing them from being garbage-collected and increasing memory usage.
psycopg: This fix resolves a potential circular import with the psycopg3 contrib.
internal: This fix resolves an issue where the tracer flare was not sent when DD_TRACE_AGENT_URL was not set, as the default URL was not used.
tracing
websocket.receive span not closing exactly when another websocket.receive span was opened.websocket.close parent should be the handshake span when configuration is disabled.ddtrace.trace.tracer.configure(...) resets the trace writer buffer, causing spans to be dropped.Code Security: Fixed a crash in the taint-aware modulo aspect when formatting SQLAlchemy objects whose __repr__ can raise (e.g., inside complex CASE expressions).
LLM Observability: Properly parse DD_TAGS onto LLM Observability span events' tags.
sampling
_dd.p.dm=-3) during distributed tracing header extraction.ssi, crashtracker: This fix resolves an issue where crashtracker receiver binary was not available in an injected environment.
Estimated end-of-life date, accurate to within three months: 09-2026 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 09-2026 See the support level definitions for more information.
python -m and deny py_compile.Estimated end-of-life date, accurate to within three months: 09-2026 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 09-2026 See the support level definitions for more information.
DD_GIT_COMMIT_SHA and DD_GIT_REPOSITORY_URL are defined before using the git command.Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
__repr__ on SpanPointer objected raised an AttributeError. This caused aws_lambdas to crash when debug logging was enabled.Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
Versions 3.12.1 through 3.12.4 did not include the full set of updates originally delivered in 3.12.0. As a result, several enhancements — including the critical profiling fix described below — were not present in those patch releases.
With 3.12.5, all updates are now fully aligned and included. We recommend upgrading directly to 3.12.5 to ensure you have the complete set of improvements and fixes.
asyncpg: Fix the error "Error: expected pool connect callback to return an instance of 'asyncpg.connection.Connection', got 'ddtrace.contrib.internal.
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
DD_TRACE_WEBSOCKET_MESSAGES_SEPARATE_TRACES is disabled.Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
pytest plugin would hold a reference to test exceptions beyond the end of the test, preventing them from being garbage-collected and increasing memory usage.Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
openai: Resolves an issue in the OpenAI integration where asyncio.CancelledError was not caught or re-raised.
LLM Observability: Properly parse DD_TAGS onto LLM Observability span events' tags.
ddtrace.settings.__init__ imports are deprecated and will be removed in version 4.0.0.
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
AI Guard is an upcoming Datadog security product currently under active design and development. Note: The Python SDK is released as a technical preview. Functionality and APIs are subject to change, and backward compatibility is not guaranteed at this stage.
DD_PROFILING_TIMELINE_ENABLED=True). This provides visualization of profiling data with timing information.ddtrace.settings.__init__ imports are deprecated and will be removed in version 4.0.0.HTTPPropagator.inject method. HTTPPropagator.inject(context=...) should be used to inject headers instead.DD_TRACE_WEBSOCKET_MESSAGES_ENABLED, which replaces DD_TRACE_WEBSOCKET_MESSAGES.CIVisibility tracer is kept separate from the global ddtrace tracer, which helps avoid interference between test and non-test tracer configurations. This mode is currently experimental, and can be enabled by setting the environment variable DD_CIVISIBILITY_USE_BETA_WRITER to true.kickoff/kickoff_async calls, including tracing internal flow method execution.SpanProcessor return None.DD_MCP_DISTRIBUTED_TRACING=False for both the client and server.route parameter was not being correctly handled in the Django path function.skipif marker with the condition passed as a keyword argument (or not provided at all) would cause the test to be reported as failed, in particular when the flaky or pytest-rerunfailures were also used.patch() with no arguments, and thus patch_all(), breaks the integration.bytes.OpenAI/AsyncOpenAI client would result in an AttributeError.pydantic-ai-slim >= 0.4.4 would fail. See this issue <https://github.com/DataDog/dd-trace-py/issues/14161>_ for more details.DD_TRACE_SAMPLING_RULES='\[{"resource": null, "sample_rate": 1}\]' will be equivalent to DD_TRACE_SAMPLING_RULES='\[{"sample_rate": 1}\]'.list_topics call in the Kafka integration could hang indefinitely. The integration now sets a 1-second timeout on `list_topics</span> calls and caches both successful cluster ID results and failures (with a 5-minute retry interval) to prevent repeated slow metadata queries.google-genai would result in no output messages on the LLM Observability llm span.ModuleNotFoundError errors when patching langgraph>=0.6.0openai>=1.66.0,<1.66.2 would result in an AttributeError.HTTPPropagator.inject method to help diagnose issues with sampling decisions.DD_PROFILING_MAX_EVENTS is deprecated and does nothing. Use DD_PROFILING_HEAP_SAMPLE_SIZE to control sampling frequency of the memory profiler.ddtrace.settings.__init__ imports are deprecated and will be removed in version 4.0.0.
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
AI Guard is an upcoming Datadog security product currently under active design and development. Note: The Python SDK is released as a technical preview. Functionality and APIs are subject to change, and backward compatibility is not guaranteed at this stage.
DD_PROFILING_TIMELINE_ENABLED=True). This provides visualization of profiling data with timing information.ddtrace.settings.__init__ imports are deprecated and will be removed in version 4.0.0.HTTPPropagator.inject method. HTTPPropagator.inject(context=...) should be used to inject headers instead.DD_TRACE_WEBSOCKET_MESSAGES_ENABLED, which replaces DD_TRACE_WEBSOCKET_MESSAGES.CIVisibility tracer is kept separate from the global ddtrace tracer, which helps avoid interference between test and non-test tracer configurations. This mode is currently experimental, and can be enabled by setting the environment variable DD_CIVISIBILITY_USE_BETA_WRITER to true.kickoff/kickoff_async calls, including tracing internal flow method execution.SpanProcessor return None.DD_MCP_DISTRIBUTED_TRACING=False for both the client and server.route parameter was not being correctly handled in the Django path function.skipif marker with the condition passed as a keyword argument (or not provided at all) would cause the test to be reported as failed, in particular when the flaky or pytest-rerunfailures were also used.patch() with no arguments, and thus patch_all(), breaks the integration.bytes.OpenAI/AsyncOpenAI client would result in an AttributeError.pydantic-ai-slim >= 0.4.4 would fail. See this issue <https://github.com/DataDog/dd-trace-py/issues/14161>_ for more details.DD_TRACE_SAMPLING_RULES='\[{"resource": null, "sample_rate": 1}\]' will be equivalent to DD_TRACE_SAMPLING_RULES='\[{"sample_rate": 1}\]'.list_topics call in the Kafka integration could hang indefinitely. The integration now sets a 1-second timeout on `list_topics</span> calls and caches both successful cluster ID results and failures (with a 5-minute retry interval) to prevent repeated slow metadata queries.google-genai would result in no output messages on the LLM Observability llm span.ModuleNotFoundError errors when patching langgraph>=0.6.0openai>=1.66.0,<1.66.2 would result in an AttributeError.HTTPPropagator.inject method to help diagnose issues with sampling decisions.DD_PROFILING_MAX_EVENTS is deprecated and does nothing. Use DD_PROFILING_HEAP_SAMPLE_SIZE to control sampling frequency of the memory profiler.ddtrace.settings.__init__ imports are deprecated and will be removed in version 4.0.0.
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
AI Guard is an upcoming Datadog security product currently under active design and development. Note: The Python SDK is released as a technical preview. Functionality and APIs are subject to change, and backward compatibility is not guaranteed at this stage.
DD_PROFILING_TIMELINE_ENABLED=True). This provides visualization of profiling data with timing information.ddtrace.settings.__init__ imports are deprecated and will be removed in version 4.0.0.HTTPPropagator.inject method. HTTPPropagator.inject(context=...) should be used to inject headers instead.DD_TRACE_WEBSOCKET_MESSAGES_ENABLED, which replaces DD_TRACE_WEBSOCKET_MESSAGES.CIVisibility tracer is kept separate from the global ddtrace tracer, which helps avoid interference between test and non-test tracer configurations. This mode is currently experimental, and can be enabled by setting the environment variable DD_CIVISIBILITY_USE_BETA_WRITER to true.SpanProcessor return None.DD_MCP_DISTRIBUTED_TRACING=False for both the client and server.route parameter was not being correctly handled in the Django path function.skipif marker with the condition passed as a keyword argument (or not provided at all) would cause the test to be reported as failed, in particular when the flaky or pytest-rerunfailures were also used.patch() with no arguments, and thus patch_all(), breaks the integration.bytes.OpenAI/AsyncOpenAI client would result in an AttributeError.pydantic-ai-slim >= 0.4.4 would fail. See this issue <https://github.com/DataDog/dd-trace-py/issues/14161>_ for more details.DD_TRACE_SAMPLING_RULES='\[{"resource": null, "sample_rate": 1}\]' will be equivalent to DD_TRACE_SAMPLING_RULES='\[{"sample_rate": 1}\]'.list_topics call in the Kafka integration could hang indefinitely. The integration now sets a 1-second timeout on `list_topics</span> calls and caches both successful cluster ID results and failures (with a 5-minute retry interval) to prevent repeated slow metadata queries.google-genai would result in no output messages on the LLM Observability llm span.ModuleNotFoundError errors when patching langgraph>=0.6.0openai>=1.66.0,<1.66.2 would result in an AttributeError.HTTPPropagator.inject method to help diagnose issues with sampling decisions.DD_PROFILING_MAX_EVENTS is deprecated and does nothing. Use DD_PROFILING_HEAP_SAMPLE_SIZE to control sampling frequency of the memory profiler.ddtrace.settings.__init__ imports are deprecated and will be removed in version 4.0.0.
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
AI Guard is an upcoming Datadog security product currently under active design and development. Note: The Python SDK is released as a technical preview. Functionality and APIs are subject to change, and backward compatibility is not guaranteed at this stage.
DD_PROFILING_TIMELINE_ENABLED=True). This provides visualization of profiling data with timing information.ddtrace.settings.__init__ imports are deprecated and will be removed in version 4.0.0.HTTPPropagator.inject method. HTTPPropagator.inject(context=...) should be used to inject headers instead.DD_TRACE_WEBSOCKET_MESSAGES_ENABLED, which replaces DD_TRACE_WEBSOCKET_MESSAGES.CIVisibility tracer is kept separate from the global ddtrace tracer, which helps avoid interference between test and non-test tracer configurations. This mode is currently experimental, and can be enabled by setting the environment variable DD_CIVISIBILITY_USE_BETA_WRITER to true.SpanProcessor return None.DD_MCP_DISTRIBUTED_TRACING=False for both the client and server.route parameter was not being correctly handled in the Django path function.skipif marker with the condition passed as a keyword argument (or not provided at all) would cause the test to be reported as failed, in particular when the flaky or pytest-rerunfailures were also used.patch() with no arguments, and thus patch_all(), breaks the integration.bytes.OpenAI/AsyncOpenAI client would result in an AttributeError.pydantic-ai-slim >= 0.4.4 would fail. See this issue <https://github.com/DataDog/dd-trace-py/issues/14161>_ for more details.DD_TRACE_SAMPLING_RULES='\[{"resource": null, "sample_rate": 1}\]' will be equivalent to DD_TRACE_SAMPLING_RULES='\[{"sample_rate": 1}\]'.list_topics call in the Kafka integration could hang indefinitely. The integration now sets a 1-second timeout on `list_topics</span> calls and caches both successful cluster ID results and failures (with a 5-minute retry interval) to prevent repeated slow metadata queries.google-genai would result in no output messages on the LLM Observability llm span.ModuleNotFoundError errors when patching langgraph>=0.6.0openai>=1.66.0,<1.66.2 would result in an AttributeError.HTTPPropagator.inject method to help diagnose issues with sampling decisions.DD_PROFILING_MAX_EVENTS is deprecated and does nothing. Use DD_PROFILING_HEAP_SAMPLE_SIZE to control sampling frequency of the memory profiler.Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
pytest plugin would hold a reference to test exceptions beyond the end of the test, preventing them from being garbage-collected and increasing memory usage.asyncio.CancelledError was not caught or re-raised.google-genai would result in no output messages on the LLM Observability llm span.ModuleNotFoundError errors when patching langgraph>=0.6.0openai>=1.66.0,<1.66.2 would result in an AttributeError.AAP: resolves a bug where ASGI middleware would not catch the BlockingException raised by AAP because it was aggregated in an ExceptionGroup
route parameter was not being correctly handled in the Django path function.bytes.OpenAI/AsyncOpenAI client would result in an AttributeError.this issue <https://github.com/DataDog/dd-trace-py/issues/14161>_ for more details.Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
skipif marker with the condition passed as a keyword argument (or not provided at all) would cause the test to be reported as failed, in particular when the flaky or pytest-rerunfailures were also used.Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
patch() with no arguments, and thus patch_all(), breaks the integration.CI Visibility: The freezegun integration is deprecated and will be removed in 4.0.0. The freezegun integration is not necessary anymore for the correc…
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
pytest does not require coverage.py as a dependency anymore.freezegun integration is deprecated and will be removed in 4.0.0. The freezegun integration is not necessary anymore for the correct reporting of test durations and timestamps.submit_evaluation_for() for submitting boolean metrics in LLMObs evaluation metrics, using metric_type="boolean". This enables tracking binary evaluation results such as toxicity detection and content appropriateness in your LLM application workflow.toolResult content blocks are formatted as tool messages on LLM Observability llm spans' inputs.mcp.client.session.ClientSession.call_tool and mcp.server.fastmcp.tools.tool_manager.ToolManager.call_tool methods in the MCP SDK.DD_METRICS_OTEL_ENABLED must be set to true and the application must include its own OTLP metrics exporter.DD_ASGI_OBFUSCATE_404_RESOURCE is enabled (disabled by default).ddtrace-run or import ddtrace.auto. To disable this feature, set the environment variable DD_LOGS_INJECTION to False.CI Visibility
freezegun would not work with tests defined in unittest classes.flaky or pytest-rerunfailures would cause the test results not to be reported correctly to Datadog. With this change, those plugins can be used with ddtrace, and test results will be reported to Datadog, but Test Optimization advanced features such as Early Flake Detection and Auto Test Retries will not be available when such plugins are used.pytest, leading to I/O operation on closed file errors at the end of the test session.pytest-xdist.AAP: This fix resolves an issue where the FastAPI body extraction was not functioning correctly in asynchronous contexts for large bodies, leading to missing security events. The timeout for reading request body chunks has been set to 0.1 seconds to ensure timely processing without blocking the event loop. This can be configured using the DD_FASTAPI_ASYNC_BODY_TIMEOUT_SECONDS environment variable.
litellm: This fix resolves an issue where potentially sensitive parameters were being tagged as metadata on LLM Observability spans. Now, metadata tags are based on an allowlist instead of a denylist.
lib-injection: Fix a bug preventing the Single Step Instrumentation (SSI) telemetry forwarder from completing when debug logging was enabled.
litellm: This fix resolves an issue where potentially sensitive parameters were being tagged as metadata on LLM Observability spans. Now, metadata tags are based on an allowlist instead of a denylist.
LLM Observability
AttributeError while parsing NoneType streamed chunk deltas.AttributeError.AttributeError while parsing NoneType streamed chunk deltas.langgraph>=0.3.22.langchain libraries raised a ValueError.dynamic instrumentation: improve support for function probes with frameworks and applications that interact with the Python garbage collector (e.g. synapse).
logging: Fix issue when dd.* properties were not injected onto logging records unless DD_LOGS_ENABLED=true env var was set (default value is structured). This issue causes problems for non-structured loggers which set their own format string instead of having ddtrace set the logging format string for you.
azure_functions: This fix resolves an issue where a function that consumes a list of service bus messages throws an exception when instrumented.
profiling
tracing
@tracer.wrap() decorator failed to preserve the decorated function's return type, returning Any instead of the original return type.freezegun was in use. With this change, the freezegun integration is not necessary anymore.LONG_MAX caused traces to fail to send.Code Security (IAST)
mysqlsh (MySQL Shell) reassigns globals with a custom object. This can interfere with analysis or instrumentation logic.CI Visibility: The freezegun integration is deprecated and will be removed in 4.0.0. The freezegun integration is not necessary anymore for the correc…
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
pytest does not require coverage.py as a dependency anymore.freezegun integration is deprecated and will be removed in 4.0.0. The freezegun integration is not necessary anymore for the correct reporting of test durations and timestamps.submit_evaluation_for() for submitting boolean metrics in LLMObs evaluation metrics, using metric_type="boolean". This enables tracking binary evaluation results such as toxicity detection and content appropriateness in your LLM application workflow.toolResult content blocks are formatted as tool messages on LLM Observability llm spans' inputs.mcp.client.session.ClientSession.call_tool and mcp.server.fastmcp.tools.tool_manager.ToolManager.call_tool methods in the MCP SDK.DD_METRICS_OTEL_ENABLED must be set to true and the application must include its own OTLP metrics exporter.DD_ASGI_OBFUSCATE_404_RESOURCE is enabled (disabled by default).ddtrace-run or import ddtrace.auto. To disable this feature, set the environment variable DD_LOGS_INJECTION to False.CI Visibility
freezegun would not work with tests defined in unittest classes.flaky or pytest-rerunfailures would cause the test results not to be reported correctly to Datadog. With this change, those plugins can be used with ddtrace, and test results will be reported to Datadog, but Test Optimization advanced features such as Early Flake Detection and Auto Test Retries will not be available when such plugins are used.pytest, leading to I/O operation on closed file errors at the end of the test session.pytest-xdist.AAP: This fix resolves an issue where the FastAPI body extraction was not functioning correctly in asynchronous contexts for large bodies, leading to missing security events. The timeout for reading request body chunks has been set to 0.1 seconds to ensure timely processing without blocking the event loop. This can be configured using the DD_FASTAPI_ASYNC_BODY_TIMEOUT_SECONDS environment variable.
litellm: This fix resolves an issue where potentially sensitive parameters were being tagged as metadata on LLM Observability spans. Now, metadata tags are based on an allowlist instead of a denylist.
lib-injection: Fix a bug preventing the Single Step Instrumentation (SSI) telemetry forwarder from completing when debug logging was enabled.
litellm: This fix resolves an issue where potentially sensitive parameters were being tagged as metadata on LLM Observability spans. Now, metadata tags are based on an allowlist instead of a denylist.
LLM Observability
AttributeError while parsing NoneType streamed chunk deltas.AttributeError.AttributeError while parsing NoneType streamed chunk deltas.langgraph>=0.3.22.langchain libraries raised a ValueError.dynamic instrumentation: improve support for function probes with frameworks and applications that interact with the Python garbage collector (e.g. synapse).
logging: Fix issue when dd.* properties were not injected onto logging records unless DD_LOGS_ENABLED=true env var was set (default value is structured). This issue causes problems for non-structured loggers which set their own format string instead of having ddtrace set the logging format string for you.
azure_functions: This fix resolves an issue where a function that consumes a list of service bus messages throws an exception when instrumented.
profiling
tracing
@tracer.wrap() decorator failed to preserve the decorated function's return type, returning Any instead of the original return type.freezegun was in use. With this change, the freezegun integration is not necessary anymore.LONG_MAX caused traces to fail to send.Code Security (IAST)
mysqlsh (MySQL Shell) reassigns globals with a custom object. This can interfere with analysis or instrumentation logic.CI Visibility: The freezegun integration is deprecated and will be removed in 4.0.0. The freezegun integration is not necessary anymore for the correc…
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
pytest does not require coverage.py as a dependency anymore.freezegun integration is deprecated and will be removed in 4.0.0. The freezegun integration is not necessary anymore for the correct reporting of test durations and timestamps.submit_evaluation_for() for submitting boolean metrics in LLMObs evaluation metrics, using metric_type="boolean". This enables tracking binary evaluation results such as toxicity detection and content appropriateness in your LLM application workflow.toolResult content blocks are formatted as tool messages on LLM Observability llm spans' inputs.mcp.client.session.ClientSession.call_tool and mcp.server.fastmcp.tools.tool_manager.ToolManager.call_tool methods in the MCP SDK.DD_METRICS_OTEL_ENABLED must be set to true and the application must include its own OTLP metrics exporter.DD_ASGI_OBFUSCATE_404_RESOURCE is enabled (disabled by default).ddtrace-run or import ddtrace.auto. To disable this feature, set the environment variable DD_LOGS_INJECTION to False.freezegun would not work with tests defined in unittest classes.flaky or pytest-rerunfailures would cause the test results not to be reported correctly to Datadog. With this change, those plugins can be used with ddtrace, and test results will be reported to Datadog, but Test Optimization advanced features such as Early Flake Detection and Auto Test Retries will not be available when such plugins are used.pytest, leading to I/O operation on closed file errors at the end of the test session.pytest-xdist.DD_FASTAPI_ASYNC_BODY_TIMEOUT_SECONDS environment variable.AttributeError while parsing NoneType streamed chunk deltas.AttributeError.AttributeError while parsing NoneType streamed chunk deltas.langgraph>=0.3.22.langchain libraries raised a ValueError.dd.* properties were not injected onto logging records unless DD_LOGS_ENABLED=true env var was set (default value is structured). This issue causes problems for non-structured loggers which set their own format string instead of having ddtrace set the logging format string for you.@tracer.wrap() decorator failed to preserve the decorated function's return type, returning Any instead of the original return type.freezegun was in use. With this change, the freezegun integration is not necessary anymore.LONG_MAX caused traces to fail to send.mysqlsh (MySQL Shell) reassigns globals with a custom object. This can interfere with analysis or instrumentation logic.CI Visibility: The freezegun integration is deprecated and will be removed in 4.0.0. The freezegun integration is not necessary anymore for the correc…
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
pytest does not require coverage.py as a dependency anymore.freezegun integration is deprecated and will be removed in 4.0.0. The freezegun integration is not necessary anymore for the correct reporting of test durations and timestamps.submit_evaluation_for() for submitting boolean metrics in LLMObs evaluation metrics, using metric_type="boolean". This enables tracking binary evaluation results such as toxicity detection and content appropriateness in your LLM application workflow.toolResult content blocks are formatted as tool messages on LLM Observability llm spans' inputs.mcp.client.session.ClientSession.call_tool and mcp.server.fastmcp.tools.tool_manager.ToolManager.call_tool methods in the MCP SDK.DD_METRICS_OTEL_ENABLED must be set to true and the application must include its own OTLP metrics exporter.DD_ASGI_OBFUSCATE_404_RESOURCE is enabled (disabled by default).ddtrace-run or import ddtrace.auto. To disable this feature, set the environment variable DD_LOGS_INJECTION to False.freezegun would not work with tests defined in unittest classes.flaky or pytest-rerunfailures would cause the test results not to be reported correctly to Datadog. With this change, those plugins can be used with ddtrace, and test results will be reported to Datadog, but Test Optimization advanced features such as Early Flake Detection and Auto Test Retries will not be available when such plugins are used.pytest-xdist.DD_FASTAPI_ASYNC_BODY_TIMEOUT_SECONDS environment variable.AttributeError while parsing NoneType streamed chunk deltas.AttributeError.langgraph>=0.3.22.langchain libraries raised a ValueError.dd.* properties were not injected onto logging records unless DD_LOGS_ENABLED=true env var was set (default value is structured). This issue causes problems for non-structured loggers which set their own format string instead of having ddtrace set the logging format string for you.@tracer.wrap() decorator failed to preserve the decorated function's return type, returning Any instead of the original return type.freezegun was in use. With this change, the freezegun integration is not necessary anymore.LONG_MAX caused traces to fail to send.mysqlsh (MySQL Shell) reassigns globals with a custom object. This can interfere with analysis or instrumentation logic.Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
skipif marker with the condition passed as a keyword argument (or not provided at all) would cause the test to be reported as failed, in particular when the flaky or pytest-rerunfailures were also used.<!-- -->
<!-- -->
dynamic instrumentation: improve support for function probes with frameworks and applications that interact with the Python garbage collector (e.g. sy
@tracer.wrap() decorator failed to preserve the decorated function's return type, returning Any instead of the original return type.mysqlsh (MySQL Shell) reassigns globals with a custom object. This can interfere with analysis or instrumentation logic.logging: Fix issue when dd.* properties were not injected onto logging records unless DD_LOGS_ENABLED=true env var was set (default value is structure
dd.* properties were not injected onto logging records unless DD_LOGS_ENABLED=true env var was set (default value is structured). This issue causes problems for non-structured loggers which set their own format string instead of having ddtrace set the logging format string for you.This fix resolves an issue where using Test Optimization together with external retry plugins such as flaky or pytest-rerunfailures would cause the te
flaky or pytest-rerunfailures would cause the test results not to be reported correctly to Datadog. With this change, those plugins can be used with ddtrace, and test results will be reported to Datadog, but Test Optimization advanced features such as Early Flake Detection and Auto Test Retries will not be available when such plugins are used.pytest-xdist.langgraph>=0.3.22.The DD_DYNAMIC_INSTRUMENTATION_UPLOAD_FLUSH_INTERVAL environment variable has been deprecated in favor of DD_DYNAMIC_INSTRUMENTATION_UPLOAD_INTERVAL_S…
DD_DYNAMIC_INSTRUMENTATION_UPLOAD_FLUSH_INTERVAL environment variable has been deprecated in favor of DD_DYNAMIC_INSTRUMENTATION_UPLOAD_INTERVAL_SECONDS. The old environment variable will be removed in a future major release.escaped and timestamp arguments in the record_exception method are deprecated and will be removed in version 4.0.0.DD_RUNTIME_METRICS_RUNTIME_ID_ENABLED to enable runtime metrics for tagging runtime metrics with the current runtime ID. This is useful for tracking runtime metrics across multiple processes. Previously, this was DD_TRACE_EXPERIMENTAL_RUNTIME_ID_ENABLED.azure_servicebus: Add support for Azure Service Bus producers.azure_functions: Adds messaging span attributes for service bus triggersazure_functions: Add distributed tracing support for Service Bus triggers.ddtrace-api: Adds patching of ddtrace_api.tracer.set_tags to the ddtrace_api integrationinvoke_agent calls. If bedrock agents tracing is enabled, the internal bedrock traces will be converted and submitted as LLM Observability spans.DD_LLMOBS_INSTRUMENTED_PROXY_URLS environment variable or by enabling LLM Observability with the instrumented_proxy_urls argument. Spans sent to a proxy URL will now show up as workflow spans instead of LLM spans.google_genai: Introduces tracing support for Google's Generative AI SDK for Python's generate_content and generate_content_stream methods.
See the docs for more information.pydantic_ai: Introduces tracing support for PydanticAI's Agent.iter and Tool.run methods.
See the docs for more information.TypeError in encoding when truncating a large bytes object.os.system or subprocess.Popen could return the wrong exception type.for v in g: yield v during the wrapping of generator functions where full bidrectional communication with the sub-generator via yield from g was appropriate. See PEP380 for an explanation of how these two generator uses differ.@tracer.wrap() decorator failed to preserve return values from generator functions, causing StopIteration.value to be None instead of the actual returned value.rq: Enable parsing distributed tracing metadata in perform jobPYTEST_ADDOPTS environment variable.DD_LLMOBS_ML_APP missing errors when LLM Observability was disabled.export_span was incorrect.langchain: Resolved an AttributeError that could occur when async tasks are cancelled during agenerate calls.try: for CPython 3.11 and later.ddtrace.tracer.get_log_correlation_context() returns the expected log correlation attributes (e.g., dd.trace_id, dd.span_id, dd.service instead of trace_id, span_id, service). This change aligns the method's output with the attributes in ddtrace log-correlation docs.ddtrace.tracer.get_log_correlation_context() would return the service name of the current span instead of the global service name.Tracing:
Profiling:
DD_PROFILING_EXPORT_LIBDD_ENABLED configuration variable is removed. As a result of this change, profiling for 32-bit Linux is not supported. Please file an issue or open a support ticket if you need profiling for 32-bit Linux.Single Step Instrumentation:
DD_TRACE_SAFE_INSTRUMENTATION_ENABLED. This change ensures that instrumentation patching is only applied to for supported versions of packages, leaving unsupported package versions unpatched.pydantic_ai: Introduces tracing support for PydanticAI's Agent.iter and Tool.run methods. See the docs for more information.
LLM Observability
pydantic_ai: Introduces tracing support for PydanticAI's Agent.iter and Tool.run methods.
See the docs for more information.Tracing
azure_functions: Adds messaging span attributes for service bus triggersThe escaped and timestamp arguments in the record_exception method are deprecated and will be removed in version 4.0.0.
escaped and timestamp arguments in the record_exception method are deprecated and will be removed in version 4.0.0.LLMObs:
google_genai: Introduces tracing support for Google's Generative AI SDK for Python's generate_content and generate_content_stream methods.
See the docs for more information.Tracing:
azure_servicebus: Add support for Azure Service Bus producers.azure_functions: Add distributed tracing support for Service Bus triggers.ddtrace-api: Adds patching of ddtrace_api.tracer.set_tags to the ddtrace_api integrationCI Visibility:
PYTEST_ADDOPTS environment variable.Dynamic Instrumentation:
try: for CPython 3.11 and later.Tracing:
@tracer.wrap() decorator failed to preserve return values from generator functions, causing StopIteration.value to be None instead of the actual returned value.rq: Enable parsing distributed tracing metadata in perform jobLLMObs:
langchain: Resolved an AttributeError that could occur when async tasks are cancelled during agenerate calls.Logging:
ddtrace.tracer.get_log_correlation_context() returns the expected log correlation attributes (e.g., dd.trace_id, dd.span_id, dd.service instead of trace_id, span_id, service). This change aligns the method's output with the attributes in ddtrace log-correlation docs.ddtrace.tracer.get_log_correlation_context() would return the service name of the current span instead of the global service name.The DD_DYNAMIC_INSTRUMENTATION_UPLOAD_FLUSH_INTERVAL environment variable has been deprecated in favor of DD_DYNAMIC_INSTRUMENTATION_UPLOAD_INTERVAL_S…
DD_DYNAMIC_INSTRUMENTATION_UPLOAD_FLUSH_INTERVAL environment variable has been deprecated in favor of DD_DYNAMIC_INSTRUMENTATION_UPLOAD_INTERVAL_SECONDS. The old environment variable will be removed in a future major release.DD_RUNTIME_METRICS_RUNTIME_ID_ENABLED to enable runtime metrics for tagging runtime metrics with the current runtime ID. This is useful for tracking runtime metrics across multiple processes. Previously, this was DD_TRACE_EXPERIMENTAL_RUNTIME_ID_ENABLED.invoke_agent calls. If bedrock agents tracing is enabled, the internal bedrock traces will be converted and submitted as LLM Observability spans.DD_LLMOBS_INSTRUMENTED_PROXY_URLS environment variable or by enabling LLM Observability with the instrumented_proxy_urls argument. Spans sent to a proxy URL will now show up as workflow spans instead of LLM spans.TypeError in encoding when truncating a large bytes object.os.system or subprocess.Popen could return the wrong exception type.for v in g: yield v during the wrapping of generator functions where full bidrectional communication with the sub-generator via yield from g was appropriate. See PEP380 for an explanation of how these two generator uses differ.DD_LLMOBS_ML_APP missing errors when LLM Observability was disabled.export_span was incorrect.Tracing:
Profiling:
DD_PROFILING_EXPORT_LIBDD_ENABLED configuration variable is removed. As a result of this change, profiling for 32-bit Linux is not supported. Please file an issue or open a support ticket if you need profiling for 32-bit Linux.Single Step Instrumentation:
DD_TRACE_SAFE_INSTRUMENTATION_ENABLED. This change ensures that instrumentation patching is only applied to for supported versions of packages, leaving unsupported package versions unpatched.One column per quarter.