NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
PyPI, #486 most downloaded on PyPI
Datadog APM client library
Latest stable
4.17.1
Last release today
17 Nov 2026
Ships on a steady schedule
a new release about every 2 weeks
Nearly every release is documented
notes for 60 of the last 60 stable releases
5 versions withdrawn
withdrawn after publishing
10 years old
870 releases · first in 2017
Drag across the ruler to compare, or tap a release to open it.
AI Guard: Adds SDS (Sensitive Data Scanner) findings to AI Guard spans, enabling visibility into sensitive data detected in LLM inputs and outputs.
<!-- -->
LLM Observability: Adds support for DeepEval evaluations in LLM Observability Experiments by allowing users to pass a DeepEval evaluation (which either inherents from BaseMetric or BaseConversationalMetric) in an LLM Obs Experiment.
Example:
from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCaseParams
from ddtrace.llmobs import LLMObs
correctness_metric = GEval(
name="Correctness",
criteria="Determine whether the actual output is factually correct based on the expected output.",
evaluation_steps=[
"Check whether the facts in 'actual output' contradicts any facts in 'expected output'",
"You should also heavily penalize omission of detail",
"Vague language, or contradicting OPINIONS, are OK"
],
evaluation_params=[LLMTestCaseParams.INPUT, LLMTestCaseParams.ACTUAL_OUTPUT, LLMTestCaseParams.EXPECTED_OUTPUT],
async_mode=True
)
dataset = LLMObs.create_dataset(
dataset_name="<DATASET_NAME>",
description="<DATASET_DESCRIPTION>",
records=[RECORD_1, RECORD_2, RECORD_3, ...]
)
def my_task(input_data, config):
return input_data["output"]
def my_summary_evaluator(inputs, outputs, expected_outputs, evaluators_results):
return evaluators_results["Correctness"].count(True)
experiment = LLMObs.experiment(
name="<EXPERIMENT_NAME>",
task=my_task,
dataset=dataset,
evaluators=[correctness_metric],
summary_evaluators=[my_summary_evaluator], # optional, used to summarize the experiment results
description="<EXPERIMENT_DESCRIPTION>."
)
result = experiment.run()
<!-- -->
run() with row count, run count, per-evaluator stats, and error counts.<!-- -->
max_retries and retry_delay parameters to experiment.run() for retrying failed tasks and evaluators. Example: experiment.run(max_retries=3, retry_delay=lambda attempt: 2 ** attempt).contextvars) could cause use-after-free or double-free crashes (SIGSEGV) inside libddwaf. A per-context lock now serializes WAF calls on the same context.<!-- -->
ddtrace.internal.wrapping.context.BaseWrappingContext.<!-- -->
pytest-html and other third-party reporting plugins caused by the ddtrace pytest plugin using a non-standard dd_retry test outcome for retry attempts. The outcome is now set to rerun, which is the standard value used by pytest-rerunfailures and recognized by reporting plugins.<!-- -->
RuntimeError: generator didn't yield in the Symbol DB remote config subscriber when the process has no writable temporary directory.<!-- -->
<module> in flame graphs has been fixed.Your coding agent can read these notes before it upgrades. Set up the MCP server →
Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
Span.parent_id will change from Optional[int] to int in v5.0.0.This introduces LLMObs.get_prompt() to retrieve managed prompts from Datadog's Prompt Registry. The method returns a ManagedPrompt object with a format()
method for variable substitution. Prompt updates propagate to running applications within the cache TTL (default: 60 seconds).
Use with annotation_context or annotate to correlate prompts with LLM spans:
prompt = LLMObs.get_prompt("greeting")
variables = {"user": "Alice"}
with LLMObs.annotation_context(prompt=prompt.to_annotation_dict(**variables)):
openai.chat.completions.create(messages=prompt.format(**variables))
experiments propagate canonical_ids from dataset records to the corresponding experiments span when present. The canonical_ids are only guaranteed to be available after calling pull_dataset.
LLMObs.create_dataset supports a bulk_upload parameter to control data uploading behavior. Both LLMObs.create_dataset and LLMObs.create_dataset_from_csv supports users specifying the deduplicate parameter.
Subset of dataset records can now be pulled with tags by using the tags argument to LLMObs.pull_dataset, provided in a list of strings of key value pairs: LLMObs.pull_dataset(dataset_name="my-dataset", tags=["env:prod", "version:1.0"])
LLMObs.create_dataset.AttributeError on openai-agents >= 0.8.0 caused by the removal of AgentRunner._run_single_turn.gevent module unnecessarily even when the profiler was not enabled.kafka_cluster_id tag to Kafka offset/backlog tracking for confluent-kafka. Previously, cluster ID was only included in DSM checkpoint edge tags (produce/consume) but missing from offset commit and produce offset backlogs. This ensures correct attribution of backlog data to specific Kafka clusters when multiple clusters share topic names.Estimated end-of-life date, accurate to within three months: 06-2027 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 06-2027 See the support level definitions for more information.
CI Visibility: This fix resolves an issue where a failure response from the /search_commits endpoint caused the git metadata upload to fall back to sending the full 30-day commit history instead of aborting. This fallback could trigger cascading write load on the backend. The upload now aborts when search_commits fails, matching the behavior when the /packfile upload itself fails.
Fixed a race condition with internal periodic threads that could have caused a rare crash when forking.
Estimated end-of-life date, accurate to within three months: 06-2027 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 06-2027 See the support level definitions for more information.
<!-- -->
os._exit, SIGKILL, segfault) caused buffered test events to be lost. To enable eager flushing, set DD_TRACE_PARTIAL_FLUSH_MIN_SPANS=1.Estimated end-of-life date, accurate to within three months: 06-2027 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 06-2027 See the support level definitions for more information.
<!-- -->
Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
CI Visibility: Fixes an issue where HTTP 429 (Too Many Requests) responses from the Datadog backend were treated as non-retriable errors, causing CI visibility data to be dropped when the backend applied rate limiting. The backend connector now retries on 429 responses and respects the X-RateLimit-Reset header when present to determine the retry delay.
internal: A bug preventing certain periodic threads of ddtrace (like the profile uploader) from triggering in fork-heavy applications has been fixed.
profiling: Fixes an issue where setting an unlimited stack size (ulimit -s unlimited) on Linux caused the stack profiler sampling thread to fail to start, resulting in empty CPU and wall-time profiles. #17132
Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
RuntimeError that occurred when the git binary was not available. Git metadata upload is now skipped gracefully with a warning instead of aborting pytest startup.<!-- -->
tool_search_tool_regex.<!-- -->
<!-- -->
RuntimeError during forks.<!-- -->
Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
pytest-rerunfailures and flaky were silently overridden by the ddtrace plugin. With this change, external rerun plugins will now drive retries as expected when Auto Test Retries and Early Flake Detection features are both disabled, otherwise our retry mechanism takes precedence and a warning is emitted.<!-- -->
<!-- -->
<!-- -->
RuntimeError could be raised when iterating over the context._meta dictionary while creating spans or generating distributed traces.Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
profiling: This fix resolves an issue where the lock profiler's wrapper class did not support PEP 604 type union syntax (e.g., asyncio.Condition | None). This was causing a TypeError at import time for libraries such as kopf that use union type annotations at class definition time.
Fix for potential crashes at process shutdown due to incorrect detection of the VM finalization state when stopping periodic worker threads.
Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
cache_creation_input_tokens and cache_read_input_tokens were not captured when using the LiteLLM integration with providers that support prompt caching (e.g., Anthropic, OpenAI, Deepseek).Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
We have identified a bug where workloads relying on fork could encounter crashes post-fork due to a race condition. We are currently working on a fix.
AAP: Fixes a memory corruption issue where concurrent calls to the WAF on the same request context from multiple threads (e.g. an asyncio event loop and a thread pool executor inheriting the same context via contextvars) could cause use-after-free or double-free crashes (SIGSEGV) inside libddwaf. A per-context lock now serializes WAF calls on the same context.
CI Visibility: Fixed an incompatibility with pytest-html and other third-party reporting plugins caused by the ddtrace pytest plugin using a non-standard dd_retry test outcome for retry attempts. The outcome is now set to rerun, which is the standard value used by pytest-rerunfailures and recognized by reporting plugins.
Template strings (t-strings) are now tracked for taint propagation, enabling detection of security vulnerabilities when user input is embedded in temp…
Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
We have identified a bug where workloads relying on fork could encounter crashes post-fork due to a race condition. We are currently working on a fix.
DD_CIVISIBILITY_USE_BETA_WRITER option), and also contains performance and memory usage improvements. A beta version of the plugin had been available since v4.2.0, and could be enabled via the DD_PYTEST_USE_NEW_PLUGIN environment variable. This new version is now the default, and the environment variable can be used to revert to the previous plugin if used with false or 0 values.tracing
DD_TRACE_128_BIT_TRACEID_GENERATION_ENABLED is deprecated and will be removed in version 5.0.0. 128-bit trace ID generation will become mandatory.The tracer parameter is deprecated in the following functions and class methods and will be removed in version 5.0.0:
trace_appTraceMiddleware.__init__TracePlugin.__init__TraceMiddleware.__init__TraceMiddleware.__init__get_traced_cachetrace_engineWSGIMiddleware.__init__The ddtrace.trace.tracer singleton is always used.
uvloop with asyncio.query, ClaudeSDKClient.query, and ClaudeSDKClient.receive_messages methods. See the docs for more information.start_as_current_span decorator on asynchronous functions. Requires opentelemetry-api>=1.24.LLMObs.async_experiment() method for running experiments with async task functions and mixed sync/async evaluators.StringCheckEvaluator: Performs string comparison operations (equals, not equals, contains, case-insensitive contains).RegexMatchEvaluator: Validates output against regex patterns with search, match, or fullmatch modes.LengthEvaluator: Validates output length constraints by characters, words, or lines.JSONEvaluator: Validates JSON syntax and optionally checks for required keys.SemanticSimilarityEvaluator: Measures semantic similarity between output and expected output using embedding vectors.json metric type in evaluation metrics. Users can now submit dict values as evaluation metrics using LLMObs.submit_evaluation() with metric_type="json". Additionally, experiment evaluators that return dict values are automatically detected as json metric type.LLMJudge evaluator for automated evaluation of LLM outputs using another LLM as the judge. Supports OpenAI and Anthropic providers with boolean, score, categorical, and custom JSON schema output formats.site-packages directory is now added as the last entry in the PYTHONPATH environment variable (it previously was added before the last entry).asyncio Tasks has been fixed."<N frame(s) omitted>" entries in profiling data and unbounded memory growth in the memory profiler.--skip-atexit flag and registered Python atexit handlers regardless. This caused profiler cleanup code to run during process shutdown even when --skip-atexit was set, leading to crashes and hangs in uwsgi workers.SIGTERM or SIGINT signals.SIGTERM or SIGINT signals.noopener and noreferrer link tags in the Datadog link in the footer of the App and API Protection HTML blocking response template.
This could previously trigger a "reverse tabnabbing" vulnerability finding from other security analysis tools.DD_EXPERIMENTAL_PROPAGATE_PROCESS_TAGS_ENABLED was enabled.Template strings (t-strings) are now tracked for taint propagation, enabling detection of security vulnerabilities when user input is embedded in temp…
Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
DD_CIVISIBILITY_USE_BETA_WRITER option), and also contains performance and memory usage improvements. A beta version of the plugin had been available since v4.2.0, and could be enabled via the DD_PYTEST_USE_NEW_PLUGIN environment variable. This new version is now the default, and the environment variable can be used to revert to the previous plugin if used with false or 0 values.tracing
DD_TRACE_128_BIT_TRACEID_GENERATION_ENABLED is deprecated and will be removed in version 5.0.0. 128-bit trace ID generation will become mandatory.The tracer parameter is deprecated in the following functions and class methods and will be removed in version 5.0.0:
trace_appTraceMiddleware.__init__TracePlugin.__init__TraceMiddleware.__init__TraceMiddleware.__init__get_traced_cachetrace_engineWSGIMiddleware.__init__The ddtrace.trace.tracer singleton is always used.
uvloop with asyncio.query, ClaudeSDKClient.query, and ClaudeSDKClient.receive_messages methods. See the docs for more information.start_as_current_span decorator on asynchronous functions. Requires opentelemetry-api>=1.24.LLMObs.async_experiment() method for running experiments with async task functions and mixed sync/async evaluators.StringCheckEvaluator: Performs string comparison operations (equals, not equals, contains, case-insensitive contains).RegexMatchEvaluator: Validates output against regex patterns with search, match, or fullmatch modes.LengthEvaluator: Validates output length constraints by characters, words, or lines.JSONEvaluator: Validates JSON syntax and optionally checks for required keys.SemanticSimilarityEvaluator: Measures semantic similarity between output and expected output using embedding vectors.json metric type in evaluation metrics. Users can now submit dict values as evaluation metrics using LLMObs.submit_evaluation() with metric_type="json". Additionally, experiment evaluators that return dict values are automatically detected as json metric type.LLMJudge evaluator for automated evaluation of LLM outputs using another LLM as the judge. Supports OpenAI and Anthropic providers with boolean, score, categorical, and custom JSON schema output formats.site-packages directory is now added as the last entry in the PYTHONPATH environment variable (it previously was added before the last entry).asyncio Tasks has been fixed."<N frame(s) omitted>" entries in profiling data and unbounded memory growth in the memory profiler.--skip-atexit flag and registered Python atexit handlers regardless. This caused profiler cleanup code to run during process shutdown even when --skip-atexit was set, leading to crashes and hangs in uwsgi workers.SIGTERM or SIGINT signals.SIGTERM or SIGINT signals.noopener and noreferrer link tags in the Datadog link in the footer of the App and API Protection HTML blocking response template.
This could previously trigger a "reverse tabnabbing" vulnerability finding from other security analysis tools.DD_EXPERIMENTAL_PROPAGATE_PROCESS_TAGS_ENABLED was enabled.Template strings (t-strings) are now tracked for taint propagation, enabling detection of security vulnerabilities when user input is embedded in temp…
Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
tracing
DD_TRACE_128_BIT_TRACEID_GENERATION_ENABLED is deprecated and will be removed in version 5.0.0. 128-bit trace ID generation will become mandatory.The tracer parameter is deprecated in the following functions and class methods and will be removed in version 5.0.0:
trace_appTraceMiddleware.__init__TracePlugin.__init__TraceMiddleware.__init__TraceMiddleware.__init__get_traced_cachetrace_engineWSGIMiddleware.__init__The ddtrace.trace.tracer singleton is always used.
uvloop with asyncio.StringCheckEvaluator: Performs string comparison operations (equals, not equals, contains, case-insensitive contains).RegexMatchEvaluator: Validates output against regex patterns with search, match, or fullmatch modes.LengthEvaluator: Validates output length constraints by characters, words, or lines.JSONEvaluator: Validates JSON syntax and optionally checks for required keys.SemanticSimilarityEvaluator: Measures semantic similarity between output and expected output using embedding vectors.json metric type in evaluation metrics. Users can now submit dict values as evaluation metrics using LLMObs.submit_evaluation() with metric_type="json". Additionally, experiment evaluators that return dict values are automatically detected as json metric type.LLMJudge evaluator for automated evaluation of LLM outputs using another LLM as the judge. Supports OpenAI and Anthropic providers with boolean, score, categorical, and custom JSON schema output formats.site-packages directory is now added as the last entry in the PYTHONPATH environment variable (it previously was added before the last entry).asyncio Tasks has been fixed."<N frame(s) omitted>" entries in profiling data and unbounded memory growth in the memory profiler.--skip-atexit flag and registered Python atexit handlers regardless. This caused profiler cleanup code to run during process shutdown even when --skip-atexit was set, leading to crashes and hangs in uwsgi workers.SIGTERM or SIGINT signals.SIGTERM or SIGINT signals.noopener and noreferrer link tags in the Datadog link in the footer of the App and API Protection HTML blocking response template.
This could previously trigger a "reverse tabnabbing" vulnerability finding from other security analysis tools.DD_EXPERIMENTAL_PROPAGATE_PROCESS_TAGS_ENABLED was enabled.Template strings (t-strings) are now tracked for taint propagation, enabling detection of security vulnerabilities when user input is embedded in temp…
Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
tracing
DD_TRACE_128_BIT_TRACEID_GENERATION_ENABLED is deprecated and will be removed in version 5.0.0. 128-bit trace ID generation will become mandatory.The tracer parameter is deprecated in the following functions and class methods and will be removed in version 5.0.0:
trace_appTraceMiddleware.__init__TracePlugin.__init__TraceMiddleware.__init__TraceMiddleware.__init__get_traced_cachetrace_engineWSGIMiddleware.__init__The ddtrace.trace.tracer singleton is always used.
uvloop with asyncio.StringCheckEvaluator: Performs string comparison operations (equals, not equals, contains, case-insensitive contains).RegexMatchEvaluator: Validates output against regex patterns with search, match, or fullmatch modes.LengthEvaluator: Validates output length constraints by characters, words, or lines.JSONEvaluator: Validates JSON syntax and optionally checks for required keys.SemanticSimilarityEvaluator: Measures semantic similarity between output and expected output using embedding vectors.json metric type in evaluation metrics. Users can now submit dict values as evaluation metrics using LLMObs.submit_evaluation() with metric_type="json". Additionally, experiment evaluators that return dict values are automatically detected as json metric type.LLMJudge evaluator for automated evaluation of LLM outputs using another LLM as the judge. Supports OpenAI and Anthropic providers with boolean, score, categorical, and custom JSON schema output formats.site-packages directory is now added as the last entry in the PYTHONPATH environment variable (it previously was added before the last entry)."<N frame(s) omitted>" entries in profiling data and unbounded memory growth in the memory profiler.--skip-atexit flag and registered Python atexit handlers regardless. This caused profiler cleanup code to run during process shutdown even when --skip-atexit was set, leading to crashes and hangs in uwsgi workers.SIGTERM or SIGINT signals.SIGTERM or SIGINT signals.noopener and noreferrer link tags in the Datadog link in the footer of the App and API Protection HTML blocking response template.
This could previously trigger a "reverse tabnabbing" vulnerability finding from other security analysis tools.DD_EXPERIMENTAL_PROPAGATE_PROCESS_TAGS_ENABLED was enabled.Template strings (t-strings) are now tracked for taint propagation, enabling detection of security vulnerabilities when user input is embedded in temp…
Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
tracing
DD_TRACE_128_BIT_TRACEID_GENERATION_ENABLED is deprecated and will be removed in version 5.0.0. 128-bit trace ID generation will become mandatory.The tracer parameter is deprecated in the following functions and class methods and will be removed in version 5.0.0:
trace_appTraceMiddleware.__init__TracePlugin.__init__TraceMiddleware.__init__TraceMiddleware.__init__get_traced_cachetrace_engineWSGIMiddleware.__init__The ddtrace.trace.tracer singleton is always used.
StringCheckEvaluator: Performs string comparison operations (equals, not equals, contains, case-insensitive contains).RegexMatchEvaluator: Validates output against regex patterns with search, match, or fullmatch modes.LengthEvaluator: Validates output length constraints by characters, words, or lines.JSONEvaluator: Validates JSON syntax and optionally checks for required keys.SemanticSimilarityEvaluator: Measures semantic similarity between output and expected output using embedding vectors.json metric type in evaluation metrics. Users can now submit dict values as evaluation metrics using LLMObs.submit_evaluation() with metric_type="json". Additionally, experiment evaluators that return dict values are automatically detected as json metric type.LLMJudge evaluator for automated evaluation of LLM outputs using another LLM as the judge. Supports OpenAI and Anthropic providers with boolean, score, categorical, and custom JSON schema output formats.site-packages directory is now added as the last entry in the PYTHONPATH environment variable (it previously was added before the last entry)."<N frame(s) omitted>" entries in profiling data and unbounded memory growth in the memory profiler.--skip-atexit flag and registered Python atexit handlers regardless. This caused profiler cleanup code to run during process shutdown even when --skip-atexit was set, leading to crashes and hangs in uwsgi workers.SIGTERM or SIGINT signals.Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
We have identified a bug where profiles emitted by ddtrace v4.4.* do not contain tags injected by the Datadog Trace Agent (like pod_name, kube_namespace, etc.)
This issue is fixed in ddtrace v4.5.0.
We’re currently investigating an issue impacting services using ddtrace versions v4.1.* -> v4.4.* with profiling turned on. Our engineering teams are actively working on a permanent fix. In the meantime, disabling the memory profiler in the ddtrace configuration has been identified as a temporary workaround, DD_PROFILING_MEMORY_ENABLED=false.
BaseEvaluator class, providing more flexibility and structure for implementing evaluation logic. The EvaluatorContext stores the context of the evaluation, including the dataset record and span information. Additionally, class-based summary evaluators are supported via BaseSummaryEvaluator, which receives a SummaryEvaluatorContext containing aggregated inputs, outputs, expected outputs, and per-row evaluation results.DD_TRACE_LOG_LEVEL to control the ddtrace logger level, following the log levels available in the logging module.pathlib.Path.open() for App and API Protection Exploit Prevention.DD_TRACE_TORNADO_ENABLED=true or DD_PATCH_MODULES=tornado:trueFallbackStreamWrapper (introduced for mid-stream fallback support) that caused an AttributeError when attempting to access the .handler attribute. The integration now gracefully handles both the original response format and wrapped responses by falling back to ddtrace's own stream wrapping when needed.asyncio task stacks could contain duplicated frames when the task was on-CPU is now fixed. The stack now correctly shows each frame only once.gevent.joinall is called.StreamedRunResult.stream_responses() method which was introduced in pydantic-ai==0.8.1. This was leading to agent spans not being finished.LLMObs.experiment was overly constrained due to the use of an invariant List type. The argument now uses the covariant Sequence type, allowing users to pass in a list of evaluator functions with narrower return type.Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
BaseEvaluator class, providing more flexibility and structure for implementing evaluation logic. The EvaluatorContext stores the context of the evaluation, including the dataset record and span information. Additionally, class-based summary evaluators are supported via BaseSummaryEvaluator, which receives a SummaryEvaluatorContext containing aggregated inputs, outputs, expected outputs, and per-row evaluation results.DD_TRACE_LOG_LEVEL to control the ddtrace logger level, following the log levels available in the logging module.pathlib.Path.open() for App and API Protection Exploit Prevention.DD_TRACE_TORNADO_ENABLED=true or DD_PATCH_MODULES=tornado:trueFallbackStreamWrapper (introduced for mid-stream fallback support) that caused an AttributeError when attempting to access the .handler attribute. The integration now gracefully handles both the original response format and wrapped responses by falling back to ddtrace's own stream wrapping when needed.asyncio task stacks could contain duplicated frames when the task was on-CPU is now fixed. The stack now correctly shows each frame only once.gevent.joinall is called.StreamedRunResult.stream_responses() method which was introduced in pydantic-ai==0.8.1. This was leading to agent spans not being finished.LLMObs.experiment was overly constrained due to the use of an invariant List type. The argument now uses the covariant Sequence type, allowing users to pass in a list of evaluator functions with narrower return type.Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
BaseEvaluator class, providing more flexibility and structure for implementing evaluation logic. The EvaluatorContext stores the context of the evaluation, including the dataset record and span information. Additionally, class-based summary evaluators are supported via BaseSummaryEvaluator, which receives a SummaryEvaluatorContext containing aggregated inputs, outputs, expected outputs, and per-row evaluation results.DD_TRACE_LOG_LEVEL to control the ddtrace logger level, following the log levels available in the logging module.FallbackStreamWrapper (introduced for mid-stream fallback support) that caused an AttributeError when attempting to access the .handler attribute. The integration now gracefully handles both the original response format and wrapped responses by falling back to ddtrace's own stream wrapping when needed.asyncio task stacks could contain duplicated frames when the task was on-CPU is now fixed. The stack now correctly shows each frame only once.gevent.joinall is called.StreamedRunResult.stream_responses() method which was introduced in pydantic-ai==0.8.1. This was leading to agent spans not being finished.LLMObs.experiment was overly constrained due to the use of an invariant List type. The argument now uses the covariant Sequence type, allowing users to pass in a list of evaluator functions with narrower return type.Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
BaseEvaluator class, providing more flexibility and structure for implementing evaluation logic. The EvaluatorContext stores the context of the evaluation, including the dataset record and span information. Additionally, class-based summary evaluators are supported via BaseSummaryEvaluator, which receives a SummaryEvaluatorContext containing aggregated inputs, outputs, expected outputs, and per-row evaluation results.FallbackStreamWrapper (introduced for mid-stream fallback support) that caused an AttributeError when attempting to access the .handler attribute. The integration now gracefully handles both the original response format and wrapped responses by falling back to ddtrace's own stream wrapping when needed.asyncio task stacks could contain duplicated frames when the task was on-CPU is now fixed. The stack now correctly shows each frame only once.StreamedRunResult.stream_responses() method which was introduced in pydantic-ai==0.8.1. This was leading to agent spans not being finished.Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
gevent.joinall is called.Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
Tracer.data_streams_processor is deprecated and will be removed in v5.0.0. Use ddtrace.data_streams.data_streams_processor() instead.
Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
⚠️ An issue was detected with Memory Profiling in this release. ⚠️ An issue was detected with Datastreams Monitoring in this release.
httpx library from 0.17.1 to 0.25.0. If you are using httpx versions prior to 0.25.0, upgrade to httpx>=0.25.0 to ensure compatibility with ddtrace.Tracer.data_streams_processor is deprecated and will be removed in v5.0.0. Use ddtrace.data_streams.data_streams_processor() instead.threading.Condition locking type profiling in Python. The Lock profiler now provides visibility into threading.Condition usage, helping identify contention in multi-threaded applications using condition variables.EvaluatorResult class that evaluator functions can return to provide additional fields beyond just the evaluation value. This enables evaluators to include reasoning, assessment, metadata, and tags alongside the evaluation result.RuntimeError caused by capturing the __dict__ attribute of an object that mutates during iteration.PicklingError when using multiprocessing.Manager() with lock profiling enabled on Python 3.14+. Python 3.14 changed the default multiprocessing start method from fork to forkserver, which requires all objects to be pickleable. The lock profiler's wrappers now properly support pickle serialization.tracer=None.Tracer.data_streams_processor is deprecated and will be removed in v5.0.0. Use ddtrace.data_streams.data_streams_processor() instead.
Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
httpx library from 0.17.1 to 0.25.0. If you are using httpx versions prior to 0.25.0, upgrade to httpx>=0.25.0 to ensure compatibility with ddtrace.Tracer.data_streams_processor is deprecated and will be removed in v5.0.0. Use ddtrace.data_streams.data_streams_processor() instead.threading.Condition locking type profiling in Python. The Lock profiler now provides visibility into threading.Condition usage, helping identify contention in multi-threaded applications using condition variables.EvaluatorResult class that evaluator functions can return to provide additional fields beyond just the evaluation value. This enables evaluators to include reasoning, assessment, metadata, and tags alongside the evaluation result.RuntimeError caused by capturing the __dict__ attribute of an object that mutates during iteration.PicklingError when using multiprocessing.Manager() with lock profiling enabled on Python 3.14+. Python 3.14 changed the default multiprocessing start method from fork to forkserver, which requires all objects to be pickleable. The lock profiler's wrappers now properly support pickle serialization.tracer=None.Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
FallbackStreamWrapper (introduced for mid-stream fallback support) that caused an AttributeError when attempting to access the .handler attribute. The integration now gracefully handles both the original response format and wrapped responses by falling back to ddtrace's own stream wrapping when needed.asyncio task stacks could contain duplicated frames when the task was on-CPU is now fixed. The stack now correctly shows each frame only once.Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
Bug Fixes
The Hooks class (config. .hooks) is deprecated and will be removed in v5.0. All hook methods (register(), on(), deregister(), emit()) are now no-op an…
Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
Hooks class (config.<integration>.hooks) is deprecated and will be removed in v5.0. All hook methods (register(), on(), deregister(), emit()) are now no-op and no longer affect span behavior. To interact with spans, use ddtrace.trace_utils.get_current_span() or ddtrace.trace_utils.get_current_root_span() instead.initialize requests and their responses on modelcontextprotocol/python-sdk servers.source:otel tagging for evaluations when OpenTelemetry (OTel) tracing is enabled when DD_TRACE_OTEL_ENABLED=true is set. This tag allows the backend to wait for OTel span conversion before processing evaluations.DD_MCP_CAPTURE_INTENT environment variable.asyncio.BoundedSemaphore lock type profiling in Python Lock Profiler.asyncio.Condition locking type profiling in Python. The Lock profiler now provides visibility into asyncio.Condition usage, helping identify contention in async applications using condition variables.asyncio.Semaphore lock type profiling in Python Lock Profiler.asyncio.shield.asyncio.TaskGroup.aws.httpapi span name for v2 apis when the API Gateway sets the x-dd-proxy header to aws-httpapi. Additionally, the tag http.route and the resource name of the span now contains the api resource path instead of the path when propagated with the x-dd-proxy-resource-path header.DD_TRACE_REMOVE_INTEGRATION_SERVICE_NAMES_ENABLED support, which was previously ignored.https:// scheme prefix as part of the http.url tag; this caused the entire url to be parsed as the http path.isDefined to result in an evaluation error.DD_EXPERIMENTAL_FLAGGING_PROVIDER_ENABLED=true), ensuring remote configuration is received before process forking occurs.None metadata in Ray job submission caused a crash.AttributeError when calling tag_agent_manifest.annotation_context blocks caused annotations to fail after the first operation in subsequent contexts. Previously, the trace context created by the first annotation_context remained active after exiting, causing the second context to reuse a stale context ID. This resulted in annotations not being applied to spans after the first batch call in the second annotation_context block.activate_distributed_headers() where distributed requests missing a LLM Observability trace ID would be incorrectly propagated twice.TypeError during profiling. One example of this is neo4j's AsyncRLock, which inherits from asyncio.Lock: https://github.com/neo4j/neo4j-python-driver/blob/6.x/src/neo4j/_async_compat/concurrency.py#L45uvloop and forking has been resolved._psutil_linux.abi3.so file in an injected environment.The Hooks class (config. .hooks) is deprecated and will be removed in v5.0. All hook methods (register(), on(), deregister(), emit()) are now no-op an…
Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
Hooks class (config.<integration>.hooks) is deprecated and will be removed in v5.0. All hook methods (register(), on(), deregister(), emit()) are now no-op and no longer affect span behavior. To interact with spans, use ddtrace.trace_utils.get_current_span() or ddtrace.trace_utils.get_current_root_span() instead.initialize requests and their responses on modelcontextprotocol/python-sdk servers.source:otel tagging for evaluations when OpenTelemetry (OTel) tracing is enabled when DD_TRACE_OTEL_ENABLED=true is set. This tag allows the backend to wait for OTel span conversion before processing evaluations.DD_MCP_CAPTURE_INTENT environment variable.asyncio.BoundedSemaphore lock type profiling in Python Lock Profiler.asyncio.Condition locking type profiling in Python. The Lock profiler now provides visibility into asyncio.Condition usage, helping identify contention in async applications using condition variables.asyncio.Semaphore lock type profiling in Python Lock Profiler.asyncio.shield.asyncio.TaskGroup.aws.httpapi span name for v2 apis when the API Gateway sets the x-dd-proxy header to aws-httpapi. Additionally, the tag http.route and the resource name of the span now contains the api resource path instead of the path when propagated with the x-dd-proxy-resource-path header.DD_TRACE_REMOVE_INTEGRATION_SERVICE_NAMES_ENABLED support, which was previously ignored.https:// scheme prefix as part of the http.url tag; this caused the entire url to be parsed as the http path.isDefined to result in an evaluation error.DD_EXPERIMENTAL_FLAGGING_PROVIDER_ENABLED=true), ensuring remote configuration is received before process forking occurs.None metadata in Ray job submission caused a crash.AttributeError when calling tag_agent_manifest.annotation_context blocks caused annotations to fail after the first operation in subsequent contexts. Previously, the trace context created by the first annotation_context remained active after exiting, causing the second context to reuse a stale context ID. This resulted in annotations not being applied to spans after the first batch call in the second annotation_context block.activate_distributed_headers() where distributed requests missing a LLM Observability trace ID would be incorrectly propagated twice.TypeError during profiling. One example of this is neo4j's AsyncRLock, which inherits from asyncio.Lock: https://github.com/neo4j/neo4j-python-driver/blob/6.x/src/neo4j/_async_compat/concurrency.py#L45uvloop and forking has been resolved._psutil_linux.abi3.so file in an injected environment.The Hooks class (config. .hooks) is deprecated and will be removed in v5.0. All hook methods (register(), on(), deregister(), emit()) are now no-op an…
Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
Hooks class (config.<integration>.hooks) is deprecated and will be removed in v5.0. All hook methods (register(), on(), deregister(), emit()) are now no-op and no longer affect span behavior. To interact with spans, use ddtrace.trace_utils.get_current_span() or ddtrace.trace_utils.get_current_root_span() instead.initialize requests and their responses on modelcontextprotocol/python-sdk servers.source:otel tagging for evaluations when OpenTelemetry (OTel) tracing is enabled when DD_TRACE_OTEL_ENABLED=true is set. This tag allows the backend to wait for OTel span conversion before processing evaluations.DD_MCP_CAPTURE_INTENT environment variable.asyncio.BoundedSemaphore lock type profiling in Python Lock Profiler.asyncio.Condition locking type profiling in Python. The Lock profiler now provides visibility into asyncio.Condition usage, helping identify contention in async applications using condition variables.asyncio.Semaphore lock type profiling in Python Lock Profiler.asyncio.shield.asyncio.TaskGroup.aws.httpapi span name for v2 apis when the API Gateway sets the x-dd-proxy header to aws-httpapi. Additionally, the tag http.route and the resource name of the span now contains the api resource path instead of the path when propagated with the x-dd-proxy-resource-path header.DD_TRACE_REMOVE_INTEGRATION_SERVICE_NAMES_ENABLED support, which was previously ignored.https:// scheme prefix as part of the http.url tag; this caused the entire url to be parsed as the http path.isDefined to result in an evaluation error.DD_EXPERIMENTAL_FLAGGING_PROVIDER_ENABLED=true), ensuring remote configuration is received before process forking occurs.None metadata in Ray job submission caused a crash.AttributeError when calling tag_agent_manifest.annotation_context blocks caused annotations to fail after the first operation in subsequent contexts. Previously, the trace context created by the first annotation_context remained active after exiting, causing the second context to reuse a stale context ID. This resulted in annotations not being applied to spans after the first batch call in the second annotation_context block.activate_distributed_headers() where distributed requests missing a LLM Observability trace ID would be incorrectly propagated twice.TypeError during profiling. One example of this is neo4j's AsyncRLock, which inherits from asyncio.Lock: https://github.com/neo4j/neo4j-python-driver/blob/6.x/src/neo4j/_async_compat/concurrency.py#L45uvloop and forking has been resolved._psutil_linux.abi3.so file in an injected environment.The Hooks class (config. .hooks) is deprecated and will be removed in v5.0. All hook methods (register(), on(), deregister(), emit()) are now no-op an…
Estimated end-of-life date, accurate to within three months: 07-2027 See the support level definitions for more information.
Hooks class (config.<integration>.hooks) is deprecated and will be removed in v5.0. All hook methods (register(), on(), deregister(), emit()) are now no-op and no longer affect span behavior. To interact with spans, use ddtrace.trace_utils.get_current_span() or ddtrace.trace_utils.get_current_root_span() instead.initialize requests and their responses on modelcontextprotocol/python-sdk servers.source:otel tagging for evaluations when OpenTelemetry (OTel) tracing is enabled when DD_TRACE_OTEL_ENABLED=true is set. This tag allows the backend to wait for OTel span conversion before processing evaluations.asyncio.BoundedSemaphore lock type profiling in Python Lock Profiler.asyncio.Condition locking type profiling in Python. The Lock profiler now provides visibility into asyncio.Condition usage, helping identify contention in async applications using condition variables.asyncio.Semaphore lock type profiling in Python Lock Profiler.aws.httpapi span name for v2 apis when the API Gateway sets the x-dd-proxy header to aws-httpapi. Additionally, the tag http.route and the resource name of the span now contains the api resource path instead of the path when propagated with the x-dd-proxy-resource-path header.DD_TRACE_REMOVE_INTEGRATION_SERVICE_NAMES_ENABLED support, which was previously ignored.https:// scheme prefix as part of the http.url tag; this caused the entire url to be parsed as the http path.isDefined to result in an evaluation error.DD_EXPERIMENTAL_FLAGGING_PROVIDER_ENABLED=true), ensuring remote configuration is received before process forking occurs.None metadata in Ray job submission caused a crash.AttributeError when calling tag_agent_manifest.TypeError during profiling. One example of this is neo4j's AsyncRLock, which inherits from asyncio.Lock: https://github.com/neo4j/neo4j-python-driver/blob/6.x/src/neo4j/_async_compat/concurrency.py#L45uvloop and forking has been resolved.Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
anthropic: Handle empty content field when tracing the beta client.
LLM Observability: Resolves an issue in activate_distributed_headers() where distributed requests missing a LLM Observability trace ID would be incorrectly propagated twice.
profiling: Fixes a bug where code that sub-classes our wrapped locks crashes with TypeError during profiling. One example of this is neo4j's AsyncRLock, which inherits from asyncio.Lock: https://github.com/neo4j/neo4j-python-driver/blob/6.x/src/neo4j/_async_compat/concurrency.py#L45
profiling: This fix resolves a race condition leading to incorrect stacks being reported for asyncio parent/child Tasks (e.g. when using asyncio.gather).
profiling: Fixes a PicklingError when using multiprocessing.Manager() with lock profiling enabled on Python 3.14+. Python 3.14 changed the default multiprocessing start method from fork to forkserver, which requires all objects to be pickleable. The lock profiler's wrappers now properly support pickle serialization.
Estimated end-of-life date, accurate to within three months: 07-2027 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 07-2027 See the support level definitions for more information.
_psutil_linux.abi3.so file in an injected environment.<!-- -->
annotation_context blocks caused annotations to fail after the first operation in subsequent contexts. Previously, the trace context created by the first annotation_context remained active after exiting, causing the second context to reuse a stale context ID. This resulted in annotations not being applied to spans after the first batch call in the second annotation_context block.<!-- -->
<!-- -->
uvloop and forking has been resolved.<!-- -->
<!-- -->
Estimated end-of-life date, accurate to within three months: 07-2027 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 07-2027 See the support level definitions for more information.
None metadata in Ray job submission caused a crash.Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
DD_CODE_ORIGIN_FOR_SPANS_ENABLED=true is setDeprecated support for Tornado versions older than v6.1. Use Tornado v6.1 or later.
⚠️ An issue was detected with memory profiling in this release. Please consider upgrading to v4.1.3 or newer
Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
DD_CODE_ORIGIN_FOR_SPANS_ENABLED=true is setdataset_name, project_name, project_id, experiment_name.ExperimentResult class' rows and summary_evaluations attributes are deprecated and will be removed in the next major release. ExperimentResult.rows/summary_evaluations attributes will only store the results of the first run iteration for multi-run experiments. Use the ExperimentResult.runs attribute instead to access experiment results and summary evaluations.threading.BoundedSemaphore locking type profiling in Python. The implementation follows the same approach as threading.Semaphore, properly handling internal lock detection to prevent double-counting of the underlying threading.Lock object.threading.Semaphore locking type profiling in Python. The Lock profiler now detects and marks "internal" Lock objects, i.e. those that are part of implementation of higher-level locking types. One example of such higher-level primitive is threading.Semaphore, which is implemented with threading.Condition, which itself uses threading.Lock internally. Marking internal lock as "internal" will prevent it from being sampled, ensuring that the high-level (e.g. Semaphore) sample is processed.process_id tag to profiles. The value of this tag is the current process ID (PID).asyncio.wait.codeobject.co_qualname in memory profiler and lock profiler flamegraphs for Python 3.11+. Stack profiler has already been using this. This aligns the user experience across different profile types.asyncio.as_completed util in the Profiler.asyncio.wait in the Profiler. This makes it possible to track dependencies between Tasks/Coroutines that await/are awaited through asyncio.wait.client.beta.messages.create() and client.beta.messages.stream()). This feature requires Anthropic client version 0.37.0 or higher.aiokafka>=0.9.0. See the aiokafka<https://ddtrace.readthedocs.io/en/stable/integrations.html#aiokafka> documentation for more information.thread=False is no longer required when performing monkey-patching with gevent via gevent.monkey.patch_all.responses endpoint (available in OpenAI SDK >= 1.87.0).runs argument, to assess the true performance of an experiment in the face of the non determinism of LLMs. Use the new ExperimentResult class' runs attribute to access the results and summary evaluations by run iteration.RunnableLambda instances._Py_DumpTracebackThreads function is not available.IndexError in partial flush when the finished span counter was out of sync with actual finished spans.DD_TRACE_PARTIAL_FLUSH_MIN_SPANS values less than 1 now default to 1 with a warning.ray.init() at the top of their scripts were not properly instrumented, resulting in incomplete traces. To ensure full tracing capabilities, use ddtrace-run when starting your Ray cluster: DD_PATCH_MODULES="ray:true,aiohttp:false,grpc:false,requests:false" ddtrace-run ray start --head.gsutil toolLLMObs.export_span() would raise when LLMObs is disabled.self was being annotated as an input parameter using LLM Observability function decorators.LLMObs.annotation_context() properties (tags, prompt, and name) were not applied to subsequent LLM operations within the same context block. This occurred when multiple sequential operations (such as Langchain batch calls with structured outputs) were performed, causing only the first operation to receive the annotations.AttributeError when trying to access the name or description attributes of a tool.opentelemetry.trace.get_current_span() or NonRecordingSpan. Spans are now kept and appear in the UI unless explicitly dropped by the Agent or sampling rules.frame.f_locals while trying to retrieve class name of a PyFrameObject.asyncio.gather).Deprecated support for Tornado versions older than v6.1. Use Tornado v6.1 or later.
Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
dataset_name, project_name, project_id, experiment_name.ddtrace.contrib.tornado module. Configure tracing using environment variables and import ddtrace.auto instead.ExperimentResult class' rows and summary_evaluations attributes are deprecated and will be removed in the next major release. ExperimentResult.rows/summary_evaluations attributes will only store the results of the first run iteration for multi-run experiments. Use the ExperimentResult.runs attribute instead to access experiment results and summary evaluations.threading.BoundedSemaphore locking type profiling in Python. The implementation follows the same approach as threading.Semaphore, properly handling internal lock detection to prevent double-counting of the underlying threading.Lock object.threading.Semaphore locking type profiling in Python. The Lock profiler now detects and marks "internal" Lock objects, i.e. those that are part of implementation of higher-level locking types. One example of such higher-level primitive is threading.Semaphore, which is implemented with threading.Condition, which itself uses threading.Lock internally. Marking internal lock as "internal" will prevent it from being sampled, ensuring that the high-level (e.g. Semaphore) sample is processed.process_id tag to profiles. The value of this tag is the current process ID (PID).asyncio.wait.codeobject.co_qualname in memory profiler and lock profiler flamegraphs for Python 3.11+. Stack profiler has already been using this. This aligns the user experience across different profile types.asyncio.as_completed util in the Profiler.asyncio.wait in the Profiler. This makes it possible to track dependencies between Tasks/Coroutines that await/are awaited through asyncio.wait.client.beta.messages.create() and client.beta.messages.stream()). This feature requires Anthropic client version 0.37.0 or higher.aiokafka>=0.9.0. See the aiokafka<https://ddtrace.readthedocs.io/en/stable/integrations.html#aiokafka> documentation for more information.thread=False is no longer required when performing monkey-patching with gevent via gevent.monkey.patch_all.responses endpoint (available in OpenAI SDK >= 1.87.0).runs argument, to assess the true performance of an experiment in the face of the non determinism of LLMs. Use the new ExperimentResult class' runs attribute to access the results and summary evaluations by run iteration.RunnableLambda instances._Py_DumpTracebackThreads function is not available.IndexError in partial flush when the finished span counter was out of sync with actual finished spans.DD_TRACE_PARTIAL_FLUSH_MIN_SPANS values less than 1 now default to 1 with a warning.ray.init() at the top of their scripts were not properly instrumented, resulting in incomplete traces. To ensure full tracing capabilities, use ddtrace-run when starting your Ray cluster: DD_PATCH_MODULES="ray:true,aiohttp:false,grpc:false,requests:false" ddtrace-run ray start --head.gsutil toolLLMObs.export_span() would raise when LLMObs is disabled.self was being annotated as an input parameter using LLM Observability function decorators.LLMObs.annotation_context() properties (tags, prompt, and name) were not applied to subsequent LLM operations within the same context block. This occurred when multiple sequential operations (such as Langchain batch calls with structured outputs) were performed, causing only the first operation to receive the annotations.AttributeError when trying to access the name or description attributes of a tool.opentelemetry.trace.get_current_span() or NonRecordingSpan. Spans are now kept and appear in the UI unless explicitly dropped by the Agent or sampling rules.frame.f_locals while trying to retrieve class name of a PyFrameObject.asyncio.gather).Deprecated support for Tornado versions older than v6.1. Use Tornado v6.1 or later.
Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
dataset_name, project_name, project_id, experiment_name.ddtrace.contrib.tornado module. Configure tracing using environment variables and import ddtrace.auto instead.ExperimentResult class' rows and summary_evaluations attributes are deprecated and will be removed in the next major release. ExperimentResult.rows/summary_evaluations attributes will only store the results of the first run iteration for multi-run experiments. Use the ExperimentResult.runs attribute instead to access experiment results and summary evaluations.threading.BoundedSemaphore locking type profiling in Python. The implementation follows the same approach as threading.Semaphore, properly handling internal lock detection to prevent double-counting of the underlying threading.Lock object.threading.Semaphore locking type profiling in Python. The Lock profiler now detects and marks "internal" Lock objects, i.e. those that are part of implementation of higher-level locking types. One example of such higher-level primitive is threading.Semaphore, which is implemented with threading.Condition, which itself uses threading.Lock internally. Marking internal lock as "internal" will prevent it from being sampled, ensuring that the high-level (e.g. Semaphore) sample is processed.process_id tag to profiles. The value of this tag is the current process ID (PID).asyncio.wait.codeobject.co_qualname in memory profiler and lock profiler flamegraphs for Python 3.11+. Stack profiler has already been using this. This aligns the user experience across different profile types.asyncio.as_completed util in the Profiler.asyncio.wait in the Profiler. This makes it possible to track dependencies between Tasks/Coroutines that await/are awaited through asyncio.wait.client.beta.messages.create() and client.beta.messages.stream()). This feature requires Anthropic client version 0.37.0 or higher.aiokafka>=0.9.0. See the aiokafka<https://ddtrace.readthedocs.io/en/stable/integrations.html#aiokafka> documentation for more information.thread=False is no longer required when performing monkey-patching with gevent via gevent.monkey.patch_all.responses endpoint (available in OpenAI SDK >= 1.87.0).runs argument, to assess the true performance of an experiment in the face of the non determinism of LLMs. Use the new ExperimentResult class' runs attribute to access the results and summary evaluations by run iteration.RunnableLambda instances._Py_DumpTracebackThreads function is not available.IndexError in partial flush when the finished span counter was out of sync with actual finished spans.DD_TRACE_PARTIAL_FLUSH_MIN_SPANS values less than 1 now default to 1 with a warning.ray.init() at the top of their scripts were not properly instrumented, resulting in incomplete traces. To ensure full tracing capabilities, use ddtrace-run when starting your Ray cluster: DD_PATCH_MODULES="ray:true,aiohttp:false,grpc:false,requests:false" ddtrace-run ray start --head.gsutil toolLLMObs.export_span() would raise when LLMObs is disabled.self was being annotated as an input parameter using LLM Observability function decorators.LLMObs.annotation_context() properties (tags, prompt, and name) were not applied to subsequent LLM operations within the same context block. This occurred when multiple sequential operations (such as Langchain batch calls with structured outputs) were performed, causing only the first operation to receive the annotations.AttributeError when trying to access the name or description attributes of a tool.opentelemetry.trace.get_current_span() or NonRecordingSpan. Spans are now kept and appear in the UI unless explicitly dropped by the Agent or sampling rules.frame.f_locals while trying to retrieve class name of a PyFrameObject.asyncio.gather).Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
activate_distributed_headers() where distributed requests missing a LLM Observability trace ID would be incorrectly propagated twice.<!-- -->
TypeError during profiling.
One example of this is neo4j's AsyncRLock, which inherits from asyncio.Lock:
https://github.com/neo4j/neo4j-python-driver/blob/6.x/src/neo4j/_async_compat/concurrency.py#L45<!-- -->
asyncio.gather).Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
<!-- -->
uvloop and forking has been resolved.<!-- -->
profiling: This fix resolves an issue where memory profiler module fails to load when the system doesn't have libatomic installed.
profiling: This fix ensures the profiler now correctly tracks dependencies between Tasks and Coroutines that are awaiting or being awaited via asyncio.wait.
Fix exposure event deduplication to use (flag_key, subject_id) as cache key instead of (flag_key, variant_key, allocation_key). This ensures different
Fixes critical memory safety issue in IAST when used with forked worker processes (MCP servers with Gunicorn and Uvicorn). Workers previously crashed
Code Security:
tracing:
LLM Observability:
AttributeError when trying to access the name or description attributes of a tool.AAP:
lib-injection:
gsutil toolprofiling:
frame.f_locals while trying to retrieve class name of a PyFrameObject.To find which of these your code relies on, follow the "deprecation warnings" instructions here.
Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
This is a major-version release that contains many backwards-incompatible changes to public APIs. To find which of these your code relies on, follow the "deprecation warnings" instructions here.
dd-trace-py now includes an OpenFeature provider implementation, enabling feature flag evaluation through the OpenFeature API.
This integration is under active design and development. Functionality and APIs are experimental and may change without notice. For more information, see the Datadog documentation at https://docs.datadoghq.com/feature_flags/#overview
ddtrace.Pin object with mongoengine. With this change, the ddtrace library no longer directly supports mongoengine. Mongoengine will be supported through the pymongo integration.pytest_benchmark and pytest_bdd integrations. These plugins are now supported by the regular pytest integration.DD_DYNAMIC_INSTRUMENTATION_UPLOAD_FLUSH_INTERVAL variable.DD_EXCEPTION_DEBUGGING_ENABLED variable.Span.set_tag_str has been removed, use Span.set_tag instead.Span.set_struct_tag has been removed.Span.get_struct_tag has been removed.Span._pprint has been removedSpan.finished setter was removed, please use Span.finish() method instead.Tracer.on_start_span method has been removed.Tracer.deregister_on_start_span method has been removed.ddtrace.trace.Pin has been removed.Span.finish_with_ancestors was removed with no replacement.Span.set_tag typing is now set_tag(key: str, value: Optional[str] = None) -> NoneSpan.get_tag typing is now get_tag(key: str) -> Optional[str]Span.set_tags typing is now set_tags(tags: dict[str, str]) -> NoneSpan.get_tags typing is now get_tags() -> dict[str, str]Span.set_metric typing is now set_metric(key: str, value: int | float) -> NoneSpan.get_metric typing is now get_metric(key: str) -> Optional[int | float]Span.set_metrics typing is now set_metrics(metrics: Dict[str, int | float]) -> NoneSpan.get_metrics typing is now get_metrics() -> dict[str, int | float]Span.record_exception's timestamp and escaped parameters are removedLLMObs.annotate(), LLMObs.export_span(), LLMObs.submit_evaluation(), LLMObs.inject_distributed_headers(), and LLMObs.activate_distributed_headers() now raise exceptions instead of logging. LLM Observability auto-instrumentation is not affected.LLMObs.submit_evaluation_for() has been removed. Please use LLMObs.submit_evaluation() instead for submitting evaluations. To migrate:
LLMObs.submit_evaluation_for(...) users: rename to LLMObs.submit_evaluation(...)LLMObs.submit_evaluation_for(...) users: rename the span_context argument to span, i.e. LLMObs.submit_evaluation(span_context={"span_id": ..., "trace_id": ...}, ...) to LLMObs.submit_evaluation(span={"span_id": ..., "trace_id": ...}, ...)DD_PROFILING_STACK_V2_ENABLED is now removed.freezegun integration is now removed.opentracer packagegoogle_generativeai integration has been removed as the google_generativeai library has reached end-of-life.
As an alternative, you can use the recommended google_genai library and corresponding integration instead.tiktoken library, and instead
will default to having their token counts estimated if not explicitly provided in the OpenAI response object. To guarantee accurate streamed token metrics, set stream_options={"include_usage": True} in the OpenAI request.DD_DJANGO_TRACING_MINIMAL now defaults to true). Django ORM, cache, and template instrumentation are disabled by default to eliminate duplicate span creation since library integrations for database drivers (psycopg, MySQLdb, sqlite3), cache clients (redis, memcached), template renderers (Jinja2), and other supported libraries continue to be traced. This reduces performance overhead by removing redundant Django-layer instrumentation. To restore all Django instrumentation, set DD_DJANGO_TRACING_MINIMAL=false, or enable individual features using DD_DJANGO_INSTRUMENT_DATABASES=true, DD_DJANGO_INSTRUMENT_CACHES=true, and DD_DJANGO_INSTRUMENT_TEMPLATES=true.DD_DJANGO_INSTRUMENT_DATABASES=true (default false), database instrumentation now merges Django-specific tags into database driver spans created by supported integrations (psycopg, sqlite3, MySQLdb, etc.) instead of creating duplicate Django database spans. If the database cursor is not already wrapped by a supported integration, Django wraps it and creates a span. This change reduces overhead and duplicate spans while preserving visibility into database operations.ddtrace.settings package. Environment variables should be used to adjust settings.HttpPropagator.injectDEFAULT_RUNTIME_METRICS_INTERVAL.ddtrace.contrib.tornado module. Configure tracing using environment variables and import ddtrace.auto instead.urllib3 and requests. It does not require enabling APM instrumentation for urllib3 anymore.threading.RLock (reentrant lock) profiling. The Lock profiler now tracks both threading.Lock and threading.RLock usage, providing comprehensive lock contention visibility for Python applications.version argument to LLMObs.pull_datasetversion and latest_version to provide information on the version of the dataset that is being worked with and the latest global version of the dataset, respectivelyLock, RLock, Event).HTTPS_PROXY.assessment argument in submit_evaluation().langchain integration would incorrectly mark Azure OpenAI calls as duplicate llm operations even if the openai integration was enabled.DD_PROFILING_API_TIMEOUT doesn't have any effect, and is marked to be removed in upcoming 4.0 release. New environment variable DD_PROFILING_API_TIMEOUT_MS is introduced to configure timeout for uploading profiles to the backend. The default value is 10000 ms (10 seconds)_acquire method of the Lock profiler (note: this only occurs when assertions are enabled.)version field. The version field is now omitted unless explicitly set by the user.
assessment now refers to whether the evaluation itself passes or fails according to your application, rather than the validity of the evaluation result.
The langchain integration will trace Azure OpenAI spans as workflow spans if there is an equivalent llm span from the openai integration.IndexError.async for model in client.models.list()) caused a TypeError: 'async for' requires an object with __aiter__ method, got coroutine. See issue #14574.KeyError exceptions in test runs when gevent is detected within the environment.ray.init().ray.data._internal to the module denylist.ValueError: Formatting field not found in record: 'dd.service'.ResourceWarning in multiprocess scenarios.wrapt library dependency from the Lock Profiler implementation, improving performance and reducing overhead during lock instrumentation.To find which of these your code relies on, follow the "deprecation warnings" instructions here.
Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
This is a major-version release that contains many backwards-incompatible changes to public APIs. To find which of these your code relies on, follow the "deprecation warnings" instructions here.
dd-trace-py now includes an OpenFeature provider implementation, enabling feature flag evaluation through the OpenFeature API.
This integration is under active design and development. Functionality and APIs are experimental and may change without notice. For more information, see the Datadog documentation at https://docs.datadoghq.com/feature_flags/#overview
ddtrace.Pin object with mongoengine. With this change, the ddtrace library no longer directly supports mongoengine. Mongoengine will be supported through the pymongo integration.pytest_benchmark and pytest_bdd integrations. These plugins are now supported by the regular pytest integration.DD_DYNAMIC_INSTRUMENTATION_UPLOAD_FLUSH_INTERVAL variable.DD_EXCEPTION_DEBUGGING_ENABLED variable.Span.set_tag_str has been removed, use Span.set_tag instead.Span.set_struct_tag has been removed.Span.get_struct_tag has been removed.Span._pprint has been removedSpan.finished setter was removed, please use Span.finish() method instead.Tracer.on_start_span method has been removed.Tracer.deregister_on_start_span method has been removed.ddtrace.trace.Pin has been removed.Span.finish_with_ancestors was removed with no replacement.Span.set_tag typing is now set_tag(key: str, value: Optional[str] = None) -> NoneSpan.get_tag typing is now get_tag(key: str) -> Optional[str]Span.set_tags typing is now set_tags(tags: dict[str, str]) -> NoneSpan.get_tags typing is now get_tags() -> dict[str, str]Span.set_metric typing is now set_metric(key: str, value: int | float) -> NoneSpan.get_metric typing is now get_metric(key: str) -> Optional[int | float]Span.set_metrics typing is now set_metrics(metrics: Dict[str, int | float]) -> NoneSpan.get_metrics typing is now get_metrics() -> dict[str, int | float]Span.record_exception's timestamp and escaped parameters are removedLLMObs.annotate(), LLMObs.export_span(), LLMObs.submit_evaluation(), LLMObs.inject_distributed_headers(), and LLMObs.activate_distributed_headers() now raise exceptions instead of logging. LLM Observability auto-instrumentation is not affected.LLMObs.submit_evaluation_for() has been removed. Please use LLMObs.submit_evaluation() instead for submitting evaluations. To migrate:
LLMObs.submit_evaluation_for(...) users: rename to LLMObs.submit_evaluation(...)LLMObs.submit_evaluation_for(...) users: rename the span_context argument to span, i.e. LLMObs.submit_evaluation(span_context={"span_id": ..., "trace_id": ...}, ...) to LLMObs.submit_evaluation(span={"span_id": ..., "trace_id": ...}, ...)DD_PROFILING_STACK_V2_ENABLED is now removed.freezegun integration is now removed.opentracer packagegoogle_generativeai integration has been removed as the google_generativeai library has reached end-of-life.
As an alternative, you can use the recommended google_genai library and corresponding integration instead.tiktoken library, and instead
will default to having their token counts estimated if not explicitly provided in the OpenAI response object. To guarantee accurate streamed token metrics, set stream_options={"include_usage": True} in the OpenAI request.DD_DJANGO_TRACING_MINIMAL now defaults to true). Django ORM, cache, and template instrumentation are disabled by default to eliminate duplicate span creation since library integrations for database drivers (psycopg, MySQLdb, sqlite3), cache clients (redis, memcached), template renderers (Jinja2), and other supported libraries continue to be traced. This reduces performance overhead by removing redundant Django-layer instrumentation. To restore all Django instrumentation, set DD_DJANGO_TRACING_MINIMAL=false, or enable individual features using DD_DJANGO_INSTRUMENT_DATABASES=true, DD_DJANGO_INSTRUMENT_CACHES=true, and DD_DJANGO_INSTRUMENT_TEMPLATES=true.DD_DJANGO_INSTRUMENT_DATABASES=true (default false), database instrumentation now merges Django-specific tags into database driver spans created by supported integrations (psycopg, sqlite3, MySQLdb, etc.) instead of creating duplicate Django database spans. If the database cursor is not already wrapped by a supported integration, Django wraps it and creates a span. This change reduces overhead and duplicate spans while preserving visibility into database operations.ddtrace.settings package. Environment variables should be used to adjust settings.HttpPropagator.injectDEFAULT_RUNTIME_METRICS_INTERVAL.ddtrace.contrib.tornado module. Configure tracing using environment variables and import ddtrace.auto instead.urllib3 and requests. It does not require enabling APM instrumentation for urllib3 anymore.threading.RLock (reentrant lock) profiling. The Lock profiler now tracks both threading.Lock and threading.RLock usage, providing comprehensive lock contention visibility for Python applications.version argument to LLMObs.pull_datasetversion and latest_version to provide information on the version of the dataset that is being worked with and the latest global version of the dataset, respectivelyLock, RLock, Event).HTTPS_PROXY.assessment argument in submit_evaluation().langchain integration would incorrectly mark Azure OpenAI calls as duplicate llm operations even if the openai integration was enabled.DD_PROFILING_API_TIMEOUT doesn't have any effect, and is marked to be removed in upcoming 4.0 release. New environment variable DD_PROFILING_API_TIMEOUT_MS is introduced to configure timeout for uploading profiles to the backend. The default value is 10000 ms (10 seconds)_acquire method of the Lock profiler (note: this only occurs when assertions are enabled.)version field. The version field is now omitted unless explicitly set by the user.
assessment now refers to whether the evaluation itself passes or fails according to your application, rather than the validity of the evaluation result.
The langchain integration will trace Azure OpenAI spans as workflow spans if there is an equivalent llm span from the openai integration.IndexError.async for model in client.models.list()) caused a TypeError: 'async for' requires an object with __aiter__ method, got coroutine. See issue #14574.KeyError exceptions in test runs when gevent is detected within the environment.ray.init().ray.data._internal to the module denylist.ValueError: Formatting field not found in record: 'dd.service'.ResourceWarning in multiprocess scenarios.wrapt library dependency from the Lock Profiler implementation, improving performance and reducing overhead during lock instrumentation.To find which of these your code relies on, follow the "deprecation warnings" instructions here.
Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
This is a major-version release that contains many backwards-incompatible changes to public APIs. To find which of these your code relies on, follow the "deprecation warnings" instructions here.
dd-trace-py now includes an OpenFeature provider implementation, enabling feature flag evaluation through the OpenFeature API.
This integration is under active design and development. Functionality and APIs are experimental and may change without notice. For more information, see the Datadog documentation at https://docs.datadoghq.com/feature_flags/#overview
ddtrace.Pin object with mongoengine. With this change, the ddtrace library no longer directly supports mongoengine. Mongoengine will be supported through the pymongo integration.pytest_benchmark and pytest_bdd integrations. These plugins are now supported by the regular pytest integration.DD_DYNAMIC_INSTRUMENTATION_UPLOAD_FLUSH_INTERVAL variable.DD_EXCEPTION_DEBUGGING_ENABLED variable.Span.set_tag_str has been removed, use Span.set_tag instead.Span.set_struct_tag has been removed.Span.get_struct_tag has been removed.Span._pprint has been removedSpan.finished setter was removed, please use Span.finish() method instead.Tracer.on_start_span method has been removed.Tracer.deregister_on_start_span method has been removed.ddtrace.trace.Pin has been removed.Span.finish_with_ancestors was removed with no replacement.Span.set_tag typing is now set_tag(key: str, value: Optional[str] = None) -> NoneSpan.get_tag typing is now get_tag(key: str) -> Optional[str]Span.set_tags typing is now set_tags(tags: dict[str, str]) -> NoneSpan.get_tags typing is now get_tags() -> dict[str, str]Span.set_metric typing is now set_metric(key: str, value: int | float) -> NoneSpan.get_metric typing is now get_metric(key: str) -> Optional[int | float]Span.set_metrics typing is now set_metrics(metrics: Dict[str, int | float]) -> NoneSpan.get_metrics typing is now get_metrics() -> dict[str, int | float]Span.record_exception's timestamp and escaped parameters are removedLLMObs.annotate(), LLMObs.export_span(), LLMObs.submit_evaluation(), LLMObs.inject_distributed_headers(), and LLMObs.activate_distributed_headers() now raise exceptions instead of logging. LLM Observability auto-instrumentation is not affected.LLMObs.submit_evaluation_for() has been removed. Please use LLMObs.submit_evaluation() instead for submitting evaluations. To migrate:
LLMObs.submit_evaluation_for(...) users: rename to LLMObs.submit_evaluation(...)LLMObs.submit_evaluation_for(...) users: rename the span_context argument to span, i.e. LLMObs.submit_evaluation(span_context={"span_id": ..., "trace_id": ...}, ...) to LLMObs.submit_evaluation(span={"span_id": ..., "trace_id": ...}, ...)DD_PROFILING_STACK_V2_ENABLED is now removed.freezegun integration is now removed.opentracer packagegoogle_generativeai integration has been removed as the google_generativeai library has reached end-of-life.
As an alternative, you can use the recommended google_genai library and corresponding integration instead.tiktoken library, and instead
will default to having their token counts estimated if not explicitly provided in the OpenAI response object. To guarantee accurate streamed token metrics, set stream_options={"include_usage": True} in the OpenAI request.DD_DJANGO_TRACING_MINIMAL now defaults to true). Django ORM, cache, and template instrumentation are disabled by default to eliminate duplicate span creation since library integrations for database drivers (psycopg, MySQLdb, sqlite3), cache clients (redis, memcached), template renderers (Jinja2), and other supported libraries continue to be traced. This reduces performance overhead by removing redundant Django-layer instrumentation. To restore all Django instrumentation, set DD_DJANGO_TRACING_MINIMAL=false, or enable individual features using DD_DJANGO_INSTRUMENT_DATABASES=true, DD_DJANGO_INSTRUMENT_CACHES=true, and DD_DJANGO_INSTRUMENT_TEMPLATES=true.DD_DJANGO_INSTRUMENT_DATABASES=true (default false), database instrumentation now merges Django-specific tags into database driver spans created by supported integrations (psycopg, sqlite3, MySQLdb, etc.) instead of creating duplicate Django database spans. If the database cursor is not already wrapped by a supported integration, Django wraps it and creates a span. This change reduces overhead and duplicate spans while preserving visibility into database operations.ddtrace.settings package. Environment variables should be used to adjust settings.HttpPropagator.injectDEFAULT_RUNTIME_METRICS_INTERVAL.urllib3 and requests. It does not require enabling APM instrumentation for urllib3 anymore.threading.RLock (reentrant lock) profiling. The Lock profiler now tracks both threading.Lock and threading.RLock usage, providing comprehensive lock contention visibility for Python applications.version argument to LLMObs.pull_datasetversion and latest_version to provide information on the version of the dataset that is being worked with and the latest global version of the dataset, respectivelyLock, RLock, Event).HTTPS_PROXY.assessment argument in submit_evaluation().langchain integration would incorrectly mark Azure OpenAI calls as duplicate llm operations even if the openai integration was enabled.DD_PROFILING_API_TIMEOUT doesn't have any effect, and is marked to be removed in upcoming 4.0 release. New environment variable DD_PROFILING_API_TIMEOUT_MS is introduced to configure timeout for uploading profiles to the backend. The default value is 10000 ms (10 seconds)_acquire method of the Lock profiler (note: this only occurs when assertions are enabled.)version field. The version field is now omitted unless explicitly set by the user.
assessment now refers to whether the evaluation itself passes or fails according to your application, rather than the validity of the evaluation result.
The langchain integration will trace Azure OpenAI spans as workflow spans if there is an equivalent llm span from the openai integration.IndexError.async for model in client.models.list()) caused a TypeError: 'async for' requires an object with __aiter__ method, got coroutine. See issue #14574.KeyError exceptions in test runs when gevent is detected within the environment.ray.init().ray.data._internal to the module denylist.ValueError: Formatting field not found in record: 'dd.service'.ResourceWarning in multiprocess scenarios.wrapt library dependency from the Lock Profiler implementation, improving performance and reducing overhead during lock instrumentation.To find which of these your code relies on, follow the "deprecation warnings" instructions here.
Estimated end-of-life date, accurate to within three months: 05-2027 See the support level definitions for more information.
This is a major-version release that contains many backwards-incompatible changes to public APIs. To find which of these your code relies on, follow the "deprecation warnings" instructions here.
ddtrace.Pin object with mongoengine. With this change, the ddtrace library no longer directly supports mongoengine. Mongoengine will be supported through the pymongo integration.pytest_benchmark and pytest_bdd integrations. These plugins are now supported by the regular pytest integration.DD_DYNAMIC_INSTRUMENTATION_UPLOAD_FLUSH_INTERVAL variable.DD_EXCEPTION_DEBUGGING_ENABLED variable.Span.set_tag_str has been removed, use Span.set_tag instead.Span.set_struct_tag has been removed.Span.get_struct_tag has been removed.Span._pprint has been removedSpan.finished setter was removed, please use Span.finish() method instead.Tracer.on_start_span method has been removed.Tracer.deregister_on_start_span method has been removed.ddtrace.trace.Pin has been removed.Span.finish_with_ancestors was removed with no replacement.Span.set_tag typing is now set_tag(key: str, value: Optional[str] = None) -> NoneSpan.get_tag typing is now get_tag(key: str) -> Optional[str]Span.set_tags typing is now set_tags(tags: dict[str, str]) -> NoneSpan.get_tags typing is now get_tags() -> dict[str, str]Span.set_metric typing is now set_metric(key: str, value: int | float) -> NoneSpan.get_metric typing is now get_metric(key: str) -> Optional[int | float]Span.set_metrics typing is now set_metrics(metrics: Dict[str, int | float]) -> NoneSpan.get_metrics typing is now get_metrics() -> dict[str, int | float]Span.record_exception's timestamp and escaped parameters are removedLLMObs.annotate(), LLMObs.export_span(), LLMObs.submit_evaluation(), LLMObs.inject_distributed_headers(), and LLMObs.activate_distributed_headers() now raise exceptions instead of logging. LLM Observability auto-instrumentation is not affected.LLMObs.submit_evaluation_for() has been removed. Please use LLMObs.submit_evaluation() instead for submitting evaluations. To migrate:
LLMObs.submit_evaluation_for(...) users: rename to LLMObs.submit_evaluation(...)LLMObs.submit_evaluation_for(...) users: rename the span_context argument to span, i.e. LLMObs.submit_evaluation(span_context={"span_id": ..., "trace_id": ...}, ...) to LLMObs.submit_evaluation(span={"span_id": ..., "trace_id": ...}, ...)DD_PROFILING_STACK_V2_ENABLED is now removed.freezegun integration is now removed.opentracer packagegoogle_generativeai integration has been removed as the google_generativeai library has reached end-of-life.
As an alternative, you can use the recommended google_genai library and corresponding integration instead.tiktoken library, and instead
will default to having their token counts estimated if not explicitly provided in the OpenAI response object. To guarantee accurate streamed token metrics, set stream_options={"include_usage": True} in the OpenAI request.DD_DJANGO_TRACING_MINIMAL now defaults to true). Django ORM, cache, and template instrumentation are disabled by default to eliminate duplicate span creation since library integrations for database drivers (psycopg, MySQLdb, sqlite3), cache clients (redis, memcached), template renderers (Jinja2), and other supported libraries continue to be traced. This reduces performance overhead by removing redundant Django-layer instrumentation. To restore all Django instrumentation, set DD_DJANGO_TRACING_MINIMAL=false, or enable individual features using DD_DJANGO_INSTRUMENT_DATABASES=true, DD_DJANGO_INSTRUMENT_CACHES=true, and DD_DJANGO_INSTRUMENT_TEMPLATES=true.DD_DJANGO_INSTRUMENT_DATABASES=true (default false), database instrumentation now merges Django-specific tags into database driver spans created by supported integrations (psycopg, sqlite3, MySQLdb, etc.) instead of creating duplicate Django database spans. If the database cursor is not already wrapped by a supported integration, Django wraps it and creates a span. This change reduces overhead and duplicate spans while preserving visibility into database operations.ddtrace.settings package. Environment variables should be used to adjust settings.HttpPropagator.injectDEFAULT_RUNTIME_METRICS_INTERVAL.urllib3 and requests. It does not require enabling APM instrumentation for urllib3 anymore.threading.RLock (reentrant lock) profiling. The Lock profiler now tracks both threading.Lock and threading.RLock usage, providing comprehensive lock contention visibility for Python applications.version argument to LLMObs.pull_datasetversion and latest_version to provide information on the version of the dataset that is being worked with and the latest global version of the dataset, respectivelyLock, RLock, Event).HTTPS_PROXY.assessment argument in submit_evaluation().langchain integration would incorrectly mark Azure OpenAI calls as duplicate llm operations even if the openai integration was enabled.DD_PROFILING_API_TIMEOUT doesn't have any effect, and is marked to be removed in upcoming 4.0 release. New environment variable DD_PROFILING_API_TIMEOUT_MS is introduced to configure timeout for uploading profiles to the backend. The default value is 10000 ms (10 seconds)_acquire method of the Lock profiler (note: this only occurs when assertions are enabled.)version field. The version field is now omitted unless explicitly set by the user.
assessment now refers to whether the evaluation itself passes or fails according to your application, rather than the validity of the evaluation result.
The langchain integration will trace Azure OpenAI spans as workflow spans if there is an equivalent llm span from the openai integration.IndexError.async for model in client.models.list()) caused a TypeError: 'async for' requires an object with __aiter__ method, got coroutine. See issue #14574.KeyError exceptions in test runs when gevent is detected within the environment.ray.init().ray.data._internal to the module denylist.ValueError: Formatting field not found in record: 'dd.service'.ResourceWarning in multiprocess scenarios.wrapt library dependency from the Lock Profiler implementation, improving performance and reducing overhead during lock instrumentation.tracing: Fixes a security vulnerability, as described in GHSA-3633-r77q-h3v3 .
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
contextvars) could cause use-after-free or double-free crashes (SIGSEGV) inside libddwaf. A per-context lock now serializes WAF calls on the same context.Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
gevent.joinall is called.Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
isDefined to result in an evaluation error.<!-- -->
TypeError during profiling. One example of this is neo4j's AsyncRLock, which inherits from asyncio.Lock: https://github.com/neo4j/neo4j-python-driver/blob/6.x/src/neo4j/_async_compat/concurrency.py#L45LLM Observability: Warning logs for incorrect usage of the LLM Observability SDK are deprecated and will be replaced with raised errors in ddtrace>=4.…
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
ddtrace>=4.0.0.<!-- -->
<!-- -->
profiling: This fix resolves an issue where memory profiler module fails to load when the system doesn't have libatomic installed.
profiling: This fix ensures the profiler now correctly tracks dependencies between Tasks and Coroutines that are awaiting or being awaited via asyncio.wait.
Resolves a potential deadlock when forking.
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
Deprecated support for Tornado versions older than v6.1. Use Tornado v6.1 or later.
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
ddtrace.contrib.tornado module. Configure tracing using environment variables and import ddtrace.auto instead.AAP: This fix resolves an issue where the appsec layer was not compatible anymore with the lambda/serverless version of the tracer.
Code Security: Fixes critical memory safety issue in IAST when used with forked worker processes (MCP servers with Gunicorn and Uvicorn). Workers previously crashed with segmentation faults due to stale PyObject pointers in native taint maps after fork.
Dynamic instrumentation: fix issue with line probes matching the wrong source file when multiple source files from different Python path entries share the same name.
Exception replay: ensure exception information is captured when exceptions are raised by the GraphQL client library.
Lib-injection: do not inject into the gsutil tool
LLM Observability: Fixes an issue where the Google ADK integration would throw an AttributeError when trying to access the name or description attributes of a tool.
Profiling:
frame.f_locals while trying to retrieve class name of a PyFrameObject.Tracing:
IndexError in partial flush when the finished span counter was out of sync with actual finished spans.DD_TRACE_PARTIAL_FLUSH_MIN_SPANS values less than 1 now default to 1 with a warning.Span.finished setter is deprecated, use Span.finish() method instead.
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
Span.finished setter is deprecated, use Span.finish() method instead.Span.finish_with_ancestors() is deprecated with no alternative.ExperimentResult class' rows and summary_evaluations attributes are deprecated and will be removed in the next major release. ExperimentResult.rows/summary_evaluations attributes will only store the results of the first run iteration for multi-run experiments. Use the ExperimentResult.runs attribute instead to access experiment results and summary evaluations.runs argument, to assess the true performance of an experiment in the face of the non determinism of LLMs. Use the new ExperimentResult class' runs attribute to access the results and summary evaluations by run iteration.Lock, RLock, Event).ResourceWarning in multiprocess scenarios.wrapt library dependency from the Lock Profiler implementation, improving performance and reducing overhead during lock instrumentation.Span.finished setter is deprecated, use Span.finish() method instead.
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
Span.finished setter is deprecated, use Span.finish() method instead.Span.finish_with_ancestors() is deprecated with no alternative.ExperimentResult class' rows and summary_evaluations attributes are deprecated and will be removed in the next major release. ExperimentResult.rows/summary_evaluations attributes will only store the results of the first run iteration for multi-run experiments. Use the ExperimentResult.runs attribute instead to access experiment results and summary evaluations.runs argument, to assess the true performance of an experiment in the face of the non determinism of LLMs. Use the new ExperimentResult class' runs attribute to access the results and summary evaluations by run iteration.Lock, RLock, Event).ResourceWarning in multiprocess scenarios.wrapt library dependency from the Lock Profiler implementation, improving performance and reducing overhead during lock instrumentation.Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
CI Visibility: this fix resolves an issue where repo tags would be fetched while unshallowing to extract commit metadata, causing performance issues for repos with a large number of tags.
LLM Observability: fix support for HTTPS_PROXY.
Error Tracking: Modifies the way exception events are stored such that the exception id is stored instead of the exception object, to prevent TypeErrors with custom exception objects.
Span.set_struct_tag is deprecated and will be removed in v4.0.0 with no direct replacement.
Estimated end-of-life date, accurate to within three months: 08-2026 See the support level definitions for more information.
tiktoken library, and insteadstream_options={"include_usage": True} in the OpenAI request.Span.set_struct_tag is deprecated and will be removed in v4.0.0 with no direct replacement.Span.get_struct_tag is deprecated and will be removed in v4.0.0 with no direct replacement.Span.set_tag_str is deprecated and will be removed in version 4.0.0.
As an alternative to Span.set_tag_str, you can use Span.set_tag instead.DD_PROFILING_STACK_V2_ENABLED=false will no longer have an effect starting in 4.0.urllib3 and requests. It does not require enabling APM instrumentation for urllib3 anymore.threading.RLock (reentrant lock) profiling. The Lock profiler now tracks both threading.Lock and threading.RLock usage, providing comprehensive lock contention visibility for Python applications.version argument to LLMObs.pull_datasetversion and latest_version to provide information on the version of the dataset that is being worked with and the latest global version of the dataset, respectivelyversion field. The version field is now omitted unless explicitly set by the user.IndexError.assessment argument in submit_evaluation().
assessment now refers to whether the evaluation itself passes or fails according to your application, rather than the validity of the evaluation result.langchain integration would incorrectly mark Azure OpenAI calls as duplicate llm operations even if the openai integration was enabled.
The langchain integration will trace Azure OpenAI spans as workflow spans if there is an equivalent llm span from the openai integration.async for model in client.models.list()) caused a TypeError: 'async for' requires an object with __aiter__ method, got coroutine. See issue #14574.KeyError exceptions in test runs when gevent is detected within the environment.ray.init().ray.data._internal to the module denylist._acquire method of the Lock profiler (note: this only occurs when assertions are enabled.)DD_PROFILING_API_TIMEOUT doesn't have any effect, and is marked to be removed in upcoming 4.0 release. New environment variable DD_PROFILING_API_TIMEOUT_MS is introduced to configure timeout for uploading profiles to the backend. The default value is 10000 ms (10 seconds)ValueError: Formatting field not found in record: 'dd.service'.One column per quarter.