ddtrace

Upgrade Notes

• <div id="remove-unsupported-pylons">

  This removes the pylons integration, which does not support Python 3.

  </div>

Deprecation Notes

• aioredis: The aioredis integration is deprecated and will be removed in a future version. As an alternative to the aioredis integration, you can use the redis integration with redis>=4.2.0.

New Features

• ASM: dependency telemetry metrics now will only report dependencies actually in use (imported) and will also report new imported modules periodically.

• ASM: This introduces Threat Monitoring and Blocking on FastAPI.

  • IP Blocking and all input addresses are supported on requests and responses

  - Custom Blocking This does not contain user blocking specific features yet.

• tracing: Introduces support for OpenTracing Baggage Items with HTTP Propagation. Enable this support by DD_TRACE_PROPAGATION_HTTP_BAGGAGE_ENABLED=true. The Context._set_baggage_item and Context._get_baggage_item internal methods are provided for manual modifications to the Baggage Items. These API changes are subject to change.

• dynamic instrumentation: Add support for more built-in container types, such as defaultdict, frozenset, OrderedDict and Counter.

• Vulnerability Management for Code-level (IAST): Adds Python 3.12 compatibility

• Optionally lazy loads and disables Instrumentation Telemetry. Users must explicitly set "DD_INSTRUMENTATION_TELEMETRY_ENABLED=false".

• tracer: Add support for remotely setting the trace sample rate from the Datadog UI. This functionality is enabled by default when using ddtrace-run and library injection. To enable it when using the library manually, use ddtrace.config.enable_remote_config().

Bug Fixes

• tracer: tag spans that have been sampled due to an Agent sampling configuration.
• lambda: This change disables the use of multiprocessing.queue in Lambda, because it is not supported in Lambda
• langchain: This fix resolves a crash that could occur during embedding when no embeddings are found.
• Fix a regression with the support for gevent that could have occurred if some products, like ASM, telemetry, were enabled.
• kafka: Resolves TypeError raised by serializing producers and deserializing consumers when the message.key tag is set on spans.
• dynamic instrumentation: Fix an issue that caused the instrumented application to fail to start if a non-standard module was imported.
• openai: This fix resolves an issue where tagging image inputs in the chat completions endpoint resulted in attribute errors.
• openai: This fix resolves an issue where requesting raw API responses from openai>=1.0 resulted in attribute errors while tagging.
• profiling: Fix an issue that prevented threading locks from being traced when using gevent.
• profiling: Fix a segmentation fault with CPython 3.12 when sampling thread stacks.
• pylibmc: Fixes an issue where using ddtrace-run or ddtrace.patch_all() with DD_TRACE_ENABLED=False would break with get, gets, and get_multi operations on pylibmc Clients.
• tracing: This fix resolves an issue where concurrent mutations to the context._meta dict caused <span class="title-ref">RuntimeError: dictionary changed size during iteration</span>.
• django: Resolves AttributeError raised by traced StreamingHttpResponse.
• Vulnerability Management for Code-level (IAST): This fix resolves an issue where certain aspects incorrectly expected at least one argument, leading to an IndexError when none were provided. The solution removes this constraint and incorporates regression tests for stability assurance.
• Vulnerability Management for Code-level (IAST): Cookies vulnerabilities are only reported if response cookies are insecure.
• Vulnerability Management for Code-level (IAST): Fix propagation error on .format string method.
• requests: Updates the resource names of requests.requests spans to include the method and path of the request.
• propagation: This fix resolves an issue where a Context generated from extracted headers could lack a span_id or trace_id, leading SpanLink encoding errors.
• psycopg: This fix resolves an issue where a circular import of the psycopg library could cause a crash during monkeypatching.
• psycopg: This fix resolves an issue where exceptions originating from asynchronous Psycopg cursors were not propagated up the call stack.
• redis: This fix resolves an issue where the yaaredis and aredis integrations imported code from the redis integration, causing a circular import error.
• tracing: Resolves trace encoding errors raised when DD_TRACE_API_VERSION is set to v0.5 and a BufferFull Exception is raised by the TraceWriter. This fix ensures span fields are not overwritten and reduces the frequency of 4XX errors in the trace agent.

Other Changes

• tracing: Upgrades the trace encoding format to v0.5. This change improves the performance of encoding and sending spans.

---

New Features

• propagation: 128-bit trace ids are now used by default for propagation. Previously the default was 64-bit. This change is backwards compatible with tracers that still use 64-bit trace ids and should not cause any breaking behavior.
• Adds DSM pathway.hash tag to spans when DSM is enabled. This allows traces from the instrumented service to show up in the DSM traces tab.
• propagation: When the tracer is configured to extract and inject tracecontext, the tracer will propagate the tracestate values from other vendors so long as the traceparent trace-id matches the first found trace context, regardless of propagator configuration order. To disable this behavior DD_TRACE_PROPAGATION_EXTRACT_FIRST=true can be set.
• opentelemetry: Map reserved OpenTelemetry attributes to Datadog span model.
• opentelemetry: datadog operation name from semantic conventions
• propagation: If a valid context is extracted from headers, and the following extracted trace context's trace_ids do not match the valid context's, then add a span link to the root span to represent the broken propagation.
• tracing: This change treats spans that terminated with sys.exit(0) as successful non-error spans.
• tracing: This introduces the DD_TRACE_SPAN_TRACEBACK_MAX_SIZE environment variable, allowing the maximum size of tracebacks included on spans to be configured.

Bug Fixes

• CI Visibility: fixes the fact that the GITHUB_SERVER_URL environment variable was not being sanitized for credentials
• dynamic instrumentation: Needs to update the pubsub instance when the application forks because the probe mechanism should run in the child process. For that, DI needs the callback as the method of an instance of Debugger, which lives in the child process.
• CI Visibility: Fixes an issue where a ValueError was raised when using different path drives on Windows
• Fixes an issue where ddtrace could not be installed from source when using setuptools>=69 due to a change in the license field.
• tracing: Fixes an issue where the thread responsible for sending traces is killed due to concurrent dictionary modification.
• structlog: Fixes TypeError raised when ddtrace log processor is configured with a tuple
• Vulnerability Management for Code-level (IAST): Generates cookies vulnerabilities report if IAST is enabled. Before this fix, Cookies vulnerabilities were only generated if both IAST and Appsec were enabled.
• Vulnerability Management for Code-level (IAST): This fix resolves an issue where, at AST patching to replace code with IAST aspects, passing the original function/method as an extra parameter for accurate patching unintentionally triggers side effects in methods obtained from an expression (like decode in file.read(n).decode()), resulting in unexpected multiple calls to the expression (file.read(n) in the example).
• Vulnerability Management for Code-level (IAST): This fix eliminates some reference leaks and C-API usage when IAST reports a vulnerability and calls get_info_frame.
• kafka: This fix resolves an issue where calls to confluent_kafka's produce method with key=None would cause an exception to be raised.
• tracing: This fix resolves an issue where ddtrace's signal handlers could cause Flask apps not to respond correctly to SIGINT.
• logging: A log handler is automatically added to the ddtrace logger upon ddtrace import, when not using ddtrace-run. This can lead to duplicate logging if users add additional loggers and do not explicitly modify the ddtrace logger. This fix adds a feature flag that can be used to toggle this behavior off DD_TRACE_LOG_STREAM_HANDLER which defaults to true.

---

Upgrade Notes

• The wrapt and psutil packages are vendored to help users avoid building these packages if wheels were not available for a given platform. This reverses a change released in v2.0.0.

New Features

• CI Visibility: adds ITR support for unittest
• CI Visibility: adds start/end line support for pytest test spans
• CI Visibility: adds start/end line source file data to unittest test spans
• aiohttp: This introduces basic tracing of streaming responses that stay open long after the <span class="title-ref">on_prepare</span> signal has been sent.
• CI Visibility: introduce pytest hooks for modifying the module, suite, and test naming logic
• CI Visibility: add support for AWS Codepipeline to CI env var gathering
• datastreams: this change adds message payload size metrics and aggregations for Kafka.
• structlog: Wraps get_logger function in order to add datadog injection processor regardless of configuration
• openai: This adds support for openai v1.
• Source Code: filters Git repo URLs from env vars and setuptools
• logbook: This introduces log correlation for the logbook library. Refer to logbook-docs <ddtrace.contrib.logbook> for more details.
• loguru: This introduces log correlation for the loguru library. Refer to loguru-docs <ddtrace.contrib.loguru> for more details.
• openai: This adds support for tagging function call arguments when using OpenAI's function calling feature.
• Adds ARM64 support for Single-Step instrumentation
• structlog: This introduces log correlation for the structlog library. Refer to structlog-docs <ddtrace.contrib.structlog> for more details.
• celery: Adds Python 3.11 and 3.12 support for the celery integration.

Known Issues

• ASM: fix a body read problem on some corner case where passing empty content length makes wsgi.input.read() blocks.

Bug Fixes

• Application Security Management (ASM): fix a body read error when Transfer-Encoding: chunked header is sent
• CI Visibility: fixes an issue where class-based test methods with the same name across classes would be considered duplicates, and cause one (or more) tests to be dropped from results, by adding --ddtrace-include-class-name as an optional flag (defaulting to false) to prepend the class name to the test name.
• CI Visibility: fixes a crash where the unittest integration would try to enable coverage when tests are run even if the Intelligent Test Runner is not enabled.
• data_streams: This fix resolves an issue where tracing would crash if a kafka client produced a message with no key or value.
• CI: fixes an issue which prevented the library from filtering user credentials for SSH Git repository URLs
• dynamic instrumentation: fix an issue that caused function probes on the same module to fail to instrument and be reported in the ERROR status in the UI if the module was not yet imported.
• Use a unique default service name across all the products provided by the library when one is not given via the configuration interface.
• sampling: This fix reverts a refactor which affected how the tracer handled the trace-agent's recommended trace sampling rates, leading to an unintended increase in traces sampled.
• tracing: Fixes a msgpack import error when DD_TRACE_API is set to v0.5
• fix(profiling): numeric type exception in memalloc When pushing allocation samples, an exception was being thrown due to a float being passed instead of an integer. We now cast the ceiled value to an integer.
• CI Visibility: fixes unittest data not being initialized properly
• CI Visibility: fixes an issue where just importing <span class="title-ref">unittest</span> enabled CIVisibility and potentially caused unexpected logs and API requests
• Vulnerability Management for Code-level (IAST): This fix addresses AST patching issues where custom functions or methods could be replaced by aspects with differing argument numbers, causing runtime errors as a result. Furthermore, it addresses a case during patching where the module is inadvertently passed as the first argument to the aspect.
• Vulnerability Management for Code-level (IAST): Fix potential string id collisions that could cause false positives with non tainted objects being marked as tainted.
• IAST: This fix resolves an issue where JSON encoder would throw an exception while encoding a tainted dict or list.
• Vulnerability Management for Code-level (IAST): This fix resolves an issue where SimpleJSON encoder would throw an exception while encoding a tainted dict or list.
• ASM: add support for psycopg2 adapt mechanism to LazyTaintList, preventing a ProgrammingError when using psycopg2 with IAST.
• tracing: This fix resolves an issue where unserializable tracer attributes caused crashes when DD_TRACE_DEBUG was set.
• This fix resolves an issue where confluent_kafka's SerializingProducer and DeserializingConsumer classes were incorrectly patched, causing crashes when these classes are in use with Datadog patching.
• langchain: This fix resolves an issue with tagging pydantic <span class="title-ref">SecretStr</span> type api keys.
• lib injection: Fix permissions error raised when non-root users copy single step instrumentation files.
• redis: The Datadog Agent removes command arguments from the resource name. However there are cases, like compressed keys, where this obfuscation cannot correctly remove command arguments. To safeguard that situation, the resource name set by the tracer will only be the command (e.g. SET) with no arguments. To retain the previous behavior and keep arguments in the span resource, with the potential risk of some command arguments not being fully obfuscated, set DD_REDIS_RESOURCE_ONLY_COMMAND=false.

Other Changes

• tags: Previously DD_TRACE_X_DATADOG_TAGS_MAX_LENGTH had a max limit setting of 512 characters. This change removes that limit but keeps the default at 512.

---

Prelude

The Datadog APM Python team is happy to announce the release of v2.0.0 of ddtrace. This release drops support for Python 2.7, 3.5, and 3.6. This release adds support for Python 3.12.

<div class="important">

<div class="title">

Important

</div>

If you are on version of Python not supported by v2, we will continue to maintain the ddtrace v1 with bug fixes.

</div>

<div class="note">

<div class="title">

Note

</div>

Before upgrading to v2.0.0, we recommend users install ddtrace~=1.20.0 and enable deprecation warnings. All removals to the library interface and environment variables in v2 were deprecated in the 1.x release line.

</div>

<div class="note">

<div class="title">

Note

</div>

The changes to environment variables apply only to the configuration of the ddtrace library and not the Datadog Agent.

</div>

Upgrading summary

Functionality changes

The default logging configuration functionality of ddtrace has been changed to avoid conflicting with application logging configurations. DD_CALL_BASIC_CONFIG has been removed and the ddtrace logger will log to stdout by default, or a log file as specified using DD_TRACE_LOG_FILE.

Setting the environment variable DD_TRACE_PROPAGATION_STYLE='b3', which previously enabled b3multi now enables b3 single header. b3 single header still works but is deprecated for b3. Simplified: b3 used to enable b3multi, but now enables b3 single header to better align with Opentelemetry's terms.

Removed deprecated environment variables

These environment variables have been removed. In all cases the same functionality is provided by other environment variables and replacements are provided as recommended actions for upgrading.

Variable	Replacement	Note
DD_GEVENT_PATCH_ALL	None	📝<remove-dd-gevent-patch-all>
DD_AWS_TAG_ALL_PARAMS	None	📝<remove-aws-tag-all-params>
DD_REMOTECONFIG_POLL_SECONDS	DD_REMOTE_CONFIG_POLL_INTERVAL_SECONDS	📝<rename-remote-config-poll-seconds>
DD_CALL_BASIC_CONFIG	None	📝<remove-basic-config>
DD_TRACE_OBFUSCATION_QUERY_STRING_PATERN	DD_TRACE_OBFUSCATION_QUERY_STRING_REGEXP	📝<remove-trace-obfuscation-query-string-pattern>

Removed deprecated library interfaces

These methods and module attributes have been removed. Where the same functionality is provided by a different public method or module attribute, a recommended action is provided for upgrading. In a few limited cases, because the interface was no longer used or had been moved to the internal interface, it was removed and so no action is provided for upgrading.

Module	Method/Attribute	Note
ddtrace.constants	APPSEC_ENABLED	📝<remove-appsec-private-constants>
	APPSEC_JSON	📝<remove-appsec-private-constants>
	APPSEC_EVENT_RULE_VERSION	📝<remove-appsec-private-constants>
	APPSEC_EVENT_RULE_ERRORS	📝<remove-appsec-private-constants>
	APPSEC_EVENT_RULE_LOADED	📝<remove-appsec-private-constants>
	APPSEC_EVENT_RULE_ERROR_COUNT	📝<remove-appsec-private-constants>
	APPSEC_WAF_DURATION	📝<remove-appsec-private-constants>
	APPSEC_WAF_DURATION_EXT	📝<remove-appsec-private-constants>
	APPSEC_WAF_TIMEOUTS	📝<remove-appsec-private-constants>
	APPSEC_WAF_VERSION	📝<remove-appsec-private-constants>
	APPSEC_ORIGIN_VALUE	📝<remove-appsec-private-constants>
	APPSEC_BLOCKED	📝<remove-appsec-private-constants>
	IAST_JSON	📝<remove-appsec-private-constants>
	IAST_ENABLED	📝<remove-appsec-private-constants>
	IAST_CONTEXT_KEY	📝<remove-appsec-private-constants>
ddtrace.contrib.fastapi.patch	span_modifier	📝<remove-fastapi-starlette-span-modifier>
	aggregate_resources	📝<remove-fastapi-starlette-aggregate-resources>
ddtrace.contrib.starlette.patch	span_modifier	📝<remove-fastapi-starlette-span-modifier>
	aggregate_resources	📝<remove-fastapi-starlette-aggregate-resources>
	get_resource	📝<remove-fastapi-starlette-span-modifier>
ddtrace.contrib.grpc.constants	GRPC_PORT_KEY	📝<remove-grpc-port-key>
ddtrace.ext.cassandra	ROW_COUNT	📝<remove-cassandra-row-count>
ddtrace.ext.mongo	ROWS	📝<remove-mongo-row-count>
ddtrace.ext.sql	ROWS	📝<remove-sql-row-count>
ddtrace.filters	TraceCiVisibilityFilter	📝<remove-trace-ci-visibility-filter>
ddtrace.tracer	DD_LOG_FORMAT	📝<remove-dd-log-format>

Upgrade Notes

• <div id="remove-dd-gevent-patch-all">

  DD_GEVENT_PATCH_ALL is removed. There is no special configuration necessary to make ddtrace work with gevent if using ddtrace-run.

  </div>

• <div id="remove-aws-tag-all-params">

  DD_AWS_TAG_ALL_PARAMS is removed. The boto/botocore/aiobotocore integrations no longer collect all API parameters by default.

  </div>

• <div id="rename-remote-config-poll-seconds">

  DD_REMOTECONFIG_POLL_SECONDS is removed. Use the environment variable DD_REMOTE_CONFIG_POLL_INTERVAL_SECONDS instead.

  </div>

• <div id="remove-appsec-private-constants">

  APPSEC_ENABLED, APPSEC_JSON, APPSEC_EVENT_RULE_VERSION, APPSEC_EVENT_RULE_ERRORS, APPSEC_EVENT_RULE_LOADED, APPSEC_EVENT_RULE_ERROR_COUNT, APPSEC_WAF_DURATION, APPSEC_WAF_DURATION_EXT, APPSEC_WAF_TIMEOUTS, APPSEC_WAF_VERSION, APPSEC_ORIGIN_VALUE, APPSEC_BLOCKED, IAST_JSON, IAST_ENABLED, IAST_CONTEXT_KEY are removed. This should not affect existing code as these deprecated ASM constants were meant for private use only.

  </div>

• <div id="remove-fastapi-starlette-span-modifier">

  ddtrace.contrib.starlette.get_resource, ddtrace.contrib.starlette.span_modifier, and ddtrace.contrib.fastapi.span_modifier are removed. The starlette and fastapi integrations now provide the full route and not just the mounted route for sub-applications.

  </div>

• <div id="remove-fastapi-starlette-aggregate-resources">

  ddtrace.contrib.starlette.config['aggregate_resources'] and ddtrace.contrib.fastapi.config['aggregate_resources'] are removed. The starlette and fastapi integrations no longer have the option to aggregate_resources, as it now occurs by default.

  </div>

• <div id="remove-grpc-port-key">

  ddtrace.contrib.grpc.constants.GRPC_PORT_KEY is removed. Use ddtrace.ext.net.TARGET_PORT instead.

  </div>

• <div id="remove-cassandra-row-count">

  ddtrace.ext.cassandra.ROW_COUNT is removed. Use ddtrace.ext.db.ROWCOUNT instead.

  </div>

• <div id="remove-mongo-row-count">

  ddtrace.ext.mongo.ROW_COUNT is removed. Use ddtrace.ext.db.ROWCOUNT instead.

  </div>

• <div id="remove-sql-row-count">

  ddtrace.ext.sql.ROW_COUNT is removed. Use ddtrace.ext.db.ROWCOUNT instead.

  </div>

• <div id="remove-trace-ci-visibility-filter">

  ddtrace.filters.TraceCiVisibilityFilter is removed.

  </div>

• <div id="remove-dd-log-format">

  ddtrace.tracer.DD_LOG_FORMAT is removed. As an alternative, please follow the log injection formatting as provided in the log injection docs.

  </div>

• <div id="remove-basic-config">

  DD_CALL_BASIC_CONFIG is removed. There is no special configuration necessary to replace DD_CALL_BASIC_CONFIG. The ddtrace logger will log to stdout by default or additionally to a file specified by DD_TRACE_LOG_FILE.

  </div>

• <div id="remove-trace-obfuscation-query-string-pattern">

  DD_TRACE_OBFUSCATION_QUERY_STRING_PATTERN is removed. Use DD_TRACE_OBFUSCATION_QUERY_STRING_REGEXP instead.

  </div>

New Features

• Adds support for Python 3.12.

Known Issues

• aiohttp: Python 3.12 is not supported.
• aiohttp-jinja: Python 3.12 is not supported.
• aiobotocore: Python 3.12 is not supported.
• asm: IAST for Python 3.12 is not supported.
• flask-caching: Python 3.12 is not supported.
• openai/langchain: Python 3.12 is not supported.
• opentelemetry-api: Python 3.12 is not supported.
• opentracing: Python 3.12 is not supported.
• pyramid: Python 3.12 is not supported.
• pynamodb: Python 3.12 is not supported.
• redis/redis-py-cluster: Python 3.12 is not supported.

---