ddtrace

New Features

• tracing: Adds support for 128 bit trace ids for b3 and w3c distributing tracing headers.
• pytest: Adds the DD_CIVISIBILITY_AGENTLESS_ENABLED environment variable to configure the CIVisibility service to use an agent-less test reporting CIVisibilityWriter. Note that the CIVisibility service will use regular agent reporting by default.
• sci: Extracts and sends git metadata from environment variables DD_GIT_REPOSITORY_URL, DD_GIT_COMMIT_SHA, or from the python package specified in the DD_MAIN_PACKAGE. This feature can be disabled by setting DD_TRACE_GIT_METADATA_ENABLED=False.
• otel: Adds support for the OpenTelemetry Tracing API. Please refer to the docs <ddtrace.opentelemetry> for more details.

Bug Fixes

• tracing: Ensure datadog headers propagate 128 bit trace ids when DD_TRACE_128_BIT_TRACEID_GENERATION_ENABLED=False
• aws_lambda: Fix AttributeError raised when ddtrace.patch_all(), or ddtrace.patch(aws_lambda=True), is set on user handler.
• aws_lambda: Fix AttributeError raised when extracting context from arguments.
• aws_lambda: Fix AttributeError raised when callable handlers are traced.
• dynamic instrumentation: Fixed an issue with expressions in metric probes that prevented them from being evaluated.
• Prevent exceptions when autoreloading modules that directly or indirectly import ddtrace with the iPython autoreload extension.
• profiling: Corrects accounting of wall and CPU time for gevent tasks within the main Python thread.
• profiling: Fixed an issue with the memory collector where a segmentation fault could occur during shutdown.
• lib-injection: The ddtrace package is now provided via the Docker image rather than relying on a run-time pip install. This solves issues like containers blocking network requests, installation overhead during application startup, permissions issues with the install.

---

Deprecation Notes

• ASM: Several deprecated ASM constants that were added to the public API will be removed. This should not affect existing code as they were meant for private use only.

New Features

• tracing: Adds support for 128 bit trace ids. To generate and propagate 128 bit trace ids using Datadog distributed tracing headers set the following configuration: DD_TRACE_128_BIT_TRACEID_GENERATION_ENABLED=True. Support for B3 and W3C distributed tracing headers will be added in a future change.
• aiohttp: Add missing component meta tag to aiohttp server spans.
• redis: Adds tracing support for <span class="title-ref">redis.cluster.RedisCluster</span>.
• celery: Adds automatic tracing of the celery.beat scheduling service to the celery integration.
• kafka: Adds instrumentation support for confluent-kafka>=1.7. See the confluent-kafka<https://ddtrace.readthedocs.io/en/stable/integrations.html#kafka> documentation for more information.
• dynamic instrumentation: introduced support for dynamic span probes.
• Adds source code integration with setuptools build metadata. This enables traces and profiles to be automatically tagged with git metadata to track deployments in Datadog.

Bug Fixes

• tracing: This fix resolves an issue where making a sampling decision before the env span tag had been set caused sample rate data from the Datadog Agent to be ignored.
• ASM: make track_custom_event() also set appsec.events.<custom_event>.track which was missing.
• django: Fixes an issue where http.route was only set if use_handler_resource_format and use_legacy_resource_format were set to False.
• tracing: This fix resolves an issue where a very long string as a span attribute would cause that span not to be delivered. It replaces string span attributes larger than DD_TRACE_WRITER_BUFFER_SIZE_BYTES (which as of this version defaults to 8388608) with a small string containing debug information and not containing any of the original attribute string.
• ASM: Resolves installation issues with compiling native code on Windows and unknown platforms.
• aws_lambda: Fixes a RecursionError which is raised when aws lambda signal handlers are wrapped infinitely. This caused lambdas to crash on startup.
• botocore: Fix TypeError raised by injecting trace context into Kinesis messages.
• dynamic instrumentation: Fix a bug where the dynamic instrumentation would stop injecting function probes after the first failed one.
• dynamic instrumentation: This change fixes a bug whereby probes that have been disabled/removed from the front-end would not be removed by the client library.
• futures: Resolves an issue that prevents tasks from being submitted to a thread pool executor when gevent is used (e.g. as a worker class for gunicorn or celery).
• propagation: This fix resolves an issue where previously W3C tracestate propagation could not handle whitespace. With this fix whitespace is now removed for incoming and outgoing requests.
• httplib: Fixes an issue with patching of http client upon import
• Ensure DD_REMOTE_CONFIGURATION_ENABLED environment variable disables remote config if set to False

Other Changes

• aws_lambda: Updates how <span class="title-ref">DD_APM_FLUSH_DEADLINE_MILLISECONDS</span> is used. Previously, we would set the deadline as the environment variable value, if set. Now, when the remaining time in an AWS Lambda invocation is less than <span class="title-ref">DD_APM_FLUSH_DEADLINE_MILLISECONDS</span>, the tracer will attempt to submit the current active spans and all finished spans. the value in the environment variable is used to subtract from the deadline. The default is still 100ms.

---

Prelude

Application Security Management (ASM) has added Django support for blocking malicious users using one click within Datadog.

<div class="note">

<div class="title">

Note

</div>

One click blocking for ASM is currently in beta.

</div>

Deprecation Notes

• dbapi: ddtrace.ext.mongo.ROWS is deprecated. Use ddtrace.ext.db.ROWCOUNT instead.

New Features

• starlette: Add http.route tag to starlette.request spans.
• fastapi: Add http.route tag to fastapi.request spans.
• ASM: Add support for one click blocking of user ids with the Django framework using Remote Configuration Management.
• ASM: This introduces the "suspicious request blocking" feature for Django and Flask.

Known Issues

• ASM: There is a known issue with the flask support for any rule blocking on server.request.path_params. The request will be correctly blocked but the client application will be receiving and processing the suspicious request. Possible workaround: use server.request.uri.raw instead, if you want the request to be blocked before entering the flask application.

Bug Fixes

• dbapi: The dbapi integration no longer assumes that a cursor object will have a rowcount as not all database drivers implement rowcount.

• dbm: Support sql queries with the type byte.

• elasticsearch: Omit large elasticsearch.body tag values that are greater than 25000 characters to prevent traces from being too large to send.

• aws_lambda: This fix resolves an issue where existing signals were wrapped multiple times.

• profiling: Handles a race condition on process shutdown that would cause an error about a module not being started to occasionally appear in the logs.

• Fix for KeyError exceptions when when <span class="title-ref">ASM_FEATURES</span> (1-click activation) disabled all ASM products. This could cause 1-click activation to work incorrectly in some cases.

• ASM: Solve some corner cases where a Flask blocking request would fail because headers would be already sent.

• ASM: Solve the content-type not always being correct in blocking responses.

• ASM: Ensure the blocking responses have the following tags: <span class="title-ref">http.url</span>, <span class="title-ref">http.query_string</span>, <span class="title-ref">http.useragent</span>, <span class="title-ref">http.method</span>, <span class="title-ref">http.response.headers.content-type</span> and <span class="title-ref">http.response.headers.content-length</span>.

• ASM: fix memory leaks and memory corruption in the interface between ASM and the WAF library

• psycopg2: Fixes a bug with DSN parsing integration.

Other Changes

• remote_config: Change the level of remote config startup logs to debug.

---

Deprecation Notes

• gevent: DD_GEVENT_PATCH_ALL is deprecated and will be removed in the next major version. Gevent compatibility is now automatic and does not require extra configuration when running with ddtrace-run. If not using ddtrace-run, please import ddtrace.auto before calling gevent.monkey.patch_all().

Bug Fixes

• aws_lambda: Resolves an exception not being handled, which occurs when no root span is found before a lambda times out.
• gevent: This fix resolves an incompatibility between ddtrace and gevent that caused threads to hang in certain configurations, for example the profiler running in a gunicorn application's gevent worker process.

Other Changes

• ASM: The list of headers for retrieving the IP when Application Security Management is enabled or the <span class="title-ref">DD_TRACE_CLIENT_IP_ENABLED</span> environment variable is set has been updated. "Via" has been removed as it rarely contains IP data and some common vendor headers have been added. You can also set the environment variable <span class="title-ref">DD_TRACE_CLIENT_IP_HEADER</span> to always retrieve the IP from the header specified as the value.

---

Prelude

Application Security Management (ASM) has added Django support for blocking malicious IPs using one click within Datadog.

<div class="note">

<div class="title">

Note

</div>

One click blocking for ASM is currently in beta.

</div>

Application Security Management (ASM) has added Flask support for blocking malicious IPs using one click within Datadog.

<div class="note">

<div class="title">

Note

</div>

One click blocking for ASM is currently in beta.

</div>

Deprecation Notes

• grpc: Deprecates ddtrace.contrib.grpc.constants.GRPC_PORT_KEY. Use ddtrace.ext.net.TARGET_PORT instead.
• dbapi: ddtrace.ext.sql.ROWS is deprecated. Use ddtrace.ext.db.ROWCOUNT instead.
• cassandra: ddtrace.ext.cassandra.ROW_COUNT is deprecated. Use ddtrace.ext.db.ROWCOUNT instead.

New Features

• Enable traces to be sent before an impending timeout for datadog_lambda>=4.66.0. Use DD_APM_FLUSH_DEADLINE to override the default flush deadline. The default is the AWS Lambda function configured timeout limit.

• debugger: Add dynamic log probes to that generate a log message and optionally capture local variables, return value and exceptions

• tracing: Add support for enabling collecting of HTTP request client IP addresses as the http.client_ip span tag. You can set the DD_TRACE_CLIENT_IP_ENABLED environment variable to true to enable. This feature is disabled by default.

• ASM: add support for one click blocking of IPs with the Django framework using Remote Configuration Management.

• ASM: add support for one click blocking of IPs with the Flask framework using Remote Configuration Management.

• ASM: also fetch loopback IPs if client IP fetching is enabled (either via ASM or DD_TRACE_CLIENT_IP_ENABLED).

• ASM: Enable ability to remotely activate and configure ASM features. To enable, check the Python Security page in your account. Note that this is a beta feature.

• profiling: Collects endpoint invocation counts.

• dynamic instrumentation: Python 3.11 is now supported.

• graphene: Adds support for Python 3.11.

• graphql: Adds support for Python 3.11.

• httpx: Add support for httpx<0.14.0,>=0.9.0.

• tracer/span: Add Span.finish_with_ancestors method to enable the abrupt finishing of a trace in cases where the trace or application must be immediately terminated.

Known Issues

• remote config: There is a known issue with remote configuration management (RCM) when paired with gevent which can cause child processes to deadlock. If you are experiencing issues, we recommend disabling RCM with DD_REMOTE_CONFIGURATION_ENABLED=false. Note, this will disable one click activation for ASM.
• gunicorn: ddtrace-run does not work with gunicorn. To instrument a gunicorn application, follow the instructions here.

Bug Fixes

• fastapi: Previously, custom fastapi middlewares configured after application startup were not traced. This fix ensures that all fastapi middlewares are captured in the <span class="title-ref">fastapi.request</span> span.

• tracing: Pads trace_id and span_ids in b3 headers to have a minimum length of 16.

• Fix full stacktrace being sent to the log on remote config connection errors.

• httpx: Only patch httpx.AsyncClient for httpx>=0.11.0.

• tracing: This fix resolves an issue with the encoding of traces when using the v0.5 API version with the Python optimization option flag -O or the PYTHONOPTIMIZE environment variable.

• pylons: This fix resolves an issue where str.decode could cause critical unicode decode errors when ASM is enabled. ASM is disabled by default.

• gevent: This fix resolves incompatibility under 3.8>=Python<=3.10 between ddtrace-run and applications that depend on gevent, for example gunicorn servers. It accomplishes this by keeping copies that have not been monkey patched by gevent of most modules used by ddtrace. This "module cloning" logic can be controlled by the environment variable DD_UNLOAD_MODULES_FROM_SITECUSTOMIZE. Valid values for this variable are "1", "0", and "auto". "1" tells ddtrace to run its module cloning logic unconditionally, "0" tells it never to run that logic, and "auto" tells it to run module cloning logic only if gevent is accessible from the application's runtime. The default value is "0".

• lib-injection: Use package versions published to PyPI to install the library. Formerly the published image was installing the package from source using the tagged commit SHA which resulted in slow and potentially failing installs.

• profiler: Handles potential AttributeErrors which would arise while collecting frames during stack unwinding in Python 3.11.

• remote config: ensure proper validation of responses from the agent.

---

Upgrade Notes

• ASM: libddwaf upgraded to version 1.6.1 using a new library loading mechanism
• profiling: upgrades the profiler to support the v2.4 backend API for profile uploads, using a new request format.

Deprecation Notes

• DD_REMOTECONFIG_POLL_SECONDS environment variable is deprecated and will be removed in v2.0. Please use DD_REMOTE_CONFIG_POLL_INTERVAL_SECONDS instead.

New Features

• CI Visibility: Add support for CI provider buddy.works

• The component tag has been added for all auto-instrumented spans. The value of the component tag is equal to the name of the integration that produced the span.

• tracing: Adds support for IPv6 agent hostnames for <span class="title-ref">DD_AGENT_HOST</span>.

• elasticsearch: Update elasticsearch integration to add support for opensearch-py. See the elasticsearch documentation for more information.

• ASM: one click activation enabled by default using Remote Configuration Management (RCM). Set DD_REMOTE_CONFIGURATION_ENABLED=false to disable this feature.

• ASM: New Application Security Events Tracking API, starting with the functions track_user_login_success_event and track_user_login_failure_event for tracking user logins (it will also internally call set_user) and track_custom_event for any custom events. You can find these functions in the ddtrace.appsec.trace_utils module. Calling these functions will create new tags under the appsec.events namespace (appsec.events.user.login for logins) allowing you to track these events with Datadog. In the future this will be used to provide protection against account takeover attacks (ATO). Public documentation will be online soon.

• celery: Enhances context tags containing dictionaries so that their contents are sent as individual tags (issue #4771).

• tornado: Support custom error codes: https://ddtrace.readthedocs.io/en/stable/advanced_usage.html#custom-error-codes.

• CI Visibility: Support reliably linking tests to the pipeline that executed them.

Known Issues

• profiling: There is currently a known performance regression issue with the profiler's code provenance feature. Note that this feature is disabled by default and will only be enabled if DD_PROFILING_ENABLE_CODE_PROVENANCE is set to true.

Bug Fixes

• This fix improves a cryptic error message encountered during some pip install ddtrace runs under pip versions <18.
• dynamic instrumentation: remove unnecessary log line from application start up
• This fix removes unintended url parts in the http.url tag.
• botocore: Before this change, the botocore integration stripped newlines from the JSON string encoded in the data blob of Amazon Kinesis records. This change includes a terminating newline if it is present in the decoded data.
• profiling: This fix resolves an issue in Python 3.11 where a PyFrameObject strong reference count was not properly decremented in the stack collector.
• telemetry: This fix resolves an issue when we try to fetch platform.libc_ver() on an unsupported system.
• Fix for ValueError when @ is not present in network location but other part of the url.

Other Changes

• profiler: CPU overhead reduction.

---

Prelude

Initial library support has been added for Python 3.11.

<div class="note">

<div class="title">

Note

</div>

Continuous Profiler and Dynamic Instrumentation are not yet compatible and must be disabled in order to use the library with Python 3.11. Support for them will be added in a future release. To track the status, see the Support Python 3.11 issue on GitHub.

</div>

Upgrade Notes

• The default propagation style configuration changes to DD_TRACE_PROPAGATION_STYLE=tracecontext,datadog. To only support Datadog propagation and retain the existing default behavior, set DD_TRACE_PROPAGATION_STYLE=datadog.
• tracer: support for Datadog Agent v5 has been dropped. Datadog Agent v5 is no longer supported since ddtrace==1.0.0. See https://ddtrace.readthedocs.io/en/v1.0.0/versioning.html#release-support for the version support.
• Python 3.11: Continuous Profiler and Dynamic Instrumentation must be disabled as they do not current support Python 3.11.
• The configured styles in DD_TRACE_PROPAGATION_STYLE_EXTRACT are now evaluated in order to specification. To keep the previous fixed evaluation order, set: DD_TRACE_PROPAGATION_STYLE_EXTRACT=datadog,b3,b3 single header.
• tracing: upgrades the default trace API version to v0.5 for non-Windows systems. The v0.5 trace API version generates smaller payloads, thus increasing the throughput to the Datadog agent especially with larger traces.
• tracing: configuring the v0.5 trace API version on Windows machines will raise a RuntimeError due to known compatibility issues. Please see https://github.com/DataDog/dd-trace-py/issues/4829 for more details.

Deprecation Notes

• propagation: Configuration of propagation style with DD_TRACE_PROPAGATION_STYLE=b3 is deprecated and will be removed in version 2.0.0. Please use the newly added DD_TRACE_PROPAGATION_STYLE=b3multi instead.
• aws: The boto, botocore and aiobotocore integrations no longer include all API parameters by default. To retain the deprecated behavior, set the environment variable DD_AWS_TAG_ALL_PARAMS=1. The deprecated behavior and environment variable will be removed in v2.0.0.

New Features

• django: add configuration option to allow a resource format like <span class="title-ref">{method} {handler}.{url_name}</span> in projects with Django <2.2.0
• django: Adds the DD_DJANGO_INCLUDE_USER_NAME option to toggle whether the integration sets the django.user.name tag.
• Added environment variable DD_TRACE_PROPAGATION_STYLE to configure both injection and extraction propagation styles. The configured styles can be overridden with environment variables DD_TRACE_PROPAGATION_STYLE_INJECT and DD_TRACE_PROPAGATION_STYLE_EXTRACT.
• tracing: This introduces none as a supported propagator for trace context extraction and injection. When none is the only propagator listed, the corresponding trace context operation is disabled. If there are other propagators in the inject or extract list, the none propagator has no effect. For example DD_TRACE_PROPAGATION_STYLE=none
• ASM: now http.client_ip and network.client.ip will only be collected if ASM is enabled.
• tracing: Adds support for W3C Trace Context propagation style for distributed tracing. The traceparent and tracestate HTTP headers are enabled by default for all incoming and outgoing HTTP request headers. The Datadog propagation style continue to be enabled by default.
• flask: Adds support for streamed responses. Note that two additional spans: flask.application and flask.response will be generated.
• profiling: Adds support for Python 3.11.
• tracer: added support for Python 3.11.

Bug Fixes

• ASGI: response headers are correctly processed instead of ignored
• Fix issue with attrs and contextlib2 version constraints for Python 2.7.
• CGroup file parsing was fixed to correctly parse container UUID for PCF containers.
• ASM: Do not raise exceptions when failing to parse XML request body.
• ASM: fix a body read problem on some corner case where don't passing the content length makes wsgi.input.read() blocks.
• aws: We are reducing the number of API parameters that the boto, botocore and aiobotocore integrations collect as span tags by default. This change limits span tags to a narrow set of parameters for specific AWS APIs using standard tag names. To opt out of the new default behavior and collect no API parameters, set the environment variable DD_AWS_TAG_NO_PARAMS=1. To retain the deprecated behavior and collect all API parameters, set the environment variable DD_AWS_TAG_ALL_PARAMS=1.
• tracing: make ddtrace.context.Context serializable which fixes distributed tracing across processes.
• django: avoid SynchronousOnlyOperation when failing to retrieve user information.
• Remove forbiddenfruit as dependency and rollback wrapt changes where forbiddenfruit was called. IAST: Patch builtins only when IAST is enabled.
• httpx: Fixes an incompatibility from httpx==0.23.1 when the URL.raw property is not available.
• Fix error in patching functions. forbiddenfruit package has conflicts with some libraries such as asynctest. This conflict raises AttributeError exception. See issue #4484.
• tracer: This fix resolves an issue where the rate limiter used for span and trace sampling rules did not reset the time since last call properly if the rate limiter already had max tokens. This fix resets the time since last call always, which leads to more accurate rate limiting.
• Ensure that worker threads that run on start-up are recreated at the right time after fork on Python < 3.7.
• tracing: This fix resolves an issue where the DD_SERVICE_MAPPING mapped service names were not used when updating span metadata with the DD_VERSION set version string.
• wsgi: This fix resolves an issue where BaseException raised in a WSGI application caused spans to not be submitted.
• library injection: Pin the library version in the library injection image. Prior, the latest version of ddtrace would always be installed, regardless of the image version.
• Fix error in the agent response payload when the user disabled ASM in a dashboard using 1-click Remote Configuration.
• flask: add support for flask v2.3. Remove deprecated usages of flask._app_ctx_stack and flask._request_ctx_stack.
• The specification of DD_TRACE_PROPAGATION_STYLE_EXTRACT now respects the configured styles evaluation order. The evaluation order had previously been fixed and so the configured order was ignored.
• tracing: Ensures that encoding errors due to wrong span tag types will be logged. Previously, if non-text span tags were set, this resulted in v0.5 encoding errors to be output to stderr instead of to a logger.

Other Changes

• Kubernetes library injection: run commands as non-root user.
• tracing: The value of ddtrace.constants.PID has been changed from system.pid to process_id. All spans will now use the metric tag of process_id instead.
• tracing: The exception logged for writing errors no longer includes a long, unhelpful stack trace. The message now also includes the number of traces dropped and the number of retries attempted.

---