NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
PyPI
PyPI, #1733 most downloaded on PyPI
dlt is an open-source python-first scalable data loading library that does not require any backend to run.
Latest stable
1.32.0
Last release 23 days ago
29 Oct 2026
Ships on a steady schedule
a new release about every 3 weeks
Nearly every release is documented
notes for 59 of the last 60 stable releases
13 versions withdrawn
withdrawn after publishing
9 years old
187 releases · first in 2018
Drag across the ruler to compare, or tap a release to open it.
Naive scd2 timestamps are no longer read as UTC (#4547 @rudolfix)
boundary_timestamp or active_record_timestamp is sent as it is, so the database session timezone reads it. A naive active_record_timestamp may stop matching existing active records on servers whose session timezone is not UTC.refresh="drop_resources", pipeline.drop() and dlt pipeline drop attempt to drop all tables belonging to resources (#4582 @rudolfix) — They drop every table of the dropped resources that exists in the destination. Before, only tables that were marked as saw data were dropped.schema_contract that write the same table share the contract resolved for the first one, as object items always did.boundary_timestamp applies only to the load that declares it (#4547 @rudolfix) — The boundary is stored in the load package, not in the schema. dlt no longer writes or reads the x-boundary-timestamp table hint, and a leftover hint in an existing schema is ignored. This is actually a bugfix - boundary_timestamp should be ephemeral.nested_hints of a resource are a list in the trace, each entry with its path as a list of identifiers, instead of a dict keyed by path. Each entry carries load_id, extract_idx and resource_name, and nested hint columns are YAML text. A trace loaded to a destination puts them in ..._hints__nested_hints and ..._hints__nested_hints__path.Relation.columns, columns_schema and schema raise LineageFailedException when a column cannot be resolved (an unqualified column of a table outside the dataset schema, * over such a table, a non-SELECT query). Before, they returned a guessed or empty schema while execution failed anyway. Raw query relations and relations with _allow_partial_lineage return columns without data type instead.dlt.dataset.lineage.compute_columns_schema no longer accepts infer_sqlglot_schema and allow_anonymous_columns..limit()) execute as raw query. Before, they were qualified and bound.NULL literal column has no data type (was text), like on newer sqlglot.add_limit on transformers and unselected parents (#4582 @rudolfix) — A limit on a transformer is set on the resource at the top of its chain, so that resource and all its transformers get the limited pages. source.add_limit() also limits parents that are not selected, for example the filesystem resource behind readers() or a REST API parent with selected: False. Before, a limit on a transformer had no effect and unselected parents ran unlimited. sql_table(defer_table_reflect=True).add_limit(n) loads n pages instead of n - 1.source_filter and destination_scope accept value placeholders: {load_id}, {table_name}, {load_package_timestamp}, {boundary_timestamp}, {curr_date}, date parts ({YYYY}, {MM}, ...) and your own extra_placeholders. These can be constants, or functions called after extract with the resource and the table name.IN, e.g. event_day IN {days}, so you can compute the partitions to replace from the extracted data.destination_scope in nested_hints deletes by that scope instead of a subquery on the root table, so its partitions can be pruned. With every nested table scoped, root_key can be disabled.trace metrics (#4582 @rudolfix)
dlt pipeline <name> trace metrics shows files, rows and sizes of each step of the last run: by resource and table for extract, by table for normalize, and files, sizes, job time and failed jobs by table for load.file_max_items / file_max_bytes to set.replace can keep the resource state (#4575 @rudolfix) — The dict form {"disposition": "replace"} keeps the resource state by default, so incremental cursors survive a full table replace. Set exclude_resource_state: False to reset the state like the "replace" shorthand does. Closes the long-standing #2252.run_all() and run() accept interval, e.g. dlt.current.interval(), and pass it to dbt as --event-time-start / --event-time-end.load_id is passed to the package as a variable, so models can write it to _dlt_load_id.salesforce_table (dlt-hub/verified-sources#717, dlt-hub/verified-sources#725 @rudolfix)
salesforce_table(sobject=...) loads any sObject, standard or custom, like sql_table does for tables. salesforce_source keeps its 15 resources and table names and runs them in parallel.api="auto" | "bulk" | "rest").date, time, timestamp, bigint and json.incremental="auto" picks the cursor per sObject, start_date sets its initial value, and any dlt.sources.incremental is translated to SOQL WHERE / ORDER BY. add_limit is pushed down as SOQL LIMIT.queryAll and removed from the destination (include_deleted).pyarrow and python backends.issues load incrementally from start_date and merge on id. Pages of issues are bulk fetched in parallel./field. included_columns and excluded_columns select fields.issue_comments, issue_changelogs, worklogs and issue_fields.add_limit make progress.init_replication(initial_snapshot="direct") reads the tables with sql_table in the snapshot exported by the slot, without copies. "unlogged_table" makes unlogged copies.backend="pyarrow" decodes inserts column-wise: 173k rows/s against 23k before, for 500k rows from Postgres 17 into filesystem.hard_delete=False keeps deleted rows with _dlt_deleted_ts set.propagate_to_nested map step and merge staging options (#4547 @rudolfix)
propagate_to_nested in dlt.sources.helpers.transform copies root keys into nested lists of dicts, with a nesting cutoff.load.truncate_staging_merge_tables_before_load keeps the staging tables of merge tables between loads.source_filter, dedup ranks only the merge source; the trace serializes to JSON when nested_hints use tuple paths; the merge warning on a nested table no longer fires when the root uses a merge dict.trace metrics, balanced normalize and faster arrow extract: see Highlights (#4582 @rudolfix)replace keeps resource state: see Highlights (#4575 @rudolfix). replace on filesystem also deletes files in parallel.add_limit on transformers and on unselected parents (#4582 @rudolfix) — Derived REST API endpoints and filesystem readers can now be limited. dlt warns when this replaces a different limit.table_name function dispatches arrow, pandas and polars items (#4582 @rudolfix)_allow_partial_lineage works. Raw query relations compute partial lineage for columns / schema. dlt warns when untyped columns are dropped in the normalizer. Supports sqlglot >= 30.22.MONEY is handled.add_input() get a code input pointing at their source or resource function, so custom sources show an upstream in workspace lineage. Also fixes inputs lost when a resource is renamed or cloned, e.g. sql_table(...).with_name(...).("a", "bc") and ("ab", "c") collided and retired an unrelated active record.ALTER when a new schema hash is first stored (#4542 @raunak2007) — A column added to a table without data in that load was never applied, and later loads failed with a missing column.precision, scale, timezone and x-nested-type of the previous type no longer stay in the schema. Fixes TypeError: not all arguments converted on Snowflake VARCHAR after decimal -> text.insert_at after the Incremental step were moved before it in pipeline.run().where() edge cases on Relation (#4570 @rudolfix) — Filters on aggregating, grouping, DISTINCT and window relations wrap the query instead of producing invalid SQL. Filters on aliases that shadow a source column filter the alias. Typed where() on a column without data type no longer raises KeyError. Explicit joins work on table functions.TIMESTAMP WITH LOCAL TIME ZONE maps to timezone: true (#4343 @Aryan-Pardeshi)destination_fingerprint uses scheme://netloc of the bucket url (#4576 @rudolfix) — Fingerprints change for buckets whose name repeats in the path (ie. s3://data/data), bucket roots with a trailing slash, urls with a query string or inline password, upper case schemes and hf:// namespaces. s3://bucket and s3://bucket/path do not change.s3fs could resolve to 0.4.2 from 2020.discard_row does not raise KeyError and data_type contracts keep valid values.add_limit no longer counts hints-only items as pages (sql_table(defer_table_reflect=True).add_limit(1) loaded no rows).dlt.transformer accepts the dict form of write_disposition in its type hints.drop_sources or drop_resources recreates dropped tables from the import schema. Importing a schema keeps the normalizer hints of existing tables.profile is active before launchers resolve config, the workspace .gitignore no longer ignores lock files, and agent loops recognize retried tool calls on newer pydantic-ai.salesforce_table: see Highlights (dlt-hub/verified-sources#717 @rudolfix)start_date for incremental loading (dlt-hub/verified-sources#725 @rudolfix) — Also fixes changes skipped during long jobs and very large fields and records.replace reloads all issues (dlt-hub/verified-sources#720 @rudolfix) — Issues were lost from the second run on.pagefind (#4515 @zilto)/context links and aligned source counts (#4491) (@martinibach)traffic-logs dataset (#4561 @djudjuu)Your coding agent can read these notes before it upgrades. Set up the MCP server →
boundary_timestamp or active_record_timestamp is now sent as written (#4547), so the database session timezone reads it. On servers whose session timezone is not UTC, a naive active_record_timestamp may stop matching existing active records. Use a timezone-aware value.refresh="drop_resources", pipeline.drop() and dlt pipeline drop now drop every table of the dropped resources that exists in the destination, not only tables that saw data (#4582). When several pipelines share one dataset, a refresh can drop same-named tables created by the others.schema_contract that write the same table now share the contract resolved for the first one, as object items always did (#4582).source_filter and destination_scope now accept value placeholders such as {load_id}, {curr_date}, date parts and your own extra_placeholders (#4547). dlt stores the values in the load package, not the schema, so a partition that changes every load no longer creates a new schema version. A list expands for IN, and a nested table can carry its own scope to prune its partitions.
import dlt
from datetime import date
@dlt.resource(
write_disposition={
"disposition": "merge",
"strategy": "delete-insert",
"destination_scope": "date_col = {day}",
"extra_placeholders": {"day": lambda resource, table_name: resource.explicit_args["day"]},
},
)
def my_partition(day: date):
# yield the records for the given day
yield [{"id": 1, "date_col": day}]
pipeline = dlt.pipeline("daily", destination="duckdb")
pipeline.run(my_partition(date(2026, 8, 1)))
replace can keep the resource stateThe dict form {"disposition": "replace"} now keeps the resource state by default, so an incremental cursor survives a full table replace and each run loads only new records (#4575). The "replace" shorthand still resets the state. Set exclude_resource_state to False on the dict form to reset it too.
import dlt
@dlt.resource(write_disposition={"disposition": "replace"})
def clients(updated_at=dlt.sources.incremental("updated_at")):
yield [{"id": 1, "updated_at": "2026-10-01T00:00:00Z"}]
# reset the state like the "replace" shorthand does
@dlt.resource(write_disposition={"disposition": "replace", "exclude_resource_state": False})
def clients_full(updated_at=dlt.sources.incremental("updated_at")):
yield [{"id": 1, "updated_at": "2026-10-01T00:00:00Z"}]
trace metricsdlt pipeline <name> trace metrics prints the files, rows and sizes of the last run: by resource and table for extract, by table for normalize, and files, sizes, job time and failed jobs for load (#4582). Add -v to break it down by file, or --steps to choose steps. Normalize now balances files across workers by total size, not count.
dlt pipeline <pipeline_name> trace metrics
dlt pipeline <pipeline_name> trace metrics -v --steps load
The dbt runner's run_all(), run() and test() now accept a load_id passed to the package as a variable, so models can write it to _dlt_load_id (#4562). run_all() and run() also accept an interval forwarded to microbatch models as --event-time-start and --event-time-end. Each DBTNodeResult now carries rows_affected, failures and adapter_response.
import dlt
from dlt.common.storages import create_load_id
pipeline = dlt.pipeline("pipedrive", destination="duckdb", dataset_name="pipedrive_dbt")
dbt = dlt.dbt.package(pipeline, "pipedrive/dbt_pipedrive/pipedrive")
# a load id for `_dlt_load_id`, and the interval of the load for microbatch models
load_id = create_load_id()
models = dbt.run_all(interval=dlt.current.interval(), load_id=load_id)
for m in models:
print(m.model_name, m.status, m.rows_affected)
Big thanks to our newest contributors:
Full release notes
`timestamp_timezone` on the parquet writer is deprecated as a zone selector (#4033 @rudolfix): use the context timezone instead. "" still means "no UT…
dlt.common.time; pendulum>=3 required (#4033 @rudolfix)
What changed: the minimum pendulum is now 3.0.0 on all Python versions. dlt still works with pendulum as before. documented usage and
pendulum helpers work. Internally pendulum is not used.
Unofficial helpers below were removed, with no compatibility aliases.
| Removed | Replacement |
|---|---|
ensure_pendulum_datetime_utc(v) |
ensure_datetime_in_tz(v, timezone.utc) |
ensure_pendulum_datetime_non_utc(v) |
ensure_datetime(v) |
ensure_datetime_utc(v) |
ensure_datetime_in_tz(v, timezone.utc) |
PY_DATETIME_DECODERS |
DECODERS |
No more forced UTC: ensure_pendulum_datetime no longer forces UTC. It uses the context timezone, which is UTC unless you set one.
datetime (#4388 @rudolfix): modification_date and other timestamps from the fsspec layer and the filesystem source are now stdlib datetime objects (tz-aware UTC). Code that calls pendulum-only methods on them breaks.start_value, last_value and get_current_range() return pendulum.DateTime only for pendulum cursors. datetime.datetime cursors now get stdlib datetimes.lag unit now comes from incremental[date] / [datetime] or from the type of initial_value. It is no longer guessed from the string format of the values, which used to turn lag=3 into 3 seconds for ISO datetime strings.+00:00 instead of Z (#4033 @rudolfix): applies to jsonl loader files and json columns, the same with every JSON backend. Old state and packages still load.Incremental.last_value is the current cursor position (#4033 @rudolfix)
Note: this is enforcement of documented usage. last_value is the actual cursor value at the time of the call and now it works like that in every case.
start_value (no change!); after that it is the running value.lag no longer applies to it, and it keeps advancing when end_value is set.{incremental.last_value} placeholders on dependent resources.TIMESTAMP_LTZ (#4397 @rudolfix). Set use_timestamp_tz=True to keep TIMESTAMP_TZ.upsert honors hard_delete (#4305 @adrienModeo)
SchemaCorruptedException before the load.timestamp_timezone on the parquet writer is deprecated as a zone selector (#4033 @rudolfix): use the context timezone instead. "" still means "no UTC adjustment".cdc merge strategy (#4305 @adrienModeo)
merge_key partitions.cdc and upsert (#4305 @adrienModeo)
skip_unchanged_rows: True updates only rows that changed, so unchanged rows keep their _dlt_load_id. Snowflake Streams and similar consumers then see only real changes.row_version_column_name compares one version or hash column instead of all columns.source_filter and destination_scope (#4305 @adrienModeo)
source_filter chooses which loaded rows to merge. destination_scope chooses which destination rows can be deleted or retired.delete-insert, scd2 and cdc (upsert takes source_filter only).delete-insert;scd2 retiring only within a scope;{table} / {staging_table} placeholders.Relation.incremental() (#4033 @rudolfix)
MAX(cursor), so runs cover the cursor range without gaps or duplicates.range_start / range_end overrides, and JSONPath or qualified cursors.timezone column hint, on both the object and arrow paths.TimezoneContext, or with require.timezone on a job.cdc merge strategy, skip_unchanged_rows, source_filter / destination_scope: see Highlights (#4305 @adrienModeo)hard_delete column treats any non-NULL value, such as "D", as deleted. Merge options are validated when the resource is defined and again before the load.Relation.incremental() and context timezone: see Highlights (#4033 @rudolfix)Incremental helpers (#4033 @rudolfix):
with_cursor() copies an incremental with a different cursor;get_current_range(apply_lag=True) works on bound and unbound instances;advance() pins the range end;dlt.common.interval (#4033 @rudolfix): TTimeInterval is now a NamedTuple, so .start and .end work alongside tuple unpacking. croniter is now a core dependency.workload_identity_provider credential field (AWS, AZURE, GCP, OIDC). Needs snowflake-connector-python>=3.17.0.staging-optimized replace on Replicated / Shared database engines (#4392 @rudolfix)_dlt tables (#3719 @aditypan)import_folder_module in dlt.common.reflection.ref, which does not touch sys.path._dlt_loads.inserted_at and _dlt_version.inserted_at are written in UTC (#4033 @rudolfix): they were written in the loading machine's local time.UTC+02:00, which ClickHouse rejects.merge with hard_delete and no keys raised UnboundLocalError on nested tables (#4305 @adrienModeo): this also left the package partially loaded. Fixed in sql_jobs.py and the sqlalchemy merge job.lag unit follows the declared cursor type (#4406 @rudolfix)last_value stays forward-only when the cursor is exactly 0 (#4367 @chjnett)bind_query generated wrong aliases on case-folding destinations (#4359 @rudolfix)ValueError when coercing a non-finite decimal to bigint (#4348 @eeshsaxena)PipelineTrace last-step accessors typed as Optional (#4344 @bunnysayzz)Pipeline.last_trace typed as possibly None (#4469 @VioletM)orjson import was removed, and the simplejson backend gets the proper error. Adds a lint-emscripten import check.read_csv_duckdb no longer uses the global duckdb instance (#4434 @jtcurlin)DeletingResourcesNotSupported error message repaired (#4438 @simpleqt)cdc, merge conditions and the support matrix (#4305 @adrienModeo); support table formatting (#4508 @rudolfix)<DocCardList /> in the llms-txt output (#4078 @ShreyasGS)hotdata destination to the community page (#4423 @eddietejeda)noindex implementation details, add page descriptions (#4490 @martinibach)examples/ directory (#4439 @simpleqt)prek and ty; removes docs_tools and the old docs workflows.mdsmith formatting rules for docs (#4450 @zilto)test_examples.yml workflow (@zilto, direct commit 23ecc8fee)~/.dlt (#4368 @arose26)@job / @pipeline_run: allow_external_schedulers and refresh deprecated (#4033 @rudolfix)
incremental_mode="interval" | "pipeline" and refresh_propagation instead.DltDeprecationWarning. They are no longer in the typed signatures, so mypy reports errors for typed callers.TimeIntervalContext(allow_external_schedulers=False) is no longer a kill switch (#4033 @rudolfix)engine_version. Jobs that take arguments now carry inputs, so every workspace's manifest hash changes once. Expect a one-time diff on the next deploy.RequiresAccess are treated as needing full access (#4417 @rudolfix): plugin tools without the annotation are hidden from callers with a narrower grant, such as agent jobs. Interactive clients are not affected.dlthub ai toolkit install --overwrite removes stale files (#4417 @rudolfix)
.dlt/.toolkits.Relation.incremental() advances pipeline state by default (#4033 @rudolfix)
advance=False to get the old stateless filter..incremental() twice now combines both filters with AND instead of raising.Incremental.allow_external_schedulers defaults to None (#4033 @rudolfix): an explicit value on the incremental always wins, and a context-level False no longer force-disables it.AGENT.md or with a decorated function.inputs and output (JSON Schema), MCP tool groups, skills, rules and an access grant.run.agent(...), run locally with dlthub local run or on the runner.status, summary, result and a full trace.plug_agent_loop hook. Each loop has its own dependency group, installed by the runner.access declares local (read/write/execute/network), data (read/write) and context (read).local maps to a standard tool set (Read/Glob/Grep, Write/Edit, Bash, WebFetch/WebSearch) that is limited to the workspace and the temp folder.data and context are enforced by the workspace MCP server: a tool the grant doesn't cover is never shown to the model.*secrets.toml, .env) are never readable by file tools.job.last_job_result exposes the trace, token counts and entities.agent.py next to AGENT.md provides validate_input / validate_output hooks; JobAbortedException aborts a run cleanly.incremental_mode, refresh_propagation and auto_refresh_pipeline_mode, plus a [jobs] config section.dlt.current.interval can be changed from inside a job (set, update, apply_lag, apply_full_days).require.timezone runs a job in its declared timezone.run.result(...) declares a typed result.type, job_ref, object, result) to telemetry as a job_result.inputs / output JSON Schema is available on every job kind.Annotated[str, Entity("job-runs")] or entity_type: in AGENT.md. Supported types: job-runs, job, pipeline, dataset, workspace.object and expose.object_input, so the UI can offer an agent on a failed run.job.fail: / job.success: triggers accept selectors (#4417 @rudolfix): selectors expand at manifest time to every matching job. They never target the declaring job or interactive jobs.agent.model, agent.instructions, agent.verbosity, agent.max_turns and agent.max_tokens can be set per run with -c, env or toml.agent.api_key / api_url / api_version, with runtime_* twins supplied by the runtime. Azure, LiteLLM and xAI are mapped automatically.--features, --no-default-features, --access (#4417 @rudolfix): starts a server that exposes only the requested tool groups and only the tools the grant covers. Tools declare their needs with RequiresAccess..claude/dlthub/agents/<toolkit>/<name>/ (or .cursor/, .agents/), separate from native subagents..dlt/.toolkits index ships with deployments, so <toolkit>:<agent> refs resolve on the runner.job, and passed as DLT_INTERVAL_* env vars to the module, streamlit, marimo, mcp and dashboard launchers.execute_sql_query read-only check hardened (#4417 @rudolfix): it now accepts exactly one statement, and the whole statement is checked for mutating statements and host functions.RecursionError on symlink loops.dlthub ai status warning no longer mentions MCP (#4445 @AstrakhantsevaAA)timeout / grace_period examples to use the nested form (#4380 @AstrakhantsevaAA)pendulum>=3.0.0 on every Python version (#4033).+00:00. Datetimes stored inside json columns now end with +00:00 instead of Z (#4033).Incremental.last_value is the current cursor position. last_value is the actual cursor value at call time (#4033). Before the first batch it equals start_value, then the running value. lag no longer applies to it, and it keeps advancing when end_value is set. This affects rest_api {incremental.last_value} placeholders on dependent resources.start_value, last_value and get_current_range() return pendulum.DateTime only for pendulum cursors (#4406). datetime.datetime cursors now return stdlib datetimes. The lag unit comes from the declared cursor type or the type of initial_value, not from the string format of the values, which used to read lag=3 as 3 seconds for ISO strings.Relation.incremental() advances state by default. With a cursor bound to a resource, Relation.incremental() now advances the stored cursor on each run (#4033). Pass advance=False for the old stateless filter. Calling .incremental() twice on a relation combines both filters with AND.datetime. modification_date and other timestamps from the fsspec layer and the filesystem source are now stdlib datetime objects, tz-aware in UTC (#4388). Code that calls pendulum-only methods on them breaks.TIMESTAMP_LTZ. New Snowflake tables store tz-aware timestamps as TIMESTAMP_LTZ instead of TIMESTAMP_TZ (#4397). This keeps values consistent across loader file formats, since parquet could not carry offsets. Existing columns keep their type. Set use_timestamp_tz=True on the destination to keep TIMESTAMP_TZ when a consumer depends on the stored offset.hard_delete. On the Delta table format, upsert now deletes records that the hard_delete column marks as deleted and never inserts them (#4305). With nested tables, this combination raises SchemaCorruptedException before the load starts.cdc merge strategyThe new cdc merge strategy treats each load as a complete snapshot of the source system (#4305). It inserts new keys, updates changed keys, and deletes destination records that the snapshot no longer contains. It needs a unique primary_key. Supported on duckdb (1.4+), motherduck, ducklake, Snowflake, Postgres, BigQuery, Databricks, MSSQL, Fabric, Athena with Iceberg, and filesystem with Delta.
import dlt
@dlt.resource(
write_disposition={"disposition": "merge", "strategy": "cdc"},
primary_key="id",
)
def customers():
# each run yields a complete snapshot of the source table
yield [{"id": 1, "name": "ada"}, {"id": 2, "name": "alan"}]
Set skip_unchanged_rows on an upsert or cdc resource so dlt updates only the records that actually changed (#4305). An unchanged record keeps its _dlt_load_id, so a Snowflake stream or similar consumer sees only real changes. Set row_version_column_name to compare one version or hash column instead of every column.
import dlt
@dlt.resource(
write_disposition={
"disposition": "merge",
"strategy": "upsert",
"skip_unchanged_rows": True,
},
primary_key="id",
)
def customers():
yield [{"id": 1, "name": "ada"}]
Two new write-disposition options limit a merge with a SQL condition (#4305). source_filter chooses which loaded records to merge. destination_scope chooses which destination records a merge may delete or retire. Together they enable partition replace, where BigQuery and Snowflake prune partitions, and keyless delete-insert. They work with delete-insert, scd2 and cdc. upsert takes source_filter only.
import dlt
@dlt.resource(
write_disposition={
"disposition": "merge",
"strategy": "delete-insert",
"destination_scope": "day = '2026-08-01'",
},
)
def events():
# yield only the records for 2026-08-01
yield [{"id": 1, "day": "2026-08-01"}]
Relation.incremental() now pushes the cursor range down to SQL and keeps state between runs (#4033). When the cursor is bound to a resource, each run locks its range end to MAX(cursor) and advances the stored cursor, so repeated runs cover the full range without gaps or duplicates. It supports date and datetime cursors, range_start / range_end overrides, and dotted cursor paths that auto-join another table. Pass advance=False for the old stateless filter.
from datetime import datetime, timezone
import dlt
pipeline = dlt.pipeline("events_pipeline", destination="duckdb", dataset_name="events")
@dlt.resource(write_disposition="append")
def recent_events(
created_at=dlt.sources.incremental(
"created_at",
initial_value=datetime(2026, 1, 1, tzinfo=timezone.utc),
# the cursor is unique, so rows at the range end are not deferred
primary_key="created_at",
),
):
dataset = pipeline.dataset()
events = dataset.table("events").incremental(created_at)
yield dlt.mark.with_hints(
events, dlt.mark.make_hints(columns=dataset.schema.tables["events"]["columns"])
)
# each run reads only rows past the stored cursor, then advances it
pipeline.run(recent_events())
dlt can now store timestamps in a single process-wide timezone instead of always UTC (#4033). The context timezone decides how naive and tz-aware timestamps are normalized for the timezone column hint, on both the object and arrow paths. Set it for a block of code with TimezoneContext, and read the active zone with dlt.current.timezone(). The default is still UTC.
import dlt
from dlt.common.configuration.container import Container
from dlt.common.configuration.specs import TimezoneContext
@dlt.resource
def events():
yield {"id": 1, "created_at": "2026-01-01T12:00:00"}
pipeline = dlt.pipeline("my_pipeline", destination="duckdb")
with Container().injectable_context(TimezoneContext("Europe/Berlin")):
# timestamps in this run are normalized to Europe/Berlin, not UTC
pipeline.run(events())
Big thanks to our newest contributors:
Full release notes
chore(docs): sync docs lockfile without pydantic in the hub extra
chore(docs): sync docs lockfile without pydantic in the hub extra
fix(deployment): write launcher output as UTF-8 so job results and tr…
fix(deployment): write launcher output as UTF-8 so job results and tr…
clarifies color/no color output for agent loop
clarifies color/no color output for agent loop
…every retry writes its exception message. See Breaking Changes for the behavior change this implies.
Failed load packages no longer auto-abort (#3557 @anuunchin) — auto_abort_on_terminal_error defaults to False. With raise_on_failed_jobs=True the job is now retried and you get a LoadClientJobTerminalRetry instead of LoadClientJobFailed, and the package stays pending. Pipeline.drop_pending_packages() and the drop-pending-packages CLI command are deprecated in favour of abort_packages / abort-packages. This has no effect on ephemeral storage, which auto-aborts by wiping the working directory. Set the flag to True to restore the old behavior.
Filesystem layouts that omit {ext} now get the extension appended (#4220 @mattfaltyn) — the documented extension fallback was unreachable, so these layouts wrote extension-less files. They now write mocked-table.jsonl, and .jsonl.gz stays intact for compressed jobs. Pipelines already running an ext-less layout will start writing different file names.
Table prefix keeps its separator (#4283 @AnnasMazhar) — with layout="{table_name}" the prefix is now event. rather than event, which is what stops a replace of event from deleting events. A new warn_unsafe_layout_separators option (default True) warns when the separator around {table_name} can also occur inside table names; set it to False to keep a layout as it is.
Cross-destination joins (#4286 @rudolfix) — join datasets that live on different destinations. dlt attaches the foreign dataset into the query engine of the destination you read from, so the join runs in a single engine:
users = duckdb_pipeline.dataset().table("users")
orders = s3_pipeline.dataset().table("orders")
joined = users.join(orders, on="users.id = orders.user_id")
Supported for duckdb, motherduck, ducklake, lance, lancedb and filesystem, with eager and lazy materialization
Snowflake nested types (#4276 @rudolfix) — native nested-type support behind a new use_nested_types flag: automatic for parquet with nested fields, and for jsonl/dicts with an upfront columns definition via a pyarrow schema. Schema and data-type evolution both work. The user-facing interface piggy-backs on the json data type and is experimental for now.
Input/output lineage in traces (#4289 @rudolfix) — traces now carry lineage in a relational form close to OpenLineage but ingestible by dlt itself. Inputs (per resource and data location) land on the extract step, and dlt derives the outputs onto the load step.
Manual load package abort (#3557 @anuunchin) — abort_packages records what happened instead of silently deleting, and gives you the chance to inspect or retry a failed job first. Adds list_pending_retry_jobs_in_package(), fail_pending_job() and retry_failed_job() on Pipeline, a dlt pipeline <name> load-package <load-id> fail-job <job> CLI command, and an .exceptions folder in the load package where every retry writes its exception message. See Breaking Changes for the behavior change this implies.
Retryable schema migrations (#4266 @rudolfix) — a retry_schema_update helper for use with tenacity, so a failing schema migration retries with jitter and backoff instead of failing the load:
from tenacity import retry, retry_if_exception, stop_after_attempt, wait_random_exponential
from dlt.pipeline.helpers import retry_schema_update
@retry(
stop=stop_after_attempt(5),
wait=wait_random_exponential(multiplier=1, max=30),
retry=retry_if_exception(retry_schema_update()),
reraise=True,
)
def load():
return pipeline.run(chess_source(...))
_dlt tables on databricks, which made CREATE TABLE non-atomic, and adds an option to disable comment creation altogether via TBLPROPS.session_timezone can be set on clickhouse, databricks, duckdb, ducklake, postgres, redshift, snowflake and others, guarded by a new supports_session_timezone capability. Defaults are unchanged: UTC on duckdb and snowflake, where snowflake's was previously hardcoded. duckdb's TimeZone moves from global to per-connection config, so dlt no longer changes the global timezone of a duckdb connection you pass in; set session_timezone to an empty string to keep the duckdb default of the machine timezone.add_limit on the source factory (#4223 @richacode007-byte) — @dlt.source factories accept .add_limit() before instantiation, for both sync and async sources, and return the factory for chaining. Factory clones preserve the limit.dev.secrets.toml), vault identity or AWS account, region and prefix. See Breaking Changes for the trace shape change.instance key in the job require spec (#4262 @tetelio) — a job can declare its runner instance requirements as an open dict, with instance.size (for example {"size": "medium"}) read today. The legacy machine key still works and is still serialized, but now emits a DltDeprecationWarning, so existing manifests and deployments are unaffected.__repr__ for WorkspaceRunContext (#4022 @sohamwaghe) — shows the run dir and profile instead of an opaque object in a REPL or debugger.initialize_storage in schema-update error handling (#4325 @rudolfix) — CREATE SCHEMA was excluded from the retry-only-the-schema-update path.get_table_prefix_layout (#4283 @AnnasMazhar) — stops a replace of event from deleting events. See Breaking Changes for the delta and iceberg table move and the new warn_unsafe_layout_separators option.Schema.from_stored_schema, which skips migrate_schema and apply_defaults. dlt.dataset() and the dashboard schema-version loader therefore could not read schemas written by older dlt versions, raising KeyError: 'name' on any naming normalization.sge.Boolean (#4231 @chuenchen309) — bool fell into the int/float branch and produced a malformed numeric literal that crashed annotate_types() with decimal.InvalidOperation in Relation.where() and .filter().remove_columns as a sequence (#4241 @chuenchen309) — add_dlt_load_id_column silently dropped RecordBatch columns whose name is a substring of _dlt_load_id (id, load_id, load, dlt, …), because membership against a bare str is a substring test.work_group to primary (#4282 @burnash) — newer dbt-athena passes an empty value straight through.extra_placeholders in get_table_prefix (#4226 @axelray-dev) — when extra_placeholders overrode {schema_name}, replace-truncation and table listing matched zero files and silently left old data in place.{ext} (#4220 @mattfaltyn) — see Breaking Changes for the resulting file name change.OAuthJWTAuth.scopes is optional and defaults to None, but __post_init__ passed it to str.join, raising TypeError before authentication could begin.int(total) raises TypeError, not ValueError, when total_path resolves to an object or array, so pagination failed opaquely instead of raising the paginator's actionable error.has_more=true could override a stop already decided by maximum_offset, maximum_page or the response total, or re-enable a cursor paginator with no valid cursor. has_more=false can still stop pagination early.ModuleType.__dict__ usage infringing PEP 562 (#4215 @zilto)uvx dlthub-init@latest.guest and workspace developer roles, how invites and roles combine, and the safety rules for managing members.hub/getting-started/platform-tutorial.md and repoints the redirects at pipeline-operations/deployments so old URLs keep resolving.rest_api_source doing the full describe → load → dataset().pokemon.df() round trip, plus sql_database, filesystem | read_csv_duckdb and direct pandas/Polars/Arrow. Meta sections are kept intact.help wanted or good first issue./position example in an issue comment matched the workflow's trigger and the run began writing the wrong page.documentation and agent together sends two labeled webhooks and both matched. agent now arms the run, and the /position comment arm is removed.UV_SYNC_ARGS and stops uv run from resyncing prepared CI environments via UV_NO_SYNC. Also raises the GitPython minimum and keeps the SQLGlot compatibility test working across supported versions.pytest-rerunfailures filters for mssql, synapse and fabric ODBC transients and for databricks DatabaseTransientException. Benign assertion failures still fail fast.gather_metrics test on an early sleep return (#4236 @burnash)auto_abort_on_terminal_error now defaults to False. With raise_on_failed_jobs=True a terminally failed job is retried and raises LoadClientJobTerminalRetry instead of LoadClientJobFailed, and the package stays pending. drop_pending_packages is deprecated in favour of abort_packages. Set the flag back to True to restore the old behavior (#3557).{ext} now append the extension. A layout that omits {ext} previously wrote extension-less files because the documented fallback was unreachable. dlt now appends the job extension, so a table writes mocked-table.jsonl and .jsonl.gz stays intact for compressed jobs. Pipelines already running an ext-less layout will start writing different file names (#4220).layout="{table_name}" the table prefix is now event. rather than event, which stops a replace of event from deleting events. Delta and iceberg tables move to the new location. A new warn_unsafe_layout_separators option (default True) warns when a separator around the table name can occur inside table names (#4283).dlt can now join two datasets that live on different destinations (#4286). It attaches the foreign dataset into the query engine of the destination you read from, so the join runs in one engine. Supported for duckdb, motherduck, ducklake, lance, lancedb and filesystem, with eager and lazy materialization. A cross-destination join needs an explicit on.
import dlt
crm = dlt.pipeline("crm", destination="duckdb", dataset_name="crm")
events = dlt.pipeline("events", destination="filesystem", dataset_name="events")
# dlt attaches the filesystem dataset into duckdb and runs the join there
joined = crm.dataset().table("users").join(
events.dataset().table("events"),
on="users.id = events.user_id",
)
df = joined.df()
Set use_nested_types on the Snowflake destination to store nested (json) columns as native ARRAY, OBJECT and MAP types instead of a single VARIANT (#4276). dlt derives the structured type from the column's Arrow type, automatically for parquet or from a pyarrow schema you declare for jsonl. Schema and type evolution both work. The interface is experimental for now.
import dlt
import pyarrow as pa
@dlt.resource(
columns=pa.schema([
pa.field("payload", pa.struct([("tags", pa.list_(pa.int64())), ("city", pa.string())])),
])
)
def items():
yield {"id": 1, "payload": {"tags": [1, 2], "city": "Berlin"}}
snow = dlt.destinations.snowflake(use_nested_types=True)
pipeline = dlt.pipeline("my_pipeline", destination=snow)
pipeline.run(items(), loader_file_format="jsonl")
A terminally failed job no longer silently deletes its package (#3557). abort_packages records what happened and lets you inspect or retry a job first, and new Pipeline methods list_pending_retry_jobs_in_package, fail_pending_job and retry_failed_job plus a dlt pipeline <name> abort-packages CLI command drive recovery. Every retry's exception is saved in the package's .exceptions folder.
import os
import dlt
pipeline = dlt.attach("my_pipeline")
load_id = pipeline.list_normalized_load_packages()[0]
jobs = pipeline.list_pending_retry_jobs_in_package(load_id)
job_file, folder = jobs[0]
# give up on a job, keep its exception, then retry the rest
pipeline.fail_pending_job(load_id, os.path.basename(job_file))
pipeline.load()
Parallel runs against one dataset can race on CREATE TABLE and ALTER TABLE and fail. The new retry_schema_update helper marks that failure as retryable (#4266), so wrapping the run in tenacity retries the schema migration with backoff and jitter instead of failing the load. dlt re-reads the destination on each attempt and applies only what is still missing.
import dlt
from tenacity import retry, retry_if_exception, stop_after_attempt, wait_random_exponential
from dlt.pipeline.helpers import retry_schema_update
pipeline = dlt.pipeline(pipeline_name="chess", destination="duckdb", dataset_name="games")
@retry(
stop=stop_after_attempt(5),
wait=wait_random_exponential(multiplier=1, max=30),
retry=retry_if_exception(retry_schema_update()),
reraise=True,
)
def load():
return pipeline.run([{"id": 1}], table_name="games")
Traces now carry input and output lineage, in a relational form close to OpenLineage but ingestible by dlt itself (#4289). The sql_database, rest_api and filesystem sources record what each resource read onto the extract step, and dlt derives the tables written per resource onto the load step, so a run's read and write sides are captured automatically. Custom sources can record their own reads with resource.add_input().
Big thanks to our newest contributors:
auto enables orphan removal when merge key is set
auto enables orphan removal when merge key is set
Nothing published for this version
{"size": "medium"}). The legacy machine key is deprecated — it still works and serializes, but now emits a DltDeprecationWarning pointing to instance.…
instance requirement spec for jobs (#4262 @tetelio) — Jobs can now declare runner instance requirements via require.instance (an open Dict[str, Any], e.g. {"size": "medium"}). The legacy machine key is deprecated — it still works and serializes, but now emits a DltDeprecationWarning pointing to instance. Fully backward compatible.has_more=true (#4227 @mattfaltyn) — An API returning has_more=true can no longer re-enable a paginator that already hit a stop condition (maximum_offset/maximum_page, response total, or a missing cursor), preventing requests past a hard limit or without a valid cursor. Fixes #4225.total_path resolving to a JSON object or array now raises the paginator's clear "not an integer" ValueError instead of escaping as an opaque TypeError.OAuthJWTAuth with the default scopes=None no longer raises TypeError; the scope claim is omitted when no scopes are configured. Fixes #4234.remove_columns as a sequence (#4241 @chuenchen309) — add_dlt_load_id_column no longer drops RecordBatch columns whose name is a substring of _dlt_load_id (e.g. id, load), which previously produced silent NULLs or a crash. Fixes #4240.UV_SYNC_ARGS, prevents uv run from resyncing prepared CI environments (UV_NO_SYNC), bumps the GitPython minimum, and keeps the SQLGlot compatibility test green across versions.pytest-rerunfailures filters so transient ODBC/Azure SQL and Databricks connection errors get retried instead of failing the run, while benign assertion failures still fail fast.ClickHouse `staging-optimized` replace strategy (#3927 @filipesilva) — ClickHouse now supports the staging-optimized replace strategy, performing atom
staging-optimized replace strategy (#3927 @filipesilva) — ClickHouse now supports the staging-optimized replace strategy, performing atomic table swaps via EXCHANGE TABLES so a full replace is instantaneous and never leaves the destination in a half-loaded state.Relation.join() (#3868 @burnash) — The dataset relation API gains explicit join support, giving you control over join conditions when composing relations.physical_location() accessor and can_join_with rules, so dlt knows when relations that live on different destinations (or datasets) can actually be joined together.Features
enable_atomic_replace config flag makes the truncate-and-insert strategy do a single-job, metadata-preserving WRITE_TRUNCATE_DATA load from GCS staging, giving BigQuery an atomic, transactional full replace.DATA_WRITER__COMPRESSION or ParquetFormatConfiguration(compression=...).ArrowToParquetWriter (#3896 @AyushPatel101) — Safe Arrow type promotions (e.g. float32 → float64) now work across flush batches instead of crashing once data spans multiple batches.FILLRECORD) to the generated COPY command for staged loads.dlt→dlthub command proxying in workspace context (#4037 @rudolfix) — Instead of silently running a dlthub command, dlt now shows a note and points to the correct dlthub counterpart, avoiding misleading behavior.Fixes
add_dlt_id (#4187 @nizar-zerrad) — Adding _dlt_id to an empty Arrow table no longer fails with ArrowInvalid.items_count).decompose="parallel" / parallel-isolated runs so concurrent tasks no longer truncate each other's staging files.log_period (#3962 @cbility) — Stops resetting the log timer on every counter update, so logs respect the configured log_period.jsonpath-ng 1.8.0, dropping the archived ply transitive dependency flagged by security scanners (CVE)./commit skill (#4098 @lis365b) — Claude Code tooling to keep commit messages Conventional-Commit clean (no footers/emojis).Destination factory (#4170 @rudolfix)Relation.join() now takes an on= predicate, so you write the join condition yourself instead of relying on dlt schema references (#3868). This covers joins between unrelated top-level tables, self-joins, and joins across two datasets. Passing a Relation from another dlt.Dataset joins across datasets, and destination configs now declare which relations can be joined together (#3905).
import dlt
crm = dlt.pipeline("crm", destination="duckdb", dataset_name="crm_data")
sales = dlt.pipeline("sales", destination="duckdb", dataset_name="sales_data")
# explicit predicate between two tables without a dlt schema reference
by_email = crm.dataset()["users"].join(
"logins", on="users.email = logins.email", kind="left"
)
# pass a Relation from another dataset to join across datasets
users_purchases = crm.dataset()["users"].join(
sales.dataset()["purchases"], on="users.id = purchases.user_id"
)
dlt can now resolve config and secrets from AWS Secrets Manager, alongside the existing Google Secret Manager provider (#4162). Enable it with enable_aws_secrets. By default dlt looks up secrets under the dlt/ name prefix so they stay namespaced in a shared vault. The change also stops re-creating the Google secrets client on every lookup.
[providers]
enable_aws_secrets = true
[providers.aws_secrets.credentials]
aws_access_key_id = "..."
aws_secret_access_key = "..."
region_name = "eu-central-1"
staging-optimized replace strategyClickHouse now supports the staging-optimized replace strategy (#3927). It swaps the destination and staging tables with EXCHANGE TABLES, so a full replace is atomic and never leaves the table half-loaded. The database must use the Atomic or Shared engine, otherwise the load fails fast before any data is extracted.
[destination.clickhouse]
replace_strategy = "staging-optimized"
Turn on enable_atomic_replace and the truncate-and-insert strategy replaces a table with a single WRITE_TRUNCATE_DATA load job from GCS staging (#4130). The data is swapped in place, so descriptions, labels, column policy tags, primary keys, and dependent materialized views survive the refresh with no empty-table window. It requires a GCS staging destination.
import dlt
pipeline = dlt.pipeline(
"bq_pipeline",
destination=dlt.destinations.bigquery(enable_atomic_replace=True),
staging="filesystem",
)
Safe Arrow type promotions such as float32 to float64 now work across flush batches instead of crashing once the data spans more than one flush (#3896). Set arrow_concat_promote_options to default to rotate a new parquet file on a type change, or permissive to cast batches into the first schema. The default none still requires identical schemas but now fails with a clearer error.
[sources.data_writer]
arrow_concat_promote_options = "default"
Declare a job's runner resources with require.instance, an open dict the platform reads (today size) (#4262). The legacy machine key still works but now emits a deprecation warning pointing to instance.
from dlt.hub.run import job
@job(require={"instance": {"size": "medium"}})
def train():
...
A REST source paginator that has already hit a stop condition (maximum_offset, maximum_page, the response total, or a missing cursor) is no longer restarted when the API also returns has_more=true (#4227). Configured page limits and cursor checks now act as hard stops.
Big thanks to our newest contributors:
This is a patch release that allows 1.28.x dlt to use future versions of dlthub-client
This is a patch release that allows 1.28.x dlt to use future versions of dlthub-client
Dropped Python 3.9 support (#4074 @Travior) — Python 3.9 reached end-of-life on 2025-10-31. dlt no longer tests against or advertises support for 3.9.
ruff/mypy target versions were bumped accordingly. Users still on 3.9 must upgrade to 3.10+ to use this release.timestamp[ns] and times as time64[ns] (via the arrow_stream return type) instead of date64 (ms). dlt now normalizes all connectorx temporal columns to microsecond precision (the renamed cast_connectorx_temporal_columns), truncating ns→us losslessly since source DBs carry no nanoseconds.detect_datetime_format so week-date cursors round-trip (#4061 @yashs33244) — Week-date formats (YYYY-Www) were detected as %Y-W%W, which disagrees with ISO weeks at year boundaries. "2026-W01" parsed to 2025-12-29 and re-rendered as "2025-W52", silently corrupting saved incremental cursors. Now emits %G-W%V / %G-W%V-%u.0x0 characters from postgres INSERT strings (#4086 @rudolfix) — Postgres escaping allowed null (0x0) characters through; they are now stripped in the escape function. INSERT statements are also split by \n only.sql_database (#4067 @rudolfix) — User-passed metadata is now correctly consulted for cached tables in both eager and deferred reflection. Fixes #4066.ai init (#4072 @lis365b) — Codex silently drops skills whose description frontmatter exceeds 1024 chars. The Codex install path now truncates the installed SKILL.md description to the cap (with a warning) so workbench skills like dlthub-router load correctly.dlthub-init vs dlthub-start consistently in onboarding docs (#4080 @bjoaquinc)test_common.yml into 27 parallel focused jobs (3 per OS/Python combo) so wall-clock is roughly max(suite) instead of sum(suites), moves runners to blacksmith-* variants, and enables Python 3.14 in CI (running the full suite, with 3.14-gated deps snowflake-connector-python>=4.4.0 and cffi>=1.18). Also includes job/workflow renames; one functional reduction — lance_s3 non-essential tests now run only on full-suite triggers.refresh is now the recommended way to do a full refresh — the replace switch is deprecated. Also fixes #3998 and #4017.
refresh="drop_data" on Delta and persistent-catalog Iceberg no longer frees storage (#4051 @rudolfix) — Truncation is now a transactional delete that keeps the table, schema, version history, and data files (retained for time travel until vacuum). Previously the files were deleted. This corrects prior erroneous behavior, but pipelines relying on drop_data to reclaim disk space will no longer see storage freed without an explicit vacuum.replace now fully truncates empty and orphaned tables (#4010 @rudolfix) — Tables belonging to a replace resource that receive no data in a run (including nested tables, dynamic-name and variant tables) are now consistently truncated. Previously these tables could be left orphaned with stale rows surviving the reload, leaving the dataset in an inconsistent state. Pipelines that implicitly relied on that leftover data will now see those tables emptied.LanceNamespace + lance.Session across job clients; atomic single-commit-per-table writes uncommitted fragments in parallel and commits them in one version (Append/Overwrite/upsert). replace is now a single Overwrite commit so readers never see a partially-replaced table, and the namespace pool rebuilds handles on credential rotation to avoid ExpiredToken on long loads. Also fixes #3800 (Iceberg 409 Table already exists after drop_sources).replace / refresh truncation (#4010 @rudolfix) — replace resources now consistently truncate all participating tables even when a load carries no data (nested tables, dynamic names, variants included), and drop_data refresh truncates correctly on append and survives non-existent tables. refresh is now the recommended way to do a full refresh — the replace switch is deprecated. Also fixes #3998 and #4017.write_encoding option lets you choose the encoding of CSV files dlt writes (default utf-8), e.g. utf-8-sig for Excel BOM or latin-1/cp1252 for legacy importers. Set via [normalize.data_writer] write_encoding="latin-1".ExpiredToken failures on long-held connections (#4003).Features
zip path as good as pandas.--asset-url. Launcher path resolution uses find_spec instead of import_module so notebooks aren't executed before marimo run / streamlit run (~1–1.5s faster port readiness).Fixes
credential_chain secrets to survive temp-token expiry (#4021 @0ywfe) — Adds REFRESH auto so long-held sql_client connections no longer die with ExpiredToken once temporary AWS tokens rotate. Fixes #3987.Retry-After: 0 no longer triggers an immediate retry loop (#4043 @AstrakhantsevaAA) — Values ≤ 0 are treated as no actionable hint, letting tenacity's exponential backoff take over. Fixes #4036.create_indexes is enabled (#4011 @burnash) — Avoids Unity Catalog UC_REFERENTIAL_CONSTRAINT_DOES_NOT_EXIST failures when the matching primary/unique key isn't created.META_TYPE 'sqlite' (#3871 @Analect) — Splits the duckdb/sqlite branch so a duckdb:///catalog.duckdb catalog URI attaches cleanly instead of failing on PRAGMA journal_mode=WAL.__ in generated CLI docs to prevent markdown bold (#3993 @burnash)__deployment__.py file name in CLI docs (#3988 @nuetu)pytest-rerunfailures, scoped to just those two transient destinations via --only-rerun.test_multi_schema_selection flaking on slow CI.refresh="drop_data" keeps files on Delta and Iceberg. Truncating a Delta or persistent-catalog Iceberg table is now a transactional delete (#4051) that keeps its files for time travel, so refresh="drop_data" no longer frees storage. Pipelines that relied on it to reclaim disk must now run vacuum explicitly.replace truncates empty and orphaned tables. Tables of a replace resource that receive no data in a run, including nested, dynamic-name, and variant tables, are now consistently truncated (#4010). Previously they could keep stale rows across reloads, so pipelines relying on that leftover data will now see those tables emptied.You can now set the encoding of the CSV files dlt writes with a new encoding option (default utf-8) under [normalize.data_writer] (#4045). Use utf-8-sig for an Excel BOM or latin-1/cp1252 for legacy importers, and encoding_errors to control how unrepresentable characters are handled.
[normalize.data_writer]
encoding="utf-8-sig"
encoding_errors="replace"
refresh optionrefresh is now the recommended way to fully reload data. Set refresh="drop_data" to truncate the selected resources' tables and reset their incremental state, leaving other tables intact (#4010). The old write_disposition="replace" switch for a full refresh is deprecated. Truncation now reliably covers nested, dynamic-name, and variant tables.
import dlt
@dlt.resource
def users():
yield {"id": 1, "name": "alice"}
pipeline = dlt.pipeline("app", destination="duckdb")
# full refresh: truncate the resource's tables and reset incremental state
pipeline.run(users(), refresh="drop_data")
Default cloud credential chains (AWS, Azure, GCP) are now handed to the underlying access libraries, fsspec, the object_store crate, pyiceberg, and DuckDB, so each refreshes tokens through its own provider chain instead of a frozen snapshot (#4056, #4021). This stops ExpiredToken failures on long reads and loads that use temporary credentials, while static keys and explicit sessions stay frozen.
String values containing a NUL byte (the ASCII null character, 0x00), common in data from MongoDB, previously produced invalid INSERT statements on Postgres, Redshift, and DuckDB. dlt now strips NUL from escaped string literals so those rows load (#4086), and splits multi-row inserts on newlines only so values holding other line separators stay intact.
With a newer connectorx, the sql_database connectorx backend returns timestamps and times at nanosecond precision through its arrow_stream type instead of the older millisecond date64. dlt now normalizes every connectorx temporal column to the microsecond precision destinations expect, truncating losslessly (#3996).
Big thanks to our newest contributors:
Hotfix: fixes #3998 (merge with empty data after replace on incremental truncates the destination table). Upgrade for anyone on 1.27.0/1.27.1.
Hotfix: fixes #3998 (merge with empty data after replace on incremental truncates the destination table).
Upgrade for anyone on 1.27.0/1.27.1.
Nothing published for this version
…mcp, ai) into a dedicated dlt[hub] plugin. See Breaking Changes above.
workspace extra removed and dlthub command split out (#3929 @rudolfix) — The workspace extra is gone; users should install marimo, pyarrow, ibis, fastmcp, and other dependencies directly. Part of dev tooling moved to a plugin: dlt dashboard, dlt pipeline ... show, dlt pipeline ... mcp now require pip install dlt[hub]. dlt ai was moved to dlthub ai.@dlt.resource without manual conversion. Auto-detected and routed through the Arrow extraction pipeline; LazyFrames are auto-collected before conversion.databricks_adapter(my_resource, insert_api="zerobus") enables loading via the Databricks Zerobus SDK into Delta tables. API mirrors bigquery_adapter; supported for append write disposition.dlt.Relation (#3889 @burnash) — Apply incremental filters directly on dlt.Relation, enabling efficient incremental reads from datasets.dlthub command split (#3929 @rudolfix) — Reorganizes _workspace modules and splits dev tooling (dashboard, mcp, ai) into a dedicated dlt[hub] plugin. See Breaking Changes above.dlt.Relation (#3889 @burnash) — See Highlights.dlthub command split (#3929 @rudolfix) — See Highlights / Breaking Changes.lance destination REST Namespace support (experimental) (#3908 @jorritsandbrink) — Adds experimental REST Namespace support to the lance destination, currently only validated against an ephemeral in-memory Lance REST Namespace proxy.-y / --yes flag to bypass non-interactive prompts (#3910 @anuunchin) — New flag auto-accepts all confirm() prompts via a dedicated ALWAYS_CONFIRM flag in echo.py, without leaking into prompt() or text_input(). Resolves #3592.RootModel handling and preserves root-level Annotated[...] metadata after Pydantic 2.13 moved the discriminator off root_field.annotation.scd2 unmapped insert for nested tables (#3812 @deschman) — Generates an explicit column list (excluding SCD2 from/to metadata) for the staging-to-destination insert, fixing DATATYPE_MISMATCH errors on Spark/Databricks ANSI mode when schemas drift. Fixes #3811.ReplicatedMergeTree / SharedMergeTree for merge delete temp tables, preventing duplicates on replicated/shared MergeTree deployments. Closes #3797._user_agent_entry (#3935 @xodn348) — Switches DatabricksCredentials.to_connector_params() to the non-prefixed user_agent_entry, silencing the deprecation warning printed on every connection. Closes #3934.with_load_id breaking on non-root sibling branches (#3878 @Travior)max_length param (#3826 @aditypan) — Removes spurious length appending in schema naming when max_length is provided. Fixes #3816.dataset_name to job metrics in load step (#3808 @rudolfix)join usage (#3902 @Travior)fork_tests_with_secrets.yml workflow gated by the ci from fork label and a maintainer approval in the fork-ci environment. Resolves #3850.fruit_pipeline fixture to module to avoid duckdb lock race (#3932 @burnash) — Stabilizes the dashboard e2e suite.workspace extra removed and dev tooling split into a plugin. The workspace extra is gone. Install marimo, pyarrow, ibis, fastmcp, and other dependencies directly. dlt dashboard, dlt pipeline ... show, and dlt pipeline ... mcp now require pip install dlt[hub], and dlt ai moved to dlthub ai (#3929).Resources can now yield polars.DataFrame and polars.LazyFrame objects directly, with no manual conversion (#3837). dlt auto-detects them and routes them through the Arrow extraction path, collecting LazyFrames first, so Parquet loading and schema inference apply. Install with pip install dlt[polars].
import dlt
import polars as pl
@dlt.resource
def orders():
yield pl.DataFrame({"order_id": [1, 2, 3], "amount": [100.0, 200.0, 300.0]})
pipeline = dlt.pipeline("orders", destination="duckdb")
pipeline.run(orders())
The Databricks destination can now ingest into Delta tables through the Zerobus SDK instead of COPY INTO. Set insert_api="zerobus" on the destination or per resource with databricks_adapter, for the append write disposition (#3904). Configure the endpoint under [destination.databricks.zerobus].
import dlt
from dlt.destinations.adapters import databricks_adapter
@dlt.resource(write_disposition="append")
def events():
yield from [{"id": 1}, {"id": 2}]
databricks_adapter(events, insert_api="zerobus")
pipeline = dlt.pipeline("events", destination="databricks")
pipeline.run(events())
Relation.incremental() turns a dlt.sources.incremental cursor into a WHERE clause, so you read only rows in the cursor window straight from a dataset (#3889). Pass a column or table.column cursor path. A dotted path auto-joins the referenced table. It also works via dataset.table(..., incremental=...).
import dlt
from dlt.common.pendulum import pendulum
pipeline = dlt.pipeline("events", destination="duckdb")
dataset = pipeline.dataset()
cursor = dlt.sources.incremental(
"created_at",
initial_value=pendulum.datetime(2026, 1, 1, tz="UTC"),
end_value=pendulum.datetime(2026, 2, 1, tz="UTC"),
)
rows = dataset.table("events", incremental=cursor).fetchall()
-yA global -y/--yes flag auto-accepts every confirm() prompt, so destructive commands like pipeline drop and sync run unattended in CI (#3910). It differs from --non-interactive, which only picks prompt defaults and so skips actions that default to no.
dlt -y pipeline chess_pipeline drop players_games
Running merge with empty data after a replace on an incremental resource could silently truncate the destination table, a regression introduced in 1.27.0. 1.27.2 reverts the change (#3999). Anyone on 1.27.0 or 1.27.1 should upgrade.
decimal coercionCoercing a Python float into a decimal or wei column now converts through str() first, so 34.7 stores as 34.7 instead of a long IEEE 754 tail like 34.700000000000003 (#3928). For exact values, keep source data as Decimal or strings rather than floats.
Big thanks to our newest contributors:
Nothing published for this version
Nothing published for this version
Incremental external scheduler now raises instead of silently warning (#3877 @rudolfix) — Untyped/non-coercible cursor values now raise JoinSchedulerE
JoinSchedulerError; missing intervals raise ExternalSchedulerNotAvailable. Resources with allow_external_schedulers=True that previously fell back to dlt state will now fail. This is a bugfix that corrects previously incorrect behavior.dlt.Relation.join(...) (#3590 @Travior) — Adds a join() method on dlt.Relation based on the normalizer and table references, enabling fluent relational composition over datasets.TJobQueryTags is generalized to TQueryTags with a new operation field (with a compatibility export).dlt.current.interval() returns the active (start, end) interval or None, backed by an injectable TimeIntervalContext with optional allow_external_schedulers override and auto-detection from env vars / Airflow.dlt.Relation.join(...) (#3590 @Travior) — see Highlights.dlt/extract/incremental/sql.py (to_sqlglot_filter) honors timestamp_precision, supports_tz_aware_datetime_in_cast, and sqlite quirks; works on bound and unbound incrementals.start_value persisted in incremental state (#3877 @rudolfix) — Only written when rows actually arrive, so it is no longer advanced silently on empty runs.uuid_to_string PyArrow fast path (#3877 @rudolfix) — Numpy-vectorized with a pure-Python fallback; pyarrow ≥ 24 arrow.uuid extension arrays are coerced to canonical strings, and UUID columns under pyarrow < 24 also take the fast path.dlt.current.resource_metrics() counters are no longer dropped when every item is filtered out.NotRequired[T] (#3877 @rudolfix) — Via __required_keys__."dremio" dialect literal (#3877 @rudolfix) — Added to TSqlGlotDialect.Schema.unify_schemas() (#3898 @burnash) — The naming-convention check in Schema.unify_schemas() is now opt-in; also drops the max_length tests workaround.dlt.attach() when pipeline cannot be restored (#3890 @bjoaquinc) — Rewrites CannotRestorePipelineException messages to name required inputs, show a concrete dlt.attach(...) example, and offer dlt.dataset() as a lighter alternative; suppresses a redundant inner exception in tracebacks.iter_std (#3877 @rudolfix) — Reader threads swallow ValueError/OSError and always close the queue.ConfigFieldMissingException (#3877 @rudolfix)pipeline.md (#3885 @ShreyasGS) — Tense, contractions, and grammar cleanups via Harper + Vale Google Developer Docs style.allow_external_schedulers=True no longer fall back to dlt state when a scheduler interval is missing. dlt raises ExternalSchedulerNotAvailable when no interval resolves, and JoinSchedulerError when the cursor type cannot coerce to a timestamp. Provide an interval, or drop the flag (#3877).Relation.join()Relation.join() composes a SQL join from a dataset's schema references, so you navigate parent/child tables (or annotated relationships) without writing the ON clause. Call it on a base-table relation and name the table to join. Columns from the joined table are prefixed with its name, or with the alias you pass (#3590).
import dlt
pipeline = dlt.pipeline(
pipeline_name="shop", destination="duckdb", dataset_name="shop_data"
)
dataset = pipeline.dataset()
# join uses the schema's parent/child references, no ON clause needed
users_with_orders = dataset["users"].join("users__orders", alias="orders")
df = users_with_orders.select("name", "orders__order_id", "orders__total").df()
dlt.current.interval()dlt.current.interval() returns the active (start, end) window from an external scheduler, or None when none is set. Any resource can read it, even ones not using Incremental, to scope requests, validation, or logging. Drive it from Airflow, the DLT_INTERVAL_START and DLT_INTERVAL_END env vars, or by injecting a TimeIntervalContext yourself (#3877).
import dlt
@dlt.resource
def my_resource():
interval = dlt.current.interval()
if interval is not None:
start, end = interval
# scope your requests to the [start, end) window
yield {}
Snowflake query tagging now runs beyond load jobs. dlt tags sessions for storage setup, schema and state reads, schema updates, load completion, and table drops, each carrying a new operation field. Add {operation} to your query_tag template to see which dlt step a session is running (#3759).
[destination.snowflake]
query_tag='{{"operation":"{operation}", "source":"{source}", "resource":"{resource}", "table": "{table}", "load_id":"{load_id}", "pipeline_name":"{pipeline_name}"}}'
Big thanks to our newest contributors:
Nothing published for this version
Nothing published for this version
Multischema datasets (#3770 @burnash) — See Breaking Changes above. Enables sidecar schemas (e.g. data-quality quarantine tables) to live alongside th…
dataset() method and still go back to single-schema dataset by providing pipeline.default_schema when creating dataset.lance destination (#3810 @jorritsandbrink) — New destination for the Lance table format with optional vector embedding generation via lancedb. Supports local storage and s3/az/gs, uses the Lance Directory Namespace V2 spec, and supports branching. Complements the existing lancedb destination (which targets LanceDB Cloud).lance destination (#3810 @jorritsandbrink) — See Highlights.ducklake: metadata_schema ATTACH option (#3763 @sangwookWoo) — Adds metadata_schema to DuckLakeCredentials so the DuckLake metadata schema can be configured independently from ducklake_name.__main__ in orchestrators (#3784 @rudolfix) — Closes #3586._dlt_id requirement when merging arrow tables without nested tables on ClickHouse.aws_session_token to staging s3() table function (#3769 @anuunchin) — Temporary AWS credentials now work for ClickHouse staging.read_csv (#3743 @biefan) — File is opened with the requested encoding so SFTP/paramiko stacks no longer pre-decode as UTF-8.trace.asdict() now retains pipelines that fail in the sync step before extract.lance extension promotion to built-in in duckdb 1.5.PermissionError in rename_tree (#3853 @burnash) — Resolves intermittent Windows CI failures during normalize→loaded rename.llm-native-workflow.md.hf login command (#3781 @julien-c)mypy configs to pyproject.toml (#3780 @zilto) — Partially resolves #3346.pipeline.dataset() now includes every schema. pipeline.dataset() without a schema argument used to return only the default schema's tables. With multiple schemas it now includes them all by default, so queries may see extra tables or rows in shared table names. Restore the old behavior with pipeline.dataset(schema=pipeline.default_schema_name) (#3770).lance destinationLoad data into the Lance columnar format on local disk or s3, gs, and az object storage, with optional vector embeddings generated through lancedb (#3810). It uses the Lance Directory Namespace V2 catalog and supports branching. Install with pip install "dlt[lance]", then point a pipeline at it. This complements the existing lancedb destination, which targets LanceDB Cloud.
import dlt
pipeline = dlt.pipeline(
pipeline_name="movies",
destination="lance",
dataset_name="movies_db",
)
info = pipeline.run(
[{"id": 1, "title": "Blade Runner", "year": 1982}],
table_name="movies",
)
A pipeline that loads several sources now exposes all their tables through one pipeline.dataset(), so you can query across sources together (#3770). Tables that share a name across schemas have their columns merged and rows combined, with missing columns filled as NULL. Pass a schema name to scope to one source, and read its load history per schema.
import dlt
pipeline = dlt.pipeline(pipeline_name="my_pipeline", destination="duckdb")
# includes tables from every schema the pipeline loaded
dataset = pipeline.dataset()
# scope to a single schema (the pre-1.25.0 default)
one_source = pipeline.dataset(schema="github")
# load history is tracked per schema
load_ids = dataset.load_ids(schema_name="github")
Load metrics are now persisted to the load package, so a load that is interrupted and resumed keeps the metrics of already-completed jobs instead of losing them, and each job records the follow-up jobs it creates. That follow-up graph is saved into the pipeline trace, closing the long-standing #853 (#3768).
Big thanks to our newest contributors:
Custom resource metrics now stored as tables (#3718 @rudolfix) — Incremental metrics in the trace are now represented in table format. This changes th
insert-only merge strategy that performs idempotent, key-based appending: inserts records whose primary key doesn't exist in the destination while silently skipping duplicates. No updates or deletes. Supported across all SQL destinations, Delta Lake, and Iceberg.parallel and parallel-isolated decompose modes, all source components now fan out concurrently from a shared start node. Previously the first source had to complete before others could begin, adding unnecessary wall-clock time. This release also adds basic Airflow 3 support with smoke tests.replacing_merge_tree table engine type for ClickHouse that enables native deduplication and soft deletes via dedup_sort and hard_delete column hints.arrow_concat_promote_options can now be set to "default" or "permissive" instead of the hardcoded "none", enabling automatic type promotion when yielding multiple Arrow tables with slightly different inferred types.dlt pipeline info/show no longer crashes with UnknownDestinationModule on pipelines using @dlt.destination.primary_key=() to Incremental to disable deduplication is no longer silently overwritten by the resource's own primary key.md:) now raise a clear configuration error instead of a confusing connection failure.mask_columns() function supporting all sql_database backends.unfiltered_items_count and initial_unique_hashes_count no longer sit as scalar keys under a resource's custom_metrics. They now live in an incremental child table of the extract resource_metrics. Update code that reads them from the trace (#3718).insert-only merge strategyA fourth merge strategy joins delete-insert, scd2, and upsert. insert-only inserts a record when its primary key is not yet in the destination and skips the record when the key already exists, never updating or deleting. It suits append-only data like events, logs, and transactions, and gives idempotent, faster loads by skipping UPDATE operations. Works on all SQL destinations, Delta, Iceberg, and LanceDB (#3741).
import dlt
@dlt.resource(
write_disposition={"disposition": "merge", "strategy": "insert-only"},
primary_key="event_id",
)
def events():
yield [{"event_id": 1, "name": "login"}, {"event_id": 2, "name": "click"}]
ClickHouse tables can now use the replacing_merge_tree engine to deduplicate and soft-delete at the storage layer, without dlt's merge write disposition. Select it through clickhouse_adapter and load with append. The dedup_sort column hint picks the version to keep and hard_delete marks rows for removal, and dlt wires both into the engine parameters (#3366).
import dlt
from dlt.destinations.adapters import clickhouse_adapter
@dlt.resource(
write_disposition="append",
primary_key="id",
columns={
"version": {"dedup_sort": "desc", "nullable": False},
"is_deleted": {"hard_delete": True, "nullable": False},
},
)
def events(data):
yield from data
clickhouse_adapter(
events,
table_engine_type="replacing_merge_tree",
settings={"clean_deleted_rows": "Always"},
)
PipelineTasksGroup.add_run gains a serialize_first_task argument for the parallel and parallel-isolated decompose modes. Set it to False to fan out every decomposed source component concurrently from a shared start node, rather than waiting for the first component to finish. The default True keeps the old behavior, running the first component first to create the schema. This release also adds basic Airflow 3 support (#3652).
import dlt
from dlt.helpers.airflow_helper import PipelineTasksGroup
@dlt.source
def smoke_source():
@dlt.resource
def users():
yield [{"id": 1, "name": "Alice"}]
return users
tasks = PipelineTasksGroup("dlt_smoke_tasks", wipe_local_data=True)
pipeline = dlt.pipeline(pipeline_name="dlt_smoke", destination="duckdb")
tasks.add_run(
pipeline,
smoke_source(),
decompose="parallel",
serialize_first_task=False,
)
When a resource yields several Arrow tables or pandas DataFrames, dlt concatenates them and by default requires identical schemas, so any type difference across batches raises an error. Set arrow_concat_promote_options to "default" to fill missing columns with nulls, or "permissive" to also promote across type families such as int64 to double (#3701).
[data_writer]
arrow_concat_promote_options = "permissive"
DATA_WRITER__ARROW_CONCAT_PROMOTE_OPTIONS=permissive
Big thanks to our newest contributors:
Streamlit dashboard removed (#3674 @rudolfix) — The legacy Streamlit-based pipeline dashboard (dlt pipeline show) has been removed. It was a dead code
Streamlit dashboard removed (#3674 @rudolfix) — The legacy Streamlit-based pipeline dashboard (dlt pipeline show) has been removed. It was a dead code for a long time.
New sources.<name>.<key> configuration lookup path (#3626 @rudolfix) — Source configuration now supports a compact layout. When a source's section name differs from its resource/source name, dlt now also looks up sources.<name>.<key> in addition to the full sources.<section>.<name>.<key> path. For example, for a source registered under section chess_com with name chess:
# Before (still works): full qualified path
[sources.chess_com.chess]
api_key = "secret"
# New (also works now): compact path using just the source name
[sources.chess]
api_key = "secret"
# Credentials follow the same pattern:
# Full: sources.chess_com.chess.credentials.api_key
# Compact: sources.chess.credentials.api_key
This is breaking if you previously had values at sources.<name> that were unrelated to this source — they will now be resolved where they were previously ignored.
AI Workbench (#3674 @rudolfix) — New dlt ai CLI command group that turns dlt workspaces into AI-assisted development environments. Includes toolkit system for installing curated skill/rule bundles, pluggable MCP server architecture with composable features (pipeline, workspace, toolkit, secrets), and multi-agent support (Claude Code, Cursor, Codex).
Relational normalizer optimization (#3626 @rudolfix) — Major performance improvements to JSON data normalization and schema evolution: 5x faster on flat data, ~2x on nested REST API data, ~1.8x on wide nested data. ISO timestamp parsing improved 2-3x by removing timezone conversions.
Iceberg table properties (#3699 @rudolfix) — Adds support for setting Iceberg table and namespace properties via the adapter and configuration.
override_data_path option to DuckLake ATTACH (#3709 @udus122) — New override_data_path configuration option that appends OVERRIDE_DATA_PATH true to the ATTACH statement, allowing the current DATA_PATH to override the path stored in catalog metadata.PageNumberPaginatorConfig, OffsetPaginatorConfig, and JSONResponseCursorPaginatorConfig.os.path.commonprefix() with os.path.commonpath() in FileStorage.is_path_in_storage() to correctly validate path containment using path segments instead of characters.dev_mode flag in pipeline local state so it persists across dlt.attach() calls. Detects dev→non-dev transitions and resets working folder cleanly.HF_ENDPOINT env var for card operations.start_out_of_range flag with range_start="open" (#3708 @AyushPatel101) — Correctly sets start_out_of_range=True when a row's cursor value equals start_value with range_start="open", fixing delayed can_close() in descending-order pipelines.dataset_name=None (#3710 @Travior) — Handles the case where dataset_name is None in LanceDBSqlClient.create_view, preventing None prefix in view names.dlt pipeline show now generates and launches the marimo-based workspace dashboard instead, which requires marimo to be installed.The dltHub AI Harness ships as the dlthub ai command group, installed with pip install dlt[hub] (#3674). Run dlthub ai init to configure a coding agent (Claude Code, Cursor, or Codex) with dlt rules and skills, install curated toolkits with dlthub ai toolkit, and run a pluggable MCP server with dlthub ai mcp. In the base package, dlt ai redirects here.
pip install "dlt[hub]"
dlthub ai init --agent claude
Set Iceberg table properties per resource with iceberg_adapter(table_properties=...), or as defaults for every table via iceberg_table_properties on the filesystem destination (#3699). Namespace properties use iceberg_namespace_properties. Properties apply only when the table or namespace is first created, and adapter values win over destination defaults on conflicting keys.
import dlt
from dlt.destinations.adapters import iceberg_adapter
@dlt.resource(table_format="iceberg")
def my_data():
yield [{"id": 1, "value": "a"}]
iceberg_adapter(
my_data,
table_properties={"write.format.default": "parquet"},
)
pipeline = dlt.pipeline("iceberg_props", destination="filesystem")
pipeline.run(my_data)
dlt's relational normalizer and schema evolution are substantially faster (#3626): the maintainers benchmark roughly 5x on flat data, about 2x on nested REST API data, about 1.8x on wide nested data, and 2 to 3 times faster ISO timestamp parsing. Every pipeline that normalizes JSON benefits automatically.
When dlt runs inside a Databricks notebook without explicit server_hostname or http_path, it now derives server_hostname from the workspace URL and builds http_path from the notebook's own cluster, instead of defaulting to a SQL warehouse (#3667). Warehouse discovery remains the fallback when no cluster context is available.
Big thanks to our newest contributors:
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Remove outdated Motherduck troubleshooting (#3683 @elviskahoro) — Removed read-only database troubleshooting section for deprecated DuckDB versions.
hf protocol support to the filesystem destination, enabling direct loading to Hugging Face datasets. Closes #1227.feat(workspace): add default exclude patterns for file selector (#3661 @canassa) — WorkspaceFileSelector now ships with DEFAULT_EXCLUDES (.git/, .venv
WorkspaceFileSelector now ships with DEFAULT_EXCLUDES (.git/, .venv/, __pycache__/, node_modules/, etc.) so well-known non-deployable paths are always excluded, even without a .gitignore.ignore_file_found attribute to WorkspaceFileSelector (#3663 @canassa) — Consumers can now check whether the configured ignore file (e.g. .gitignore) was actually found.utils.py and dlt_dashboard.py into focused modules with simplified UI across all sections.sse for http-stream transport for built-in MCP servers and annotates pipeline trace schema.sources.<name>.api_key..to_mermaid() now handles columns missing the data_type field instead of crashing.select_sequential_consistency to fix flaky tests caused by ClickHouse's eventual consistency model.dlthub changes.shutil docs.dlthub metrics section; update checks (#3641 @zilto)docs/ linting in one make command (#3666 @anuunchin) — Introduces an overarching lint target in the docs Makefile. Resolves #3642.source.md file.make test-load-local-p (#3645 @tetelio) — Convenience make install target for local load tests on duckdb and filesystem.Pydantic v1 support removed (#3572 @anuunchin) — All Pydantic v1 compatibility code has been removed. The codebase now requires Pydantic v2 only.
data_type contract semantic change (#3572 @anuunchin @rudolfix) — The data_type contract now applies to full data type (ie. precision, nullability), not only to variant columns (data type change). Users with data_type: freeze who relied on changing nullable/precision/scale on existing columns will now be blocked.merge_columns now removes compound properties (#3431 @anuunchin) — Previously merge_columns was purely additive, which caused compound properties like merge_key to be incorrectly replaced rather than properly merged. The function now correctly removes compound properties that should be removed.RootModel types (validation of event streams with various event types), schema contracts properly separate resource-defined vs data-derived hints, Pydantic model columns bypass contract checks when authoritative. Supports Pydantic models on arrow and model items with full schema contract enforcement. Prepares for Pydantic v3.ALTER TABLE ... SWAP for staging-optimized replace strategy on Snowflake, eliminating table downtime during data replacement.sql_database (#3595 @rudolfix) — Register custom TableLoader implementations as named backends. ConnectorX backend ported as PoC; ADBC and paginated loader implemented as test cases.llms.txt and Markdown docs generation (#3635 @rudolfix) — Generates llms.txt index and Markdown versions of docs pages with a "View Markdown" navigation option, making the docs LLM-friendly.rest_api: parallelized dependent resources (#3574 @Shadesfear) — Add parallelized flag to dependent resources (transformers) so child resource fetches run concurrently.dlt.Relation: filter by load_id (#3547 @zilto) — Filter dataset relations by load ID (experimental).dlt.Relation: flatten logic and improve typing (#3578 @zilto) — Remove dynamic methods; explicit return types for .df(), .arrow(), etc.SourceFactory (#3636 @rudolfix) — Add preprocessor hooks to dlt.source factory for modifying source instances.engine_kwargs for sql_database/sql_table sources (#3414 @tetelio) — Pass SQLAlchemy engine arguments directly to create_engine() for sources.DECFLOAT columns via the SQLAlchemy backend.query_result_bucket now optional (#3566 @arel) — Omit or set to None when using Athena's managed results bucket.extra_credentials for S3 (#2888 @warje) — Adds extra_credentials config for role-based S3 authentication._dlt_load_id written as dict on MSSQL + ADBC (#3584 @rudolfix)CREATE OR REPLACE for merge temp tables (#3589 @rudolfix)read_csv_duckdb respects filename=True (#3606 @karlanka)sql_database (#3638 @rudolfix)dlt init (#3615 @rudolfix)ibis-framework, remove sqlglot constraint (#3621 @Travior)pytest-xdist with fully isolated workers..zed/ directory (#3633 @zilto)data_type contract now covers full column type. The data_type schema contract now applies to changes in precision, scale, nullability, and timezone on existing complete columns, not only to variant columns (#3572). Pipelines with data_type set to freeze that changed these properties on existing columns will now be blocked.merge_columns previously merged additively, so compound hints like merge_key and primary_key were added rather than replaced (#3431). Redefining these hints on an already extracted resource now replaces the previous configuration instead of accumulating old and new keys.The Pydantic support was reworked (#3572). A RootModel wrapping a discriminated union validates a single resource that emits different item types and dispatches each to its own table, resolving the variant by its discriminator field. Mark the model authoritative with DltConfig, then route items with dlt.mark.with_table_name.
from typing import ClassVar, Literal, Union
from typing_extensions import Annotated
from pydantic import BaseModel, Field, RootModel
from dlt.common.libs.pydantic import DltConfig
import dlt
class EventBase(BaseModel):
kind: str
id: int
class ClickEvent(EventBase):
kind: Literal["click"]
element_id: str
class PurchaseEvent(EventBase):
kind: Literal["purchase"]
amount: float
EventUnion = Annotated[Union[ClickEvent, PurchaseEvent], Field(discriminator="kind")]
class Event(RootModel[EventUnion]):
dlt_config: ClassVar[DltConfig] = {"is_authoritative_model": True}
TABLE_MAP = {"click": "click_events", "purchase": "purchase_events"}
@dlt.resource(name="events", columns=Event, schema_contract={"data_type": "discard_row"})
def event_stream(items):
for item in items:
yield dlt.mark.with_table_name(item, TABLE_MAP[item["kind"]])
rest_apiSet parallelized to True on a dependent resource (transformer) so child fetches for many parent items run concurrently in dlt's thread pool instead of one after another (#3574). All pages for a single parent are collected in memory before being yielded, so expect higher memory use for parents with many child pages.
import dlt
from dlt.sources.rest_api import rest_api_source
source = rest_api_source({
"client": {"base_url": "https://api.example.com"},
"resources": [
"posts",
{
"name": "post_comments",
"parallelized": True,
"endpoint": {"path": "posts/{resources.posts.id}/comments"},
},
],
})
dataset.table() now takes a load_ids argument to return only rows from specific loads, joining nested tables back to their root when needed (#3547). List available loads with dataset.load_ids() and get the newest with dataset.latest_load_id(). The feature is experimental.
import dlt
pipeline = dlt.pipeline("my_pipeline", destination="duckdb")
dataset = pipeline.dataset()
all_loads = dataset.load_ids() # every load id in the dataset
recent_users = dataset.table("users", load_ids=all_loads[-1:])
df = recent_users.df()
With enable_atomic_swap set on the Snowflake destination, the staging-optimized replace strategy swaps production and staging tables using ALTER TABLE ... SWAP WITH instead of cloning them (#3540). The swap is near-instant, so tables stay available during replacement. Each table keeps its existing permissions, and dlt does not drop the old staging table.
import dlt
pipeline = dlt.pipeline(
pipeline_name="my_pipeline",
destination=dlt.destinations.snowflake(enable_atomic_swap=True),
)
When a source's section differs from its name, for example after .clone(name=..., section=...), dlt now accepts a compact sources.<name>.<key> config path in addition to the full sources.<section>.<name>.<key> (#3664). The full path and the section path still take precedence.
# compact layout: just the source name
[sources.my_db.credentials]
password = "..."
# full layout: section + name, takes precedence
[sources.my_db_module.my_db.credentials]
password = "..."
hf:// protocolThe filesystem destination now supports the hf:// protocol, loading directly to Hugging Face dataset repositories (#3669). Install dlt[hf], point bucket_url at hf://datasets/<namespace>, and dlt creates one repo per dataset, commits each table atomically, and writes a dataset card with per-table subsets (#3689) for the dataset viewer.
[destination.filesystem]
bucket_url = "hf://datasets/my-org"
[destination.filesystem.credentials]
hf_token = "hf_..."
import dlt
@dlt.resource
def my_data():
yield [{"id": 1, "name": "Alice"}, {"id": 2, "name": "Bob"}]
pipeline = dlt.pipeline(
pipeline_name="my_pipeline",
destination="filesystem",
dataset_name="my_dataset",
)
pipeline.run(my_data(), write_disposition="append")
Big thanks to our newest contributors:
Nothing published for this version
This release adds several interesting improvements and many bugfixes. Lancedb destination now uses duckdb extension to let you query lance tables with
This release adds several interesting improvements and many bugfixes. Lancedb destination now uses duckdb extension to let you query lance tables with SQL, ibis or sqlglot via our standard .dataset() interface. We introduced several iceberg-relates improvements (catalog support, s3 tables for Athena, advanced partitioning). There's also new fabric destination and additional options in `clickhouse_adapter. Finally: we have test environment for Oracle and we stared to fix Oracle related bugs.
SqlClientBase - query lance tables with ibis, sqlglot or raw SQL by @zilto in https://github.com/dlt-hub/dlt/pull/3527athena destination by @jorritsandbrink in https://github.com/dlt-hub/dlt/pull/3434fabric destination (by @mattiasthalen) by @jorritsandbrink in https://github.com/dlt-hub/dlt/pull/3535clickhouse_adapter extensions by @jorritsandbrink in https://github.com/dlt-hub/dlt/pull/3511Full Changelog: https://github.com/dlt-hub/dlt/compare/1.20.0...1.21.0
feat: implement ConfigurationFileSelector by @ivasio in https://github.com/dlt-hub/dlt/pull/3418
filesystem by @ivasio in https://github.com/dlt-hub/dlt/pull/3339JSONResponseCursorPaginator by @segetsy in https://github.com/dlt-hub/dlt/pull/3374Full Changelog: https://github.com/dlt-hub/dlt/compare/1.19.1...1.20.0
Nothing published for this version
Nothing published for this version
fixes arrow import in sql_database by @rudolfix in https://github.com/dlt-hub/dlt/pull/3411
Full Changelog: https://github.com/dlt-hub/dlt/compare/1.19.0...1.19.1
Feat: support return_type = arrow_stream for connectorx backend by @ivasio in https://github.com/dlt-hub/dlt/pull/3218
Schema.to_mermaid() by @zilto in https://github.com/dlt-hub/dlt/pull/3364snowflake clustering key modifications by @jorritsandbrink in https://github.com/dlt-hub/dlt/pull/3365with_table_name and other functions available through `dlt.pip… by @hello-world-bfree in https://github.com/dlt-hub/dlt/pull/3318query_adapter_callback by @dat-a-man in https://github.com/dlt-hub/dlt/pull/3253data_quality concept page by @zilto in https://github.com/dlt-hub/dlt/pull/3341dlt/extract/hints.py module by @luqmansen in https://github.com/dlt-hub/dlt/pull/3332Full Changelog: https://github.com/dlt-hub/dlt/compare/1.18.2...1.19.0
You can now return results from sql_database queries as an Arrow stream when using the ConnectorX backend.
This allows large query results to be processed incrementally, avoiding loading the full result set into memory.
Example:
from dlt.sources.sql_database import sql_database
db = sql_database(
backend="connectorx",
backend_kwargs={
"return_type": "arrow_stream", # new in 1.19
},
)
By default, ConnectorX returns PyArrow tables. Arrow streaming must be explicitly enabled.
The dashboard now includes a visual history of pipeline runs, making it easier to inspect run status, duration, and changes over time.
This provides a clearer overview of pipeline health and helps diagnose failures faster.
dlt can now ingest Parquet files into SQL databases (MSSQL, MySQL, and SQLite) using ADBC.
When an ADBC driver is available, Parquet loading is enabled automatically and becomes the preferred method. This delivers a 10×–100× speedup compared to INSERT-based loading and is more reliable than CSV fallbacks.
If needed, you can explicitly revert to INSERT loading:
pipeline.run(
iter(()), # your data generator
loader_file_format="insert_values",
)
Schema.to_mermaid()You can now export any dlt schema as a Mermaid diagram for quick visualization in documentation, pull requests, or onboarding materials.
schema_mermaid = pipeline.default_schema.to_mermaid()
Schemas can also be exported from the CLI and rendered natively in tools like GitHub Markdown and Notion.
Snowflake destinations now support updating clustering keys using column hints.
Clustering changes are applied when a table alteration is triggered (for example, when a new column is added), making it easier to tune clustering for large tables without recreating them.
Example:
@dlt.resource(table_name="events")
def events():
yield {"event_id": 1, "country": "DE"}
events.apply_hints(columns=[{"name": "event_id", "cluster": True}])
pipeline.run(events())
Big thanks to our newest contributors:
resolves "default" limit via ibis options by @rudolfix in https://github.com/dlt-hub/dlt/pull/3273
Full Changelog: https://github.com/dlt-hub/dlt/compare/1.18.1...1.18.2
fixes git import and enables tests by @rudolfix in https://github.com/dlt-hub/dlt/pull/3262
Full Changelog: https://github.com/dlt-hub/dlt/compare/1.18.0...1.18.1
A few cool feats in this release that also need your attention if you use them:
A few cool feats in this release that also need your attention if you use them:
databricks destination now supports iceberg table format, cluster and partition hints. note that previously they were ignored and mixing of cluster and partition is not allowed. in super rare cases you can get error messages that were not there previouslydlt to shut down load pools gracefully (previously we were raising exceptions). with console attached double CTRL-C will raise immediately. we also support graceful shutdowns for pipelines that run in thread pools. this is a behavioral change compared to 1.17.0dlt.pipeline(destination="warehouse") and via dlt.destination which now serves both as decorator and a factory, overall super helpful when you go from dev to production but in rare cases you may get linter and runtime error on custom destinations @dlt.destination that were not using kwargs to pass options to decoratorRelation got to_ibis method that works both on tables and queries and which uses dlt as a backed for ibis so you can read data from them_workspace modules where all code for cli, dashboards, mcp (and other things that you typically not use in runtime). there were changes in internal private interfaces. as a user you do not to worry about itdlt.Relation API and create bound Ibis tables by @zilto in https://github.com/dlt-hub/dlt/pull/3179pandas deps by @zilto in https://github.com/dlt-hub/dlt/pull/3157_workspace module by @rudolfix in https://github.com/dlt-hub/dlt/pull/3215TTableReference by @zilto in https://github.com/dlt-hub/dlt/pull/3093cluster by in bigquery on alter statements by @adrian-173 in https://github.com/dlt-hub/dlt/pull/3239.table(..., table_type="ibis") with .to_ibis() by @zilto in https://github.com/dlt-hub/dlt/pull/3225_workspace module by @rudolfix in https://github.com/dlt-hub/dlt/pull/3215pyproject.toml and reduce verbosity by @zilto in https://github.com/dlt-hub/dlt/pull/3205Full Changelog: https://github.com/dlt-hub/dlt/compare/1.17.1...1.18.0
@dlt.destination factoryYou can now reference destinations by name directly from your workspace configuration. This makes it easy to switch between development, staging, and production environments without changing pipeline code.
If your .dlt/config.toml contains:
[destination.custom_name]
destination_type = "duckdb"
You can activate it like this:
import dlt
@dlt.resource
def my_data():
yield {"id": 1}
pipeline = dlt.pipeline(destination="custom_name")
pipeline.run(my_data)
The destination type and all related settings are resolved automatically from the configuration.
The Databricks destination now supports the Iceberg table format, along with full support for partitioning and clustering hints.
Previously, these hints were silently ignored. With dlt 1.18, they are now enforced, and invalid combinations (such as mixing partitioning with clustering) are rejected early to prevent incorrect table layouts.
Example:
import dlt
from dlt.destinations.adapters import databricks_adapter
pipeline = dlt.pipeline(pipeline_name="to_databricks", destination="databricks")
@dlt.resource
def my_data():
for i in range(10):
yield {
"event_id": i,
"year": 2024,
"month": (i % 3) + 1,
"customer_id": i % 5,
}
# Enable clustering
databricks_adapter(my_data, cluster="AUTO")
# Enable partitioning
databricks_adapter(my_data, partition=["year", "month"])
# Use Iceberg table format
databricks_adapter(my_data, table_format="ICEBERG")
pipeline.run(my_data)
Signal handling in pipelines has been fully reworked. Pipelines now shut down gracefully when interrupted, instead of raising exceptions.
Ctrl+C triggers a clean shutdownThis applies to both process-based and threaded loaders and makes running long or production workloads much safer.
add_metrics()You can now attach custom metric collectors directly to resources. This allows you to track things like batch counts, API pages, or any other extraction-level signal without modifying the data itself.
Example: counting how many batches a resource produces:
import dlt
def batch_counter(items, meta, metrics):
metrics["batch_count"] = metrics.get("batch_count", 0) + 1
@dlt.resource
def purchases():
for i in range(3):
yield [{"id": i}]
purchases = purchases.add_metrics(batch_counter)
pipeline = dlt.pipeline("metrics_demo", destination="duckdb")
load_info = pipeline.run(purchases)
trace = pipeline.last_trace
load_id = load_info.loads_ids[0]
resource_metrics = trace.last_extract_info.metrics[load_id][0]["resource_metrics"]["purchases"] # ty: ignore
print("Custom metrics:", resource_metrics.custom_metrics)
Output:
Custom metrics: {'batch_count': 3}
Custom metrics are stored alongside performance and transform statistics under resource_metrics and are available in traces and dashboards.
The filesystem source can now read files directly from HTTP URLs, in addition to local paths and object storage such as S3.
This makes it easy to load publicly hosted datasets without setting up extra infrastructure.
Example:
import dlt
from dlt.sources.filesystem import filesystem, read_csv
pipeline = dlt.pipeline("metrics_demo", destination="duckdb")
fs = filesystem(bucket_url="https://example.com/data/", file_glob="*.csv")
pipeline.run(fs | read_csv())
Big thanks to our newest contributors:
Nothing published for this version
This patch release mostly addresses bugs and inconsistencies found in new ducklake destination. The most significant change was to rename `catalog_nam
This patch release mostly addresses bugs and inconsistencies found in new ducklake destination. The most significant change was to rename catalog_name to ducklake_name in destination.ducklake configuration in https://github.com/dlt-hub/dlt/pull/3153
postgresql by postgres in ATTACH (ducklake) by @zilto in https://github.com/dlt-hub/dlt/pull/3148read_parquet(use_arrow: bool) by @zilto in https://github.com/dlt-hub/dlt/pull/3149CONTRIBUTING.md by @jorritsandbrink in https://github.com/dlt-hub/dlt/pull/3135test_track_anon_event test by @rudolfix in https://github.com/dlt-hub/dlt/pull/3152Full Changelog: https://github.com/dlt-hub/dlt/compare/1.17.0...1.17.1
dashboard: fixes file opener on WSL by @rudolfix in https://github.com/dlt-hub/dlt/pull/3076
None if child table exists by @sh-rp in https://github.com/dlt-hub/dlt/pull/3048BaseOperator in airflow_helper.py (#2601) by @ianedmundson1 in https://github.com/dlt-hub/dlt/pull/3043filesystem via Incremental sort_order https://github.com/dlt-hub/dlt/pull/2737sql_client.raise_database_error creates circular __cause__ dependency by @anuunchin in https://github.com/dlt-hub/dlt/pull/3111ducklake destination (all buckets and catalog combinations supported) by @zilto in https://github.com/dlt-hub/dlt/pull/3015json and timestamp without timezone in pg_replication @anuunchin https://github.com/dlt-hub/verified-sources/pull/657sql_database and filesystem with examples and additional tests by @rudolfix in https://github.com/dlt-hub/dlt/pull/2737ducklake destination documentation by @rudolfix in https://github.com/dlt-hub/dlt/pull/3015Full Changelog: https://github.com/dlt-hub/dlt/compare/1.16.0...1.17.0
You can now use the DuckLake destination — supporting all bucket and catalog combinations. It’s a great fit for lightweight data lakes and local development setups.
You can now collect custom metrics directly inside your resources and transform steps. This makes it easy to track things like page counts, skipped rows, or API calls — right where the data is extracted.
Use dlt.current.resource_metrics() to store custom values while your resource runs. These metrics are automatically merged into the pipeline trace and visible in the run summary.
Example:
import dlt
from dlt.sources.helpers.rest_client import RESTClient
from dlt.sources.helpers.rest_client.paginators import JSONLinkPaginator
client = RESTClient(
base_url="https://pokeapi.co/api/v2",
paginator=JSONLinkPaginator(next_url_path="next"),
data_selector="results",
)
@dlt.resource
def get_pokemons():
metrics = dlt.current.resource_metrics()
metrics["page_count"] = 0
for page in client.paginate("/pokemon", params={"limit": 100}):
metrics["page_count"] += 1
yield page
pipeline = dlt.pipeline("get_pokemons", destination="duckdb")
load_info = pipeline.run(get_pokemons)
print("Custom metrics:", pipeline.last_trace.last_extract_info.metrics) # ty: ignore
Custom metrics are grouped together with performance and transform stats under resource_metrics, so you can view them easily in traces or dashboards.
When working with large datasets, you can now limit how much data a resource loads using the upgraded add_limit method. This is perfect for sampling a few records to preview your data or test transformations faster.
Example:
import itertools
import dlt
# Load only the first 10 items from an infinite stream
r = dlt.resource(itertools.count(), name="infinity").add_limit(10)
You can also:
Count rows instead of yields:
my_resource().add_limit(10, count_rows=True)
Or stop extraction after a specific time in seconds:
my_resource().add_limit(max_time=10)
It’s a simple but powerful way to test pipelines quickly without pulling millions of rows.
You can now use incremental loading with the filesystem source even easier — ideal for tracking updated or newly added files in S3 or local folders.
dlt detects file changes (using fields like modification_date) and loads only what’s new.
Example:
import dlt
from dlt.sources.filesystem import filesystem, read_parquet
filesystem_resource = filesystem(
bucket_url="s3://my-bucket/files",
file_glob="**/*.parquet",
incremental=dlt.sources.incremental("modification_date")
)
pipeline = dlt.pipeline("my_pipeline", destination="duckdb")
pipeline.run((filesystem_resource | read_parquet()).with_name("table_name"))
You can also split large incremental loads into smaller chunks:
row_order, files_per_page, or add_limit().This makes it easy to backfill large file collections efficiently and resume incremental updates without reloading everything.
When working with huge tables, you can now split incremental loads into smaller chunks or partition backfills into defined ranges. This makes data appear faster and allows you to retry only failed chunks instead of reloading everything.
If your source returns data in a deterministic order (for example, ordered by created_at), you can combine incremental with add_limit() to process batches sequentially:
import dlt
from dlt.sources.sql_database import sql_table
pipeline = dlt.pipeline("split_load", destination="duckdb")
messages = sql_table(
table="chat_message",
incremental=dlt.sources.incremental(
"created_at",
row_order="asc", # required for split loading
range_start="open" # disables deduplication
),
)
# Load one-minute chunks until done
while not pipeline.run(messages.add_limit(max_time=60)).is_empty:
pass
You can also load large datasets in parallel partitions using initial_value and end_value. Each range runs independently, helping you rebuild large tables safely and efficiently.
Together, these methods make incremental loading more flexible and robust for both testing and production-scale pipelines.
Big thanks to our newest contributors:
Improved timestamp handling, please carefully read https://dlthub.com/docs/general-usage/schema#handling-of-timestamp-and-time-zones. For some edge ca
dlt.Relation and dlt.Datasetdlt dashboard to see it. Learn more about how it works here: https://dlthub.com/docs/general-usage/dashboardMissingDependencyException should inherit ImportError by @zilto in https://github.com/dlt-hub/dlt/pull/2977dlt.Schema.to_dot() graphviz export by @zilto in https://github.com/dlt-hub/dlt/pull/2959dlt.Pipeline.__repr__ by @zilto in https://github.com/dlt-hub/dlt/pull/3022dlt.Dataset and dlt.Relation by @zilto in https://github.com/dlt-hub/dlt/pull/3059ruff check for linting by @zilto in https://github.com/dlt-hub/dlt/pull/2967__marimo__/ folders by @zilto in https://github.com/dlt-hub/dlt/pull/3008Full Changelog: https://github.com/dlt-hub/dlt/compare/1.15.0...1.16.0
Our free education courses are now part of the official documentation and can be launched directly in Google Colab.
Try them here!
This release brings a major cleanup and unification of how dlt handles timestamps and timezones. Both tz-aware and naive timestamps are now processed consistently across normalizers and destinations. dlt now also preserves the exact timestamp type when using incremental cursors, preventing subtle mismatches during reloads.
It also fixes several tricky edge cases — such as nanosecond precision in MSSQL — and ensures that all time types behave as documented (always naive, in UTC). Destinations now explicitly declare which timestamp formats they support, so dlt adjusts automatically.
Example:
@dlt.resource(
name='my_table',
columns={
"my_column": {
"data_type": "timestamp",
"timezone": True}}
)
def my_resource():
...
# Output:
# naive timestamp → UTC tz-aware
# "2024-05-01 12:00:00" → "2024-05-01 12:00:00+00:00"
# tz-aware timestamp with timezone=False → UTC converted, then naive
# "2024-05-01 12:00:00+02:00" → "2024-05-01 10:00:00"
The result: predictable, precise timestamp behavior across all sources, transformations, and destinations.
New beta dashboard
The Streamlit app is now replaced by the dlt Dashboard. Run dlt dashboard to explore your pipelines with an improved interface and real-time insights. Learn more here.
Export schema graphs with dlt.Schema.to_dot(). More about schema export here.
Big thanks to our newest contributors:
This version will add .gz extensions to files that are compressed. That includes filesystem destinations, internal working directory and staging locat
This version will add .gz extensions to files that are compressed. That includes filesystem destinations, internal working directory and staging locations used to feed other destinations. A few practical hints:
filesystem destination will continue storing files without gz extension and they are not affected by the change (existing datasets will retain their behavior where this extension is not added for backwards compatibility).gz extension, also if dlt is configured to keep data in stageparquet files.has_more boolean flag logic to RESTClient OffsetPaginator by @michaelconan in https://github.com/dlt-hub/dlt/pull/2817__repr__ for @dlt.transformation by @zilto in https://github.com/dlt-hub/dlt/pull/2940Schema.to_dbml(), auto export schemas in dbml format by @zilto in https://github.com/dlt-hub/dlt/pull/2929arrow2 with arrow backend for connectorx, enables newest connectorx versions by @zilto in https://github.com/dlt-hub/dlt/pull/2933streamed_exec in delta merge upsert by @anuunchin in https://github.com/dlt-hub/dlt/pull/2961Full Changelog: https://github.com/dlt-hub/dlt/compare/1.14.1...1.15.0
.gz extensions now added to compressed filesPR: #2835
Starting with v1.15, all compressed files will include the .gz extension by default. This applies to:
Notes
.parquet files.gz) for backward compatibility.gz extension, also if dlt is configured to keep data in stage.More details → File Compression in Filesystem Destination
PR: #2929
You can now export your pipeline schema in DBML format, ready for visualization in DBML frontends.
# Generate a string that can be rendered in a DBML frontend
dbml_str = pipeline.default_schema.to_dbml()
This includes:
has_more flag in paginatorsPR: #2817
Some APIs provide a has_more boolean instead of total or offset limits.
You can now configure paginators to use that flag.
from dlt.sources.helpers.rest_client.paginators import OffsetPaginator, PageNumberPaginator
# Paginator starts at offset=0, increments by `limit`
# Stops automatically when response["has_more"] == False
offset_paginator = OffsetPaginator(
offset=0,
limit=2,
has_more_path="has_more" # JSONPath to the boolean flag
)
# Paginator starts at page=1, increments by page
# Stops automatically when response["has_more"] == False
pn_paginator = PageNumberPaginator(
page=1,
has_more_path="has_more" # JSONPath to the boolean in response
)
PR: #2917
Pagination parameters can now be placed in the request body (e.g., for POST requests).
from dlt.sources.helpers.rest_client.paginators import OffsetPaginator, PageNumberPaginator
# We tell dlt to place pagination values INSIDE the request JSON body
offset_paginator = OffsetPaginator(
offset=0,
limit=3,
offset_body_path="offset", # put 'offset' in request.json["offset"]
limit_body_path="limit", # put 'limit' in request.json["limit"]
)
pn_paginator = PageNumberPaginator(
page=1,
page_body_path="page", # put 'page' in request.json["page"]
)
PR: #2674
New support for comments, tags, and constraints in Databricks Unity Catalog tables.
[destination.databricks]
# Add PRIMARY KEY and FOREIGN KEY constraints to tables
create_indexes = true
Use the adapter for column- and table-level hints:
from dlt.destinations.adapters import databricks_adapter
databricks_adapter(
my_resource,
table_comment="Event data",
table_tags=["pii", {"cost_center": "12345"}],
column_hints={
"user_id": {
"column_comment": "User ID",
"column_tags": ["pii"]
}
},
)
PR: #2951
Example of customizing a type mapper for Trino →
PR: #2961
By default, dlt runs Delta Lake upserts in streamed mode.
New option: deltalake_streamed_exec
Now you can explicitly choose whether to run Delta merge operations in streamed mode or not.
true (default): safe for very large datasets, lower memory usage.false: enables Delta to use table statistics and pruning (better performance when partitions and stats exist).[destination.filesystem]
deltalake_streamed_exec = false
PR: #2789
Python 3.14 is currently in beta4. We now provide experimental support for Python 3.14 to make it easier for projects depending on dlt to prepare for the upgrade.
arrow2 to arrowPR: #2933
ConnectorX dropped the arrow2 backend in v0.4.2, which was dlt’s default.
dlt now defaults to the arrow backend, with minor adjustments for behavioral differences, and relaxed ConnectorX version constraints.
PR: #2937
The AI command now supports all major IDEs, automatically creating the correct rule files for each AI editor/agent. This ensures seamless integration whether you’re using Cursor, Copilot, Windsurf, Claude, or other supported editors.
Big thanks to our newest contributors:
Breaking Changes If you used pipeline.dataset() and used ibis syntax to write queries please read below:
Breaking Changes
If you used pipeline.dataset() and used ibis syntax to write queries please read below:
pipeline.dataset() - with Ibis Expressions has slightly changed, you will need to update your existing codebase if you are using Ibis Expressions. If you were using the Dataset without having Ibis installed, or without using any Ibis features, no changes are needed.
Protocols by @zilto in https://github.com/dlt-hub/dlt/pull/2870Full Changelog: https://github.com/dlt-hub/dlt/compare/1.12.3...1.14.1
Nothing published for this version
Nothing published for this version
Extend CSV quoting options in CsvWriter by @burnash in https://github.com/dlt-hub/dlt/pull/2810
Full Changelog: https://github.com/dlt-hub/dlt/compare/1.12.3...1.13.0
Nothing published for this version
(feat) allows to add SQL statements to schema migration executed after tables were created/altered by @rudolfix in https://github.com/dlt-hub/dlt/pull
str and repr to dataset and relation by @rudolfix in https://github.com/dlt-hub/dlt/pull/2796@dlt.transformation to __all__ by @zilto in https://github.com/dlt-hub/dlt/pull/2797Nothing published for this version
This is a prelease of dlt and our first build with the uv package manager.
This is a prelease of dlt and our first build with the uv package manager.
Quality of Life (fixing annoying little things)
Quality of Life (fixing annoying little things)
import dlt by @djudjuu in https://github.com/dlt-hub/dlt/pull/2707LIMIT env variable which was skipped before@dlt.source
def source():
@dlt.resource(write_disposition="merge", primary_key="_id")
def documents(access_token=dlt.secrets.value, limit=10):
yield from generate_json_like_data(access_token, limit)
return documents
⚠️ Still we do not recommend to define parametrized inner resources.
dlt always wraps resources in generators so your return will be converted to yield.DltResource from a resource function you must explicitly type the return value:@dlt.resource
def rv_resource(name: str) -> DltResource:
return dlt.resource([1, 2, 3], name=name, primary_key="value")
wrap and unwrap of functions. our decorators preserve both typing and runtime signature of decorated functions. makefun got removed.Incremental initializes from another Incremental as native value, it copies original type correctlydlt.resource can define configuration section (also using lambdas)Bugfixes and improvements
delete-insert merge to decrease query cost in https://github.com/dlt-hub/dlt/pull/2721Chores & tech debt
We switch to uv in the coming days and:
__repr__() for public interface by @zilto in https://github.com/dlt-hub/dlt/pull/2630dataset for testing now)load_id col in _dlt_loads table by @zilto in https://github.com/dlt-hub/dlt/pull/2729🧪 Upgrades to data access
x-annotation hints are propagatedSqlModel represent SQL query and is processed in extract, normalize and loaded in load stepscalar() on data access expressions ie.# get latest processed package id
max_load_id = pipeline.dataset()._dlt_loads.load_id.max().scalar()
🧪 Cool experimental stuff:
Check out our new embedded pipeline explorer app
dlt pipeline <name> show --marimo
dlt pipeline <name> show --marimo --edit
use edit option to enable Notebook/edit mode in Marimo + very cool Ibis dataset explorer
We updated contribution guidelines
Full Changelog: https://github.com/dlt-hub/dlt/compare/1.11.0...1.12.0
This update includes substantial improvements to developer experience and stability, as well as bug fixes. Here's a breakdown of what's new:
PR: #2636
Overview
"<configure me>", or timestamps like 1768-07-21T02:56:00Z).Example placeholders
[credentials]
token = "<configure me>"
started_at = "1768-07-21T02:56:73Z" # random date
Example warning
Placeholder value encountered when resolving config or secret:
resolved_key: host, value: <configure me>, section: sources.sql_database.credentials.host
Most likely, this comes from init-command, which creates basic templates for non-complex configs and secrets. The provider to adjust is secrets.toml at one of these locations:
- /home/me/project/.dlt/secrets.toml
- /home/me/.dlt/secrets.toml
None-only columnsPR: #2633
Overview
None, dlt can’t infer its type.Example warning
The column `my_column` in table `my_table` did not receive any data during this load. Therefore, its type couldn't be inferred.
Unless a type hint is provided, the column will not be materialized in the destination.
Recommended fix
@dlt.resource(columns={"my_column": {"data_type": "text"}})
def my_data():
yield {"my_column": None}
Before
standalone=True argument.@dlt.resource(standalone=True)
def foo(access_token=dlt.secrets.value):
yield from generate_json_like_data(access_token)
[sources.foo]
access_token = "your_token"
Now
@dlt.resource functions behave identically.standalone=True is ignored and no longer required.sources.source_module.resource_name this is canonical form.Cleaner usage
@dlt.resource
def foo(access_token=dlt.secrets.value):
yield from generate_json_like_data(access_token)
For inner resources:
@dlt.source
def bar():
@dlt.resource
def foo(access_token=dlt.secrets.value):
yield from generate_json_like_data(access_token)
return foo
[sources.bar.foo]
access_token = "your_token"
return data (instead of yield)Overview
Example
@dlt.resource
def numbers():
return [{"key1": 1, "key2": 2}]
returnis automatically wrapped into a generator internally.
:::warning Not recommended for large-scale use due to readability and stream control. :::
PR: #2756
OR condition (n² complexity)delete-insert strategy, the generated SQL used:DELETE FROM main_table WHERE EXISTS (
SELECT 1 FROM staging_table
WHERE main.pk1 = staging.pk1 AND main.pk2 = staging.pk2
OR main.merge_key = staging.merge_key
);
OR logic in EXISTS, especially on large datasets.New approach
DELETE queries, executed in the same transaction:DELETE ... WHERE EXISTS (JOIN ON primary_key);
DELETE ... WHERE EXISTS (JOIN ON merge_key);
:::info ⚡ Improves performance from 18 hours → 4 seconds on large datasets. :::
delete-insert operations, further reducing merge cost.delete-insert.@dlt.resource(
primary_key="id",
write_disposition={
"disposition": "merge",
"strategy": "delete-insert",
"deduplicated": False
}
)
def data_resource():
yield from iter(()) # your data generator
upsert merge strategy addedPR: #2671
Overview
merge with upsert semantics, similar to Delta Lake.:::warning Known limitations due to current pyiceberg behavior:
parquet via ADBCPR: #2685
@dlt.source # your source
def crm():
yield ...
pipeline = dlt.pipeline(destination="postgres")
pipeline.run(crm, loader_file_format="parquet")
Nothing published for this version
Nothing published for this version
> We yanked this release from PyPI after discovering that the minimum allowed version of sqlglot could prevent dlt from being imported. This release h
[!IMPORTANT]
We yanked this release from PyPI after discovering that the minimum allowed version ofsqlglotcould preventdltfrom being imported. This release has been replaced by version1.12.1, which includes the same release notes.
Quality of Life (fixing annoying little things)
import dlt by @djudjuu in https://github.com/dlt-hub/dlt/pull/2707LIMIT env variable which was skipped before@dlt.source
def source():
@dlt.resource(write_disposition="merge", primary_key="_id")
def documents(access_token=dlt.secrets.value, limit=10):
yield from generate_json_like_data(access_token, limit)
return documents
⚠️ Still we do not recommend to define parametrized inner resources.
dlt always wraps resources in generators so your return will be converted to yield.DltResource from a resource function you must explicitly type the return value:@dlt.resource
def rv_resource(name: str) -> DltResource:
return dlt.resource([1, 2, 3], name=name, primary_key="value")
wrap and unwrap of functions. our decorators preserve both typing and runtime signature of decorated functions. makefun got removed.Incremental initializes from another Incremental as native value, it copies original type correctlydlt.resource can define configuration section (also using lambdas)Bugfixes and improvements
Chores & tech debt
We switch to uv in the coming days and:
__repr__() for public interface by @zilto in https://github.com/dlt-hub/dlt/pull/2630dataset for testing now)load_id col in _dlt_loads table by @zilto in https://github.com/dlt-hub/dlt/pull/2729🧪 Upgrades to data access
x-annotation hints are propagatedSqlModel represent SQL query and is processed in extract, normalize and loaded in load stepscalar() on data access expressions ie.# get latest processed package id
max_load_id = pipeline.dataset()._dlt_loads.load_id.max().scalar()
🧪 Cool experimental stuff:
Check out our new embedded pipeline explorer app
dlt pipeline <name> show --marimo
dlt pipeline <name> show --marimo --edit
use edit option to enable Notebook/edit mode in Marimo + very cool Ibis dataset explorer
We updated contribution guidelines
Full Changelog: https://github.com/dlt-hub/dlt/compare/1.11.0...1.12.0
Nothing published for this version
Nothing published for this version
One column per quarter.