NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
PyPI
PyPI, #3103 most downloaded on PyPI
Orchestrate your dbt projects in Airflow
Latest stable
1.16.0
Last release 19 days ago
03 Nov 2026
Release timing varies
gaps range from 9 days to 3 months
Nearly every release is documented
notes for 60 of the last 60 stable releases
7 versions withdrawn
withdrawn after publishing
4 years old
214 releases · first in 2022
Drag across the ruler to compare, or tap a release to open it.
Add AIConfig (experimental, it may change without a deprecation period), an opt-in diagnosis of failed dbt tasks: on their last attempt, an LLM is ask…
Breaking Changes
sys.path under InvocationMode.DBT_RUNNER, PYTHONPATH under InvocationMode.SUBPROCESS). If you keep a dbt plugin whose module name starts with dbt_ inside the DAGs folder, it will no longer be discovered: move it out of the DAGs folder, or set enable_dags_folder_exclusion_from_dbt=False (env var AIRFLOW__COSMOS__ENABLE_DAGS_FOLDER_EXCLUSION_FROM_DBT) to restore the previous behaviour. This fixes dbt importing DAG files named dbt_*.py as plugins, which leaked or duplicated DAGs across unrelated Cosmos DAGs in-process and could crash the dbt command in subprocess mode. See #1673 and #2893.marts or marts+) now resolve against each node's fully qualified name (fqn) under LoadMode.DBT_MANIFEST, as dbt does. Review bare select and exclude entries before upgrading: an exclude that was a silent no-op, or that matched too broadly, now removes exactly the dbt-selected nodes and their descendants, so tasks can disappear from an existing DAG. A graph operator on a folder or package root (e.g. post_processing+ or fhir_dbt_utils+) now resolves nodes where it previously matched nothing, and a token equal to the model-paths directory (models), a filesystem ancestor, or a single middle folder segment no longer matches, because those are not an fqn prefix. See #2962 and #2978.ExecutionMode.AIRFLOW_ASYNC is no longer supported on Apache Airflow 2.x. This change was necessary in order to support this execution mode in Airflow 3.4. Users who rely on AIRFLOW_ASYNC and cannot upgrade to Airflow 3 should stay on an earlier Cosmos release. See more information on #3050./cosmos route requires a logged-in Airflow user with read access to the Website view, matching the Airflow 2 plugin; unauthenticated requests receive HTTP 401 and users without that permission receive HTTP 403. Previously these routes were served without authentication, exposing model SQL and schema metadata to anyone who could reach the API server. As a result, opening dbt docs from the Airflow UI now requires Airflow 3.1.1 or later (Astro Runtime 3.1-3 or later): the Airflow 3.1.0 UI discards its session cookie after login, so on Airflow 3.1.0 (including Astro Runtime 3.1-1 and 3.1-2) the docs return HTTP 401 and the plugin logs a warning. The plugin still loads on Airflow 3.1.0, and bearer-token API access to the docs still works. FastAPI's /openapi.json, /docs and /redoc under /cosmos are disabled, and the plugin registers no sub-application or menu entry when dbt_docs_projects is empty. See #3081.Behaviour Changes
--vars and --args dbt flags as one-line flow YAML (for example {var: '2'}) instead of block YAML, because dbt OSS rejects block YAML that contains quoted strings, for example var: '2'. dbt 1.x parses both forms to the same values, but the flag text in task logs and rendered commands differs. See #3099.RenderConfig.exclude is now resolved against the full project graph instead of against the subset RenderConfig.select already narrowed. Graph operators traverse outwards from the nodes their selector matches, so when select had already dropped those root nodes the traversal had nothing to start from and shared ancestors or descendants were silently kept. For example, given a parent tagged test and its child, select=["+child"], exclude=["+tag:test"] previously rendered both parent and child, because +tag:test was evaluated after +child had already been applied; it now renders only child. This matches dbt, which evaluates both filters against the whole graph and subtracts afterwards. See #2976.ProjectConfig now keeps the models/seeds/snapshots relative paths it was configured with even when it has no dbt_project_path of its own, and LoadMode.CUSTOM falls back to them when the project root is supplied through RenderConfig/ExecutionConfig instead. Previously those relative paths were discarded in that split configuration and LegacyDbtProject silently crawled its defaults, so a project declaring e.g. models_relative_paths="custom/models" rendered no models at all; it now crawls custom/models as configured. Projects that set dbt_project_path on ProjectConfig are unaffected. See #2977.LoadMode.DBT_MANIFEST, RenderConfig.exclude and a YAML selector's exclude now apply to test nodes: tests matched by the exclude (for example resource_type:test or tag:<x>) are no longer rendered, where previously they were added back next to their model. With TestBehavior.AFTER_EACH, a model whose tests are all excluded is rendered as a plain task instead of a run and test group, so its task ids change (for example model_a.run becomes model_a_run). The tests of excluded models also leave DbtGraph.filtered_nodes. Test nodes also keep their own tags in addition to the tags of every parent, so select tag:<x> matches a test tagged <x> whose parent is not. An exclude now also removes every test that depends on an excluded node, as dbt's default indirect selection does, so a multi-parent test goes with any excluded parent while the other parents stay. Selecting by a test's own tag now renders that test where the test behavior allows it: a detached task for a multi-parent test when should_detach_multiple_parents_tests=True, or the single test task with TestBehavior.AFTER_ALL; previously nothing was rendered. A source whose tests are all excluded no longer counts as having tests, so with source_rendering_behavior=WITH_TESTS_OR_FRESHNESS it is rendered only if it has a freshness check. See #2981 and #3080.emit_datasets is now a template field on the dbt operators, so emission can be decided per DAG run:operator_args={"emit_datasets": "{{ dag_run.run_type != 'backfill' }}"} stops backfills from triggering downstreamoperator_args now takes precedence over RenderConfig.emit_datasets (which remains aoperator_args value was silently discarded, sooperator_args={"emit_datasets": False} now takes effect. See #2996.dbt_project_path it appends guidance to set the path relative to the DAG file (e.g. (Path(__file__).parent / "dbt/my_dbt_project").absolute().as_posix()). This makes failures caused by versioned DAG bundles (e.g. on Astronomer) relocating the project between DAG parsing and task execution self-explanatory in the logs. The missing-project case in create_symlinks now raises CosmosValueError instead of a bare FileNotFoundError. See #2921.ExecutionMode.WATCHER (including the Kubernetes and GKE variants), the aggregated test status that the producer pushes to XCom under <model_unique_id>_tests_status is now a dict (status, passed_count, failed_count, total_count and up to five failed_tests unique_ids) instead of the "pass" or "fail" string, and the test sensors log that summary. The sensors and the trigger still accept the string, so DAG runs in flight during the upgrade are unaffected; anything reading that XCom key directly must now read status from the dict. See #2750.SourceRenderingBehavior.WITH_TESTS_OR_FRESHNESS, a source that has tests but no freshness checks now renders its freshness task as an EmptyOperator instead of a DbtSource task that always ended with "Nothing to do" and failed when warn-error is enabled, matching the existing SourceRenderingBehavior.ALL path. The EmptyOperator keeps the dbt owner and does not receive operator_args or per-node operator_kwargs. The test tasks are unchanged. See #3044.ExecutionMode.WATCHER consumers of a node that dbt skipped for an upstream failure now run the node on attempt 1 once the producer has finished, instead of failing and waiting one retry_delay for attempt 2. Previously every level of the graph below a failed model added one retry_delay. Test consumers whose model did not succeed do the same. This applies only when the task has retries set, and either watcher_dbt_retry_queue is unset or the consumer already runs on it. Such a node now gets retries + 1 real dbt runs instead of retries. See #3023 and #3047.ExecutionMode.WATCHER producer keeps its per-node XCom backup in the Airflow task store instead of an Airflow Variable when enable_watcher_reliable_retry is True, split across keys that roll over at 8 KB so each push rewrites only the open key. Variable writes look the key up in every configured secrets backend first, so with an external backend the producer task ran between about 3 and 6 times faster in our benchmark, depending on how many backends are configured. With False the Variable flush from the retry callback is unchanged. A producer that finds no backup on a retry logs a warning when the task store was in use. On Airflow 3.4 and later, clearing the producer from the UI or the REST API discards that backup unless "Keep task state" is ticked; see the watcher execution mode guide. See #2771 and #3076.TestBehavior.BUILD, a rendered dbt source that has tests is now a task group (<source>.source and<source>.test) instead of a single <source>_source task, so references to the old task id need updating.staging or the project name) and fqn: no longer select sources under LoadMode.DBT_MANIFEST. Before, a source matched when the folder of its YAML file or its package matched, so with source rendering on, a bare folder or package selector also rendered the source and the models that only read it. Use source: to select a source; package: and path: still select sources. See #3147.dbt-oss installed in the Airflow environment and openlineage-integration-common 1.43.0 or older, local operators now emit datasets (assets); before, they emitted none. emit_datasets is enabled by default, so DAGs scheduled on the assets of dbt models can be triggered by dbt OSS runs; set emit_datasets=False to keep the previous behaviour. With openlineage-integration-common 1.44.0 or later, dbt OSS already emitted datasets and this does not change. On Airflow 2.10.5, a version where registering new datasets is only partially supported, emitting datasets can fail with FlushError: Can't flush None value found in collection DatasetModel.aliases. See #3111.ExecutionMode.WATCHER now emits the datasets of models, seeds and snapshots when the manifest has no URI for them, as with dbt OSS, which writes manifest.json when the run ends, and dbt Fusion. The producer builds the URI from the database, schema and alias in the relation of the node event, and the manifest still wins when it has one. With emit_datasets=False nothing changes, and neither do the Kubernetes and GKE watchers. See #3129.Features
ExecutionMode.WATCHER test sensors by @michal-mrazek in #2750config.materialized: metric_view on Databricks, semantic_view on Snowflake). When enable_semantic_metric_view_reclassification is set (env var AIRFLOW__COSMOS__ENABLE_SEMANTIC_METRIC_VIEW_RECLASSIFICATION), these models are reclassified from model to DbtResourceType.SEMANTIC_METRIC_VIEW and rendered via a dedicated DbtSemanticMetricView* operator with a semantic_metric_run task suffix. Disabled by default. See the "Managing semantic metric views" guide by @corsettigyg in #2885ExecutionMode.WATCHER by @jroachgolf84 in #2844ExecutionMode.WATCHER_KUBERNETES by @MichaelRBlack in #2595ProjectConfig by @pankajastro in #2875retries parameter to the gcs_upload callback by @TTMichaelA in #2871emit_datasets templatable via operator_args by @tatiana in #2996config.group and group: selectors by @biswasbiplob in #2980AIConfig (experimental, it may change without a deprecation period), an opt-in diagnosis of failed dbt tasks: on their last attempt, an LLM is asked for the root cause and a fix, which are added after the dbt error. It uses apache-airflow-providers-common-ai (pip install astronomer-cosmos[ai], for Python 3.11+ and Airflow 2.11+), works with ExecutionMode.LOCAL and ExecutionMode.VIRTUALENV, masks secrets, and lets you set what is sent and how much it can spend. See the AI config reference by @corsettigyg in #3018dbt-oss extra (pip install astronomer-cosmos[dbt-oss]) by @pankajastro in #3130DbtDocsS3Operator, DbtDocsAzureStorageOperator and DbtDocsGCSOperator now upload the docs site written by dbt OSS, which has no catalog.json, with the content type of each file. The files uploaded for dbt 1 are unchanged by @pankajastro in #3119Enhancements
on_failure_callback by @pankajastro in #2845boto3 for Kubernetes dbt docs S3 uploads by @jx2lee in #2957ExecutionMode.WATCHER producer failure performance by keeping the per-node XCom backup in the Airflow task store on Airflow 3.3 and later by @tatiana in #3076Bug Fixes
RuntimeError on load. Previously, with [core] lazy_load_plugins = False (as on Astro Runtime), every Airflow CLI command logged a "Failed to import plugin cosmos" traceback, and the plugin's listeners were never registered by @jedcunningham in #3055LoadMode.DBT_LS now runs dbt ls through a Python subprocess when dbt OSS (dbt-core, dbt-oss or dbt at version 2) is installed in the Airflow environment, even under the default InvocationMode.DBT_RUNNER. dbt 2's dbtRunner returns only node names for dbt ls --output json, which previously rendered an empty DAG. The fallback runs dbt_executable_path and is logged at INFO level; dbt deps still runs through dbtRunner by @tatiana in #3060on_warning_callback not being called after deferral in Kubernetes mode by @mathiasgout in #3062ExecutionMode.WATCHER consumer never reaching the fallback when its node errors early by @pankajkoti in #3011ExecutionMode.WATCHER dataset emission being silently disabled when profiles.yml uses dbt Jinja filters by @goingforstudying-ctrl in #2950AttributeError masking the real dbt error on run-operation failure by @barking-code in #2983env_vars in the Kubernetes operator env merge by @pankajkoti in #3030ExecutionMode.WATCHER producer on dbt (oss) 2.0 by @fpiped in #2998EmptyOperator for freshness tasks of test-only sources, which now carry the dbt owner when owner inheritance is enabled by @JH0917 in #3044LoadMode.DBT_MANIFEST by matching the node's fqn by @pankajkoti in #2978exclude under LoadMode.DBT_MANIFEST by @tatiana and @ege-arhan in #3080exclude graph selectors against the full project graph by @00yhj22-debug in #2976ProjectConfig relative paths when the project root comes from RenderConfig by @00yhj22-debug in #2977ExecutionMode.WATCHER consumers waiting one retry_delay per level below a failure by @biswasbiplob in #3047run-operation failure with dbt < 1.12 by @barking-code in #2986run-operation on dbt 2 by @pankajastro in #3089cosmos.constants is imported before Airflow by @tatiana in #3087TestBehavior.BUILD by @developer-rpai in #3037ExecutionMode.WATCHER and enable the watcher tests on dbt-oss, so a capitalised Warn or Error from dbt 2 is recognised by the source sensor and the default freshness callback by @pankajastro in #3088--vars and --args dbt flags as one-line flow YAML, because dbt OSS rejects block YAML that contains quoted strings by @pankajkoti in #3099openlineage-integration-common is 1.43.0 or older (Airflow's constraints pin 1.27.0 with Airflow 2.10.5, 1.37.0 with 3.1.0 and 1.43.0 with 3.1.7). dbt 2 leaves args.profiles_dir out of run_results.json. Those library versions read it without a fallback, so parsing the OpenLineage events raised a KeyError and the task got no inlets or outlets (1.44.0 and later use the --profiles-dir lookup, and Cosmos does not change their run results). Cosmos now adds the profiles directory (--profiles-dir or DBT_PROFILES_DIR) to the run results of the task's temporary project before parsing, only when dbt did not write it. A dbt 2 executable used with a dbt 1.x environment is not covered. dbt Fusion used with a dbt 2 environment is covered by @pankajastro in #3111InvocationMode.SUBPROCESS, Cosmos now reads the text output of dbt OSS for dbt tests. DbtTestLocalOperator now calls on_warning_callback with the warned tests in test_names and test_results, and the exception for a failing dbt test, from any local operator, names the failing tests, with the number of failed rows when dbt 2 reports it. Previously the parsers only knew the dbt 1.x lines (Done. PASS=.. WARN=N .. TOTAL=N, Warning in test <name>, Failure in test <name>), so on dbt 2 the warning count was 0, the callback never fired, and the exception had no test details. The Kubernetes-based test operators (Kubernetes, GKE and EKS) now also read the dbt 2 Summary: .. | N warn line, so their on_warning_callback fires by @pankajastro in #3108ExecutionMode.WATCHER now recognises the nodes that dbt OSS skips because an upstream node failed. dbt 2 does not emit SkippingDetails or LogSkipBecauseError for them, only MarkSkippedChildren, so their consumers ended SKIPPED and were never built after the upstream recovered on a retry. Cosmos now reads the MarkSkippedChildren event that dbt 2 emits for each node it skips and treats the node as it does for dbt 1, including the nodes below a failed test and below an excluded node by @pankajastro in #3124, see #2698 and #3023InvocationMode.DBT_RUNNER now matches dbt node statuses case-insensitively when it collects warnings. dbt OSS reports a source freshness warning as Warn, which did not match warn, so DbtSourceLocalOperator called on_warning_callback with empty test_names and test_results. dbt 2 source freshness results have no message, so their status is reported in test_results instead. dbt 1 is unchanged by @pankajastro in #3123ExecutionMode.WATCHER now emits the datasets of models, seeds and snapshots when the manifest has no URI for them, as with dbt-oss and dbt Fusion by @pankajastro in #3129fqn: selectors under LoadMode.DBT_MANIFEST, as dbt does by @pankajastro in #3147--no-static-parser flag under InvocationMode.DBT_RUNNER when dbt OSS is installed. dbt 2 has no static parser, so the flag logged NoLongerSupportedOption (dbt1700) on every command, which failed the task when warn_error=True by @pankajastro in #3144ExecutionMode.WATCHER producer. dbt 2 reports node_path relative to the project instead of the models folder, so the compiled SQL was never found by @pankajastro in #3145InvocationMode.SUBPROCESS, read the source freshness warnings that dbt OSS prints as Warned source <name> (...), so DbtSourceLocalOperator calls on_warning_callback with the source in test_names and the line dbt printed in test_results by @pankajastro in #3146on_warning_callback runs for a stale source by @pankajastro in #3156InvocationMode.SUBPROCESS, list the [error] [<Name> (dbt<code>)] blocks of failed models, macros and stale sources of dbt OSS in the exception of a failed command by @pankajastro in #3148InvocationMode.DBT_RUNNER, report a dbt OSS failure that is not tied to one node, such as a failing run-operation macro, in one line instead of listing every node of the project by @pankajastro in #3149assets/ and info_schema/ files of the dbt OSS docs site in the Airflow 2 plugin, which loaded as a blank page before by @pankajastro in #3155assets/ and info_schema/ files of the dbt OSS docs site in the Airflow 3 plugin, which loaded as a blank page before by @pankajastro in #3154Docs
:ref: role by @pankajastro in #2873ProjectConfig by @pankajastro in #2876AGENTS.md test matrix to pyproject.toml instead of hardcoding versions by @pankajastro in #2887on_failure_callback access to dbt test failures by @pankajastro in #2846Note truncated.
Your coding agent can read these notes before it upgrades. Set up the MCP server →
Pre-release candidate of Cosmos 1.16.0 .
Pre-release candidate of Cosmos 1.16.0.
Second release candidate. It adds 14 milestone PRs merged since
1.16.0rc1: more dbt 2 (dbt-oss) fixes, the new minimum Python (3.11) and Apache Airflow (2.11) versions, and docs. The head branch was synced withmainso this cut contains all of them. Please try it and report any problems before the 1.16.0 GA. See the release PR for full provenance, the complete 178-PR changelog, and the test plan: #3073.
staging or the project name) and fqn: no longer select sources under LoadMode.DBT_MANIFEST. Before, a source matched when the folder of its YAML file or its package matched, so with source rendering on, a bare folder or package selector also rendered the source and the models that only read it. Use source: to select a source; package: and path: still select sources. See #3147.fqn: selectors under LoadMode.DBT_MANIFEST, as dbt does by @pankajastro in #3147--no-static-parser flag under InvocationMode.DBT_RUNNER when dbt 2 (dbt-oss) is installed. dbt 2 has no static parser, so the flag logged NoLongerSupportedOption (dbt1700) on every command, which failed the task when warn_error=True by @pankajastro in #3144dbt-oss) models in the ExecutionMode.WATCHER producer. dbt 2 reports node_path relative to the project instead of the models folder, so the compiled SQL was never found by @pankajastro in #3145InvocationMode.SUBPROCESS, read the source freshness warnings that dbt 2 (dbt-oss) prints as Warned source <name> (...), so DbtSourceLocalOperator calls on_warning_callback with the source in test_names and the line dbt printed in test_results by @pankajastro in #3146dbt-oss) source freshness warnings in the Kubernetes and GKE operators, so on_warning_callback runs for a stale source by @pankajastro in #3156InvocationMode.SUBPROCESS, list the [error] [<Name> (dbt<code>)] blocks of failed models, macros and stale sources of dbt 2 (dbt-oss) in the exception of a failed command by @pankajastro in #3148InvocationMode.DBT_RUNNER, report a dbt 2 (dbt-oss) failure that is not tied to one node, such as a failing run-operation macro, in one line instead of listing every node of the project by @pankajastro in #3149assets/ and info_schema/ files of the dbt 2 (dbt-oss) docs site in the Airflow 2 plugin, which loaded as a blank page before by @pankajastro in #3155assets/ and info_schema/ files of the dbt 2 (dbt-oss) docs site in the Airflow 3 plugin, which loaded as a blank page before by @pankajastro in #3154enable_dataset_alias value that Airflow 3 rejects in the scheduling guide by @jedcunningham in #3153Milestone: Cosmos 1.16.0
Full changelog: astronomer-cosmos-v1.16.0rc1...astronomer-cosmos-v1.16.0rc2
Pre-release candidate of Cosmos 1.16.0 .
Pre-release candidate of Cosmos 1.16.0.
First release candidate. It adds 29 milestone PRs merged since the third alpha, mostly dbt 2 (dbt-oss) support and the dbt-oss CI jobs. The head branch was synced with
mainso this cut contains all of them. Please try it and report any problems before the 1.16.0 GA. See the release PR for full provenance, the complete 164-PR changelog, and the test plan: #3073.
--vars and --args dbt flags as one-line flow YAML (for example {var: '2'}) instead of block YAML, because dbt 2 rejects block YAML that contains quoted strings, for example var: '2'. dbt 1.x parses both forms to the same values, but the flag text in task logs and rendered commands differs. See #3099.dbt-oss) installed in the Airflow environment and openlineage-integration-common 1.43.0 or older, local operators now emit datasets (assets); before, they emitted none. emit_datasets is enabled by default, so DAGs scheduled on the assets of dbt models can be triggered by dbt 2 runs; set emit_datasets=False to keep the previous behaviour. With openlineage-integration-common 1.44.0 or later, dbt 2 already emitted datasets and this does not change. On Airflow 2.10.5, a version where registering new datasets is only partially supported, emitting datasets can fail with FlushError: Can't flush None value found in collection DatasetModel.aliases. See #3111.ExecutionMode.WATCHER now emits the datasets of models, seeds and snapshots when the manifest has no URI for them, as with dbt-oss, which writes manifest.json when the run ends, and dbt Fusion. The producer builds the URI from the database, schema and alias in the relation of the node event, and the manifest still wins when it has one. With emit_datasets=False nothing changes, and neither do the Kubernetes and GKE watchers. See #3129.dbt-oss extra (pip install astronomer-cosmos[dbt-oss]) by @pankajastro in #3130DbtDocsS3Operator, DbtDocsAzureStorageOperator and DbtDocsGCSOperator now upload the docs site written by dbt-oss, which has no catalog.json, with the content type of each file. The files uploaded for dbt 1 are unchanged by @pankajastro in #3119--vars and --args dbt flags as one-line flow YAML, because dbt 2 rejects block YAML that contains quoted strings by @pankajkoti in #3099dbt-oss) is installed in the Airflow environment and openlineage-integration-common is 1.43.0 or older (Airflow's constraints pin 1.27.0 with Airflow 2.10.5, 1.37.0 with 3.1.0 and 1.43.0 with 3.1.7). dbt 2 leaves args.profiles_dir out of run_results.json. Those library versions read it without a fallback, so parsing the OpenLineage events raised a KeyError and the task got no inlets or outlets (1.44.0 and later use the --profiles-dir lookup, and Cosmos does not change their run results). Cosmos now adds the profiles directory (--profiles-dir or DBT_PROFILES_DIR) to the run results of the task's temporary project before parsing, only when dbt did not write it. A dbt 2 executable used with a dbt 1.x environment is not covered. dbt Fusion used with a dbt 2 environment is covered by @pankajastro in #3111InvocationMode.SUBPROCESS, Cosmos now reads the text output of dbt 2 (dbt-oss) for dbt tests. DbtTestLocalOperator now calls on_warning_callback with the warned tests in test_names and test_results, and the exception for a failing dbt test, from any local operator, names the failing tests, with the number of failed rows when dbt 2 reports it. Previously the parsers only knew the dbt 1.x lines (Done. PASS=.. WARN=N .. TOTAL=N, Warning in test <name>, Failure in test <name>), so on dbt 2 the warning count was 0, the callback never fired, and the exception had no test details. The Kubernetes-based test operators (Kubernetes, GKE and EKS) now also read the dbt 2 Summary: .. | N warn line, so their on_warning_callback fires by @pankajastro in #3108ExecutionMode.WATCHER now recognises the nodes that dbt 2 (dbt-oss) skips because an upstream node failed. dbt 2 does not emit SkippingDetails or LogSkipBecauseError for them, only MarkSkippedChildren, so their consumers ended SKIPPED and were never built after the upstream recovered on a retry. Cosmos now reads the MarkSkippedChildren event that dbt 2 emits for each node it skips and treats the node as it does for dbt 1, including the nodes below a failed test and below an excluded node by @pankajastro in #3124, see #2698 and #3023InvocationMode.DBT_RUNNER now matches dbt node statuses case-insensitively when it collects warnings. dbt 2 (dbt-oss) reports a source freshness warning as Warn, which did not match warn, so DbtSourceLocalOperator called on_warning_callback with empty test_names and test_results. dbt 2 source freshness results have no message, so their status is reported in test_results instead. dbt 1 is unchanged by @pankajastro in #3123ExecutionMode.WATCHER now emits the datasets of models, seeds and snapshots when the manifest has no URI for them, as with dbt-oss and dbt Fusion by @pankajastro in #3129ubuntu-24.04 by @pankajastro in #3134Milestone: Cosmos 1.16.0
Full changelog: astronomer-cosmos-v1.16.0a3...astronomer-cosmos-v1.16.0rc1
Add AIConfig (experimental, it may change without a deprecation period), an opt-in diagnosis of failed dbt tasks: on their last attempt, an LLM is ask…
Pre-release alpha of Cosmos 1.16.0.
Refresh of
a2with 15 additional milestone PRs merged since the second alpha. The head branch was synced withmainagain so this cut actually contains all of them. See the release PR for full provenance, the complete 135-PR changelog, and the test plan: #3073.
RenderConfig.exclude is now resolved against the full project graph instead of against the subset RenderConfig.select already narrowed. Graph operators traverse outwards from the nodes their selector matches, so when select had already dropped those root nodes the traversal had nothing to start from and shared ancestors or descendants were silently kept. For example, given a parent tagged test and its child, select=["+child"], exclude=["+tag:test"] previously rendered both parent and child, because +tag:test was evaluated after +child had already been applied; it now renders only child. This matches dbt, which evaluates both filters against the whole graph and subtracts afterwards. See #2976.ProjectConfig now keeps the models/seeds/snapshots relative paths it was configured with even when it has no dbt_project_path of its own, and LoadMode.CUSTOM falls back to them when the project root is supplied through RenderConfig/ExecutionConfig instead. Previously those relative paths were discarded in that split configuration and LegacyDbtProject silently crawled its defaults, so a project declaring e.g. models_relative_paths="custom/models" rendered no models at all; it now crawls custom/models as configured. Projects that set dbt_project_path on ProjectConfig are unaffected. See #2977.LoadMode.DBT_MANIFEST, RenderConfig.exclude and a YAML selector's exclude now apply to test nodes: tests matched by the exclude (for example resource_type:test or tag:<x>) are no longer rendered, where previously they were added back next to their model. With TestBehavior.AFTER_EACH, a model whose tests are all excluded is rendered as a plain task instead of a run and test group, so its task ids change (for example model_a.run becomes model_a_run). The tests of excluded models also leave DbtGraph.filtered_nodes. Test nodes also keep their own tags in addition to the tags of every parent, so select tag:<x> matches a test tagged <x> whose parent is not. An exclude now also removes every test that depends on an excluded node, as dbt's default indirect selection does, so a multi-parent test goes with any excluded parent while the other parents stay. Selecting by a test's own tag now renders that test where the test behavior allows it: a detached task for a multi-parent test when should_detach_multiple_parents_tests=True, or the single test task with TestBehavior.AFTER_ALL; previously nothing was rendered. A source whose tests are all excluded no longer counts as having tests, so with source_rendering_behavior=WITH_TESTS_OR_FRESHNESS it is rendered only if it has a freshness check. See #2981 and #3080.ExecutionMode.WATCHER consumers of a node that dbt skipped for an upstream failure now run the node on attempt 1 once the producer has finished, instead of failing and waiting one retry_delay for attempt 2. Previously every level of the graph below a failed model added one retry_delay. Test consumers whose model did not succeed do the same. This applies only when the task has retries set, and either watcher_dbt_retry_queue is unset or the consumer already runs on it. Such a node now gets retries + 1 real dbt runs instead of retries. See #3023 and #3047.ExecutionMode.WATCHER producer keeps its per-node XCom backup in the Airflow task store instead of an Airflow Variable when enable_watcher_reliable_retry is True, split across keys that roll over at 8 KB so each push rewrites only the open key. Variable writes look the key up in every configured secrets backend first, so with an external backend the producer task ran between about 3 and 6 times faster in our benchmark, depending on how many backends are configured. With False the Variable flush from the retry callback is unchanged. A producer that finds no backup on a retry logs a warning when the task store was in use. On Airflow 3.4 and later, clearing the producer from the UI or the REST API discards that backup unless "Keep task state" is ticked; see the watcher execution mode guide. See #2771 and #3076.TestBehavior.BUILD, a rendered dbt source that has tests is now a task group (<source>.source and <source>.test) instead of a single <source>_source task, so references to the old task id need updating. The source tests, which this mode never executed before, now run and can fail the DAG. See #1683 and #3037.AIConfig (experimental, it may change without a deprecation period), an opt-in diagnosis of failed dbt tasks: on their last attempt, an LLM is asked for the root cause and a fix, which are added after the dbt error. It uses apache-airflow-providers-common-ai (pip install astronomer-cosmos[ai], for Python 3.11+ and Airflow 2.11+), works with ExecutionMode.LOCAL and ExecutionMode.VIRTUALENV, masks secrets, and lets you set what is sent and how much it can spend. See the AI config reference by @corsettigyg in #3018ExecutionMode.WATCHER producer failure performance by keeping the per-node XCom backup in the Airflow task store on Airflow 3.3 and later by @tatiana in #3076exclude under LoadMode.DBT_MANIFEST by @tatiana and @ege-arhan in #3080exclude graph selectors against the full project graph by @00yhj22-debug in #2976ProjectConfig relative paths when the project root comes from RenderConfig by @00yhj22-debug in #2977ExecutionMode.WATCHER consumers waiting one retry_delay per level below a failure by @biswasbiplob in #3047run-operation failure with dbt < 1.12 by @barking-code in #2986run-operation on dbt 2 by @pankajastro in #3089cosmos.constants is imported before Airflow by @tatiana in #3087TestBehavior.BUILD by @developer-rpai in #3037ExecutionMode.WATCHER and enable the watcher tests on dbt-oss, so a capitalised Warn or Error from dbt 2 is recognised by the source sensor and the default freshness callback by @pankajastro in #3088on_failure_callback access to dbt test failures by @pankajastro in #2846Milestone: Cosmos 1.16.0
Full changelog: astronomer-cosmos-v1.16.0a2...astronomer-cosmos-v1.16.0a3
The plugin-discovery ( #2893 ) and bare-selector ( #2978 ) changes are now listed under Breaking Changes in the release PR. See the release PR for ful…
Pre-release alpha of Cosmos 1.16.0.
Refresh of
a1with 15 additional milestone PRs merged since the first alpha, plus changes from review. The head branch was synced withmainso this cut actually contains all of them. The plugin-discovery (#2893) and bare-selector (#2978) changes are now listed under Breaking Changes in the release PR. See the release PR for full provenance, the complete 120-PR changelog, and the test plan: #3073.
/cosmos route requires a logged-in Airflow user with read access to the Website view, matching the Airflow 2 plugin; unauthenticated requests receive HTTP 401 and users without that permission receive HTTP 403. Previously these routes were served without authentication, exposing model SQL and schema metadata to anyone who could reach the API server. As a result, opening dbt docs from the Airflow UI now requires Airflow 3.1.1 or later (Astro Runtime 3.1-3 or later): the Airflow 3.1.0 UI discards its session cookie after login, so on Airflow 3.1.0 (including Astro Runtime 3.1-1 and 3.1-2) the docs return HTTP 401 and the plugin logs a warning. The plugin still loads on Airflow 3.1.0, and bearer-token API access to the docs still works. FastAPI's /openapi.json, /docs and /redoc under /cosmos are disabled, and the plugin registers no sub-application or menu entry when dbt_docs_projects is empty. See #3081.ExecutionMode.WATCHER (including the Kubernetes and GKE variants), the aggregated test status that the producer pushes to XCom under <model_unique_id>_tests_status is now a dict (status, passed_count, failed_count, total_count and up to five failed_tests unique_ids) instead of the "pass" or "fail" string, and the test sensors log that summary. The sensors and the trigger still accept the string, so DAG runs in flight during the upgrade are unaffected; anything reading that XCom key directly must now read status from the dict. See #2750.SourceRenderingBehavior.WITH_TESTS_OR_FRESHNESS, a source that has tests but no freshness checks now renders its freshness task as an EmptyOperator instead of a DbtSource task that always ended with "Nothing to do" and failed when warn-error is enabled, matching the existing SourceRenderingBehavior.ALL path. The EmptyOperator keeps the dbt owner and does not receive operator_args or per-node operator_kwargs. The test tasks are unchanged. See #3044.ExecutionMode.WATCHER test sensors by @michal-mrazek in #2750config.group and group: selectors by @biswasbiplob in #2980ExecutionMode.WATCHER producer on dbt (oss) 2.0 by @fpiped in #2998EmptyOperator for freshness tasks of test-only sources, which now carry the dbt owner when owner inheritance is enabled by @JH0917 in #30442.0-dbt-fusion by @pankajastro in #3043OSSUSER in OSS.COSMOS_CI with a key pair by @tatiana in #3075main by @tatiana in #3077test-runtime-alpha run name and summary by @jedcunningham in #3078Milestone: Cosmos 1.16.0
Full changelog: astronomer-cosmos-v1.16.0a1...astronomer-cosmos-v1.16.0a2
Raised the minimum supported dbt Core version to 1.8. dbt Core 1.5, 1.6, and 1.7 have reached End of Life per the dbt Labs support policy and are no l
Breaking Changes
ExecutionMode.AIRFLOW_ASYNC is no longer supported on Apache Airflow 2.x. Cosmos now builds the async operator at runtime in a way that works on Airflow 3.4, which leaves the operator not fully initialised on Airflow 2.9 and unpicklable on Airflow 2.x. Users who run AIRFLOW_ASYNC and cannot upgrade to Airflow 3 should stay on an earlier Cosmos release. See #3050.Behaviour Changes
emit_datasets is now a template field on the dbt operators, so emission can be decided per DAG run: operator_args={"emit_datasets": "{{ dag_run.run_type != 'backfill' }}"} stops backfills from triggering downstream asset-scheduled DAGs. operator_args now takes precedence over RenderConfig.emit_datasets (which remains a parse-time boolean and is not templatable). Previously an operator_args value was silently discarded, so operator_args={"emit_datasets": False} now takes effect. See #2996.InvocationMode.DBT_RUNNER (in-process, via sys.path) and InvocationMode.SUBPROCESS (via PYTHONPATH). Previously dbt-core imported DAG files named dbt_*.py as a side effect of plugin discovery, which leaked or duplicated DAGs across unrelated Cosmos DAGs when running in-process and could crash the dbt command in subprocess mode. This is enabled by default. As a result, a genuine dbt plugin named dbt_* placed inside the DAGs folder will no longer be discovered; set enable_dags_folder_exclusion_from_dbt=False (env var AIRFLOW__COSMOS__ENABLE_DAGS_FOLDER_EXCLUSION_FROM_DBT) to restore the previous behaviour. See #1673 and #2893.dbt_project_path it appends guidance to set the path relative to the DAG file (e.g. (Path(__file__).parent / "dbt/my_dbt_project").absolute().as_posix()). This makes failures caused by versioned DAG bundles (e.g. on Astronomer) relocating the project between DAG parsing and task execution self-explanatory in the logs. The missing-project case in create_symlinks now raises CosmosValueError instead of a bare FileNotFoundError. See #2921.LoadMode.DBT_MANIFEST, following dbt's default fqn selector (leaf name, or an ordered folder path from the project root). A graph operator on a bare folder or package root (e.g. post_processing+ or fhir_dbt_utils+) now resolves nodes where it previously matched nothing. Bare matching is also stricter: a token equal to the model-paths directory (models), a filesystem ancestor, or a single middle folder segment no longer matches, because those are not an fqn prefix. This mainly affects exclude: bare exclude entries that were previously a silent no-op or matched too broadly now remove exactly the dbt-selected nodes and their descendants, so review them before upgrading. See #2962.Features
config.materialized: metric_view on Databricks, semantic_view on Snowflake). When enable_semantic_metric_view_reclassification is set (env var AIRFLOW__COSMOS__ENABLE_SEMANTIC_METRIC_VIEW_RECLASSIFICATION), these models are reclassified from model to DbtResourceType.SEMANTIC_METRIC_VIEW and rendered via a dedicated DbtSemanticMetricView* operator with a semantic_metric_run task suffix. Disabled by default. See the "Managing semantic metric views" guide by @corsettigyg in #2885ExecutionMode.WATCHER. by @jroachgolf84 in #2844ExecutionMode.WATCHER_KUBERNETES. by @MichaelRBlack in #2595ProjectConfig. by @pankajastro in #2875retries parameter to the gcs_upload callback. by @TTMichaelA in #2871emit_datasets templatable via operator_args. by @tatiana in #2996Enhancements
on_failure_callback by @pankajastro in #2845boto3 for Kubernetes dbt docs S3 uploads by @jx2lee in #2957Bug Fixes
RuntimeError on load. Previously, with [core] lazy_load_plugins = False (as on Astro Runtime), every Airflow CLI command logged a "Failed to import plugin cosmos" traceback, and the plugin's listeners were never registered. by @jedcunningham in #3055LoadMode.DBT_MANIFEST by matching the node's fqn, so <folder>+ and <package>+ behave like their path: and package: equivalents and like dbt ls. Previously the graph selector resolved its root by exact node name only, and bare-identifier matching keyed off the absolute file path. by @pankajkoti in #2978LoadMode.DBT_LS now runs dbt ls through a Python subprocess when dbt 2 (dbt-core, dbt-oss or dbt 2.x) is installed in the Airflow environment, even under the default InvocationMode.DBT_RUNNER. dbt 2's dbtRunner returns only node names for dbt ls --output json, which previously rendered an empty DAG. The fallback runs dbt_executable_path and is logged at INFO level; dbt deps still runs through dbtRunner. by @tatiana in #3060on_warning_callback not being called after deferral in Kubernetes mode by @mathiasgout in #3062ExecutionMode.WATCHER consumer never reaching the fallback when its node errors early by @pankajkoti in #3011ExecutionMode.WATCHER dataset emission being silently disabled when profiles.yml uses dbt Jinja filters by @goingforstudying-ctrl in #2950AttributeError masking the real dbt error on run-operation failure by @barking-code in #2983env_vars in the Kubernetes operator env merge by @pankajkoti in #3030Docs
:ref: role by @pankajastro in #2873ProjectConfig by @pankajastro in #2876AGENTS.md test matrix to pyproject.toml instead of hardcoding versions by @pankajastro in #2887Others
manifest.json for altered_jaffle_shop up to date in tests by @shantanutheone in #3063no-any-return error in _resolve_env_var by @tatiana in #2904create_test_task_metadata complexity by @tatiana in #2933github/codeql-action updates in a single Dependabot PR by @tatiana in #2928Milestone: Cosmos 1.16.0
Avoid deprecated airflow.exceptions.AirflowSkipException import on Airflow 3.1+ by @pankajastro in #2796
Behaviour Changes
These changes adjust observable behaviour of the Kubernetes-based operators and of argument
validation in async execution mode. None of them break the public Cosmos API, but users relying
on the previous behaviour should review before upgrading.
ENTRYPOINT byarguments and any user-supplied cmds isdbt now require an explicit cmds=["dbt"].compiled_sql, freshness) via operator_args inCosmosValueError at DAG parse time instead of an opaqueTypeError. See #2868.Bug Fixes
env_var() Jinja when deriving ExecutionMode.WATCHER dataset namespace by @pankajastro in #2879airflow.exceptions.AirflowSkipException import on Airflow 3.1+ by @pankajastro in #2796operator_args path by @pankajastro in #2868ExecutionMode.WATCHER multi-parent test aggregation, so test sensors no longer hang by @hellowithchicken in #2901operator_kwargs under TestBehavior.AFTER_EACH by @tatiana in #2936on_warning_callback passed via operator_args by @tatiana in #2940--full-refresh overrides by @pankajastro in #2917Docs
ENTRYPOINT handling for Kubernetes modes by @pankajkoti in #2906ExecutionMode.WATCHER by @pankajastro in #2884contributing.rst by @tatiana in #2859Others
source_pruning_dag integration test by seeding the database first by @tatiana in #2927Honor on_warning_callback passed via operator_args by @tatiana in #2940
Bug fixes
on_warning_callback passed via operator_args by @tatiana in #2940--full-refresh overrides by @pankajastro in #2917Credit all parent models in ExecutionMode.WATCHER multi-parent test aggregation, so test sensors no longer hang by @hellowithchicken in #2901
Bug Fixes
Docs
Avoid deprecated airflow.exceptions.AirflowSkipException import on Airflow 3.1+ by @pankajastro in #2796
Pre-release alpha of Cosmos 1.15.1, the first cut from the clean
release-1.15maintenance line (branched from thev1.15.0tag with the milestone fixes cherry-picked).Note for
a1/a2testers: those alphas were hotfix cuts frommainand carried 1.16.0 features (#2844 dedicated WATCHER queues, #2871gcs_uploadretries) that this line does not ship. Thea2watcher--full-refreshfix (#2917) is not yet in this cut; it lands before GA.
Behaviour Changes
None of these break the public Cosmos API, but review before upgrading:
ENTRYPOINT by default: the full dbt command is passed via arguments and any user-supplied cmds is preserved. Images whose entrypoint is itself dbt now require an explicit cmds=["dbt"]. This restores the pre-1.15.0 default. See #2891.compiled_sql, freshness) via operator_args in async execution mode now raises CosmosValueError at DAG parse time instead of an opaque TypeError. See #2868.Bug Fixes
env_var() Jinja when deriving ExecutionMode.WATCHER dataset namespace by @pankajastro in #2879airflow.exceptions.AirflowSkipException import on Airflow 3.1+ by @pankajastro in #2796operator_args path by @pankajastro in #2868Docs
ENTRYPOINT handling for Kubernetes modes by @pankajkoti in #2906ExecutionMode.WATCHER by @pankajastro in #2884contributing.rst by @tatiana in #2859Others
source_pruning_dag integration test by seeding the database first by @tatiana in #2927Milestone: Cosmos 1.15.1 Release PR: #2935 Full changelog: https://github.com/astronomer/astronomer-cosmos/compare/astronomer-cosmos-v1.15.0...astronomer-cosmos-v1.15.1a3
Fix WATCHER_KUBERNETES consumer fallback dropping --full-refresh on templated params by @pankajastro in https://github.com/astronomer/astronomer-cosmo
Bug Fixes
Hotfix alpha for validation. This alpha contains a single fix and is intended for the reporter of #2889 to confirm it resolves their issue before this
Hotfix alpha for validation. This alpha contains a single fix and is intended for the reporter of #2889 to confirm it resolves their issue before this is merged to main.
Bug Fixes
ENTRYPOINT by default in Kubernetes-based dbt operators (ExecutionMode.KUBERNETES, WATCHER_KUBERNETES, AWS_EKS, GCP_GKE, WATCHER_GCP_GKE). Previously, build_kube_args always set the pod container's command to ["dbt"], which overrode the image's ENTRYPOINT. Images that rely on their ENTRYPOINT to perform setup before dbt runs, such as sourcing secrets injected by a sidecar, would silently lose that setup. See #2889 and #2891.One column per quarter.