NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
PyPI · #1975 most downloaded on PyPI
Package for AWS-specific Dagster framework solid and resource components.
Last release 3 days ago
01 Oct 2026
Ships on a steady schedule
a new release about every 2 weeks
Nearly every release is documented
notes for 59 of the last 60 stable releases
4 versions withdrawn
withdrawn after publishing
7 years old
684 releases · first in 2019
The AutomationCondition.eager(), AutomationCondition.missing(), and AutomationCondition.on_cron conditions are now compatible with asset checks.
AutomationCondition.eager(), AutomationCondition.missing(), and AutomationCondition.on_cron conditions are now compatible with asset checks.AssetSelection.materializable(), which returns only assets that are materializable in an existing selection.AutomationCondition.all_deps_blocking_checks_passed condition, which can be used to prevent materialization when any upstream blocking checks have failed.code_version parameter to the @graph_asset decorator.LaunchPartitionBackfill mutation is submitted to GQL with invalid partition keys, it will now return an early PartitionKeysNotFoundError.AssetSelection.checks_for_assets now accepts AssetKeys and string asset keys, in addition to AssetsDefinitions.get_objects now has a since_last_modified that enables only fetching objects modified after a given timestamp.dagster_aws.pipes.PipesEMRCLient ) for running and monitoring AWS EMR jobs from Dagster.AutomationConditions that operate over dependencies.DatabricksPySparkStepLauncher, so that permissions are now set correctly for non-admin users.fetch_column_metadata did not work properly for models imported through dbt dependencies.DagsterK8sPipesClient.run now shows up in API docs.One column per quarter.
[experimental] AutomationCondition.eager() will now only launch runs for missing partitions which become missing _after_ the condition has been added
create_pull_request method on GithubClient that enables creating a pull request.create_ref method on GithubClient that enables creating a new branch.dlt.run(...).fetch_row_count() for both partitioned and non-partitioned assets. Thanks @kristianandre!PartitionedConfig objects can now return a RunConfig without causing a crash.AssetIn.__new__ typing for the dagster_type argument.fetch_row_count() method now works for databases returning uppercase column names. Thanks @kristianandre!GCSFileManager.read operations. Thanks @ollie-bell!JobDefinition, @job, and define_asset_job now take a run_tags parameter. If run_tags are defined, they will be attached to all runs of the job, and ta
JobDefinition, @job, and define_asset_job now take a run_tags parameter. If run_tags are defined, they will be attached to all runs of the job, and tags will not be. If run_tags is not set, then tags are attached to all runs of the job (status quo behavior). This change enables the separation of definition-level and run-level tags on jobs.DAGSTER_COMPUTE_LOG_TAIL_WAIT_AFTER_FINISH can now be used to pause before capturing logs (thanks @HynekBlaha!)kinds parameter is now available on AssetSpec.OutputContext now exposes the AssetSpec of the asset that is being stored as an output (thanks, @marijncv!)AutomationCondition on an asset, the label of this condition will now be shown in the sidebar on the Asset Details page.dagster definitions validate have been added.…the key for the IO manager used to load it. The deprecation warning for SourceAsset now references this method.
AssetSpec now has a with_io_manager_key method that returns an AssetSpec with the appropriate metadata entry to dictate the key for the IO manager used to load it. The deprecation warning for SourceAsset now references this method.max_runtime_seconds configuration option to run monitoring, allowing you to specify that any run in your Dagster deployment should terminate if it exceeds a certain runtime. Prevoiusly, jobs had to be individually tagged with a dagster/max_runtime tag in order to take advantage of this feature. Jobs and runs can still be tagged in order to override this value for an individual run.tags and a custom execution_fn on a ScheduleDefinition. Schedule tags are intended to annotate the definition and can be used to search and filter in the UI. They will not be attached to run requests emitted from the schedule if a custom execution_fn is provided. If no custom execution_fn is provided, then for back-compatibility the tags will also be automatically attached to run requests emitted from the schedule.SensorDefinition and all of its variants/decorators now accept a tags parameter. The tags annotate the definition and can be used to search and filter in the UI.dagster definitions validate command to Dagster CLI. This command validates if Dagster definitions are loadable.build_op_context in a unit test would sometimes raise a TypeError: signal handler must be signal.SIG_IGN, signal.SIG_DFL, or a callable object Exception on process shutdown.DefaultScheduleStatus.STOPPED / DefaultSensorStatus.STOPPED after performing a reset.Added --partition-range option to dagster asset materialize CLI. This option only works for assets with single-run Backfill Policies.
--partition-range option to dagster asset materialize CLI. This option only works for assets with single-run Backfill Policies..without() method to AutomationCondition.eager(), AutomationCondition.on_cron(), and AutomationCondition.on_missing() which allows sub-conditions to be removed, e.g. AutomationCondition.eager().without(AutomationCondition.in_latest_time_window()).AutomationCondition.on_missing(), which materializes an asset partition as soon as all of its parent partitions are filled in.pyproject.toml can now load multiple Python modules as individual Code Locations. Thanks, @bdart!ADLS2PickleIOManager now accepts lease_duration configuration. Thanks, @0xfabioo!sling.replicate(...).fetch_row_count().AutomationCondition.any_downstream_condition() with downstream AutoMaterializePolicy objects.process_config_and_initialize did not properly handle processing nested resource config.DepConditionWrapperCondition instead of the key that they were evaluated against.wait_for_pod check now waits until all pods are available, rather than erroneously returning after the first pod becomes available. Thanks @easontm!The AssetSpec constructor now raises an error if an invalid group name is provided, instead of an error being raised when constructing the Definitions
AssetSpec constructor now raises an error if an invalid group name is provided, instead of an error being raised when constructing the Definitions object.dagster/relation_identifier metadata is now automatically attached to assets which are stored using a DbIOManager.DbtCliResource, the project_dir argument will now override the DBT_PROJECT_DIR environment variable if it exists in the local environment (thanks, @marijncv!).rows_loaded metadata (thanks, @kristianandre!).asset_selection=[] on RunRequest objects yielded from sensors using asset_selection would select all assets instead of none.assets_dbt_python example.database field set (thanks, @dargmuesli!)dbt.adapters.duckdb module when loading dbt assets without that package installed.DagsterInstance.wipe_asset_partitions.allow_missing_partitions as metadata on a downstream asset.Fixed an issue where runs in Dagster+ Serverless that materialized partitioned assets would sometimes fail with an object has no attribute '_base_path
object has no attribute '_base_path' error.statuses filter argument to the sensorsOrError GraphQL field was sometimes ignored when querying GraphQL for multiple sensors at the same time.[dagster-wandb] Removed usage of deprecated recursive parameter (Thanks, chrishiste!)
CapturedLogManager and ComputeLogManager APIs into a single base class.PipesECSClient to allow Dagster to interface with ECS tasks.dbt subprocess to wait 25 seconds for the subprocess to cleanly terminate. Previously, it would only wait 2 seconds.sdf subprocess to wait 25 seconds for the subprocess to cleanly terminate. Previously, it would only wait 2 seconds.AutomationCondition using DagsterDltTranslator.get_automation_condition() (Thanks, aksestok!)dagsterDaemon.runRetries.retryOnAssetOrOpFailure to False in the Dagster Helm chart to prevent op retries and run retries from simultaneously firing on the same failure.recursive parameter (Thanks, chrishiste!)dagster/image label when image is provided from user_defined_k8s_config. (Thanks, @HynekBlaha!)custom_user_agent, was provided by defaultPipesGlueClient.run have been deprecated and will be removed in 1.9.0. The new params argument should be used instead.Fixed an issue where viewing run logs with the latest 5.0 release of the watchdog package raised an exception.
allow_missing_partitions configuration option.This version of Dagster resulted in errors when trying to launch runs that target individual asset partitions)
This version of Dagster resulted in errors when trying to launch runs that target individual asset partitions)
PartitionsDefinitions, there will no longer be an implicit asset job __ASSET_JOB_... for each PartitionsDefinition; there will just be one with all the assets. This reduces the time it takes to load code locations with assets with many different PartitionsDefinitions.[ui] Improved performance of the Automation history view for partitioned assets
OpenAIResource (Thanks, @chasleslr!)MultiPartitionMapping with @dbt_assets (Thanks, @arookieds!)AzureBlobComputeLogManager without a secret_key (Thanks, @ion-elgreco and @HynekBlaha!)AutomationCondition and associated static constructors.PipesCloudWatchMessageReader…is no longer supported. This is motivated by the deprecation of portal.airbyte.com on August 15, 2024.
build_op_context and build_asset_context now accepts a run_tags argument.Definitions.RUN_CANCELED events now display relevant error messages.PipesCloudWatchMessageReader can consume logs from CloudWatch as pipes messages.AzureBlobComputeLogManager now supports service principals, thanks @ion-elgreco!dagster-databricks now supports databricks-sdk<=0.17.0.dagster-datahub now allows pydantic versions below 3.0.0, thanks @kevin-longe-unmind!DagsterDbtTranslator class now supports a modfiying the AutomationCondition for dbt models by overriding get_automation_condition.dagster-pandera now supports polars.replicate(...).fetch_column_metadata() method.OpenAIResource now supports organization, project and base_url for configurting the OpenAI client, thanks @chasleslr!numpy<2, thanks @judahrand!AirbyteCloudResource now supports client_id and client_secret for authentication - the api_key approach is no longer supported. This is motivated by the deprecation of portal.airbyte.com on August 15, 2024.databricks-cli and databricks_apiSlingSourceConnection, SlingTargetConnectionSlingSourceConnection, SlingTargetConnectionbuild_sling_assets, and syncdagster-pipes are replacing the now deprecated Step Launchers as the new recommended approach for executing remote Spark jobs. Three new Pipes clients…
AutomationConditions, is no longer experimental. We now recommend using this system in all cases where asset-centric orchestration is desired. A suite of built-in static constructors have been added for common usecases, such as AutomationCondition.on_missing() (which can fill in missing partitions of assets as soon as upstream data is available), and AutomationCondition.all_deps_blocking_checks_passed() (which can prevent materialization of assets until all upstream blocking checks have passed).AutomationConditions to asset checks, via the automation_condition parameter on @asset_check or AssetCheckSpec.AutomationConditions to observable source assets, via the automation_condition parameter on @observable_source_asset.AutomationCondition to execute arbitrary Python code in the context of a broader expression. This allows you to compose built-in conditions with custom business logic.target arguments on schedules and sensors are now marked stable, allowing a stable way for schedules and sensors to target asset selections without needing to define a job.dagster/table_name metadata tag, containing the fully-qualified name of the destination model, has been added for Airbyte, dlt, Fivetran and Sling assets.dagster/row_count metadata tag, containing the number of records loaded in the corresponding run, has been added for dlt and Sling assets.dagster/column_schema metadata tag, containing column schema information of the destination tables, has been added for Fivetran assets.dagster_aws.pipes.PipesGlueClientdagster_aws.pipes.PipesEMRServerlessClientdagster_aws.pipes.PipesEMRClient@asset and AssetSpec now have a kinds attribute that enables specifying labels that show up on asset nodes in the asset graph in the UI. This supersedes the compute_kind attribute.[pipes] PipesGlueClient, an AWS Glue pipes client has been added to dagster_aws.
dagster_aws.[dagster-dbt] Deprecated the log_column_level_metadata macro in favor of the new with_column_metadata API.
per_step_k8s_config configuration option to the celery_k8s_job_executor , allowing the k8s configuration of individual steps to be configured at run launch time. Thanks @alekseik1!log_column_level_metadata macro in favor of the new with_column_metadata API.load_assets_from_airbyte_project as the Octavia CLI has been deprecated.RunRequest(...) as NoneEcsRunLauncher would sometimes fail to launch runs when the include_sidecars option was set to True.total_byte_billed or total_slot_ms in the BigQuery INFORMATION_SCHEMA.JOBS table.[blueprints] When specifying an asset key in ShellCommandBlueprint, you can now use slashes as a delimiter to generate an AssetKey with multiple path
ShellCommandBlueprint, you can now use slashes as a delimiter to generate an AssetKey with multiple path components.mlflow_run_id attribute (Thanks Joe Percivall!)dagster dev was logging unexpectedly without the grpcio<1.65.0 pin.ContextVar was created in a different context error was raised when executing an async asset.multi_asset type-checker fix from @aksestok, thanks!Passing GraphDefinition as the job argument to schedules and sensors is deprecated. Derive a job from the GraphDefinition using graph_def.to_job() and…
InputContext passed to an IOManager ’s load_input function when invoking the output_value or output_for_node methods on JobExecutionResult now has the name "dummy_input_name" instead of None.DbtProject is adopted and no longer experimental. Using DbtProject helps achieve a setup where the dbt manifest file and dbt dependencies are available and up-to-date, during development and in production. Check out the API docs for more: https://docs.dagster.io/_apidocs/libraries/dagster-dbt#dagster_dbt.DbtProject.—use-dbt-project flag was introduced for the cli command dagster-dbt project scaffold. Creating a Dagster project wrapping a dbt project using that flag will include a DbtProject .DAGSTER_UI_EVENT_LOAD_CHUNK_SIZE environment variable on the Dagster webserver.RunFailureReason.START_TIMEOUT run monitoring failure reason. Thanks @jobicarter!ObserveResult objects to not be stored with the produced AssetObservation event.metadata defined on SourceAssets to be unavailable when accessed in an IOManager.@graph_asset decorator overload missing an owners argument, thanks @askvinni!execute_k8s_job when there was a transient failure while loading logs from the launched job. Thanks @piotrmarczydlo!dateutil package being installed in the default EMR python evnrionment.AutoMaterializeRule.skip_on_parent_missing rule when a parent asset had its PartitionsDefinition changed.AutomationConditions.AutomationCondition.newly_updated() would trigger on any ASSET_OBSERVATION event. Now, it only triggers when the data version on that event changes.dagster-dbt project prepare-for-deployment has been replaced by dagster-dbt project prepare-and-package.DbtProject no longer prepares the dbt manifest file and dbt dependencies in its constructor during initialization. This process has been moved to prepare_if_dev(), that can be called on the DbtProject instance after initialization. Check out the API docs for more: https://docs.dagster.io/_apidocs/libraries/dagster-dbt#dagster_dbt.DbtProject.prepare_if_dev.GraphDefinition as the job argument to schedules and sensors is deprecated. Derive a job from the GraphDefinition using graph_def.to_job() and pass this instead.dagster-plus CLI in the sidenav to correctly be dagster-cloud CLI.[ui] fixes behavior issues with jobs and asset pages introduced in 1.7.11
remove deprecated param usage in dagster-wandb integration (thanks @chris histe)!
AssetsDefinition construction, enforce single key per output namedagster/last_updated_timestamp.AutoMaterializeRule.skip_on_not_all_parents_updated_since_cron() rule gained a new dependency with a different PartitionsDefinition.source_key_prefix arg of load_assets_from_modules. (thanks @drjlin)!load_assets_from_airflow_dag no longer allows multiple tasks to materialize the same asset.dagster-cloud ci init CLI will now use the --deployment argument as the base deployment when creating a branch deployment. This base deployment will be used for Change Tracking.dbt_with_bigquery_insights now respects CLI arguments for profile configuration and also selects location / dataset from the profile when available.Performance improvements when rendering the asset graph while runs are in progress.
build_freshness_checks_for_dbt_assets which allows users to parameterize freshness checks entirely within dbt. Check out the API docs for more: https://docs.dagster.io/_apidocs/libraries/dagster-dbt#dbt-dagster-dbt.max_partitions_per_run from the job’s constituent assets.asset_tags can now be specified when building dagstermill assetsDagsterSlingTranslatordagster/storage_kind tags attachedtags passed to outs in graph_multi_asset now get correctly propagated to the resulting assets.build_metadata_bounds_checks now no longer errors when targeting metadata keys that have special characters.--read-only flag to the dagster-cloud ci branch-deployment CLI command, which returns the current branch deployment name for the current code repository branch without update the status of the branch deployment.Dagster will now display a “storage kind” tag on assets in the UI, similar to the existing compute kind. To set storage kind for an asset, set its dag
dagster/storage_kind tag.dbt retry within a try/except block to avoid unnecessary, duplicate work.AssetExecutionContext now exposes a has_partition_key_range property.owners, metadata, tags, and deps properties on AssetSpec are no longer Optional. The AssetSpec constructor still accepts None values, which are coerced to empty collections of the relevant type.docker_executor and k8s_job_executor now consider at most 1000 events at a time when loading events from the current run to determine which steps should be launched. This value can be tuned by setting the DAGSTER_EXECUTOR_POP_EVENTS_LIMIT environment variable in the run process.dagster/retry_on_asset_or_op_failure tag that can be added to jobs to override run retry behavior for runs of specific jobs. See the docs for more information.build_sensor_for_freshness_checks to describe when/why it skips evaluating freshness checks.@multi_asset_sensor.ScheduleDefinition now properly supports being passed a RunConfig object.MaterializeResult, but the function has no type annotation, previously, the IO manager would still be invoked with a None value. Now, the IO manager is not invoked.AssetSpec constructor now raises an error if an invalid owner string is passed to it.graph_multi_asset decorator, the code_version property on AssetOuts passed in used to be ignored. Now, they no longer are.dagster-cloud job launch command did not support specifying asset keys with prefixes in the --asset-key argument.group:, code location:, tag:, owner:.Nothing published for this version
Backfills created via GQL can have a custom title and description.
Definitions now has a get_all_asset_specs method, which allows iterating over properties of the defined assetsterminate_runs method to the Python GraphQL Client. (thanks @baumann-t!)DbtCliInvocation now has a .get_error() method that can be useful when using dbt.cli(..., raise_on_error=False).DynamicPartitionsDefinition (using partitions_fn) that caused a crash during job backfills.build_metadata_bounds_checks API creates asset checks which verify that numeric metadata values on asset materializations fall within min or max values. See the documentation for more information.build_sensor_for_freshness_checks and Dagster Plus. This API should now work when used with Dagster Plus.dagsterCloudAgent.additionalPodSpecConfig to the Kubernetes agent Helm chart allowing arbitrary pod configuration to be applied to the agent pod.[ui] Command clicking on nodes in the asset lineage tab will now open them in a separate tab. Same with external asset links in the asset graph.
datetime.utcfromtimestamp (thanks @dbrtly!)dbt-core==1.8.*.BackfillPolicys, each asset would get materialized in its own run, rather than grouping assets together into single run.The backfill daemon now has additional logging to document the progression through each tick and why assets are and are not materialized during each e
dagster/row_count .CloudwatchLogsHandler, ECRPublicClient, SecretsManagerResource, SSMResource thanks @jacob-white-simplisafe !TableMetadataValue, TableSchemaMetadataValue, or TableColumnLineageMetadataValue defined.BackfillPolicy of the underlying assets in the job.databricks-sdk version bumped to 0.17.0, thanks @lamalex !dagster code-server start , thanks @SanjaySiddharth !@JonathanLai2004!The Asset > Checks tab now allows you to view plots of numeric metadata emitted by your checks.
MaterializeResult, ObserveResult, or Output, you can now include tags that will be attached to the corresponding AssetMaterialization or AssetObservation event. These tags will be rendered on these events in the UI.build_last_update_freshness_checks and build_time_partition_freshness_checks APIs have been updated to be clearer.% into the asset graph’s query selector no longer crashes the UI.MetadataValue have been changed from NamedTuples to Pydantic models. NamedTuple functionality on these classes was not part of Dagster’s stable public API, but usages relying on their tuple-ness may break. For example: calling json.dumps on collections that include them.dbt-core==1.5.* has been removed, as it has reached end of life in April 2024.dagster-cloud CLI where the --deployment argument was ignored when the DAGSTER_CLOUD_URL environment variable was set.dagster-cloud-cli package wouldn’t work unless the dagster-cloud package was installed as well.TimeWindowPartitionMapping now supports the start_offset and end_offset parameters even when the upstream PartitionsDefinition is different than the d
TimeWindowPartitionMapping now supports the start_offset and end_offset parameters even when the upstream PartitionsDefinition is different than the downstream PartitionsDefinition. The offset is expressed in units of downstream partitions, so TimeWindowPartitionMapping(start_offset=-1) between an hourly upstream and a daily downstream would map each downstream partition to 48 upstream partitions – those for the same and preceding day.path metadata from UPathIOManager inputs. This eliminates the creation of ASSET_OBSERVATION events for every input on every step for the default I/O manager.owners on @graph_asset.DagsterInvalidSubsetError when trying to launch runs.dagster-cloud branch-deployment CLI, you can now specify the base deployment. The base deployment will be used for comparing assets for Change Tracking. For example, to set the base deployment to a deployment named staging: dagster-cloud branch-deployment create-or-update --base-deployment-name staging .... Note that once a Branch Deployment is created, the base deployment cannot be changed.413: Request Entity Too Large error when uploading a heartbeat to the Dagster Plus servers.@graph_asset now accepts a tags argument
@graph_asset now accepts a tags argumentload_asset_defs_from_fivetran_instance. Thanks @lamalex!Duplicate check specs errors with singular tests ingested as asset checks.source.with_resources(...)Performance improvements when loading large asset graphs in the Dagster UI.
@asset_check functions can now be invoked directly for unit testing.dagster-embedded-elt dlt resource DagsterDltResource can now be used from @op definitions in addition to assets.UPathIOManager.load_partitions has been added to assist with helping UpathIOManager subclasses deal with serialization formats which support partitioning. Thanks @danielgafni!PolarsDeltaIOManager now supports MultiPartitionsDefinition with DeltaLake native partitioning. Metadata value "partition_by": {"dim_1": "col_1", "dim_2": "col_2"} should be specified to enable this feature. Thanks @danielgafni!load_assets_from_airbyte_instance and load_assets_from_airbyte_project will now be properly propagated to the created assets.build_time_partition_freshness_checks where it would incorrectly intuit that an asset was not fresh in certain cases.PolarsDeltaIOManager no longer supports loading natively partitioned DeltaLake tables as dictionaries. They should be loaded as a single pl.DataFrame/pl.LazyFrame instead.Dagster Cloud to Dagster+ all over the docs.before_storage_id / after_storage_id values in the AssetRecordsFilter class were ignored. This has been fixed.dagster-cloud deployment alert-policies list to match the format of sync.Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
[dagster-dbt][experimental] A new cli command dagster-dbt project prepare-for-deployment has been added in conjunction with DbtProject for managing th
dagster-dbt project prepare-for-deployment has been added in conjunction with DbtProject for managing the behavior of rebuilding the manifest during development and preparing a pre-built one for production.build_last_update_freshness_checks and build_time_partition_freshness_checks has been fixed where multi_asset checks passed in would not be executable.DbtArtifacts class has been replaced by DbtProject.AssetObservation to the Software-defined Assets API referencedagster-cloud-deploy.ymlget_asset_key API documentation (thanks @aksestok!)Nothing published for this version
[community-contribution] Fixed an issue where an excessive DeprecationWarning was being issued for a ScheduleDefinition passed into the Definitions ob…
OutputContext.output_metadata.DAGSTER_EVENT_BATCH_SIZE environment variable in a code server to an integer (25 recommended, 50 max). It is only currently supported in Dagster Cloud and OSS deployments with a postgres backend.., which can occur when checks are ingested from dbt tests.DBT_INDIRECT_SELECTION will no longer be set to empty when executing dbt tests as asset checks, unless specific asset checks are excluded. dagster-dbt will no longer explicitly select all dbt tests with the dbt cli, which had caused argument length issues.{{
config(
meta={
'dagster': {
'ref': {
'name': <ref_name>,
'package': ... # Optional, if included in the ref.
'version': ... # Optional, if included in the ref.
},
}
}
)
}}
...
build_schedule_from_dbt_selection. Thanks @dragos-pop!dagster_polars. Thanks @danielgafni!datadog.api module to the DatadogClient resource, enabling direct access to API methods. Thanks @shivgupta!metadata argument on multi_asset and AssetSpec to Mapping[str, Any].s3_io_manager.dagster-polars tests by excluding Decimal types. Thanks @ion-elgreco!ScheduleDefinition passed into the Definitions object. Thanks @2Ryan09!run_id was previously private and only used for testing. It will now raise an exception.get_partition_keys_in_range on a MultiPartitionsDefinition would erroneously return partition keys that were within the one-dimensional range of alphabetically-sorted partition keys for the definition. Now, this method returns the cartesian product of partition keys within each dimension’s range. Thanks, @mst!AssetCheckExecutionContext to replace AssetExecutionContext as the type of the context param passed in to @asset_check functions. @asset_check was an experimental decorator.@classmethod decorators have been removed from dagster-embedded-slt.sling DagsterSlingTranslator@classmethod decorators have been removed from DagsterDbtTranslator.AssetSelection.keys() has been deprecated. Instead, you can now supply asset key arguments to AssetSelection.assets() .AssetExecutionContext has been simplified. Op-related methods and methods with existing access paths have been marked deprecated. For a full list of deprecated methods see this GitHub Discussion.metadata property on InputContext and OutputContext has been deprecated and renamed to definition_metadata .FreshnessPolicy is now deprecated. For monitoring freshness, use freshness checks instead. If you are using AutoMaterializePolicy.lazy(), FreshnessPolicy is still recommended, and will continue to be supported until an alternative is provided.dagster-polars documentation. Thanks @danielgafni!Fixed a bug where an asset with a dependency on a subset of the keys of a parent multi-asset could sometimes crash asset job construction.
AssetCheckResult now has a text description property. Check evaluation descriptions are shown in the Checks tab on the asset details page.
AssetCheckResult now has a text description property. Check evaluation descriptions are shown in the Checks tab on the asset details page.TimestampMetadataValue. Timestamp metadata values are represented internally as seconds since the Unix epoch. They can be constructed using MetadataValue.timestamp. In the UI, they’re rendered in the local timezone, like other timestamps in the UI.AssetSelection.checks can now accept AssetCheckKeys as well as AssetChecksDefinition.add_output_metadata or by passing to Output) is now available on HookContext under the op_output_metadata property. Thanks @JYoussouf!@asset, AssetSpec, and AssetOut now accept a tags property. Tags are key-value pairs meant to be used for organizing asset definitions. If "__dagster_no_value" is set as the value, only the key will be rendered in the UI. AssetSelection.tag allows selecting assets that have a particular tag.dagster asset materialize --select tag:department=marketingDagsterDbtTranslator.get_tags. By default, we take the dbt tags configured on your dbt models, seeds, and snapshots.DatabricksJobRunner that led to an inability to use dagster-databricks with Databricks instance pools. Thanks @smats0n!AssetKey. Thanks @maxfirman!load_assets_from_package_module that would cause a crash when any submodule had the same directory name as a dependency. Thanks @CSRessel!"columns" to "dagster/column_schema". Materializations using the old metadata key will no longer result in the Columns section of the tab being filled out.Fixed an issue where dagster dev or the Dagster UI would display an error when loading jobs created with op or asset selections.
dagster dev or the Dagster UI would display an error when loading jobs created with op or asset selections.Latency improvements to the scheduler when running many simultaneous schedules.
@multi_asset s are in use has been drastically improved.the_asset.with_attributes is called without providing a group name. Previously, the existing group name was erroneously dropped. Thanks @ion-elgreco!dbt source freshness.MetadataValue in Essentials course. Thanks @stianthaulow!tag_keys argument on DagsterInstance.get_run_tagsis no longer optional. This has been done to remove an easy way of accidentally executing an extremely expensive database operation.[ui] When viewing logs for a run, the date for a single log row is now shown in the tooltip on the timestamp. This helps when viewing a run that takes
AssetSelection.dagster-snowflake now requires snowflake-connector-python>=3.4.0@sling_assets accepts an optional name parameter for the underlying opdagster-openai library is now available.DagsterDbtTranslatorSettings called enable_duplicate_source_asset_keys that allows users to set duplicate asset keys for their dbt sources. Thanks @hello-world-bfree!ConfigurableResource.SlingConnectionResource where a blank keyword argument would be emitted as an environment variabledbt source freshness would cause an error.DbtArtifacts has been added for managing the behavior of rebuilding the manifest during development but expecting a pre-built one in production.send sample alert button now exists on both the alert policies page and in the alert policies editor to make it easier to debug and configure alerts without having to wait for an event to kick them off.[dagster-embedded-elt] Fixed a bug in the SlingConnectionResource that raised an error when connecting to a database.
SlingConnectionResource that raised an error when connecting to a database.graph_multi_assets with check_specs now support subsetting.…package. Deprecated build_sling_asset, SlingSourceConnection and SlingTargetConnection.
run_retries.retry_on_op_or_asset_failures setting that can be set to false to make run retries only occur when there is an unexpected failure that crashes the run, allowing run-level retries to co-exist more naturally with op or asset retries. See the docs for more information.dagster dev now sets the environment variable DAGSTER_IS_DEV_CLI allowing subprocesses to know that they were launched in a development context.@multi_observable_source_asset decorator enables defining a set of assets that can be observed together with the same function.@sling_assets and Resource SlingConnectionResource have been added for the [dagster-embedded-elt.sling](http://dagster-embedded-elt.sling) package. Deprecated build_sling_asset, SlingSourceConnection and SlingTargetConnection.QueuedRunCoordinator.Dagster officially supports Python 3.12.
dagster-polars has been added as an integration. Thanks @danielgafni!@dbt_assets now supports loading projects with semantic models.@dbt_assets now supports loading projects with model versions.get_asset_key_for_model now supports retrieving asset keys for seeds and snapshots. Thanks @aksestok!@asset decorator causing a false positive in Pyright strict mode. Thanks @tylershunt!security_context field on the k8s_job_executor didn't correctly set the security context on the launched step pods. Thanks @krgn!ObserveResults with no data_version.FreshnessPolicys on observable source assets. These assets will be considered “Overdue” when the latest value for the “dagster/data_time” metadata value is older than what’s allowed by the freshness policy.Within a backfill or within auto-materialize, when submitting runs for partitions of the same assets, runs are now submitted in lexicographical order
DbtCliResource now include metadata from the dbt adapter response. This includes fields like rows_affected, query_id from the Snowflake adapter, or bytes_processed from the BigQuery adapter.k8s_job_executor.context.log.exception in a sensor or schedule did not properly capture exception information.@observable_source_asset-decorated functions can now return an ObserveResult. This allows including metadata on the observation, in addition to a data version. This is currently only supported for non-partitioned assets.AutoMaterializeRule.skip_on_not_all_parents_updated_since_cron class allows you to construct AutoMaterializePolicys which wait for all parents to be updated after the latest tick of a given cron schedule.The following methods on AssetExecutionContext have been marked deprecated, with their suggested replacements in parenthesis:
build_schedule_from_partitioned_job now supports creating a schedule from a static-partitioned job (Thanks @craustin!)PipesK8sClient will now autodetect the namespace when using in-cluster config. (Thanks @aignas!)PipesK8sClient can now inject the context in to multiple containers. (Thanks @aignas!)write_pandas method to load Pandas DataFrames in Snowflake. To support this change, the database connector was switched from SqlDbConnection to SnowflakeConnection .@ion-elgreco!)@dbt_assets and multiple dbt resources produce the same AssetKey, we now display an exception message that highlights the file paths of the misconfigured dbt resources in your dbt project.@jblawatt!)automountServiceAccountToken: false to the default service account used by the Helm chart, in order to better comply with security policies. (Thanks @MattyKuzyk!)@ruizh22!)PipesK8sClient has improved handling for init containers and additional containers. (Thanks @aignas!)last_sensor_start_time property of the SensorEvaluationContext, which would get cleared on ticks after the first tick after the sensor starts.dagster instance migrate --bigint-migration, which caused some operational errors on mysql storages.AssetExecutionContext have been marked deprecated, with their suggested replacements in parenthesis:
context.op_config (context.op_execution_context.op_config)context.node_handle (context.op_execution_context.node_handle)context.op_handle (context.op_execution_context.op_handle)context.op (context.op_execution_context.op)context.get_mapping_key (context.op_execution_context.get_mapping_key)context.selected_output_names (context.op_execution_context.selected_output_names)context.dagster_run (context.run)context.run_id (context.run.run_id)context.run_config (context.run.run_config)context.run_tags (context.run.tags)context.has_tag (key in context.run.tags)context.get_tag (context.run.tags.get(key))context.get_op_execution_context (context.op_execution_context)context.asset_partition_key_for_output (context.partition_key)context.asset_partition_keys_for_output (context.partition_keys)context.asset_partitions_time_window_for_output (context.partition_time_window)context.asset_partition_key_range_for_output (context.partition_key_range)@asset_check now has a blocking parameter. When this is enabled, if the check fails with severity ERROR then any downstream assets in the same run won’t execute.MaterializeResultEdit button next to an agent token on the Tokens tab in Org Settings to configure permissions for a particular token. You must be an Organization Admin to edit agent token permissions.Added support for the 3.0 release of the pendulum library, for Python versions 3.9 and higher.
pendulum library, for Python versions 3.9 and higher.AllPartitionMapping now supports mapping to downstream partitions, enabling asset backfills with these dependencies. Thanks @craustin!@asset_check has new fields additional_deps and additional_ins to allow dependencies on assets other than the asset being checked.dbt-core==1.4.* is now removed because the version has reached end-of-life.get_partition_keys_not_in_subset on a BaseTimeWindowPartitionsSubset that targeted a partitions definition with no partitions (e.g. a future start date) would raise an error. Now, it returns an empty list.LocalComputeLogManager on Windows could result in errors relating to invalid paths. This has been resolved. Thanks @hainenber!DatabricksJobRunner now ensures the correctdatabricks-sdk is installed. Thanks @zyd14!config.meta.description. Thanks @CapitanHeMo!PipesK8sClient can now correctly handle load_incluster_config . Thanks @aignas!agent_queue key in a dagster_cloud.yaml file incorrectly failed to validate when using the dagster-cloud ci init or dagster-cloud ci check commands during CI/CD.The warning for unloadable sensors and schedules in the Dagster UI has now been removed.
warn_if/ error_if severities1.6.0 where run status sensors did not cursor correctly when deployed on Dagster Cloud.Added experimental functionality which hides user code errors from the Dagster UI. You may enable this functionality by setting the DAGSTER_REDACT_USE
DAGSTER_REDACT_USER_CODE_ERRORS environment variable to 1.@dbt_assets now accepts a required_resource_keys argument.This replaces the experimental external_assets_from_specs API, as well as SourceAssets, which are now deprecated. Unlike SourceAssets, AssetSpecs can…
AssetSpec objects to the assets argument of Definitions, to let Dagster know about assets without associated materialization functions. This replaces the experimental external_assets_from_specs API, as well as SourceAssets, which are now deprecated. Unlike SourceAssets, AssetSpecs can be used for non-materializable assets with dependencies on Dagster assets, such as BI dashboards that live downstream of warehouse tables that are orchestrated by Dagster. [docs].Definitions objects together into a single larger Definitions object, using the new Definitions.merge API (doc). This makes it easier to structure large Dagster projects, as you can construct a Definitions object for each sub-domain and then merge them together at the top level.BackfillPolicys assigned to assets are now respected for backfills launched from jobs that target those assets.AutomationConditions to your assets to have them automatically executed in response to specific conditions (docs). These serve as a drop-in replacement and improvement over the AutoMaterializePolicy system, which is being marked as deprecated.target parameter, instead of needing to construct a job.Experimental navigation feature flag in user settings.build_metadata_bounds_checks API [doc] enables easily defining asset checks that fail if a numeric asset metadata value falls outside given bounds.PipesSubprocessClient) and its integrations with Lambda (PipesLambdaClient), Kubernetes (PipesK8sClient), and Databricks (PipesDatabricksClient) are no longer experimental.DbtProject class (docs) makes it simpler to define dbt assets that can be constructed in both development and production. DbtProject.prepare_if_dev() eliminates boilerplate for local development, and the dagster-dbt project prepare-and-package CLI can helps pull deps and generate the manifest at build time.dagster-looker package can be used to define a set of Dagster assets from a Looker project that is defined in LookML and is backed by git. See the GitHub discussion for more details.Default op/asset concurrency limits are now configurable at the deployment level, using the concurrency > default_op_concurrency_limit configuration i
concurrency > default_op_concurrency_limit configuration in your dagster.yaml (OSS) or Deployment Settings page (Dagster Cloud). In OSS, this feature first requires a storage migration (e.g. dagster instance migrate).dagster instance migrate).Nothing-typed output is returned from an asset or op, the handle_output function of the I/O manager will no longer be called. Users of most Dagster-maintained I/O managers will see no behavioral changes, but users of the In-Memory I/O manager, or custom I/O managers that store Nothing-typed outputs should reference the migration guide for more information.dagster asset materialize will now return a non-zero exit code upon failure.dagster<1.5.1, you may see the first tick of your run status sensor skip runs that completed but were not yet registered by the sensor during your upgrade. This should not be common, but to avoid any chance of that, you may consider an interim upgrade to dagster>=1.5.1,<1.6.0 first.Nothing published for this version
The SensorEvaluationContext object has two new properties: last_sensor_start_time and is_first_tick_since_sensor_start. This enables sensor evaluation
SensorEvaluationContext object has two new properties: last_sensor_start_time and is_first_tick_since_sensor_start. This enables sensor evaluation functions to vary behavior on the first tick vs subsequent ticks after the sensor has started.asset_selection argument to @sensor and SensorDefinition now accepts sequence of AssetsDefinitions, a sequences of strings, or a sequence of AssetKeys, in addition to AssetSelections.dbt-core==1.3.* has been removed.show_url_only parameter to GCSComputeLogManager that allows you to configure the compute log manager so that it displays a link to the GCS console rather than loading the logs from GCS, which can be useful if giving Dagster access to GCS credentials is undesirable.BranchingIOManagerhashlib.md5 as not used for security purposes (Thanks @jlloyd-widen!)execute_k8s_job to be aware of run-termination and op failure by deleting the executing k8s job (Thanks @Taadas!).plotly graphs) when viewing dagstermill notebooks (Thanks @aebrahim!).DbtCliResource API docs (Thanks @akan72!)report_custom_message ) to the orchestration process (get_custom_messages).[dagster-embedded-elt] Fixed an issue where EnvVars used in Sling source and target configuration would not work properly in some circumstances.
EnvVars used in Sling source and target configuration would not work properly in some circumstances.[ui] Asset graph now displays active filters.
DAGSTER_DBT_PARSE_PROJECT_ON_LOAD=1 dagster dev in a new scaffolded project from dagster-dbt project scaffold, dbt logs from creating dbt artifacts to loading the project are now silenced.connection_meta_to_group_fn argument which allows configuring loaded asset groups based on the connection’s metadata dict.QueuedRunCoordinatorDaemon has been refactored to paginate over runs when applying priority sort and tag concurrency limits. Previously, it loaded all runs into memory causing large memory spikes when many runs were enqueued.UPathIOManager has been updated to use the correct path delimiter when interacting with cloud storages from a Windows process.STEP_WORKER_STARTED event now fires before importing code in line with the other executors.EnvVar did not work properly.IAttachDifferentObjectToOpContext would pass the incorrect object to schedules and sensors.materialize_on_cron rule with dynamically partitioned assets.without_checks no longer fails by attempting to include the checks.DATABRICKS_HOST is set. Thanks @zyd14!PipesLambdaClient, an AWS Lambda pipes client has been added to dagster_aws.PipesLambdaClient with Dagster Pipes.Marked the experimental Airbyte ingestion-as-code feature as deprecated, to be removed in a future release. We suggest users interested in managing th…
MetadataValue.job metadata type, which can be used to link to a Dagster job from other objects in the UI.schema metadata set on the asset or op, I/O manager schema/ dataset configuration, key_prefix set on the asset. Previously, all methods for setting the schema/dataset were mutually exclusive, and setting more than one would raise an exception.DAGSTER_DBT_PARSE_PROJECT_ON_LOAD=1 dagster dev in a new scaffolded project from dagster-dbt project scaffold, dbt artifacts for loading the project are now created in a static target/ directory.ScheduleEvaluationContext when testing via build_schedule_context.metadata from a Failure exception is now hoisted up to the failure that culminates when retry limits are exceeded.PipesK8sClient now correctly raises on failed containers.maxCatchupRuns and maxTickRetries configuration options for the scheduler in the Helm chart.DBT_INDIRECT_SELECTION=empty.@asset(check_specs=... to not cooperate with the key_prefix argument of the load_assets_from_modules method and it’s compatriots.define_asset_job now accepts an op_retry_policy argument, which specifies a default retry policies for all of the ops in the job. (thanks Eugenio Contreras!)observable_source_asset decorator now accepts a key argument.implicit_materializations argument has been added to get_results and get_materialize_result to control whether an implicit materialization event is created or not.SlingConnectionResource to allow reusing sources and targets interoperably.build_dbt_asset_selection now also selects asset checks based on their underlying dbt tests. E.g. build_dbt_asset_selection([my_dbt_assets], dbt_select="tag:data_quality") will select the assets and checks for any models and tests tagged with ‘data_quality’.EnvVar vs. os.getenv to the Environment variables documentation.The email_on_failure sensor called deprecated methods on the context. This has been fixed
AutoMaterializeRule.materialize_on_cron() rule makes it possible to create policies which materialize assets on a regular cadence.SensorResult from a sensor no longer overwrites a cursor if it was set via the context.can_subset=True alongside assets which were upstream of some assets in the multi-asset, and downstream of others.HourlyPartitionsDefinition with a non-UTC timezone and the default format string (or any format string not including a UTC-offset), there was no way to disambiguate between the first and second instance of the repeated hour during a daylight saving time transition. Now, for the one hour per year in which this ambiguity exists, the partition key of the second instance of the hour will have the UTC offset automatically appended to it.check_specs to AssetsDefinition.from_graphdagster-dbt that caused some dbt tests to not be selected as asset checks.email_on_failure sensor called deprecated methods on the context. This has been fixedDagsterInstance.report_runless_asset_event is now public.AutoMaterializeRule.materialize_on_parent_updated now accepts an updated_parents_filter of type AutoMaterializeAssetPartitionsFilter, which allows only materializing based on updates from runs with a required set of tags.Fixed an error when trying to directly invoke a run status sensor when passing resources.
EnvVars used in Airbyte or Fivetran resources would show up as their processed values in the launchpad when loading assets from a live Fivetran or Airbyte instance.The OpExecutionContext and AssetExecutionContext now have a partition_keys property
OpExecutionContext and AssetExecutionContext now have a partition_keys propertydbt-core==1.7.* is now supported.build_schedule_from_partitioned_job was used with a job with multi-partitioned assets and the partitions_def argument wasn’t provided to define_asset_job.config.dagster-embedded-elt where sling’s updated_at parameter was set to the incorrect typeAssetCheckKeyPipesSubprocessClient now inherits the environment variables of the parent process in the launched subprocess.[dagster-k8s] The PipesK8sClient will now attempt to load the appropriate kubernetes config, and exposes arguments for controlling this process.
PipesK8sClient will now attempt to load the appropriate kubernetes config, and exposes arguments for controlling this process.load_asset_checks_from_modules functions for loading asset checks in tandem with load_assets_from_modules.dagster-dbt project scaffold now uses ~/.dbt/profiles.yml if a profiles.yml is not present in the dbt project directory.@dbt_assets now support PartitionMapping using DagsterDbtTranslator.get_partition_mapping.@dbt_assets. To enable this, add the following metadata to your dbt model’s metadata in your dbt project:meta:
dagster:
has_self_dependency: True
pydantic<2.0.0 but having pydantic-core installed.LastPartitionMapping dependency would raise an error. This has been fixed.instance is not available to load partitions error was raised.build_asset_with_blocking_check1.5.0 where instances that haven’t been migrated to the latest schema hit an error upon run deletion.exclude that didn’t match any dbt nodes was used in @dbt_assets, an error would be raised. The error is now properly handled.DbtCliResource.cli(...) in an op, AssetMaterialization's instead of Output are now emitted.report_asset_observation REST endpoint for reporting runless events is now available.Your coding agent can read these notes before it upgrades. Set up the MCP server →