NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
PyPI · #1999 most downloaded on PyPI
Package for AWS-specific Dagster framework solid and resource components.
Last release today
16 Sep 2026
Ships on a steady schedule
a new release about every 2 weeks
Nearly every release is documented
notes for 59 of the last 60 stable releases
4 versions withdrawn
withdrawn after publishing
7 years old
682 releases · first in 2019
Alert policies can now be set on assets + asset checks (currently experimental). Check out the alerting docs for more information.
--live-data-poll-rate that allows configuring how often the UI polls for new asset data when viewing the asset graph, asset catalog, or overview assets page. It defaults to 2000 ms.report_asset_materialization REST API endpoint for creating external asset materialization events. This is available in cloud as well.@dbt_assets decorator now accepts a backfill_policy argument, for controlling how the assets are backfilled.@dbt_assets decorator now accepts a op_tags argument, for passing tags to the op underlying the produced AssetsDefinition.get_materialize_result & get_asset_check_result to PipesClientCompletedInvocationacryl-datahub pin in the dagster-datahub package has been removed.PipesDatabricksClient now performs stdout/stderr forwarding from the Databricks master node to Dagster.dagster-dbt-cloud CLI.K8sRunLauncher. See the docs for more information.Previously, the asset backfill page would display negative counts if failed partitions were manually re-executed. This has been fixed.
Fixed an issue where the run list dialog for viewing the runs occupying global op concurrency slots did not expand to fit the content size.
Fixed an issue where selecting a partition would clear the launchpad and typing in the launchpad would clear the partition selection
Fixed various issues with the asset-graph displaying the wrong graph
The IO manager’s handle_output method is no longer invoked when observing an observable source asset.
[ui] Fixed an issue where the run config dialog could not be scrolled.
[pipes] Fixed an issue in the PipesDockerClient with parsing logs fetched via the docker client.
[external assets] Fixed an issue in external_assets_from_specs where providing multiple specs would error
[external assets] Correct copy in tooltip to explain why Materialize button is disabled on an external asset.
dagster-pipes version used in the external process.is_dagster_pipes_process has been removed from the dagster-pipes package.One column per quarter.
Previously, asset backfills targeting assets with multi-run backfill policies would raise a "did not submit all run requests" error. This has been fix
[dagster-dbt] Enabling asset checks using dbt project metadata has been deprecated.
EnvVar utility will now raise an exception if it is used outside of the context of a Dagster resource or config class. The get_value() utility will retrieve the value outside of this context.DbtCliResource, an explicit target_path can now be specified.DagsterDbtTranslator and DagsterDbtTranslatorSettings: see the docs for more information.Asset lineage graph UI revamp, to make it easier to visualize and navigate large graphs
MaterializeResult, AssetSpec, and AssetDep now marked stable – These APIs, introduced in Dagster 1.5, were previously marked experimental. They offer a more straightforward way of defining assets when you don’t want to use I/O managers.PipesLambdaClient [guide].report_custom_message and get_custom_messages.[ui] Global Asset Graph performance improvement - the first time you load the graph it will be cached to disk and any subsequent load of the graph sho
AssetExecutionContext is now a subclass of OpExecutionContext, not a type alias. The codedef my_helper_function(context: AssetExecutionContext): ...
@op
def my_op(context: OpExecutionContext):
my_helper_function(context)
will cause type checking errors. To migrate, update type hints to respect the new subclassing.
AssetExecutionContext cannot be used as the type annotation for @ops run in @jobs. To migrate, update the type hint in @op to OpExecutionContext. @ops that are used in @graph_assets may still use the AssetExecutionContext type hint.## old
@op
def my_op(context: AssetExecutionContext): ...
## correct
@op
def my_op(context: OpExecutionContext): ...
backfill_policy=BackfillPolicy.single_run() to your assets.has_dynamic_partition implementation has been optimized. Thanks @edvardlindelof!stream_to_asset_map argument to build_airbyte_assets to support the Airbyte prefix setting with special characters. Thanks @chollinger93!DatabricksPysparkStepLauncher fails to get logs when job_run doesn’t have cluster_id at root level. Thanks @PadenZach!dagster_cloud.dagster_insights module that contains utilities for capturing and submitting external metrics about data operations to Dagster Cloud via an api. Dagster Cloud Insights is a soon-to-be released feature that shows improves visibility into usage and cost metrics such as run duration and Snowflake credits in the Cloud UI.Deprecated ExpectationResult. This will be made irrelevant by upcoming data quality features.
ExpectationResult. This will be made irrelevant by upcoming data quality features.SensorResult.The deps parameter for @asset and @multi_asset now supports directly passing @multi_asset definitions. If an @multi_asset is passed to deps, dependenc
deps parameter for @asset and @multi_asset now supports directly passing @multi_asset definitions. If an @multi_asset is passed to deps, dependencies will be created on every asset produced by the @multi_asset.dagster instance migrate --bigint-migration.DbtCliResource now validates at definition time that its project_dir and profiles_dir arguments are directories that respectively contain a dbt_project.yml and profiles.yml.policy_id for new clusters when using the databricks_pyspark_step_launcher (thanks @zyd14!)dagster-webserver command was not indicating which port it was using in the command-line output.MaterializeResult has been added as a new return type to be used in @asset / @multi_asset materialization functionsAdded a new tooltip to asset runs to either view the asset list or lineage
dagster-ext module along with subprocess, docker, databricks, and k8s pod integrations are now available. Read more at https://github.com/dagster-io/dagster/discussions/16319. Note that the module is temporarily being published to PyPI under dagster-ext-process, but is available in python as import dagster_ext.@asset_check or AssetChecksDefinition.check_specs argument to @graph_multi_asset@graph_asset that would raise an error about nonexistant checksOpExecutionContext.add_output_metadata can now be called multiple times per output.
OpExecutionContext.add_output_metadata can now be called multiple times per output.@janosroden !)29 2 *) no longer cause exceptions in the UI or daemon.observable_source_assets with different partition definitions existed in the same code location, runs targeting those assets could fail to launch. This has been fixed.start_date of their underlying PartitionsDefinition could result in runs being launched for partitions that no longer existed. This has been fixed.@ldnicolasmay!)DagsterDbtTranslator did not properly invoke get_auto_materialize_policy and get_freshness_policy for load_assets_from_dbt_project.@graph_asset support. This can be used to implement blocking checks, by raising an exception if the check fails.@multi_asset subsetting, so only checks which target assets in the subset will execute.AssetCheckSpecs will now cause an error at definition time if they target an asset other than the one they’re defined on.run_monitoring > free_slots_after_run_end_secondsNothing published for this version
Nothing published for this version
The context object now has an asset_key property to get the AssetKey of the current asset.
context object now has an asset_key property to get the AssetKey of the current asset.dagster dev and dagster-daemon run commands now include a --log-level argument that allows you to customize the logger level threshold.AirbyteResource now includes a poll_interval key that allows you to configure how often it checks an Airbyte sync’s status.DatabricksStepLauncher to fail. Thanks @zyd14!workspace and volumes init scripts in the databricks client. Thanks @zyd14!Asset checks are now displayed in the asset graph and sidebar.
[Breaking] Asset check severity is now set at runtime on AssetCheckResult instead of in the @asset_check definition. Now you can define one check that either errors or warns depending on your check logic. ERROR severity no longer causes the run to fail. We plan to reintroduce this functionality with a different API.
[Breaking] @asset_check now requires the asset= argument, even if the asset is passed as an input to the decorated function. Example:
@asset_check(asset=my_asset)
def my_check(my_asset) -> AssetCheckResult: ...
[Breaking] AssetCheckSpec now takes asset= instead of asset_key=, and can accept either a key or an asset definition.
[Bugfix] Asset checks now work on assets with key_prefix set.
[Bugfix] Execution failure asset checks are now displayed correctly on the checks tab.
DbtCliResource in custom asset/op to API docs.Dagster code servers now wait to shut down until any calls that they are running have finished, preventing them from stopping while in the middle of e
dagster execute job cli now accepts —-op-selection (thanks @silent-lad!)AssetsDefinition instances.@graph and @job now work again, fixing a regression introduced in 1.4.5.ins argument to graph_asset is now respected correctly.dagster dev failed on startup when the DAGSTER_GRPC_PORT `environment variable was set in the environment.deps arguments for an asset can now be specified as an iterable instead of a sequence, allowing for sets to be passed.securityContext setting now applies correctly to all init containers (thanks @maowerner!)AutoMaterializeRule.skip_on_not_all_parents_updated that enforces that an asset can only be materialized if all parents have been materialized since the asset's last materialization.AutoMaterializeRule.skip_on_parent_missing –which is already part of the behavior of the default auto-materialize policy.dagster-dbt project scaffold.[dagster-webserver] Fixed an issue that broke loading static files on Windows.
[dagster-webserver] Fixed an issue that caused some missing icons in the UI.
The ConfigurablePickledObjecS3IOManager will continue to be available but is considered deprecated in favor of S3PickleIOManager. There is no change i…
@partitioned_config decorator has been added for defined configuration for partitioned jobs. Thanks @danielgafni!ConfigurablePickledObjectS3IOManager has been renamed S3PickleIOManager for simplicity. The ConfigurablePickledObjecS3IOManager will continue to be available but is considered deprecated in favor of S3PickleIOManager. There is no change in the functionality of the I/O manager.ConfigurablePickledObjectADLS2IOManager has been renamed ADLS2PickleIOManager for simplicity. The ConfigurablePickledObjectADLS2IOManager will continue to be available but is considered deprecated in favor of ADLS2PickleIOManager. There is no change in the functionality of the I/O manager.DbtCliResource, the exception message now includes a link to the dbt.log produced. This log file can be inspected for debugging.ConfigurablePickledObjectGCSIOManager has been renamed GCSPickleIOManager for simplicity. The ConfigurablePickledObjecGCSIOManager will continue to be available but is considered deprecated in favor of GCSPickleIOManager. There is no change in the functionality of the I/O manager.DagsterInvariantViolationError when executing a multi-asset where both assets have self-dependencies on earlier partitions.--debug flag raised an exception in the Dagster framework.stderr which aligns with Python’s logging defaults.BackfillPolicy, thanks @ruizh22!timeout argument when constructing a DagsterGraphQLClient.DbtCliResource and DbtCliResource.cli(...).InputContext and OutputContext have been fixed. Thanks @Sergey Mezentsev!Added a respect_materialization_data_versions option to auto materialization. It can enabled in dagster.yaml with
Added a respect_materialization_data_versions option to auto materialization. It can enabled in dagster.yaml with
auto_materialize:
respect_materialization_data_versions: True
This flag may be changed or removed in the near future.
ops or assets with multiple outputs that are all required and return type None/ Nothing will interpret an explicitly or implicitly returned value None
None/ Nothing will interpret an explicitly or implicitly returned value None to indicate that all outputs were successful.skip_reason argument to the constructor of SensorResult now accepts a string in addition to a SkipReason.step_k8s_config field to k8s_job_executor that allows you to customize the raw Kubernetes config for each step in a job. See the docs for more information.dagster-dbt project scaffold now creates the scaffold in multiple files:
constants.py contains a reference to your manifest and dbt project directoryassets.py contains your initial dbt assets definitionsdefinitions.py contains the code to load your asset definitions into the Dagster UIschedules.py contains an optional schedule to add for your dbt assetsget_auto_materialize_policy and get_freshness_policy to DagsterDbtTranslator.load_assets_from_fivetran_instance.SSHResource would warn when allow_host_key_change was set. Now known hosts are always loaded from the system hosts file, and the allow_host_key_change parameter is ignored.@graph_multi_asset now has an API docs entry.GCSComputeLogManager example in the Dagster Instance reference is now correct.@dbt_assets for the following use-cases:
--vars@graph_asset now takes a config parameter equivalent to the parameter on @graph.
@graph_asset now takes a config parameter equivalent to the parameter on @graph.dynamic_partitions_store argument to DynamicPartitionsDefinition for multi-partition run properly with dynamic partitions (Thanks @elzzz!). partitionsByAssets to backfillParams for ranged partition backfill (Thanks @ruizh22!).dbt-core==1.6 has been added.DbtCliResource now supports configuring profiles_dir.restart_policy on k8s_job_op (Thanks @Taadas!).authenticator to SnowflakePandasIOManager, which allows specifying the authentication mechanism to use (Thanks @pengw0048!).AutoMaterializePolicy with assets that had at least one source asset parent and at least one non-source asset parent. This has been fixed.AutoMaterializePolicy to a time-partitioned asset downstream of an unpartitioned asset, the latest partition would only ever be materialized a single time, rather than updating in response to any parent updates. This has been fixed.StaticPartitionsDefinition containing many thousands of partitions could take a significant amount of time.Unexpected exception error would be raised when scaffolding a pull request on a repository with no profiles.yml. This behavior has been updated to raise a more descriptive error message on the repo selection page.agentReplicas config setting on the helm chart has been renamed to isolatedAgents. In order to use this config setting, your user code dagster version needs to be 1.4.3 or greater.[ui] When viewing a run for auto-materialized assets, show a tag with information about the assets that were materialized.
dagster-dbt project scaffold didn’t create a project directory with all the scaffolded files.SpecificPartitionsPartitionMapping with auto-materialization.max_materializations_per_minute on an AutoMaterializePolicy to a non-positive number. This will now result in an error.upath_io_manager from @harrylojames; thank you!dagster project scaffold and the new dagster-dbt APIs.The experimental agent config setting agent_replicashas been deprecated in favor of a new name isolated_agents (agentReplicas --> isolatedAgents in th…
dagster-dbt project scaffold on a dbt project directory, if a profiles.yml exists in the root of the directory, its contents are used to add dbt adapter packages to the scaffolded setup.py.max_concurrent field has been changed from 0 to None to more clearly signal its intent. A value of 0 is still interpreted as the sentinel value which dynamically allocates max_concurrent based on detected CPU count.Definitions, so that jobs are able to override the IO manager used.EnvVars in a FivetranResource would not be evaluated when loading assets from the Fivetran instance.EnvVars in an AirbyteResource would not be evaluated when loading assets from the Airbyte resource.DbtCliResource, DbtCliInvocation, @dbt_assets, DagsterDbtTranslator, dagster-dbt project scaffoldagent_replicashas been deprecated in favor of a new name isolated_agents (agentReplicas --> isolatedAgents in the helm chart). Upgrading to the new name requires all code locations to be on 1.4.3 or greater.Fixes a bug in dagster-dbt that was preventing it from correctly materializing subselections of dbt asset.
dagster-dbt that was preventing it from correctly materializing subselections of dbt asset.Fixes a bug in dagster-dbt that was preventing it efficiently loading dbt projects from a manifest.
dagster-dbt that was preventing it efficiently loading dbt projects from a manifest.Improved ergonomics for execution dependencies in assets - We introduced a set of APIs to simplify working with Dagster that don't use the I/O manager
Improved ergonomics for execution dependencies in assets - We introduced a set of APIs to simplify working with Dagster that don't use the I/O manager system for handling data between assets. I/O manager workflows will not be affected.
AssetDep type allows you to specify upstream dependencies with partition mappings when using the deps parameter of @asset and AssetSpec.MaterializeResult can be optionally returned from an asset to report metadata about the asset when the asset handles any storage requirements within the function body and does not use an I/O manager.AssetSpec has been added as a new way to declare the assets produced by @multi_asset. When using AssetSpec, the multi_asset does not need to return any values to be stored by the I/O manager. Instead, the multi_asset should handle any storage requirements in the body of the function.Asset checks (experimental) - You can now define, execute, and monitor data quality checks in Dagster [docs].
@asset_check decorator, as well as the check_specs argument to @asset and @multi_asset enable defining asset checks.Auto materialize customization (experimental) - AutoMaterializePolicies can now be customized [docs].
AutoMaterializeRules which determine if an asset should be materialized or skipped.PipesSubprocessClient, PipesDocketClient, PipesK8sClient, PipesDatabricksClientopen_pipes_session. One can augment existing invocations rather than replacing them wholesale.We’ve deprecated the non_argument_deps parameter of @asset and @multi_asset in favor of a new deps parameter. The new parameter makes it clear that th…
non_argument_deps parameter of @asset and @multi_asset in favor of a new deps parameter. The new parameter makes it clear that this is a first-class way of defining dependencies, makes code more concise, and accepts AssetsDefinition and SourceAsset objects, in addition to the strs and AssetKeys that the previous parameter accepted.UPathIOManager can now be extended to load multiple partitions asynchronously (Thanks Daniel Gafni!).dagster-user-deployments.deployments.[...].readinessProbe.enabled=false.non_argument_deps in favor of deps, build_airbyte_assets now accepts a deps parameter.non_argument_deps in favor of deps, define_dagstermill_asset now accepts a deps parameter.dagster-dbt project scaffold has been provided to scaffold a Dagster code location for an existing dbt project.StaticPartitionsDefinition will now raise an error.AutoMaterializePolicy's to not materialize missing assets.--path-prefix.build_asset_reconciliation_sensor (Experimental) has been removed. It was deprecated in 1.3 in favor of AutoMaterializePolicy.asset_key(s) properties on AssetIn and AssetDefinition have been removed in favor of key(s). These APIs were deprecated in 1.0.root_input_manager and RootInputManagerDefinition have been removed in favor of input_manager and InputManagerDefinition. These APIs were deprecated in 1.0.event_metadata_fn parameter on create_dagster_pandas_dataframe_type has been removed in favor of metadata_fn.@dbt_assets and DbtCliResource. See the migration guide for details.
dbt-rpc has been removed.DbtCloudResourceV2 has been removed.DbtCli has been renamed to DbtCliResource. Previously, DbtCliResource was a class alias for DbtCliClientResource.load_assets_from_dbt_project and load_assets_from_dbt_manifest now default to use_build=True.load_assets_from_dbt_project and load_assets_from_dbt_manifest has changed. Rather than assigning a group name using the model’s subdirectory, a group name will be assigned using the dbt model’s dbt group.node_info_to_definition_metadata_fn for load_assets_from_dbt_project and load_assets_from_dbt_manifest now overrides metadata instead of adding to it.load_assets_from_dbt_project and load_assets_from_dbt_manifest now must be specified using keyword arguments.DbtCliResource with load_assets_from_dbt_project and load_assets_from_dbt_manifest, stdout logs from the dbt process will now appear in the compute logs instead of the event logs.dagit python package is deprecated and will be removed in 2.0 in favor of dagster-webserver. See the migration guide for details.dagit → dagsterWebserveringress.dagit → ingress.dagsterWebserveringress.readOnlyDagit → ingress.readOnlyDagsterWebservertag:GetResources. Without this policy, the ECS Agent will log a deprecation warning and fall back to its old behavior (listing all ECS services in the cluster and then listing each service's tags).DbtCliClientResource, dbt_cli_resource and DbtCliOutput are now being deprecated in favor of DbtCliResource.load_assets_from_dbt_project and load_assets_from_dbt_manifest are now deprecated in favor of other options. See the migration for details.Nothing published for this version
Nothing published for this version
Fixes a bug in dagster project from-example that was preventing it from downloading examples correctly.
dagster project from-example that was preventing it from downloading examples correctly.[dagster-dbt] The use of the state_path parameter with DbtManifestAssetSelection has been deprecated, and will be removed in the next minor release.
--name argument is now optional when running dagster project from-example.@asset(key=...).AssetKey now has a with_prefix method.AutoMaterializePolicys with large numbers of partitions.dagster instance migrate now prints information about changes to the instance database schema.dagster-cloud-agent helm chart now supports setting K8s labels on the agent deployment.[ui] Fixed an issue that prevented filtering by date on the job-specific runs tab.
[ui] “F” key with modifiers (alt, ctrl, cmd, shift) no longer toggles the filter menu on pages that support filtering.
[ui] Fix empty states on Runs table view for individual jobs, to provide links to materialize an asset or launch a run for the specific job, instead of linking to global pages.
[ui] When a run is launched from the Launchpad editor while an editor hint popover is open, the popover remained on the page even after navigation. This has been fixed.
[ui] Fixed an issue where clicking on the zoom controls on a DAG view would close the right detail panel for selected nodes.
[ui] Fixed an issue shift-selecting assets with multi-component asset keys.
[ui] Fixed an issue with the truncation of the asset stale causes popover.
When using a TimeWindowPartitionMapping with a start_offset or end_offset specified, requesting the downstream partitions of a given upstream partition would yield incorrect results. This has been fixed.
When using AutoMaterializePolicys with observable source assets, in rare cases, a second run could be launched in response to the same version being observed twice. This has been fixed.
When passing in hook_defs to define_asset_job, if any of those hooks had required resource keys, a missing resource error would surface when the hook was executed. This has been fixed.
Fixed a typo in a documentation URL in dagster-duckdb-polars tests. The URL now works correctly.
DbtManifest to fetch asset keys of sources and models: DbtManifest.get_asset_key_for_model, DbtManifest.get_asset_key_for_source. These methods are utilities for defining python assets as dependencies of dbt assets via @asset(key=manifest.get_asset_key_for_model(...).state_path parameter with DbtManifestAssetSelection has been deprecated, and will be removed in the next minor release.grpcio package (for dagster) has been removed.PartitionMapping have been removed. Defining custom partition mappings has been unsupported since 1.1.7.build_airbyte_assets. Thanks @guy-rvvup!Assets with lazy auto-materialize policies are no longer auto-materialized if they are missing but don’t need to be materialized in order to help down
meta.dagster.asset_key. This field takes in a list of strings that are used as the components of the generated AssetKey.version: 2
models:
- name: users
config:
meta:
dagster:
asset_key: ["my", "custom", "asset_key"]
meta.dagster.group. This field takes in a string that is used as the Dagster group for the generated software-defined asset corresponding to the dbt model.version: 2
models:
- name: users
config:
meta:
dagster:
group: "my_group"
dagster-msteams and dagster-mlflow packages could be installed with incompatible versions of the dagster package due to a missing pin.dagster-daemon run command sometimes kept code server subprocesses open longer than it needed to, making the process use more memory.@observable_source_assets with AutoMaterializePolicies, it was possible for downstream assets to get “stuck”, not getting materialized when other upstream assets changed, or for multiple down materializations to be kicked off in response to the same version being observed multiple times. This has been fixed.@dbt_assets
project_dir and target_path in DbtCliTask are converted from type str to type pathlib.Path.stdout.Dagster is now compatible with a breaking change introduced in croniter==1.4.0. Users of earlier versions of Dagster can pin croniter<1.4.
dagster field under +meta configuration. The following are equivalent:Before:
version: 2
models:
- name: users
config:
dagster_freshness_policy:
maximum_lag_minutes: 60
cron_schedule: "0 9 * * *"
dagster_auto_materialize_policy:
type: "lazy"
After:
version: 2
models:
- name: users
config:
meta:
dagster:
freshness_policy:
maximum_lag_minutes: 60
cron_schedule: "0 9 * * *"
auto_materialize_policy:
type: "lazy"
Added support for Pythonic Config classes to the @configured API, which makes reusing op and asset definitions easier:
class GreetingConfig(Config):
message: str
@op
def greeting_op(config: GreetingConfig):
print(config.message)
class HelloConfig(Config):
name: str
@configured(greeting_op)
def hello_op(config: HelloConfig):
return GreetingConfig(message=f"Hello, {config.name}!")
Added AssetExecutionContext to replace OpExecutionContext as the context object passed in to @asset functions.
TimeWindowPartitionMapping now contains an allow_nonexistent_upstream_partitions argument that, when set to True, allows a downstream partition subset to have nonexistent upstream parents.
Unpinned the alembic dependency in the dagster package.
[ui] A new “Assets” tab is available from the Overview page.
[ui] The Backfills table now includes links to the assets that were targeted by the backfill.
croniter==1.4.0. Users of earlier versions of Dagster can pin croniter<1.4.@run_failure_sensor.py.typed was missing in the dagster-graphql package. Thanks @Tanguy-LeFloch!AutoMaterializePolicys will now be cleared after 1 week.@dbt_assets:
profile and target can now be customized on the DbtCli resource.partial_parse.msgpack is detected in the target directory of your dbt project, it is now copied into the target directories created by DbtCli to take advantage of partial parsing.@dbt_assets can now be customized by overriding DbtManifest.node_info_to_metadata.AssetMaterializations.serverK8sConfig.containerConfig.name did not actually change the container name.Fixed an issue in the 1.3.8 release where the Dagster Cloud agent would sometimes fail to start up with an import error.
1.3.8 release where the Dagster Cloud agent would sometimes fail to start up with an import error.Nothing published for this version
Multipartitioned assets with one time dimension can now depend on earlier partitions of themselves.
define_asset_job now accepts a hooks argument.sqlalchemy==2.xadditionalInstanceConfig key that allows you to supply additional configuration to the Dagster instance.EcsRunLauncher now uses a different task definition family for each job, instead of registering a new task definition revision each time a different job is launched.EcsRunLauncher now includes a run_ecs_tags config key that lets you configure tags on the launched ECS task for each run.SkipReason, the SkipReason would be ignored. This has been fixed.dagster code-server start CLI, instead of only reloading your code when you initiate a reload from the Dagster UI.metadata parameter to define_asset_job (Thanks Elliot2718!)poll_sync_run to handle the “preparing” status from the Census API (Thanks ldnicolasmay!)@observable_source_asset-decorated functions can now return a DataVersionsByPartition to record versions for partitions.@dbt_assets
DbtCliTask's created by invoking DbtCli.cli(...) now have a method .is_successful(), which returns a boolean representing whether the underlying CLI process executed the dbt command successfully.@dbt_assets can now be customized by overriding DbtManifest.node_info_to_description.@dbt_assets.server_ecs_tags and run_ecs_tags that apply to each service or task created by the agent. See the docs for more information.instance.get_run_partition_data in Dagster Cloud.Adding a .env file in the working directory when running dagster dev can now be used for Dagster system variables like DAGSTER_HOME or environment var
.env file in the working directory when running dagster dev can now be used for Dagster system variables like DAGSTER_HOME or environment variables referenced in your dagster.yaml file using an env: key. Previously, setting a .env file only worked for environment variables referenced in your Dagster code.submit_job_execution can now take in a RunConfig object. Previously, it could only take a Python dictionary with the run configuration.jobs argument to Definitions./instance/runs path instead of the new /runs.create_databricks_run_now_op, thanks @srggrs!Added an experimental dagster code-server start command that can be used to launch a code server, much like dagster api grpc. Unlike dagster api grpc,
dagster code-server start command that can be used to launch a code server, much like dagster api grpc. Unlike dagster api grpc, however, dagster code-server start runs the code in a subprocess, so it can reload code from the Dagster UI without needing to restart the command. This can be useful for jobs that load code from some external source and may want to reload job definitions without restarting the process.sensors.num_submit_workers key to dagster.yaml that can be used to decrease latency when a sensor emits multiple run requests within a single tick. See the docs for more information.k8s_job_executor can now be used to launch each step of a job in its own Kubernetes, pod, even if the Dagster deployment is not using the K8sRunLauncher to launch each run in its own Kubernetes pod.io_manager_def param on an asset.maxResumeRunAttempts to null in the helm chart would cause it to be set to a default value of 3 instead of disabling run retries.k8s_job_executor would sometimes fail with a 409 Conflict error after retrying the creation of a Kubernetes pod for a step, due to the job having already been created during a previous attempt despite raising an error.op_name was passed to load_assets_from_dbt_manifest, and a select parameter was specified, a suffix would be appended to the desired op name.dagit would lead to JavaScript bundle loading errors.DbtCli, in dagster_dbt.cli. This new resource only support dbt-core>=1.4.0.@dbt_assets in dagster_dbt.asset_decorator that allows you to specify a compute function for a selected set of dbt assets that loaded as an AssetsDefinition.A new max_materializations_per_minute parameter (with a default of 1) to AutoMaterializationPolicy.eager() and AutoMaterializationPolicy.lazy() allows
max_materializations_per_minute parameter (with a default of 1) to AutoMaterializationPolicy.eager() and AutoMaterializationPolicy.lazy() allows you to set bounds on the volume of work that may be automatically kicked off for each asset. To restore the previous behavior, you can explicitly set this limit to None.DailyPartitionsDefinition, HourlyPartitionsDefinition, WeeklyPartitionsDefinition, and MonthlyPartitionsDefinition now support and end_date attribute.AutoMaterializePolicys or build_asset_reconciliation_sensor, a single new data version from an observable source asset could trigger multiple runs of the downstream assets. This has been fixed.EnvVar and IntEnvVar within raw run config - although they just returned the name of the env var rather than retrieve its value. This has been fixed to error directly.* was being interpreted as glob pattern, rather than as a dbt selector argument. We now explicitly set the default selection pattern as fqn:*.dagster-cloud serverless deploy did not create a unique image tag if the --image tag was not specified.op_name on load_assets_from_dbt_project and load_assets_from_dbt_manifest (thanks @wkeifenheim!)ExecuteJobResult was renamed to JobExecutionResult (ExecuteJobResult remains a deprecated alias)
TimeWindowPartitionMapping objects are now current-time aware. Subsequently, only upstream/downstream partitions existent at the current time are returned.ExecuteJobResult was renamed to JobExecutionResult (ExecuteJobResult remains a deprecated alias)AssetSelection.key_prefixes method allows matching asset keys starting with a provided prefix.ConfigurablePickledObjectADLS2IOManager that uses pythonic configDataProcResource follows the Pythonic resource system. The existing dataproc_resource remains supported.SnowflakeIOManager now supports private_keys that have been base64 encoded to avoid issues with newlines in the private key. Non-base64 encoded keys are still supported. See the SnowflakeIOManager documentation for more information on base64 encoded private keys.kvs, instance_info, and daemon_hearbeats table for existing Postgres storage instances that were migrating from before 1.2.2. This should unblock users relying on the existence of a primary key constraint for replication.SensorResult evaluation where multipartitioned run requests containing a dynamic partition added in a dynamic partitions request object would raise an invalid partition key error.TimeWindowPartitionMapping.ExecuteJobResult is now a deprecated alias for the new name, JobExecutionResult.airbyte_resource to load_assets_from_connections , you may now provide an instance of the AirbyteResource class, rather than just airbyte_resource.configured(...) (thanks @joel-olazagasti!)dagster-celery CLI accepted an inconsistent configuration format - it now matches the same format as the celery_executor. Thanks @boenshao!ecs_timeout in your ECS user code launcher config to extend how long the ECS agent polls for new code servers to start. Extending this timeout is useful if your code server takes an unusually long time to start up - for example, because it uses a very large image.load_assets_from_package_module and the other core load_assets_from_ methods now accept a source_key_prefix argument, which allows applying a key pref
load_assets_from_package_module and the other core load_assets_from_ methods now accept a source_key_prefix argument, which allows applying a key prefix to all the source assets that are loaded.
OpExecutionContext now has an asset_partitions_time_window_for_input method.
RunFailureSensorContext now has a get_step_failure_events method.
The Pythonic resource system now supports a set of lifecycle hooks which can be used to manage setup and teardown:
class MyAPIClientResource(ConfigurableResource):
api_key: str
_internal_client: MyAPIClient = PrivateAttr()
def setup_for_execution(self, context):
self._internal_client = MyAPIClient(self.api_key)
def get_all_items(self):
return self._internal_client.items.get()
Added support for specifying input and output config on ConfigurableIOManager.
QueuedRunCoordinator and SubmitRunContext are now exposed as public dagster exports.
[ui] Downstream cross-location dependencies of all source assets are now visible on the asset graph. Previously these dependencies were only displayed if the source asset was defined as a regular asset.
[ui] A new filtering experience is available on the Runs page after enabling feature flag “Experimental Runs table view with filtering”.
[dagster-aws] Allow the S3 compute log manager to specify a show_url_only: true config option, which will display a URL to the S3 file in dagit, instead of the contents of the log file.
[dagster-aws] PickledObjectS3IOManager now fully supports loading partitioned inputs.
[dagster-azure] PickedObjectADLS2IOManager now fully supports loading partitioned inputs.
[dagster-gcp] New GCSResource and ConfigurablePickledObjectGCSIOManager follow the Pythonic resource system. The existing gcs_resource and gcs_pickle_io_manager remain supported.
[dagster-gcp] New BigQueryResource follows the Pythonic resource system. The existing bigquery_resource remains supported.
[dagster-gcp] PickledObjectGCSIOManager now fully supports loading partitioned inputs.
[dagster-postgres] The event watching implementation has been moved from listen/notify based to the polling watcher used by MySQL and SQLite.
[dagster-slack] Add monitor_all_repositories to make_slack_on_run_failure_sensor, thanks @danielgafni!
[dagster-snowflake] New SnowflakeResource follows the Pythonic resource system. The existing snowflake_resource remains supported.
AssetMetadataValue.value that would cause an infinite recursion error.volumes and volumeMounts values have been added to the agent helm chart.load_assets_from_airbyte_instance and load_assets_from_airbyte_project now take a connection_to_auto_materialize_policy_fn for setting AutoMaterializePolicys on Airbyte assetsAdded performance improvements for yielding time-partitioned run requests.
NumReplicas parameter on the agent template in CloudFormation, or the dagsterCloudAgent.replicas field in Helm.enableZeroDowntimeDeploys parameter to true in the CloudFormation stack for your agent.AssetsDefinition.from_graph, as well as the@graph_asset and @graph_multi_asset decorators now support specifying AutoMaterializePolicys.async def ops/assets no longer prematurely finalize async generators during execution.build_asset_reconciliation_sensor) could incorrectly launch new runs for partitions that already had an in-progress run. This has been fixed.run_request_for_partition now throws an error. Instead, users should yield directly instantiated run requests via RunRequest(partition_key=...).graph_asset and graph_multi_asset now support specifying resource_defs directly (thanks @kmontag42)!node_info_to_auto_materialize_policy_fn param added to load_assets_from_dbt_* functions. (thanks @askvinni)!partition_key field to RunStatusSensorContext (thanks @pdstrnadJC)!latest_materialization_records_by_key method.allPartitions argument is provided.Performance improvements when evaluating time-partitioned run requests within sensors and schedules.
non_argument_deps → deps – We’ve deprecated the non_argument_deps parameter of @asset and @multi_asset in favor of a new deps parameter. The new param…
AutoMaterializePolicy. It’s located under Assets → Select an asset with an AutoMaterializePolicy → Auto-materialize history tab.non_argument_deps parameter of @asset and @multi_asset in favor of a new deps parameter. The new parameter makes it clear that this is a first-class way of defining dependencies, makes code more concise, and accepts AssetsDefinition and SourceAsset objects, in addition to the strs and AssetKeys that the previous parameter accepted.DynamicPartitionsDefinition and SensorResult are no longer marked experimental.@observable_source_asset decorator now accepts an auto_observe_interval_minutes parameter. If the asset daemon is turned on, then the observation function will automatically be run at this interval. Downstream assets with eager auto-materialize policies will automatically run if the observation function indicates that the source asset has changed. [docs]dagit package is deprecated in favor of the dagster-webserver package.@dbt_assets decorator allows much more control over how Dagster runs your dbt project. [docs]dagster-dbt project scaffold command line interface makes it easy to create files and directories for a Dagster project that wraps an existing dbt project.get_asset_key_for_model and get_asset_key_for_source utilities make it easy to specify dependencies between upstream dbt assets and downstream non-dbt assets. And you can now more easily specify dependencies between dbt models and upstream non-dbt assets by specifying Dagster asset keys in the dbt metadata for dbt sources.[previously deprecated, 0.15.0] metadata_entries arguments to event constructors have been removed. While MetadataEntry still exists and will only be…
AthenaClientResource, ECRPublicResource, RedshiftClientResource, S3Resource, S3FileManagerResource, ConfigurablePickledObjectS3IOManager, SecretsManagerResource follow Pythonic resource system. The existing APIs remain supported.DatadogResource follows Pythonic resource system. The existing datadog_resource remains supported.GEContextResource follows Pythonic resource system. The existing ge_context_resource remains supported.GithubResource follows Pythonic resource system. The existing github_resource remains supported.MSTeamsResource follows Pythonic resource system. The existing msteams_resource remains supported.SlackResource follows Pythonic resource system. The existing slack_resource remains supported.pdb.set_trace no longer worked when running Dagster locally using dagster dev or dagit.@asset or Out caused an error to be thrown.pipelines or jobs could lead to blank pages.Definitions are now automatically bound to jobs. This will only result in a change in behavior if you a) have a job with no "io_manager" defined in its resource_defs and b) have supplied an IOManager with key "io_manager" to the resource_defs argument of your Definitions. Prior to 1.3.0, this would result in the job using the default filesystem-based IOManager for the key "io_manager". In 1.3.0, this will result in the "io_manager" supplied to your Definitions being used instead. The BindResourcesToJobs wrapper, introduced in 1.2 to simulate this behavior, no longer has any effect.CeleryK8sRunLauncher is now the same namespace as the Helm chart, instead of the default namespace. To restore the previous behavior, you can set the celeryK8sRunLauncher.jobNamespace field to the string default.SnowflakePandasIOManager has historically converted all timestamp data to strings before storing it in Snowflake. Now, it will instead ensure that timestamp data has a timezone, and if not, attach the UTC timezone. This allows the timestamp data to be stored as timestamps in Snowflake. If you have been storing timestamp data using the SnowflakePandasIOManager you can set the store_timestamps_as_strings=True configuration to continue storing timestamps as strings. For more information, and instructions for migrating Snowflake tables to use timestamp types, see the Migration Guide.Changes to experimental APIs
RunConfig to many APIs which previously would only accept a config dictionary.ResourceDefinition.execution config when constructing a RunConfig object.minutes_late and previous_minutes_late properties on the experimental FreshnesPolicySensorContext have been renamed to minutes_overdue and previous_minutes_overdue, respectively.Removal of deprecated APIs
metadata_entries arguments to event constructors have been removed. While MetadataEntry still exists and will only be removed in 2.0, it is no longer passable to any Dagster public API — users should always pass a dictionary of metadata values instead.latest_materialization_records_by_key function.BigQueryPandasIOManager and BigQueryPySparkIOManager.SnowflakePandasIOManager and SnowflakePySparkIOManager.Fixed a GraphQL resolution error which occurred when retrieving metadata for step failures in the event log.
The run_request_for_partition method on JobDefinition and UnresolvedAssetJobDefinition is now deprecated and will be removed in 2.0.0. Instead, direct…
materialize and materialize_to_memory now both accept a selection argument that allows specifying a subset of assets to materialize.MultiPartitionsDefinition is no longer marked experimental.MultiPartitionsDefinitions with a time dimension.load_assets_from_package_module and similar methods now accept a freshness_policy, which will be applied to all loaded assets.build_sensor_context and build_multi_asset_sensor_context can now take a Definitions object in place of a RepositoryDefinitions3_resource now accepts use_ssl and verify configurations.selection argument on define_asset_job.MultiPartitionsDefinition.MultiPartitionsMapping (experimental) is loaded.TimeWindowPartitionMappings to not be bubbled up to the UI.pdb stopped working when using dagster dev.AssetMaterialization objects with a null AssetKey.TimeWindowPartitionsDefinition with a non-standard cron schedule, and also provided a minute_of_hour or similar argument in build_schedule_from_partitioned_job. Dagster would silently create the wrong cron expression. It now raises an error.PartitionsDefinition. These partitions are now ignored.PersistentAirflowDatabase where versions of airflow from 2.0.0 till 2.3.0 would not use the correct connection environment variable name.poll_sync_run function ofdagster-census that prevented polling from working correctly (thanks @ldincolasmay!)run_request_for_partition method on JobDefinition and UnresolvedAssetJobDefinition is now deprecated and will be removed in 2.0.0. Instead, directly instantiate a run request with a partition key via RunRequest(partition_key=...).The environment_vars argument to ScheduleDefinition is deprecated (the argument is currently non-functional; environment variables no longer need to b…
AssetsDefinition.to_source_assets to method convert a set of assets to SourceAsset objects.MultiPartitionsDefinitions.mlflow pin from the dagster-mlflow package.When using build_asset_reconciliation_sensor, in some cases duplicate runs could be produced for the same partition of an asset. This has been fixed.
When using Pythonic configuration for resources, aliased field names would cause an error. This has been fixed.
Fixed an issue where context.asset_partitions_time_window_for_output threw an error when an asset was directly invoked with build_op_context.
[dagster-dbt] In some cases, use of ephemeral dbt models could cause the dagster representation of the dbt dependency graph to become incorrect. This has been fixed.
[celery-k8s] Fixed a bug that caused JSON deserialization errors when an Op or Asset emitted JSON that doesn't represent a DagsterEvent.
Fixed an issue where launching a large backfill while running dagster dev would sometimes fail with a connection error after running for a few minutes.
Fixed an issue where dagster dev would sometimes hang when running Dagster code that attempted to read in input via stdin.
Fixed an issue where runs that take a long time to import code would sometimes continue running even after they were stopped by run monitoring for taking too long to start.
Fixed an issue where AssetSelection.groups() would simultaneously select both source and regular assets and consequently raise an error.
Fixed an issue where BindResourcesToJobs would raise errors encapsulating jobs which had config specified at definition-time.
Fixed Pythonic config objects erroring when omitting optional values rather than specifying None.
Fixed Pythonic config and resources not supporting Enum values.
DagsterInstance.local_temp and DagsterInstance.ephemeral now use object instance scoped local artifact storage temporary directories instead of a shared process scoped one, removing a class of thread safety errors that could manifest on initialization.
Improved direct invocation behavior for ops and assets which specify resource dependencies as parameters, for instance:
class MyResource(ConfigurableResource):
pass
@op
def my_op(x: int, y: int, my_resource: MyResource) -> int:
return x + y
my_op(4, 5, my_resource=MyResource())
[dagster-azure] Fixed an issue with an AttributeError being thrown when using the async DefaultAzureCredential (thanks @mpicard)
[ui] Fixed an issue introduced in 1.2.3 in which no log levels were selected by default when viewing Run logs, which made it appear as if there were no logs at all.
environment_vars argument to ScheduleDefinition is deprecated (the argument is currently non-functional; environment variables no longer need to be whitelisted for schedules)--path-prefix can now be supplied via Helm chart (thanks @mpicard)Jobs defined via define_asset_job now auto-infer their partitions definitions if not explicitly defined.
define_asset_job now auto-infer their partitions definitions if not explicitly defined.define_asset_job. This allows putting them on a schedule/sensor.instance property to the HookContext object that is passed into Op Hook functions, which can be used to access the current DagsterInstance object for the hook.create_table_schema_metadata_from_dataframe function to generate a TableSchemaMetadataValue from a Pandas DataFrame. Thanks @AndyBys!dag_run configuration on the integration’s database resources.dagster asset wipe-partitions-status-cache to see the partitions displayed.1.1.18, users with a gRPC server that could not access the Dagster instance on user code deployments would see an error when launching backfills as the instance could not instantiate. This has been fixed.build_asset_reconciliation_sensor from kicking off runs of downstream assets. This has been fixed.build_asset_reconciliation_sensor is now much more performant in cases where unpartitioned assets are upstream or downstream of static-partitioned assets with a large number of partitions.jobNamespace field in the Dagster Helm chart when using the CeleryK8sRunLauncher.en-US browser locale. This has been fixed.Dagster is now tested on Python 3.11.
Dagster is now tested on Python 3.11.
Users can now opt in to have resources provided to Definitions bind to their jobs. Opt in by wrapping your job definitions in BindResourcesToJobs. This will become the default behavior in the future.
@op(required_resource_keys={"foo"})
def my_op(context)
print(context.foo)
@job
def my_job():
my_op()
defs = Definitions(
jobs=BindResourcesToJobs([my_job])
resources={"foo": foo_resource}
Added dagster asset list and dagster asset materialize commands to Dagster’s command line interface, for listing and materializing software-defined assets.
build_schedule_from_partitioned_job now accepts jobs partitioned with a MultiPartitionsDefinition that have a time-partitioned dimension.
Added SpecificPartitionsPartitionMapping, which allows an asset, or all partitions of an asset, to depend on a specific subset of the partitions in an upstream asset.
load_asset_value now supports SourceAssets.
[ui] Ctrl+K has been added as a keyboard shortcut to open global search.
[ui] Most pages with search bars now sync the search filter to the URL, so it’s easier to bookmark views of interest.
[ui] In the run logs table, the timestamp column has been moved to the far left, which will hopefully allow for better visual alignment with op names and tags.
[dagster-dbt] A new node_info_to_definition_metadata_fn to load_assets_from_dbt_project and load_assets_from_dbt_manifest allows custom metadata to be attached to the asset definitions generated from these methods.
[dagster-celery-k8s] The Kubernetes namespace that runs using the CeleryK8sRunLauncher are launched in can now be configured by setting the jobNamespace field in the Dagster Helm chart under celeryK8sRunLauncherConfig.
[dagster-gcp] The BigQuery I/O manager now accepts timeout configuration. Currently, this configuration will only be applied when working with Pandas DataFrames, and will set the number of seconds to wait for a request before using a retry.
[dagster-gcp] [dagster-snowflake] [dagster-duckdb] The BigQuery, Snowflake, and DuckDB I/O managers now support self-dependent assets. When a partitioned asset depends on a prior partition of itself, the I/O managers will now load that partition as a DataFrame. For the first partition in the dependency sequence, an empty DataFrame will be returned.
[dagster-k8s] k8s_job_op now supports running Kubernetes jobs with more than one pod (Thanks @Taadas).
AssetValueLoader.load_asset_value() didn’t load transitive resource dependencies correctly.RunConfig object with optional config arguments would lead to an error.ScheduleEvaluationContext.scheduled_execution_time to not be Optional.OpExecutionContext.partition_time_window ****(thanks @elben10).InputContext.upstream_output.log is no longer None when loading a source asset.async def ops are now invoked with asyncio.run.TimeWindowPartitionDefinition now throws an error at definition time when passed an invalid cron schedule instead of at runtime.cron_schedule_timezone inside of the config for a dbt model would not result in that property being set on the generated FreshnessPolicy. This has been fixed.GCSComputeLogManager when the session does not have permissions to generate signed urls.validate_run_config no longer accepts pipeline_def or mode arguments. These arguments refer to legacy concepts that were removed in Dagster 1.0, and since then there have been no valid values for them.Added experimental support for resource requirements in sensors and schedules. Resources can be specified using required_resource_keys and accessed through the context or specified as parameters:
@sensor(job=my_job, required_resource_keys={"my_resource"})
def my_sensor(context):
files_to_process = context.my_resource.get_files()
...
@sensor(job=my_job)
def my_sensor(context, my_resource: MyResource):
files_to_process = my_resource.get_files()
...
Fixed a bug with postgres storage where daemon heartbeats were failing on instances that had not been migrated with dagster instance migrate after upg
dagster instance migrate after upgrading to 1.2.0.Auto-materialize policies replace the asset reconciliation sensor - We significantly renovated the APIs used for specifying which assets are scheduled
build_asset_reconciliation_sensors , AutoMaterializePolicy works across code locations, as well as allow you to customize the conditions under which each asset is auto-materialized. [docs]FreshnessPolicys, Dagster’s UI now marks them as “overdue” instead of “late”.create_databricks_job_op is now deprecated. To submit one-off runs of Databricks tasks, you must now use the create_databricks_submit_run_op.
@graph_asset and @graph_multi_asset.project_fully_featured example now uses the built in DuckDB and Snowflake I/O managers.make_persistent_airflow_db_resource this enables support for Airflow features like pools and cross-dagrun state sharing. In particular retry-from-failure now works for jobs generated from Airflow DAGs.BigQueryPandasTypeHandler now uses google.bigquery.Client methods load_table_from_dataframe and query rather than the pandas_gbq library to store and fetch DataFrames.args instead of both command and args for user code deployments, allowing to include a custom ENTRYPOINT in your the Dockerfile that loads your code.protobuf<4 pin in Dagster has been removed. Installing either protobuf 3 or protobuf 4 will both work with Dagster.@graph_asset and @graph_multi_asset now support passing metadata (thanks @askvinni)!@graph_asset and @graph_multi_asset to be ignored.TableRecord.@multi_asset and other functions would register as type errors for mypy and other static analyzers.start_date would not be converted correctly causing runs to fail.dagster instance migrate:
id column to the kvs, daemon_heartbeats and instance_info tables, enforcing that all tables have a primary key.The minimum grpcio version supported by Dagster has been increased to 1.44.0 so that Dagster can support both protobuf 3 and protobuf 4. Similarly, the minimum protobuf version supported by Dagster has been increased to 3.20.0. We are working closely with the gRPC team on resolving the upstream issues keeping the upper-bound grpcio pin in place in Dagster, and hope to be able to remove it very soon.
Prior to 0.9.19, asset keys were serialized in a legacy format. This release removes support for querying asset events serialized with this legacy format. Contact #dagster-support for tooling to migrate legacy events to the supported version. Users who began using assets after 0.9.19 will not be affected by this change.
[dagster-snowflake] The execute_queryand execute_queries methods of the SnowflakeResource now have consistent behavior based on the values of the fetch_results and use_pandas_result parameters. If fetch_results is True, the standard Snowflake result will be returned. If fetch_results and use_pandas_result are True, a pandas DataFrame will be returned. If fetch_results is False and use_pandas_result is True, an error will be raised. If both are False, no result will be returned.
[dagster-snowflake] The execute_queries command now returns a list of DataFrames when use_pandas_result is True, rather than appending the results of each query to a single DataFrame.
[dagster-shell] The default behavior of the execute and execute_shell_command functions is now to include any environment variables in the calling op. To restore the previous behavior, you can pass in env={} to these functions.
[dagster-k8s] Several Dagster features that were previously disabled by default in the Dagster Helm chart are now enabled by default. These features are:
dagster/max_retries tag set. You can configure a global number of retries in the Helm chart by setting run_retries.max_retries to a value greater than the default of 0.Each of these features can be disabled in the Helm chart to restore the previous behavior.
[dagster-k8s] The experimental k8s_job_op op and execute_k8s_job functions no longer automatically include configuration from a dagster-k8s/config tag on the Dagster job in the launched Kubernetes job. To include raw Kubernetes configuration in a k8s_job_op, you can set the container_config, pod_template_spec_metadata, pod_spec_config, or job_metadata config fields on the k8s_job_op (or arguments to the execute_k8s_job function).
[dagster-databricks] The integration has now been refactored to support the official Databricks API.
create_databricks_job_op is now deprecated. To submit one-off runs of Databricks tasks, you must now use the create_databricks_submit_run_op.databricks_client resource must now begin with https://.LogicalVersion has been renamed to DataVersion and LogicalVersionProvenance has been renamed to DataProvenance.DynamicPartitionsDefinition to add, remove, and check for existence of partitions have been removed. Refer to documentation for updated API methods.MetadataEntry have been removed.DagsterTypeMaterializer, DagsterTypeMaterializerContext, and @dagster_type_materializer have been removed.PartitionScheduleDefinition has been removed.RunRecord.pipeline_run has been removed (use RunRecord.dagster_run).DependencyDefinition.solid has been removed (use DependencyDefinition.node).pipeline_run argument to build_resources has been removed (use dagster_run)iteritems usage was removed and changed to the recommended items within dagster-snowflake-pandas (thanks @sethkimmel3)!@asset_graph decorator (thanks @simonvanderveldt)!DataVersions can now be returned on OutputOpExecutionContext.get_asset_provenancePicking a github issue was not correctly rendering, this has been fixed (thanks @Sedosa)!The new @graph_asset and @graph_multi_asset decorators make it more ergonomic to define graph-backed assets.
The new @graph_asset and @graph_multi_asset decorators make it more ergonomic to define graph-backed assets.
Dagster will auto-infer dependency relationships between single-dimensionally partitioned assets and multipartitioned assets, when the single-dimensional partitions definition is a dimension of the MultiPartitionsDefinition.
A new Test sensor / Test schedule button that allows you to perform a dry-run of your sensor / schedule. Check out the docs on this functionality here for sensors and here for schedules.
[dagit] Added (back) tag autocompletion in the runs filter, now with improved query performance.
[dagit] The Dagster libraries and their versions that were used when loading definitions can now be viewed in the actions menu for each code location.
New bigquery_pandas_io_manager can store and load Pandas dataframes in BigQuery.
[dagster-snowflake, dagster-duckdb] SnowflakeIOManagers and DuckDBIOManagers can now default to loading inputs as a specified type if a type annotation does not exist for the input.
[dagster-dbt] Added the ability to use the “state:” selector
[dagster-k8s] The Helm chart now supports the full kubernetes env var spec for Dagit and the Daemon. E.g.
dagit:
env:
- name: “FOO”
valueFrom:
fieldRef:
fieldPath: metadata.uid
ruff, pyright).@multi_asset no longer accepts Out values in the dictionary passed to its outs argument. This was experimental and deprecated. Instead, use AssetOut.
FreshnessPolicy object now supports a cron_schedule_timezone argument.AssetsDefinition.from_graph now supports a freshness_policies_by_output_name parameter.@asset_sensor will now display an informative SkipReason when no new materializations have been created since the last sensor tick.AssetsDefinition now has a to_source_asset method, which returns a representation of this asset as a SourceAsset.from dagster import asset, job, op
@asset
def emails_to_send(): ...
@op
def send_emails(emails) -> None: ...
@job
def send_emails_job():
send_emails(emails_to_send.to_source_asset())
--dagit-host/-h argument to the dagster dev command to allow customization of the host where Dagit runs.input_manager_key to an asset’s input, incorrect resource config could be used when loading that input. This has been fixed.runK8sConfig in the Dagster Helm chart would not pass configuration through to pods launched using the k8s_job_executor.execute_k8s_job op downstream of a dynamic output would result in k8s jobs with duplicate names being created. This has been fixed.asset_key, asset_partitions, and asset_partitions_defs arguments on Out.@multi_asset no longer accepts Out values in the dictionary passed to its outs argument. This was experimental and deprecated. Instead, use AssetOut.top_level_resources argument to the repository decorator has been renamed to _top_level_resources to emphasize that it should not be set manually.load_asset_values now accepts resource configuration (thanks @Nintorac!)UPathIOManager, paths with the "." character in them would be incorrectly truncated, which could result in multiple distinct objects being written to the same path. This has been fixed. (Thanks @spenczar!)The partitions_fn argument of the DynamicPartitionsDefinition class is now deprecated and will be removed in 2.0.0.
PartitionsDefinitions (e.g. HourlyPartitionsDefinition, DailyPartitionsDefinition) may now have a FreshnessPolicy.load_assets_from_dbt_project or load_assets_from_dbt_manifest with dbt-core>=1.4, AssetMaterialization events will be emitted as the dbt command executes, rather than waiting for dbt to complete before emitting events.make_ephemeral_airflow_db_resource which returns a ResourceDefinition for a local only airflow database for use in migrated airflow DAGsdagster instance migrate.observable_source_asset incorrectly required its function to have a context argument.dagster-airbytedagster-dbt integration was incompatible with dbt-core>=1.4. This has been fixed.load_assets_from_dbt_cloud_job will now avoid unnecessarily generating docs when compiling a manifest for the job. Compile runs will no longer be kicked off for jobs not managed by this integration.context.asset_partition_key returned a string instead of a MultiPartitionKey. This has been fixed.k8s_job_executor would sometimes unexpectedly fail due to transient 401 errors in certain kubernetes clusters.load_assets_from_airflow_dag no longer creates airflow db resource definitions, as a user you will need to provide them on Definitions directlypartitions_fn argument of the DynamicPartitionsDefinition class is now deprecated and will be removed in 2.0.0.asset (rather than using @asset in decorator form) to create an asset. Thanks @ns-finkelstein!The dagster-airflow library as been moved to 1.x.x to denote the stability of its api's going forward.
dagster-airflow library as been moved to 1.x.x to denote the stability of its api's going forward.make_schedules_and_jobs_from_airflow_dag_bag has been added to allow for more fine grained composition of your transformed airflow DAGs into Dagster.retries and retry_delay configuration are now converted to op RetryPolicies with all make_dagster_* apis.0 5 * * mon#1 to execute on a certain day of the week each month executed every week instead.use_airflow_template_context, mock_xcom and use_ephemeral_airflow_db params have been dropped, by default all make_dagster_* apis now use a run-scoped airflow db, similiar to how use_ephemeral_airflow_db worked.make_airflow_dag has been removed.make_airflow_dag_for_operator has been removed.make_airflow_dag_containerized has been removed.airflow_operator_to_op has been removed.make_dagster_repo_from_airflow_dags_path has been removed.make_dagster_repo_from_airflow_dag_bag has been removed.make_dagster_repo_from_airflow_example_dags has been removed.${dag_id}__${task_id} from airflow_${task_id}_${unique_int}.${dag_id} from airflow_${dag_id}.Nothing published for this version
Nothing published for this version
Definitions now accepts Executor instances in its executor argument, not just ExecutorDefinitions.
@multi_asset_sensor now accepts a request_assets parameter, which allows it to directly request that assets be materialized, instead of requesting a run of a job.Definitions when using large numbers of assets or many asset jobs.build_schedule_from_partitioned_job no longer needs to have a partitions_def directly assigned to it. Instead, Dagster will infer from the partitions from the assets it targets.OpExecutionContext.asset_partition_keys_for_output no longer requires an argument to specify the default output.pyproject.toml file that were made while Dagit was running. Previously, Dagit needed to be restarted in order for such changes to be shown.get_run_record_by_id has been added to DagsterInstance to provide easier access to RunRecord objects which expose the start_time and end_time of the run.docker_container_op op and execute_docker_container_op helper function for running ops that launch arbitrary Docker containers. See the docs for more information.python:3.x-slim base image.build_asset_reconciliation_sensor could time out when evaluating ticks over large selections of assets, or assets with many partitions. A series of performance improvements should make this much less likely.run_request_for_partition in a sensor that targeted multiple jobs created via define_asset_job.dagster has been reduced.PartitionsDefinition.k8s_job_op raised an Exception when running pods with multiple containers.asset_keys and asset_selection parameters of the experimental @multi_asset_sensor decorator have been replaced with a monitored_assets parameter. This helps disambiguate them from the new request_assets parameter.StaticPartitionMapping is now serializable, thanks @AlexanderVR!build_fivetran_assets now supports group_name , thanks @toddy86!AzureBlobComputeManager now supports authentication via DefaultAzureCredential, thanks @mpicard!load_assets_from_airflow_dag that creates graph-backed, partitioned, assets based on the provided Airflow DAG.Added a pin of the sqlalchemy package to <2.0.0 due to a breaking change in that version.
* as the asset selection before materializing the assets.sqlalchemy package to <2.0.0 due to a breaking change in that version.dbt-core package to <1.4.0 due to breaking changes in that release that affected the Dagster dbt integration. We plan to remove this pin in the next release.jupyter-client package to <8.0 due to an issue with the most recent release causing hangs while running dagstermill ops.EcsRunLauncher sometimes waited much longer than intended before retrying after a failure launching a run.The nux section of dagster.yaml config has been fixed.
nux section of dagster.yaml config has been fixed.dagster dev command raised an errorYour coding agent can read these notes before it upgrades. Set up the MCP server →