NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
PyPI · #1999 most downloaded on PyPI
Package for AWS-specific Dagster framework solid and resource components.
Last release 2 days ago
16 Sep 2026
Ships on a steady schedule
a new release about every 2 weeks
Nearly every release is documented
notes for 59 of the last 60 stable releases
4 versions withdrawn
withdrawn after publishing
7 years old
682 releases · first in 2019
[ui] Allow searching across code locations with * wildcard in selection inputs for jobs and automations.
* wildcard in selection inputs for jobs and automations.AutomationCondition.all_new_executed_with_tags, which allows automation conditions to be filtered to partitions that have been materialized since the last tick from runs with certain tags. This condition can be used to require or prevent certain run tags from triggering downstream declarative automation conditions.dagster==1.11.1, partitioned_config was unintentionally removed from the public exports of the top-level dagster package. This has been fixed.trigger_job_run with no steps_override parameter.One column per quarter.
anthropic, mcp, and claude-code-sdk dependencies of dagster-dg-cli are now under a separate ai extra, allowing dagster-dg-cli to be installed without
anthropic, mcp, and claude-code-sdk dependencies of dagster-dg-cli are now under a separate ai extra, allowing dagster-dg-cli to be installed without these dependencies.AutomationCondition.all_new_updates_have_run_tags and AutomationCondition.any_new_update_has_run_tags, which allows automation conditions to be filtered to partitions that have been materialized since the last tick from runs with certain tags. This condition can be used to require or prevent certain run tags from triggering downstream declarative automation conditions. These conditions are similar to AutomationCondition.executed_with_tags, but look at all new runs since the most recent tick instead of just looking at the latest run.skippable=True and a blocking asset check to execute as long as the asset check output was produced, even if the asset output was skipped.{{ env.FOO }}) when listing the env vars used by a component. (Thanks, @edgarrmondragon!)DbIOManagers.DbtCloudWorkspaceClient that would cause errors when calling trigger_job_run with no steps_override parameter.Added inline-component command to the publicly available scaffold commands in the Dagster CLI.
inline-component command to the publicly available scaffold commands in the Dagster CLI.require_upstream_step_success config param to all executors. If {"step_dependency_config": {"require_upstream_step_success": False}} is set, this will allow downstream steps to execute immediately after all required upstream outputs have finished, even if the upstream step has not completed in its entirety yet. This can be useful particularly in cases where there are large multi-assets with downstream assets that depend on only a subset of the assets in the upstream step.logsForRun resolvers and eventConnection resolvers in the Dagster GraphQL API will now apply a default limit of 1000 to the number of logs returned from a single graphql query. The cursor field in the response can be used to continue iterating through the logs for a given run.@airbyte_assets and AirbyteWorkspaceComponent (previously AirbyteCloudWorkspaceComponent) now support Airbyte OSS and Enterprise.dagster_dg_cli package failed to import when using Python 3.9.AutomationCondition.eager() that could cause runs for materializable assets to be launched at the same time as an upstream observable source asset that had an automation condition, even if the upstream observation would not result in a new data version.attributes configuration to Component subclasses that did not inherit from Resolvable.run_keys that had already been submitted to show up in the preview result.dbt-core dependency back to dagster-dbt as it is still required for the dbt Cloud integration. If both dbt-core and dbt Fusion are installed, dagster-dbt will still prefer using dbt Fusion by default.Subclasses of Resolved now support fields of type dict[str, T].
Resolved now support fields of type dict[str, T].assetKeys argument of the assetsOrError field in the GraphQL API would return every asset instead of an empty list of assets.A param exclusions was added to time window partition definitions to support custom calendars.
exclusions was added to time window partition definitions to support custom calendars.dagster library now supports protobuf==6.xdg scaffold defs --help now shows descriptions for subcommands.dg check toml command has been added to validate your TOML configuration files.DatabricksAssetBundleComponent has been added in preview. Databricks tasks can now be represented as assets and submitted via Dagster.cli_args configuration to allow customizing the command that is run when your assets are executed.DbtCloudWorkspace resource can now be customized with the DAGSTER_DBT_CLOUD_POLL_INTERVAL and DAGSTER_DBT_CLOUD_POLL_TIMEOUT environment variables.[components] Removed deprecated and non-functional asset_post_processors fields from SlingReplicationCollectionComponent and AirflowInstanceComponent.
dg will now report multiple detected errors in a configuration file instead of failing on the first detected error.tzdata as a dependency to dagster, to ensure that declaring timezones like US/Central work in all environments.requirements.env is now opt-in for dg check yaml.DAGSTER_DBT_CLOUD_POLL_INTERVAL and DAGSTER_DBT_CLOUD_POLL_TIMEOUT environment variables can now be used to configure the polling interval and timeout for fetching data from dbt Cloud.asset_post_processors fields from SlingReplicationCollectionComponent and AirflowInstanceComponent.Allow explicit git platform selection in link_code_references_to_git, thanks @chazmo03!
platform selection in link_code_references_to_git, thanks @chazmo03!AutomationCondition.replace to not update built-in sub-conditions that did not have an explicit label, thanks @dschafer!enable_code_references was set to True on the DagsterDbtTranslatorSettings.dagster.yaml to include the nux option, thanks @dwisdom0!Static functions on classes decorated with @template_var can now optionally accept a ComponentLoadContext argument.
@template_var can now optionally accept a ComponentLoadContext argument.dg CLI capabilities to MCP clients. See the dg mcp CLI group for details.dagster-dbt package no longer has a dependency on dbt-core.dagster-dbt package to run dbt commands with no changes to their existing dagster code. This support is still in preview as the format of the log messages produced by the dbt Fusion CLI is still subject to change. Let us know if you notice any incompatibilities.PipesDatabricksServerlessClient to support Databricks Serverless jobs with Dagster pipes.ObjectMetadataValue class.Schedules now support specifying a subset of asset checks to execute in a RunRequest.
RunRequest.docs integrations cli is available for viewing an index of available integrations.refresh_and_poll inside the @tableau_assets asset decorator.defs folder.--assets, other definitions types will no longer be displayed.Introduced AssetExecutionContext.load_asset_value, which enables loading asset values from the IO manager dynamically rather than requiring asset valu
Introduced AssetExecutionContext.load_asset_value, which enables loading asset values from the IO manager dynamically rather than requiring asset values be loaded as parameters to the asset function. For example:
@dg.asset(deps=[the_asset])
def the_downstream_asset(context: dg.AssetExecutionContext):
return context.load_asset_value(dg.AssetKey("the_asset"))
Expose asset_selection parameter for submit_job_execution function in DagsterGraphQLClient, thanks @brunobbaraujo!
Large error stack traces from Dagster events will be automatically truncated if the message or stack trace exceeds 500kb. The exact value of the truncation can be overridden by setting the DAGSTER_EVENT_ERROR_FIELD_SIZE_LIMIT environment variable.
Added databento, ax, and botorch kind tags, thanks @aleewen and @CompRhys!
[dagster-k8s] Added the option to include ownerReferencess to k8s executor step jobs, ensuring that the step job and step pod are properly garbage collected if the run pod is deleted. These can be enabled by setting the enable_owner_references flag on the executor config.
[components] Added dg list component-tree command which can be used to visualize the component tree of a project.
[components] Added the ability to reference, load, and build defs for other components in the same project. In YAML, you may use the load_component_at_path and build_defs_at_path functions:
type: dagster.PythonScriptComponent
attributes:
execution:
path: my_script.py
assets:
- key: customers_export
deps:
- "{{ load_component_at_path('dbt_ingest').asset_key_for_model('customers') }}"
@graph_assets to not properly apply the AllPartitionMapping or LastPartitionMapping to dependencies, thanks @BoLiuV5!AutomationCondition.any_deps_match() or AutomationCondition.all_deps_match().create-dagster standalone executable now works on all Linux versions using glibc 2.17 or later.@dbt_assets decorator and associated APIs no longer error when parsing dbt projects that contain an owner with multiple emails.AutomationCondition.replace type signature to provide callers more information about the returned AutomationCondition, thanks @dschafer!Code location: location -> code_location: location.The dagster package now supports Python 3.13.
dagster package now supports Python 3.13.dagster-snowflake-polars package adds a SnowflakePolarsIOManager that can be used to read and write Snowflake tables using Polars.dbt_profiles.yml in the dagster-dbt tutorial (thanks @EFox2413!).dagster definitions validate will now raise an exception if there are invalid partition mappings between any assets in your asset graph (for example,
dagster definitions validate will now raise an exception if there are invalid partition mappings between any assets in your asset graph (for example, an upstream and downstream asset with time-based partitions definitions using different timezones).blocking.@graph now correctly allows omitting inputs when the destinations of an input mapping have a default value.@record classes no longer create problematic namespace conflicts with the symbol "check."dg scaffold github-actions would invoke the dg CLI with outdated parameters for serverless orgs.These supersede the LegacyFreshnessPolicy API (formerly FreshnessPolicy, deprecated in 1.10.0), as well as freshness checks.
dg CLI are now marked as GA (previously Release Candidate). The APIs are fully supported throughout all parts of the product and remain the recommended defaults for new Dagster projects.execute() and get_asset_spec() methods that can be overridden by subclasses, making it easier and more consistent to customize your components.StateBackedComponent abstract base class that enables components to persist and manage state separately from their YAML / Python configuration. This is particularly useful for integration components that need to fetch external data. State can be managed locally, in versioned storage, or via code server snapshots. Many integration components (AirbyteWorkspaceComponent, FivetranAccountComponent, PowerBIWorkspaceComponent, AirflowInstanceComponent, and DbtProjectComponent) now extend StateBackedComponent to provide better control over state management. You can check out the docs here!dg scaffold build-artifacts scaffolds Docker and configuration files needed to build and deploy your Dagster project to Dagster Cloud, with support for multiple container registries (ECR, DockerHub, GHCR, ACR, GCR).dg scaffold github-actions generates a complete GitHub Actions CI/CD workflow for deploying to Dagster Cloud, with auto-detection of Serverless vs Hybrid agents and guided setup for required secrets.LegacyFreshnessPolicy API (formerly FreshnessPolicy, deprecated in 1.10.0), as well as freshness checks.
FreshnessPolicy and apply_freshness_policy are now exported from the top-level dagster module (instead of dagster.preview.freshness ).build_.*_freshness_checks methods have been marked as superseded . Their functionality will remain unchanged, but we recommend using FreshnessPolicy s for new use cases.FreshnessDaemon now runs by default, rather than needing to be explicitly enabled via dagster.yaml settings. To turn it off, set:freshness:
enabled: false
TimeWindowPartitionsDefinition now support an exclusions parameter, which allows you to exclude specific dates/times or recurring schedules from your partition set. This is useful for implementing custom calendars that exclude weekends, holidays, or maintenance windows. Exclusions can be specified as cron strings (e.g., "0 0 * * 6" for Saturdays) or datetime objects for specific dates.Executors have been updated to accept an optional step_dependency_config parameter with a require_upstream_step_success flag. When set to False via {"step_dependency_config": {"require_upstream_step_success": False}}, downstream steps can start as soon as their required upstream outputs are available, rather than waiting for the entire upstream step to complete successfully. This is particularly useful for large multi-assets where downstream assets only depend on a subset of upstream outputs.[dagster-tableau] The deprecated get_*_asset_key methods on the DagsterTableauTranslator class have been removed.
tableau_assets decorator is now available to create the asset definitions of all materializable assets in a given Tableau workspace. These assets can be subsetted and materialized using the TableauCloudWorkspace.refresh_and_poll method.get_*_asset_key methods on the DagsterTableauTranslator class have been removed.staleStatusByPartition, staleCausesByPartition, and dataVersionByPartition fields on the graphql AssetNode would fail when called on an asset with dynamic partitions.__init__ method would raise an Exception during execution.sling dependency.SlingResource that could cause values specified with EnvVars to provide the env var name instead of the env var value to the sling replication configuration.dagster-fivetran snapshot command, allowing Fivetran workspaces to be captured to a file for faster subsequent loading.FunctionComponent, PythonScriptComponent, and UvRunComponent to make it easier to define arbitrary computations that execute assets or asset checks when invoked.partitions_def asset customization in YAML.dg scaffold defs ....partitions_def key.dg cache is no longer operative, since dg now operates in the same python environment as the projects it manipulates. Config options for the cache have been removed.load_defs entrypoint has been deprecated in favor of load_from_defs_folder, which takes a single path for a folder to load definitions from rather than a module object.asset_post_processors field on SlingReplicationCollectionComponent and AirflowInstanceComponent is no longer supported, and has been replaced with the top-level post_processors field.autoload_defs=true could not be deployed to Dagster+.@preview from Component.ComponentLoadContext is now frozen and no longer in preview.Resolved subclasses now support Enum fields.The @asset decorator now supports a hooks argument to allow for functions to be executed on asset success / failure (thanks @brunobbaraujo)!
@asset decorator now supports a hooks argument to allow for functions to be executed on asset success / failure (thanks @brunobbaraujo)!@multi_asset_check decorator now supports the pool argument.DagsterDbtTranslator class now has a get_asset_check_spec method which can be overridden to customize the AssetCheckSpecs that are produced for each individual dbt test.blocking attribute on an AssetCheckSpec would halt execution of the step as soon as any asset check failure was emitted, even if the step still had asset materializations or check evaluations to emit that were not downstream of the failed asset check. Now that this issue has been fixed, the blocking attribute on AssetCheckSpec has been made a public attribute. If you were making use of the blocking attribute on AssetCheckSpec before it was public and relying on the previous behavior, you should exit from your asset evaluation function after emitting an AssetCheckFailure from within your multi-asset in order to halt further execution of the step.GraphDefinition.to_job() would not work if an op had a custom IO manager key..allow() and .ignore() applications to not propagate through .since() automation conditions.dagster dev was sometimes failing to load code locations with a "Deadline Exceeded" error unless the --use-legacy-code-server-behavior flag was set.google-cloud-bigquery>=1.28.3 pin to correctly reflect the lowest compatible version.dg check yaml where valid component type names were rejected if they were not registered (i.e. visible from dg check components).dg create-dagster now warns when scaffolding a project or workspace if it is not the latest version.project.registry_modules configuration can now accept wildcards (e.g. foo_bar.components.*). This will register any module matching the pattern with dg.env YAML function now errors if the specified env var is unset. Default values can be provided as an additional argument: {{ env('MY_ENV_VAR', 'default') }}dg registry.PipesSubprocessScriptCollectionComponent has been removed.dg check defs and dg dev in a project folder that is part of the workspace will now only apply to that project, instead of every project in the workspace.dagster_dg_cli.plugin entry point.dg has been renamed from dagster_dg_cli.plugin to dagster_dg_cli.registry_modules (projects no longer need to define an entry point group at all).dg list plugin-modules has been renamed to dg list registry-modules.dg list defs now supports configuring output columns with the --columns/-c option.AirbyteCloudWorkspaceComponent which can be used to pull in Airbyte Cloud connections into DagsterThe database table used by the DbIOManager is now configurable via "table" output (asset) metadata key #30310
DbIOManager is now configurable via "table" output (asset) metadata key #30310use_threads=True, num_workers=4. Thanks @HynekBlaha!build_asset_check_context can be used to build asset check contexts for direct invocation.Definitions.get_all_asset_specs to only return assets directly passed in as AssetSpecs or AssetsDefinitions.selection argument from Definitions.map_asset_specs. If selection is needed use the new Definitions.map_resolved_asset_specs.Definitions.get_job_def now warns when it finds an unresolved job or no job.Definitions.get_assets_def to return an AssetsDefinition without resolving if it was passed in directly.build_schedule_from_dbt_selection now supports a selector argument, allowing you to use yaml-based selectors.dagster/deployment-name label.dg check defs and dg dev in a project folder that is part of the workspace will now only apply to that project, instead of every project in the workspace.dg list defs now supports the --assets/-a option, to show only assets matching the provided selection.dg list defs now supports a --path argument to subset the defs files shown.create-dagster workspace command now accepts the same required path argument as the create-dagster project command, instead of defaulting to a dagster-workspace subfolder of the current working directory.dagster_dg_cli.plugin to dagster_dg_cli.registry_modules (projects no longer need to define an entry point group at all).dg list plugin-modules has been renamed to dg list registry-modules.dagster_dg_cli.plugin entry point.Resolvable subclasses can now use bare dict and list as field types.Resolvable subclass will no longer change empty strings to None.@definitions-decorated functions in a single module in the defs hierarchy and they are automatically merged and incorporated into the project.@component_instance to replace @component. This allows multiple component instances in a python file.dg commands would sometimes output extra dagster_telemetry_logger lines to stdout at the end of commands.@template_var as an alternative approach for defining variables in a templating context."component.yaml" will remain supported for several weeks, but is deprecated.
BackfillPolicy is now marked as generally available (GA).@asset decorator overloads to make custom wrappers around the asset decorator easier. Thanks @jonathanunderwood!get_client() to SlackResource.Definitions and AssetDefinition will now warn if they get different AssetSpecs with the same key. This will become an exception in 1.11.load_definitions_from_modules now handle duplicate AssetSpecs.load_assets_from_fivetran_instance where assets whose asset keys have been customized using a Fivetran translator would lead to an exception.required_resource_keys in @sensor. Thanks @seyf97!Definitions and AssetDefinition will now warn if they get different AssetSpecs with the same key. This will become an exception in 1.11.--wait to the dagster-cloud job launch command that makes it wait until the launched run copmletes. Thanks @stevenayers!dagster-cloud cli failed to deploy PEX projects on python 3.12 in certain environments without setuptools already installed.dg CLI is now installed in each project's Python environment instead of as a global tool. If you previously had dg installed globally and are upgrading, first uninstall the global dg tool (uv tool uninstall dagster-dg or pip uninstall dagster-dg) and add dagster-dg-cli as a dependency to each of your projects. Newly scaffolded projects will automatically include the dg CLI going forward.create-dagster CLI has been added for scaffolding projects instead of workspaces. uvx create-dagster project has replaced dg scaffold project, and uvx create-dagster workspace has replaced dg scaffold workspace. The commands take identical arguments.dg scaffold defs <scaffolder> instead of dg scaffold <scaffolder>.component.yaml file to specify a component instance is now called "defs.yaml". "component.yaml" will remain supported for several weeks, but is deprecated.lib folder in newly scaffolded projects has been renamed to "components".dg scaffold component-type has been renamed to dg scaffold component.dg list plugins has been renamed to dg list plugin-modules and now outputs only plugin module names, not plugin objects.dg list component now lists component types instead of instances.dagster.components are now available in the top-level dagster module.@component_instance to replace @component, which allows multiple component instances in a python file.dg scaffold defs <component>, you will now be prompted with possible matches.dg scaffold defs <component>.dg launch now supports passing config files through --config/-c.Component.from_attributes_dict and Component.from_yaml_path to help with testing.dagster.components.testing.component_defs utility to help with testing components.dg are now loadable by default.dg scaffold defs inline-component can now be used to create inline components and corresponding instances.dg list defs now outputs resources.dagster-dbt cli flag --components flag now correctly finds DbtProjectComponent again.FivetranAccountComponent which can be used to pull in Fivetran connections into Dagster.load_assets_from_airbyte_instance or other APIs that return a CacheableAssetsDefinition.Nothing published for this version
Nothing published for this version
Nothing published for this version
Fixed an issue where an error was displayed in the UI while viewing run logs
typing_extensions is now pinned to >=4.11.0 instead of >=4.10.0.
typing_extensions is now pinned to >=4.11.0 instead of >=4.10.0.Enable faceted asset nodes).Display integrations marketplace). It provides easy access to the gallery of dagster-supported plugins.ExternalNotebookDataRequest GRPC call that would allow it to access files outside of the current directory.op_tags set on @observable_source_assets to be dropped from the underlying step context object when executed.dagster dev would sometimes raise a gRPC error when loading several code locations at once.BoolSource would evaluate to True.executable_path in a workspace.yaml file to run code locations in a different virtual environment would not correctly inherit the PATH of that virtual environment in the code location.OpExecutionContext was used as the type hint for the context.timeout property data type in ClientConfig to be str instead of int (thanks, @edsoncezar16!)K8sRunLauncher example (thanks, @nishan-soni!)AssetKey and AssetSpec.dg list env now displays whether env vars are configured in each Dagster Plus scope.Resolver.passthrough() to avoid processing fields on a component model.ResolvedAssetKey is now exported from dagster.components.dg init has been removed. dg scaffold project and dg scaffold workspace should be used instead.dg dev failed with a temporarily file permissions error when running on Windows. Thanks @polivbr!…object instead of a dictionary. This is a breaking change for those who have implemented a custom scaffolder.
dagster.yaml to enable submitting backfill runs in a threadpool.Enable retries from asset failure feature flag in your user settings.FivetranWorkspace resource is now marked as generally available (GA).FivetranResource resource is now deprecated. Use the new FivetranWorkspace resource instead.Scaffolder.scaffold to have the params object as an attribute of the ScaffoldRequest object instead of a dictionary. This is a breaking change for those who have implemented a custom scaffolder.dg scaffold dagster.resources path/to/resources.py.@definitions in the defs hierarchy.component.yaml file in your editor.DbtProjectComponent fields now properly evaluate templates.asset_attributes parameter to translation, in order to match our other integration components.dg dev failed with a temporary file permissions error when running on Windows. Thanks @polivbr!dagster.InitResourceContext.dagster_run has been deprecated in favor of InitResourceContext.run.
build_tableau_materializable_assets_definition.MultiPartitionsDefinition.has_partition_key() on invalid keys.AutomationCondition.initial_evaluation which could cause it to return False for an asset that went from having a condition, to having no condition at all, back to having the original condition again.DagsterDbtTranslatorSettings.enable_source_tests_as_checks returning duplicate asset checks.dagster.InitResourceContext.dagster_run has been deprecated in favor of InitResourceContext.run.Path.cwd() or os.getcwd(). Instead, you should use a path relative to the current source file when loading paths in a component, using the context.resolve_source_relative_path method (see resolve_dbt_project in DbtProjectComponent for an example).dagster.job scaffolder.DbtProjectComponent now has a translation_settings argument for adjusting DagsterDbtTranslatorSettings.DbtProjectComponent fields now properly evaluate templates.load_defs entrypoint, thanks @mattgiles!If an unselected asset check is executed during a run, the system will now warn instead of throwing a hard error.
AutomationCondition.any_deps_match or AutomationCondition.all_dep_match with an allow / ignore specified, an error will no longer be produced if the provided asset selection references an asset key that does not exist.dagster dev is allowed to use to open subprocesses when running on Windows, by setting the DAGSTER_PORT_RANGE env var to a string of the form <start>=<end> - for example "20000-30000".get_objects now returns an empty list if no new files can be found since the since_last_modified parameter. Thanks @bartcode!@dbt_assets and build_dbt_manifest_asset_selection now support a selector argument, allowing you to use yaml-based selectors.AutomationCondition.initial_evaluation which could cause it to return False for an asset that went from having a condition, to having no condition at all, back to having the original condition again.AutomationCondition.any_deps_updated() condition to evaluate to False when evaluated on a self-dependency.quickstart_aws example. Thanks @Thenkei!ResolvedAssetSpec. description and examples arguments have been added to Resolver for documenting fields on non-pydantic model based Resolvable classes.The dagster_components package has been merged into dagster and use of the dagster-components package has been deprecated. dagster-components will rem…
By partition grouping view for recent events for assets that do not have a definition in the workspace.can_subset=True to error when using dagster-pipes if not all outputs were emitted.dg will now fail with an error message if it's version is below the minimum supported version for the version of dagster in your environment.dg now checks for new versions and prompts the user to update. The check runs automatically at dg startup once per day.dg will now emit a warning prompting the user to reinstall the package if it detects an entry point in project metadata that does not show up when running in dg list plugins.dg list defs now collapses long system stacktraces #29507dagster.multi_asset scaffolderdagster.asset_check scaffolderdg list defs would crash if anything was written to stdout while loading a project's definitions.dg user config file on Unix has been moved from ~/.dg.toml to ~/.config/dg.toml. ~/.config can be overridden by setting $XDG_CONFIG_HOME.dg contents of the cache were in an outdated format, dg could crash. Now it will just rebuild the cache.pip and uv.dg CLI will now emit a warning if you are using "active" mode for your project python environment, there is a virtual environment at <project_root>/.venv, and the activated venv is not <project_root>/.venvdg setting cli.suppress_warnings is available. This takes a list of warning types to suppress.tool.dg.project.python_environment previously accepted a string, "active" or "persistent_uv". Now it accepts a table with one of two keys:
{active = true}: equivalent of previous "active"{uv_managed = true}: equivalent of previous "persistent_uv"tool.dg.project.python_environment to {active = true}. This means by default, no virtual environment or uv.lock will be created when scaffolding a new project (via dg init or dg scaffold project). You can pass --python-environment uv_managed for the old behavior.--skip-venv flag on dg scaffold project and dg init.dagster_components package has been merged into dagster and use of the dagster-components package has been deprecated. dagster-components will remain as a stub package for the next few weeks, but code should be updated to import from dagster.components instead of dagster_components.dagster-dbt project prepare-and-package cli now supports --components for handling DbtProjectComponentDbtProjectComponent has been reworked, changing both the python api and the yaml schema. dbt has been replaced with project with a slightly different schema, and asset_attributes with translation .[ui] Runs launched from the Dagster UI get a dagster/from_ui = true tag, making it easy to filter for them.
dagster/from_ui = true tag, making it easy to filter for them.AirflowFilter API for use with dagster-airlift, allows you to filter down the set of dags retrieved up front for perf improvements.can_subset=True to error when using dagster-pipes if not all outputs were emitted.state_path argument to DbtCliResource now resolves relative to the project directory as documented.DAGSTER_PIPES_K8S_CONSUME_POD_LOGS_REQUEST_TIMEOUT environment variable) and to retry up to 5 times if an error occurs while streaming logs from the launched Kubernetes pod (this value can be overridden by setting the DAGSTER_PIPES_K8S_CONSUME_POD_LOGS_RETRIES environment variable.)DbtCliInvocation.stream_raw_events. (Thanks @ross-whatnot!)dg CLI will now emit a warning if you are using "active" mode for your project python environment, there is a virtual environment at <project_root>/.venv, and the activated venv is not <project_root>/.venvdg setting cli.suppress_warnings is now available. This takes a list of warning types to suppress.tool.dg.project.python_environment previously accepted a string, "active" or "persistent_uv". Now it accepts a table with one of three keys: - {active = true}: equivalent of previous "active" - {uv_managed = true}: equivalent of previous "persistent_uv"tool.dg.project.python_environment to {active = true}. This means by default, no virtual environment or uv.lock will be created when scaffolding a new project (via dg init or dg scaffold project). You can pass --python-environment uv_managed for the old behavior.--skip-venv flag on dg scaffold project and dg init.dg scaffold project were lacking dagster as a dependency.dg plugin" guide.dg init command have changed. You may pass . as an argument to initialize a project/workspace in the CWD. See dg init --help for more details.dagster-components package (which was converted to a stub package in the last release) is no longer being published. All dagster-components functionality is now part of dagster.dagster_dg.plugin entry point group instead of dagster_dg.library. dagster_dg.library support is being kept for now for backcompatibility, but will be dropped in a few weeks.project/lib/__init__.py when scaffolding a component type.dg env list command is now dg list envdg plus env pull command is now dg plus pull env.dg list component-type command has been removed. There is a new dg list plugins with output that is a superset of dg list component-type.dg user config file on Unix is now looked for at ~/.dg.toml instead of ~/dg.toml.DbtProjectComponent has been reworked, changing both the python api and the yaml schema. dbt has been replaced with project with a slightly different schema, and asset_attributes with translation.Nothing published for this version
Nothing published for this version
A new blocking parameter has been added to build_last_update_freshness_checks and build_time_partition_freshness_checks.
blocking parameter has been added to build_last_update_freshness_checks and build_time_partition_freshness_checks.DAGSTER_GRPC_MAX_RX_BYTES and DAGSTER_GRPC_MAX_SEND_BYTES environment variables on the gRPC client and server processes.Definitions.map_asset_specs method, which allows for the transformation of properties on any AssetSpec or AssetsDefinition objects in the Definitions object which match a given asset selection.Definitions.validate_loadable and dagster definitions validate will now raise an error on assets with invalid partition mappings, like a TimeWindowPartitionMapping between two time-based partitions definitions with different timezones. Previously, these invalid partition mappings would not raise an error until they were used to launch a run.PipesK8sClient log streaming when transient networking errors occur. The default behavior of the PipesK8sClient is now to reconnect to the stream of logs every 3600 seconds (this value can be overridden by setting the DAGSTER_PIPES_K8S_CONSUME_POD_LOGS_REQUEST_TIMEOUT environment variable) and to retry up to 5 times if an error occurs while streaming logs from the launched Kubernetes pod (this value can be overridden by setting the DAGSTER_PIPES_K8S_CONSUME_POD_LOGS_RETRIES environment variable.)NOT_STARTED status instead of a STARTING status.AutomationCondition.executed_with_tags() that would cause the tag_keys argument to not be respected. Updated the display name of AutomationCondition.executed_with_tags() to contain the relevant tag_keys and tag_values.FivetranWorkspace.sync_and_poll(...). Instead, they skip and a warning message is logged.dagster-cloud job launch command when launching runs targeting large numbers of assets or asset checks.Fixed an issue with the identification of the base repository URL the DAGSTER_CLOUD_GIT_URL. The raw URL is now exposed as DAGSTER_CLOUD_RAW_GIT_URL.
dagster_components package has been merged into dagster. All symbols exported from dagster_components are now exported by dagster.dagster_dg.plugin entry point group instead of dagster_dg.library. dagster_dg.library support is being kept for now for backcompatibility, but will be dropped in a few weeks.Component.get_schema has been renamed to Component.get_model_cls. Override that instead to customize the frontend of your component. Component.get_schema will continue to work for the time being but will be removed at some point in the future.dg.toml files. dg settings in pyproject.toml are set under tool.dg, but in dg.toml they are set at the top level. A dg.toml may be used in place of pyproject.toml at either the project or workspace level.dg-scaffolded workspaces now include a dg.toml instead of pyproject.toml file.dg init or dg scaffold project now follow modern Python packaging conventions, placing the root module in a top-level src directory and use hatchling as build backend rather than setuptools.dagster.components.Model instead of getting decorated with @dataclasses.dataclass.dg scaffold dagster.asset will no longer be commented out.dg list component-type command has been removed. There is a new dg list plugins with output that is a superset of dg list component-type.dg list defs now includes infomation about asset checks.project/lib/__init__.py when scaffolding a component type.dg list defs.dg would crash when working with Python packages with an src-based layout.[ui] Fix custom time datepicker filter selection for users with custom Dagster timezone settings.
dg.toml files. dg settings in pyproject.toml are set under tool.dg, but in dg.toml they are set at the top level. A dg.toml may be used in place of pyproject.toml at either the project or workspace level.dg-scaffolded workspaces now include a dg.toml instead of pyproject.toml file.dg-scaffolded projects now place the root package in an src/ directory and use hatchling as the build backend rather than setuptools.dg would crash when working with Python packages with an src-based layout.dg check yaml now properly validates component files in nested subdirectories of the defs/ folder.dagster-components has been merged into dagster and use of the dagster-components package has been deprecated. dagster-components will remain as a stu…
AutomationCondition.initial_evaluation condition has been updated to become true for all partitions of an asset whenever the PartitionsDefinition of that asset changes, rather than whenever the structure of the condition changes.build_fivetran_assets_definitions in a single Definitions object would result in "duplicate node" errors when launching a run.key:"*substring*" instead of name:"*substring*".DefsModuleComponent has been renamed to DefsFolderComponent.dg scaffold my_project.ComponentType instead of dg scaffold component my_project.ComponentType.dagster list defs will now read environment variables from a local .env file if present when constructing the definitions.dagster-components has been merged into dagster and use of the dagster-components package has been deprecated.
dagster-components will remain as a stub package for the next few weeks, but code should be updated to import from dagster.components instead of dagster_components.DbtProjectComponent has been relocated to the dagster-dbt package,
importable as dagster_dbt.DbtProjectComponent.SlingReplicationCollectionComponent has been relocated to the dagster-sling package,
importable as dagster_sling.SlingReplicationCollectionComponent.[dagster-dbt] Replaced @validator with @field_validator in dagster_dbt/core/resource.py to prevent Pydantic deprecation warnings. (Thanks @tintamarre!…
AutomationCondition.initial_evaluation condition to become true for all partitions of an asset whenever the PartitionsDefinition of that asset changes, rather than whenever the structure of the condition changes.dagster-airbyte, dagster-dlift, dagster-dbt, dagster-fivetran, dagster-looker, dagster-powerbi, dagster-sigma, and dagster-tableau.key:"*substring*" instead of name:"*substring*".@validator with @field_validator in dagster_dbt/core/resource.py to prevent Pydantic deprecation warnings. (Thanks @tintamarre!)dagster-gcp version while you migrate config. Please see the full list of changed fields: https://gist.github.com/deepyaman/b4d562e04fe571e40037a344b7a9937ddagster-aws and/or dagster-spark version while you migrate config. Please see the full list of changed fields: https://gist.github.com/deepyaman/f358f5a70fea28d5f164aca8da3dee04use_dg_managed_environment and require_local_venv controlled the environment used when launching project subprocesses. This is now configured at the project level. The tool.dg.project.python_environment setting takes a value of either persistent_uv or active. persistent_uv will be used by default in new scaffolded projects and uses a uv-managed .venv in the project root. active is the default if no tool.dg.project.python_environment is set, and just uses the active python environment and opts out of dg management of the environment.Resolvable, has been added. This can be used to simplify the process of defining a yaml schema for your components. Instead of manually defining a manual ResolvedFrom[...] and ResolvableModel, the framework will automatically derive the model schema for you based off of the annotations of your class.@component) can now contain relative imports.dg init command now accepts optional --workspace-name and --project-name options to allow scaffolding an initial workspace and project via CLI options instead of prompts.dagster_components.dagster.DefsFolderComponent that can be used at any level of your defs/ folder to apply asset attributes to the definitions at or below that level. This was previously named dagster_components.dagster.DefsModuleComponent.Nothing published for this version
Added a new AutomationCondition.executed_with_tags() condition that makes it possible to filter for updates from runs with particular tags.
AutomationCondition.executed_with_tags() condition that makes it possible to filter for updates from runs with particular tags.AssetCheckEvaluation can now be yielded from Dagster ops to log an evaluation of an asset check outside of an asset context.kinds argument to dagster.AssetOut, allowing kinds to be specified in @multi_asset.AssetCheckEvaluations are now yielded from ops leveraging DbtCliResource.cli(...) when asset checks are included in the dbt asset lineage.SlingResource.replicate() method now takes an option stream parameter, which allows events to be streamed as the command executes, instead of waiting until it completes (thanks, @natpatchara-w!).DagsterGraphQLClient now supports an auth keyword argument, which is passed to the underlying RequestsHTTPTransport constructor.AutomationCondition.newly_requested() which could cause it to fail when nested within AutomationCondition.any_deps_match() or AutomationCondition.all_deps_match().AutomationCondition.replace() that would cause it to not effect AutomationCondition.since() conditions.dagster-airbyte, dagster-dlift, dagster-dbt, dagster-fivetran, dagster-looker, dagster-powerbi, dagster-sigma, and dagster-tableau.__pycache__ files are no longer included in the output of dg list component. (Thanks @stevenayers!)deps of an AssetSpec from yaml, multi-part asset keys are now correctly parsed. (Thanks @stevenayers!)dagster.components to dagster_dg.library.dg check yaml is now run by default before dg dev and dg check defs.Nothing published for this version
async def yield_for_execution is now supported on ConfigurableResource. An event_loop argument has been added to context builders to support direct ex
async def yield_for_execution is now supported on ConfigurableResource. An event_loop argument has been added to context builders to support direct execution.dagster dev deduplicates stacktraces when code locations fail to load, and will by default truncate them to highlight only user code frames.deltalake version has been bumped to 0.25.0; the PolarsDeltaIOManager is now using the rust engine for writing DeltaLake tables by default.AutomationCondition.since() conditions.dagster dev would fail to load code that took more than 45 seconds to import unless the --use-legacy-code-server-behavior flag was used.build_airbyte_assets_definitions function to error when attempting to modify the default group name.build_fivetran_assets_definitions function to error when attempting to modify the default group name.[ui] The asset overview tab for a partitioned asset now shows metadata and schema of the most recent materialization, not today's partition.
tags to the created RunRequest when using build_sensor_for_freshness_checks().PickledObjectGCSIOManager now replaces the underlying blob when the same asset is materialized multiple times, instead of deleting and then re-uploading the blob.Added links from pool info in run event logs to the respective pool configuration pages.
run.kinds tag. (Thanks @dragos-pop!)kinds tag. (Thanks @jjyeo!)PostgresEventLogStorage QueuePool by setting max_overflow to -1. (Thanks @axelwas!)RunRequests with the same run key within a single tick would cause two runs with the same key to be executed. Now, only the first run will be executed. (Thanks @Gw1p!)CANCELED state instead of a FAILURE state when not every requested partition has been marked as materialized or failed by the backfill.packaged_project_dir since supporting profiles_dir in DbtProject.DbIOManagers being unable to process subclasses of handled types.kinds tag. (Thanks @dragos-pop!)create_snowflake_insights_asset_and_schedule sensor using the schedule_batch_size_hrs parameter works to yield a single partition range run instead of individual runs per partition.Turned on run-blocking for concurrency keys / pools by default. For op granularity, runs are dequeued if there exists at least one op that can execute
dagster dev) has been changed from tmp to .tmp_dagster_home_. (Thanks @chazmo03!)AssetCheckResult descriptions are rendered in the table making it easier to publish a summary of check evaluation.psycopg2.errors.UndefinedColumn database error when trying to set a concurrency key without first having run dagster instance migrate.dagster dev, some changes were not reflected in the UI after pressing the "Reload Definitions" button.build_schedule_context instead of build_sensor_context.dagster instance migrate was failing for instances with non-empty concurrency limits tables.View queue criteria dialog.asset++) not including assets connected to the target asset by paths of varying distance.direct_snapshot_uploads (directSnapshotUploads in helm) which opts in to a new more efficient scheme for how definitions are handled during code location updates.event_log and dagster_event in dagster-cloud-test-infra to facilitate the creation of test data with sensible defaults for EventLogEntry and DagsterEvent objects.execution_project if defined.execution_project is defined in the dbt profile, fall back to fetching the dataset from the dbt profile's project if the dataset cannot be found in the execution_project.Fixed an issue where runs containing pool-assigned ops without limits set got stuck in the run queue.
FreshnessPolicy — A new `FreshnessPolicy` API is introduced, replacing the deprecated FreshnessPolicy API (which has been renamed to LegacyFreshnessPo…
Components, first released as Preview in 1.10.6, have reached Release Candidate status. APIs are stable, fully documented, and are the recommended, production-ready defaults for new Dagster projects.
defs.yaml, or a lightweight Component subclass in Python, lets you spin up arbitrary Dagster definitions (such as assets, resources, schedules, checks, and more), removing boilerplate while keeping every definition type-checked.Component lets you wrap any internal script or third-party tool behind a strongly-typed interface; get the same autocompletion and docs as first-party integrations.@template_var so pipeline authors edit parameters directly in YAML without touching Python. Optional inline components keep small bits of Python co-located.dg — the everything-CLI for developer experience (now stable)The dg CLI provides a single surface for scaffolding, local iteration, execution, and static analysis; introduced as Preview in 1.10.6, it has also reached Release Candidate status.
dg scaffold defs dagster.asset assets.py, dg scaffold component … to quickly generate definitions without boilerplate.dg dev spins up local instance with UI; dg launch runs jobs and assets from the CLI.dg list enumerates definitions; dg check validates YAML and Python code.dg utils provides support for Cursor/VSCode schema setup, JSON-schema export, and deep component inspection.create-dagster — one-shot project scaffoldcreate-dagster scaffolds a ready-to-run Dagster project or workspace in one command (pipx, uvx, brew, curl friendly).
create-dagster project supersedes the dagster project scaffold flow with the modern src/ + defs/ layout and a pre-wired local dg CLI, with no active Python environment required.AssetCheckEvaluation objects can now be yielded from within ops.@asset now accepts a hooks argument for success/failure callbacks.BackfillPolicy is now GAFreshnessPolicy API is introduced, replacing the deprecated FreshnessPolicy API (which has been renamed to LegacyFreshnessPolicy). The API is under active development, and will eventually also supersede freshness checks as the primary way of specifying and evaluating asset freshness. For more details, check out the GitHub announcement and the docs.FivetranWorkspace resource is now GA [docs].Fixed a bug where runs using global op concurrency would raise an exception when claiming a concurrency slot.
Adds a top-level argument pool to asset/op definitions to replace the use of op tags to specify concurrency conditions.
pool to asset/op definitions to replace the use of op tags to specify concurrency conditions.dagster definitions validate command now loads locations in-process by default, which speeds up runtime.py.typed file, which means their type annotations will be used by static analyzers. Previously a few libraries were missing this file.dagster instance migrate to run the data migration. The migration will update serialized backfill objects in the database with an end timestamp attribute computed by querying the runs launched by that backfill to determine when the last run completed.dagster instance migrate.concurrency deployment setting, along with new settings for concurrency pools.DbtProject.DagsterDltTranslator.get_* methods have been superseded in favor of DagsterDltTranslator.get_asset_spec.PipesDataprocJobClient, a Pipes client for running workloads on GCP Dataproc in Job mode.DagsterLookerLkmlTranslator.get_* methods have been superseded in favor of DagsterLookerLkmlTranslator.get_asset_spec.MultiPartitionsDefinition to a PartitionSubset.dagster dev on Windows, starting in the 1.9.10 release.None default config fields weren't being displayed in the Launchpad view.dagster-dbt 0.25.7 which would cause execution to fail when using the @dbt_assets decorator with an io_manager_key specified.>=1.27.0 to allow using dagster-cloud with protobuf versions 3 and 4.Fixed an issue where running dagster dev would fail on Windows machines.
dagster dev would fail on Windows machines.Added a new .replace() method to AutomationCondition, which allows sub-conditions to be modified in-place.
.replace() method to AutomationCondition, which allows sub-conditions to be modified in-place..allow() and .ignore() methods to the boolean AutomationCondition operators, which allow asset selections to be propagated to sub-conditions such as AutomationCondition.any_deps_match() and AutomationCondition.all_deps_match().DAGSTER_REDACT_USER_CODE_ERRORS environment variable to mask user code errors, the unmasked log lines are now written using a dagster.masked Python logger instead of being written to stderr, allowing the format of those log lines to be customized.get_partition_key() helper method that can be used on hourly/daily/weekly/monthly partitioned assets to get the partition key for any given partition definition. (Thanks @Gw1p!)task_definition_prefix argument to EcsRunLauncher, allowing the name of the task definition families for launched runs to be customized. Previously, the task definition families always started with run.PipesEMRContainersClient Dagster Pipes client for running and monitoring workloads on AWS EMR on EKS with Dagster.{"my_key": {"raw_value": 111, "type": "timestamp"}}).PipesDbfsMessageReader(include_stdio_in_messages=True).rust engine support when writing a Delta Lake table using native partitioning. (Thanks @Milias!)AutomationCondition on an observable source asset could sometimes result in invalid backfills being launched.AndAutomationCondition.without() no longer removes the condition's label.PipesEMRServerlessClient failing if a job is in the QUEUED state.SnowflakeResource resulted in an error.openai kinds tag to be "OpenAI" instead of "Open AI" in line with the OpenAI branding.[dagster-embedded-elt] the dagster-embedded-elt library is deprecated in favor of dagster-dlt and dagster-sling.
load_definitions_from_module, which can load all the assets, checks, schedules, sensors, and job objects within a module scope into a single Definitions object. Check out the documentation to learn more.DAGSTER_REDACT_USER_CODE_ERRORS environment variable to mask user code errors, the unmasked log lines are now written using a dagster.redacted_errors Python logger instead of being written to stderr, allowing the format of those log lines to be customized.croniter package is now vendored in dagster.minstral typo and updated the Mistral logo for asset kinds tag.task_definition_prefix argument to EcsRunLauncher, allowing the name of the task definition families for launched runs to be customized. Previously, the task definition families always started with run.dagster-dlt library is added. It replaces the dlt module of dagster-embedded-elt.dagster-sling library is added. It replaces the Sling module of dagster-embedded-elt.load_assets_from_modules where AssetSpec objects were being given the key_prefix instead of the source_key_prefix. Going forward, when load_specs is set to True, only the source_key_prefix will affect AssetSpec objects.arn:aws:iam: prefix.PipesEMRServerlessClient trying to get the dashboard URL for a run before it transitions to RUNNING state.@dbt_assets decorator would be ignored.show_url_only parameter on the AzureBlobComputeLogManager. (Thanks @ion-elgreco!)networkConfiguration parameters for the ECS run launcher. (Thanks @markgrin!)dagster-embedded-elt library is deprecated in favor of dagster-dlt and dagster-sling.Fixed a bug with load_assets_from_x functions where we began erroring when a spec and AssetsDefinition had the same key in a given module. We now only
load_assets_from_x functions where we began erroring when a spec and AssetsDefinition had the same key in a given module. We now only error in this case if include_specs=True.show_url_only option to True to enable the URL showing behavior.@dbt_assets decorator would be ignoredDeprecation of @multi_asset_sensor has been rolled back.
load_definitions_from_module, which can load all the assets, checks, schedules, sensors, and job objects within a module scope into a single Definitions object. Check out the documentation to learn more: https://docs.dagster.io/_apidocs/definitions#dagster.load_definitions_from_module.BackfillPolicy, or none of them did. Mixed selections are now supported.AssetSpecs may now contain a partitions_def. Different AssetSpecs passed to the same invocation of @multi_asset can now have different PartitionsDefinitions, as long as can_subset=True.title for asset backfills (not just for job backfills).airbyte_assets decorator has been added. It can be used with the AirbyteCloudWorkspace resource and DagsterAirbyteTranslator translator to load Airbyte tables for a given connection as assets in Dagster. The build_airbyte_assets_definitions factory can be used to create assets for all the connections in your Airbyte workspace.AirbyteCloudWorkspace.sync_and_poll(…) method in the definition of a @airbyte_assets decorator.ecs_executor which executes Dagster steps via AWS ECS tasks. This can be used in conjunction with ECSRunLauncher.dbt-core>=1.9 is now supported.load_looker_asset_specs and build_looker_pdt_assets_definitions are updated to accept an instance of DagsterLookerApiTranslator or custom subclass.DagsterLookerApiTranslator.get_asset_spec have been updated - the parameter looker_structure is now of type LookerApiTranslatorStructureData instead of LookerStructureData. Custom Looker API translators should be updated.load_powerbi_asset_specs has been updated to accept an instance of DagsterPowerBITranslator or custom subclass.DagsterPowerBITranslator.get_asset_spec have been updated - the parameter data is now of type PowerBITranslatorData instead of PowerBIContentData. Custom Power BI translators should be updated.load_sigma_asset_specs has been updated to accept an instance of DagsterSigmaTranslator or a custom subclass.DagsterLookerApiTranslator.get_asset_spec have been updated - the parameter data is now of type Union[SigmaDatasetTranslatorData, SigmaWorkbookTranslatorData] instead of Union[SigmaDataset, SigmaWorkbook]. Custom Looker API translators should be updated.load_tableau_asset_specs has been updated to accept an instance of DagsterTableauTranslator or custom subclass.DagsterTableauTranslator.get_asset_spec have been updated - the parameter data is now of type TableauTranslatorData instead of TableauContentData. Custom Tableau translators should be updated.dagster-airlift package erroneously introduced a dependency on dagster. This has been rectified - dagster is only required for the dagster-airlift[core] submodule.@multi_asset_sensor has been rolled back.blobStorageSnapshotUploads has been added which enables a new process for how definition snapshots are uploaded to Dagster Cloud.Updated cronitor pin to allow versions >= 5.0.1 to enable use of DayOfWeek as 7. Cronitor 4.0.0 is still disallowed. (Thanks, @joshuataylor!)
cronitor pin to allow versions >= 5.0.1 to enable use of DayOfWeek as 7. Cronitor 4.0.0 is still disallowed. (Thanks, @joshuataylor!)checkDbReadyInitContainer to optionally disable db check initContainer.kind tags. (Thanks, @dragos-pop!)Re-executions.kind tag for display in the UI.DataprocResource now receives an optional parameter labels to be attached to Dataproc clusters. (Thanks, @thiagoazcampos!)checkDbReadyInitContainer flag to the Dagster Helm chart to allow disabling the default init container behavior. (Thanks, @easontm!)build_materialize_workbook_assets_definition which can be used to build assets that run materialize schedules for a Sigma workbook.SnowflakeResource and SnowflakeIOManager both accept additional_snowflake_connection_args config. This dictionary of arguments will be passed to the snowflake.connector.connect method. This config will be ignored if you are using the sqlalchemy connector.BackfillPolicy are now evaluated correctly during backfills. Self dependent assets no longer result in serial partition submissions or disregarded upstream dependencies.PipesCloudWatchMessageReader correctly identifies streams which are not ready yet and doesn't fail on ThrottlingException. (Thanks, @jenkoian!)FivetranWorkspace.sync_and_poll(...).fetch_column_metadata().dlt_assets API docs. (Thanks, @zilto!)The automatic run retry daemon has been updated so that there is a single source of truth for if a run will be retried and if the retry has been launc
dagster/run-id, dagster/code-location, dagster/user and Dagster Cloud environment variables) typically attached to external resources are now available under DagsterRun.dagster_execution_info.SensorReturnTypesUnion is now exported for typing the output of sensor functions.@fivetran_assets decorator.load_fivetran_asset_specs has been updated to accept an instance of DagsterFivetranTranslator or custom subclass.fivetran_assets decorator was added. It can be used with the FivetranWorkspace resource and DagsterFivetranTranslator translator to load Fivetran tables for a given connector as assets in Dagster. The build_fivetran_assets_definitions factory can be used to create assets for all the connectors in your Fivetran workspace.ECSPipesClient.run now waits up to 70 days for tasks completion (waiter parameters are configurable) (Thanks @jenkoian!)load_airbyte_cloud_asset_specs function has
been added. It can be used with the AirbyteCloudWorkspace resource and DagsterAirbyteTranslator translator to load your Airbyte Cloud connection streams as external assets in Dagster.icechunk kind.ConsolidatedSqliteEventLogStorage, which is mostly used for tests.[dagster-powerbi] DagsterPowerBITranslator.get_asset_key is deprecated in favor of DagsterPowerBITranslator.get_asset_spec().key
1.6.0 may need to run dagster instance migrate to enable.map_asset_specs to enable modifying AssetSpecs and AssetsDefinitions in bulk.AssetSpec.replace_attributes and AssetSpec.merge_attributes to easily alter properties of an asset spec.PipesS3MessageReader now has a new parameter include_stdio_in_messages which enables log forwarding to Dagster via Pipes messages.log_external_stream has been added. It can be used to forward external logs to Dagster via Pipes messages.load_powerbi_asset_specs(..., use_workspace_scan=False).dagster-sigma snapshot command, allowing Sigma workspaces to be captured to a file for faster subsequent loading.DagsterExecutionStepNotFoundError errors when trying to execute an asset check step of a run launched by a backfill.owners added to AssetOuts when defining a @graph_multi_asset were not added to the underlying AssetsDefinition.& or | operators on AutomationConditions with labels would cause that label to be erased.SigmaFilter no longer fetch lineage information.DagsterPowerBITranslator.get_asset_key is deprecated in favor of DagsterPowerBITranslator.get_asset_spec().keyDagsterLookerApiTranslator.get_asset_key is deprecated in favor of DagsterLookerApiTranslator.get_asset_spec().keyDagsterSigmaTranslator.get_asset_key is deprecated in favor of DagsterSigmaTranslator.get_asset_spec().keyDagsterTableauTranslator.get_asset_key is deprecated in favor of DagsterTableauTranslator.get_asset_spec().keyNothing published for this version
Added run_id to the run_tags index to improve database performance. Run dagster instance migrate to update the index. (Thanks, @HynekBlaha!)
Added run_id to the run_tags index to improve database performance. Run dagster instance migrate to update the index. (Thanks, @HynekBlaha!)
Added icons for kind tags: Cassandra, ClickHouse, CockroachDB, Doris, Druid, Elasticsearch, Flink, Hadoop, Impala, Kafka, MariaDB, MinIO, Pinot, Presto, Pulsar, RabbitMQ, Redis, Redpanda, ScyllaDB, Starrocks, and Superset. (Thanks, @swrookie!)
Added a new icon for the Denodo kind tag. (Thanks, @tintamarre!)
Errors raised from defining more than one Definitions object at module scope now include the object names so that the source of the error is easier to determine.
[ui] Asset metadata entries like dagster/row_count now appear on the events page and are properly hidden on the overview page when they appear in the sidebar.
[dagster-aws] PipesGlueClient now attaches AWS Glue metadata to Dagster results produced during Pipes invocation.
[dagster-aws] PipesEMRServerlessClient now attaches AWS EMR Serverless metadata to Dagster results produced during Pipes invocation and adds Dagster tags to the job run.
[dagster-aws] PipesECSClient now attaches AWS ECS metadata to Dagster results produced during Pipes invocation and adds Dagster tags to the ECS task.
[dagster-aws] PipesEMRClient now attaches AWS EMR metadata to Dagster results produced during Pipes invocation.
[dagster-databricks] PipesDatabricksClient now attaches Databricks metadata to Dagster results produced during Pipes invocation and adds Dagster tags to the Databricks job.
[dagster-fivetran] Added load_fivetran_asset_specs function. It can be used with the FivetranWorkspace resource and DagsterFivetranTranslator translator to load your Fivetran connector tables as external assets in Dagster.
[dagster-looker] Errors are now handled more gracefully when parsing derived tables.
[dagster-sigma] Sigma assets now contain extra metadata and kind tags.
[dagster-sigma] Added support for direct workbook to warehouse table dependencies.
[dagster-sigma] Added include_unused_datasets field to SigmaFilter to disable pulling datasets that aren't used by a downstream workbook.
[dagster-sigma] Added skip_fetch_column_data option to skip loading Sigma column lineage. This can speed up loading large instances.
[dagster-sigma] Introduced an experimental dagster-sigma snapshot command, allowing Sigma workspaces to be captured to a file for faster subsequent loading.
dagster-airlift (experimental)dagster-airlift is coming out of stealth. See the initial Airlift RFC here, and the following documentation to learn more:
More Airflow-related content is coming soon! We'd love for you to check it out, and post any comments / questions in the #airflow-migration channel in the Dagster slack.
monitor_all_code_locations and monitored_jobs did not raise the expected error. (Thanks, @apetryla!)AutomationCondition.any_deps_match() and AutomationCondition.all_deps_match() to render incorrectly when allow_selection or ignore_selection were set.CacheableAssetsDefinitions in code locations that contained AutomationConditionsToo many open files errors for jobs with many steps.dagster-dingtalk to the list of community supported libraries.dagster-wandb (Weights and Biases) documentation. (Thanks, @matt-weingarten!)dagster-sigma documentation.Introduced a new constructor, AssetOut.from_spec, that will construct an AssetOut from an AssetSpec.
AssetOut.from_spec, that will construct an AssetOut from an AssetSpec.Column name section of the asset overview page.gcs (Google Cloud Storage) kind tag.report and semanticmodel kind tags.EcsRunLauncher. (Thanks, @zyd14!)DagsterDbtTranslator.get_code_version to customize the code version for your dbt assets. (Thanks, @Grzyblon!)PipesClientCompletedInvocation. This metadata will be attached to all materializations and asset checks stored during the pipes invocation.AutomationCondition.execution_in_progress which would cause it to evaluate to True for unpartitioned assets that were part of a run that was in progress, even if the asset itself had already been materialized.AutomationCondition.run_in_progress that would cause it to ignore queued runs.default_automation_condition_sensor to be constructed for user code servers running on dagster version < 1.9.0 even if the legacy auto_materialize: use_sensors configuration setting was set to False.0 in cases where all partitions were evaluated./ characters now work correctly with Dagster Pipes.dagster-airlift.types-sqlalchemy package is no longer included in the dagster[pyright] extra package.dagster project scaffold now has an option to create dagster projects from templates with excluded files/filepaths.
dagster project scaffold now has an option to create dagster projects from templates with excluded files/filepaths.parse_tableau_external_and_materializable_asset_specs is now available to parse a list of Tableau asset specs into a list of external asset specs and materializable asset specs.DagsterFivetranTranslator to customize assets loaded from Fivetran.dagster_snowflake.fetch_last_updated_timestamps now supports ignoring tables not found in Snowflake instead of raising an error.default_automation_condition_sensor to be constructed for user code servers running on dagster version < 1.9.0 even if the legacy auto_materialize: use_sensors configuration setting was set to False.dagster instance migrate on Dagster version 1.9.0 constructed a SQL query that exceeded the maximum allowed depth.ImportErrors are no longer raised when bigquery libraries are not installed [#25708]Nothing published for this version
Passing a custom PartitionsDefinition subclass into a Definitions object now issues an error instead of a deprecation warning.
tags parameter to @asset and AssetSpec is no longer marked as experimental.@observable_source_asset decorator now supports an automation_condition argument.AutomationCondition and associated APIs are no longer marked as experimental.use_user_code_server parameter to AutomationConditionSensorDefinition. If set, the sensor will be evaluated in the user code server (as traditional sensors are), allowing custom AutomationCondition subclasses to be evaluated.dagster instance migrate. This migration involves a schema migration to add the new columns and table, and a data migration to populate the new columns for historical backfills and runs.container_kwargs.stop_timeout can now be set when using the DockerRunLauncher or docker_executor to configure the amount of time that Docker will wait when terminating a run for it to clean up before forcibly stopping it with a SIGKILL signal.build_dbt_asset_selection.dagster no longer supports Python 3.8, which hit EOL on 2024-10-07.dagster now requires pydantic>=2.AutomationConditionSensorDefinitions will now emit backfills to handle cases where more than one partition of an asset is requested on a given tick. This allows that asset's BackfillPolicy to be respected. This feature can be disabled by setting allow_backfills to False.PartitionsDefinition subclass into a Definitions object now issues an error instead of a deprecation warning.AssetExecutionContext is no longer a subclass of OpExecutionContext. At this release, AssetExecutionContext and OpExecutionContext implement the same methods, but in the future, the methods implemented by each class may diverge. If you have written helper functions with OpExecutionContext type annotations, they may need to be updated to include AssetExecutionContext depending on your usage. Explicit calls to isinstance(context, OpExecutionContext) will now fail if context is an AssetExecutionContext.asset_selection parameter on AutomationConditionSensorDefinition has been renamed to target, to align with existing sensor APIs.freshness_policy_sensor has been removed, as it relies on the long-deprecated FreshnessPolicy API.external_assets_from_specs and external_asset_from_spec methods have been removed. Users should use AssetsDefinition(specs=[...]), or pass specs directly into the Definitions object instead.AssetKey objects can no longer be iterated over or indexed in to. This behavior was never an intended access pattern and in all observed cases was a mistake.dagster/relation_identifier metadata key has been renamed to dagster/table_name.dagster-ge now only supports great_expectations>=0.17.15. The ge_validation_op_factory API has been replaced with the API previously called ge_validation_op_factory_v3.dagster_aws.pipes.PipesGlueClient.run.dlt_dagster_translator from @dlt_assets. The dagster_dlt_translator parameter should be used instead.DataBricksPysparkStepLauncher, EmrPySparkStepLauncher, and any custom subclass of StepLauncher have been marked as deprecated, but will not be removed from the codebase until Dagster 2.0 is released, meaning they will continue to function as they currently do for the foreseeable future. Their functionality has been superseded by the interfaces provided by dagster-pipes, and so future development work will be focused there.multi_asset_sensor has been marked as deprecated, as its main use cases have been superseded by the AutomationCondition APIs. However, it will not be removed until version 2.0.0.The AutomationCondition.eager(), AutomationCondition.missing(), and AutomationCondition.on_cron conditions are now compatible with asset checks.
AutomationCondition.eager(), AutomationCondition.missing(), and AutomationCondition.on_cron conditions are now compatible with asset checks.AssetSelection.materializable(), which returns only assets that are materializable in an existing selection.AutomationCondition.all_deps_blocking_checks_passed condition, which can be used to prevent materialization when any upstream blocking checks have failed.code_version parameter to the @graph_asset decorator.LaunchPartitionBackfill mutation is submitted to GQL with invalid partition keys, it will now return an early PartitionKeysNotFoundError.AssetSelection.checks_for_assets now accepts AssetKeys and string asset keys, in addition to AssetsDefinitions.get_objects now has a since_last_modified that enables only fetching objects modified after a given timestamp.dagster_aws.pipes.PipesEMRCLient ) for running and monitoring AWS EMR jobs from Dagster.AutomationConditions that operate over dependencies.DatabricksPySparkStepLauncher, so that permissions are now set correctly for non-admin users.fetch_column_metadata did not work properly for models imported through dbt dependencies.DagsterK8sPipesClient.run now shows up in API docs.[experimental] AutomationCondition.eager() will now only launch runs for missing partitions which become missing _after_ the condition has been added
create_pull_request method on GithubClient that enables creating a pull request.create_ref method on GithubClient that enables creating a new branch.dlt.run(...).fetch_row_count() for both partitioned and non-partitioned assets. Thanks @kristianandre!PartitionedConfig objects can now return a RunConfig without causing a crash.AssetIn.__new__ typing for the dagster_type argument.fetch_row_count() method now works for databases returning uppercase column names. Thanks @kristianandre!GCSFileManager.read operations. Thanks @ollie-bell!Your coding agent can read these notes before it upgrades. Set up the MCP server →