NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
PyPI · #1454 most downloaded on PyPI
Dagster is an orchestration platform for the development, production, and observation of data assets.
Last release 6 days ago
21 Sep 2026
Ships on a steady schedule
a new release about every 2 weeks
Nearly every release is documented
notes for 54 of the last 60 stable releases
6 versions withdrawn
withdrawn after publishing
8 years old
744 releases · first in 2018
Memoization can now be used with the multiprocess, k8s, celery-k8s, and dask executors.
Nothing published for this version
Improved the error message for situations where you try a, b = my_op(), inside @graph or @job, but my_op only has a single Out.
a, b = my_op(), inside @graph or @job, but my_op only has a single Out.dbt_cloud_run_op, as well as a more flexible dbt_cloud_resource for more customized use cases. Check out the api docs to learn more!pipeline launch / job launch CLIs that would spin up an ephemeral dagster instance for the launch, then tear it down before the run actually executed. Now, the CLI will enforce that your instance is non-ephemeral.pipeline argument of the InitExecutorContext constructor has been changed to job.@asset decorator now accepts a dagster_type argument, which determines the DagsterType for the output of the asset op.build_assets_job accepts an executor_def argument, which determines the executor for the job.One column per quarter.
Nothing published for this version
google-cloud-bigquery is temporarily pinned to be prior to version 3 due to a breaking change in that version.
google-cloud-bigquery is temporarily pinned to be prior to version 3 due to a breaking change in that version.Nothing published for this version
Previously, the EcsRunLauncher tagged each ECS task with its corresponding Dagster Run ID. ECS tagging isn't supported for AWS accounts that have not
EcsRunLauncher tagged each ECS task with its corresponding Dagster Run ID. ECS tagging isn't supported for AWS accounts that have not yet migrated to using the long ARN format. Now, the EcsRunLauncher only adds this tag if your AWS account has the long ARN format enabled.k8s_job_executor and docker_executor that could result in jobs exiting as SUCCESS before all ops have run.k8s_job_executor and docker_executor that could result in jobs failing when an op is skipped.graphene is temporarily pinned to be prior to version 3 to unbreak Dagit dependencies.Nothing published for this version
[dagster-fivetran] A new dagster-fivetran integration allows you to launch Fivetran syncs and monitor their progress from within Dagster. It provides
fivetran_sync_op, as well as a more flexible fivetran_resource for more customized use cases. Check out the api docs to learn more!SourceHashVersionStrategy class has been added, which versions op and resource code. It can be provided to a job like so:from dagster import job, SourceHashVersionStrategy
@job(version_strategy=SourceHashVersionStrategy())
def my_job():
...
Workspace > Graph view.dagster job selected both pipelines and jobs. This release changes the dagster job command to select only jobs and not pipelines.Nothing published for this version
…factory function. Deprecated pipeline_name argument in favor of job_name in all the APIs.
k8s_job_executor is no longer experimental, and is recommended for production workloads. This executor runs each op in a separate Kubernetes job. We recommend this executor for Dagster jobs that require greater isolation than the multiprocess executor can provide within a single Kubernetes pod. The celery_k8s_job_executor will still be supported, but is recommended only for use cases where Celery is required (The most common example is to offer step concurrency limits using multiple Celery queues). Otherwise, the k8s_job_executor is the best way to get Kubernetes job isolation.make_dagster_job_from_airflow_dag factory function. Deprecated pipeline_name argument in favor of job_name in all the APIs.chardet library that was required due to an incompatibility with an old version of the aiohttp library, which has since been fixed.ins argument of the op decorator.slack_on_run_failure_sensor now says “Job” instead of “Pipeline” in its default message.DagsterTypeCheckDidNotPass error when a Dagster Type contained a List inside a Tuple (thanks @jan-eat!)dagstermill, dagster-pandas, dagster-airflow, etc)execute_in_process, when job does not have a top-level output.Nothing published for this version
[dagster-k8s] Fixed a bug that caused retries to occur twice with the k8s_job_executor
k8s_job_executorNothing published for this version
Updated dagstermill to better support job/op/graph changes by adding a define_dagstermill_op factory function. Also updated documentation and examples
define_dagstermill_op factory function. Also updated documentation and examples to reflect these changes.from dagster import get_dagster_logger)dbt_cli_resource, dbt_rpc_resource, and dbt_rpc_sync_resource) now support the dbt ls command: context.resources.dbt.ls().ins and outs properties to OpDefinition.resources_config argument on build_solid_context. The config argument has been renamed to solid_config.run_worker and step_worker, respectively.execute_in_processExecutor.execute(...) has changed from pipeline_context to plan_contextdagster, rather than repeating the job name. Thanks @skirino![dagster-docker] Added a new docker_executor which executes steps in separate Docker containers.
The dagster-daemon process can now detect hanging runs and restart crashed run workers. Currently
only supported for jobs using the docker_executor and k8s_job_executor. Enable this feature in your dagster.yaml with:
run_monitoring:
enabled: true
Documentation coming soon. Reach out in the #dagster-support Slack channel if you are interested in using this feature.
dagster-aws, dagster-github, and dagster-slack to reference job/op/graph APIs.Nothing published for this version
Nothing published for this version
Nothing published for this version
All dbt resources (dbt_cli_resource, dbt_rpc_resource, and dbt_rpc_sync_resource) now support the ls command.
ls command.Nothing published for this version
Intermediate storages, which were deprecated in 0.10.0, have now been removed. Refer to the “Deprecation: Intermediate Storage” section of the 0.10.0…
execution:
celery-k8s:
Nothing published for this version
You can now configure credentials for the GCSComputeLogManager using a string or environment variable instead of passing a path to a credentials file.
GCSComputeLogManager using a string or environment variable instead of passing a path to a credentials file. Thanks @silentsokolov!dagster instance migrate would run out of memory when migrating over long run histories.Nothing published for this version
Nothing published for this version
Updated click version, thanks @ashwin153!
dagster_aws.s3.sensor.get_s3_keys that would return no keys if an invalid s3 key was providedmy_log.info("foo %s", "bar") would cause errors in some scenarios.build_assets_job. The asset graph shows each node in the job’s graph with metadata about the asset it corresponds to - including asset materializations. It also contains links to upstream jobs that produce assets consumed by the job, as well as downstream jobs that consume assets produced by the job.load_assets_from_dbt_project and load_assets_from_dbt_project that would cause runs to fail if no runtime_metadata_fn argument were supplied.@asset not to infer the type of inputs and outputs from type annotations of the decorated function.@asset now accepts a compute_kind argument. You can supply values like “spark”, “pandas”, or “dbt”, and see them represented as a badge on the asset in the Dagit asset graph.Nothing published for this version
Nothing published for this version
Nothing published for this version
Note: This is a breaking change for anyone using the experimental VersionStrategy API. Instead of directly being passed solid_def and resource_def, yo…
Changed VersionStrategy.get_solid_version and VersionStrategy.get_resource_version to take in a SolidVersionContext and ResourceVersionContext, respectively. This gives VersionStrategy access to the config (in addition to the definition object) when determining the code version for memoization. (Thanks @RBrossard!).
Note: This is a breaking change for anyone using the experimental VersionStrategy API. Instead of directly being passed solid_def and resource_def, you should access them off of the context object using context.solid_def and context.resource_def respectively.
Nothing published for this version
[dagster-msteams] Introduced a new integration with Microsoft Teams, which includes a connection resource and support for sending messages to Microsof
emr_pyspark_step_launcher to fail when stderr included non-Log4J-formatted lines.applyPerUniqueValue config on the QueuedRunCoordinator to fail Helm schema validation.@asset decorator and build_assets_job APIs to construct asset-based jobs, along with Dagit support.load_assets_from_dbt_project and load_assets_from_dbt_manifest, which enable constructing asset-based jobs from DBT models.Nothing published for this version
We've deprecated output_notebook argument in define_dagstermill_solid in favor of output_notebook_name.
[dagstermill] You can now have more precise IO control over the output notebooks by specifying output_notebook_name in define_dagstermill_solid and providing your own IO manager via "output_notebook_io_manager" resource key.
output_notebook argument in define_dagstermill_solid in favor of output_notebook_name.local_output_notebook_io_manager is provided for handling local output notebook materialization.Dagit fonts have been updated.
context.log.info("foo %s", "bar") would not get formatted as expected.QueuedRunCoordinator’s tag_concurrency_limits to not be respected in some casestags argument of the @graph decorator or GraphDefinition constructor. These tags will be set on any runs of jobs are built from invoking to_job on the graph.k8s_job_executor or celery_k8s_job_executor. Use the key image inside the container_config block of the k8s solid tag.jobs argument. Each RunRequest emitted from a multi-job sensor’s evaluation function must specify a job_name.Nothing published for this version
dagster-slack has migrated off of deprecated `slackclient` (deprecated) and now uses [slack_sdk](https://slack.dev/python-slack-sdk/v3-migration/).
KubernetesRunLauncher image pull policy is now configurable in a separate field (thanks @yamrzou!).dagster-github package is now usable for GitHub Enterprise users (thanks @metinsenturk!) A hostname can now be provided via config to the dagster-github resource with the key github_hostname:execute_pipeline(
github_pipeline, {'resources': {'github': {'config': {
"github_app_id": os.getenv('GITHUB_APP_ID'),
"github_app_private_rsa_key": os.getenv('GITHUB_PRIVATE_KEY'),
"github_installation_id": os.getenv('GITHUB_INSTALLATION_ID'),
"github_hostname": os.getenv('GITHUB_HOSTNAME'),
}}}}
)
pipeline_failure_sensor and run_status_sensor queries. To take advantage of these performance gains, run a schema migration with the CLI command: dagster instance migrate.DockerRunLauncher would raise an exception when no networks were specified in its configuration.dagster-slack has migrated off of deprecated slackclient (deprecated) and now uses [slack_sdk](https://slack.dev/python-slack-sdk/v3-migration/).OpDefinition, the replacement for SolidDefinition which is the type produced by the @op decorator, is now part of the public API.daily_partitioned_config, hourly_partitioned_config, weekly_partitioned_config, and monthly_partitioned_config now accept an end_offset parameter, which allows extending the set of partitions so that the last partition ends after the current time.Nothing published for this version
Nothing published for this version
A service account can now be specified via Kubernetes tag configuration (thanks @skirino) !
Previously in Dagit, when a repository location had an error when reloaded, the user could end up on an empty page with no context about the error. Now, we immediately show a dialog with the error and stack trace, with a button to try reloading the location again when the error is fixed.
Dagster is now compatible with Python’s logging module. In your config YAML file, you can configure log handlers and formatters that apply to the entire Dagster instance. Configuration instructions and examples detailed in the docs: https://docs.dagster.io/concepts/logging/python-logging
[helm] The timeout of database statements sent to the Dagster instance can now be configured using .dagit.dbStatementTimeout.
The QueuedRunCoordinator now supports setting separate limits for each unique value with a certain key. In the below example, 5 runs with the tag (backfill: first) could run concurrently with 5 other runs with the tag (backfill: second).
run_coordinator:
module: dagster.core.run_coordinator
class: QueuedRunCoordinator
config:
tag_concurrency_limits:
- key: backfill
value:
applyLimitPerUniqueValue: True
limit: 5
.migrate.enabled=True.Nothing published for this version
Nothing published for this version
Nothing published for this version
Dagit’s dependency on graphql-ws is now pinned to < 0.4.0 to avoid a breaking change in its latest release. We expect to remove this dependency entire…
Added instance on RunStatusSensorContext for accessing the Dagster Instance from within the
run status sensors.
The inputs of a Dagstermill solid now are loaded the same way all other inputs are loaded in the framework. This allows rerunning output notebooks with properly loaded inputs outside Dagster context. Previously, the IO handling depended on temporary marshal directory.
Previously, the Dagit CLI could not target a bare graph in a file, like so:
from dagster import op, graph
@op
def my_op():
pass
@graph
def my_graph():
my_op()
This has been remedied. Now, a file foo.py containing just a graph can be targeted by the dagit
CLI: dagit -f foo.py.
When a solid, pipeline, schedule, etc. description or event metadata entry contains a markdown-formatted table, that table is now rendered in Dagit with better spacing between elements.
The hacker-news example now includes instructions on how to deploy the repository in a Kubernetes cluster using the Dagster Helm chart.
[dagster-dbt] The dbt_cli_resource now supports the dbt source snapshot-freshness command
(thanks @emilyhawkins-drizly!)
[helm] Labels are now configurable on user code deployments.
EventMetadata.asset and EventMetadata.pipeline_run in
AssetMaterialization metadata. (Thanks @ymrzkrrs and @drewsonne!)fs_io_manager, which allows
data to be passed out of the Jupyter process boundary.objects.inv is available at http://docs.dagster.io/objects.inv for other projects to link.execute_solid has been removed from the testing (https://docs.dagster.io/concepts/testing)
section. Direct invocation is recommended for testing solids.gcsfs as a dependency.create_databricks_job_solid now includes an example of how to use it.Nothing published for this version
In Dagit, the repository locations list has been moved from the Instance Status page to the Workspace page. When repository location errors are presen
context.log.info() and other similar functions now fully respect the python logging API. Concretely, log statements of the form context.log.error(“something %s happened!”, “bad”) will now work as expected, and you are allowed to add things to the “extra” field to be consumed by downstream loggers: context.log.info("foo", extra={"some":"metadata"}).config_from_files, config_from_pkg_resources, and config_from_yaml_strings have been added for constructing run config from yaml files and strings.DockerRunLauncher can now be configured to launch runs that are connected to more than one network, by configuring the networks key.env_from and volume_mounts are now properly applied to the corresponding Kubernetes run worker and job pods.end_mlflow_run_on_pipeline_finished hook now no longer errors whenever invoked.context.log calls are now not allowed. context.log.info("msg", foo="hi") should be rewritten as context.log.info("msg", extra={"foo":"hi"}).AssetMaterialization. Previously, it would still yield an AssetMaterialization where the path is a temp file path that won't exist after the notebook execution.InputContext and OutputContext now each has an asset_key that returns the asset key that was provided to the corresponding InputDefinition or OutputDefinition.Nothing published for this version
[dagster-dbt] Added a new synchronous RPC dbt resource (dbt_rpc_sync_resource), which allows you to programmatically send dbt commands to an RPC serve
dbt_rpc_sync_resource), which allows you to programmatically send dbt commands to an RPC server, returning only when the command completes (as opposed to returning as soon as the command has been sent).k8s_job_executor now adds to the secrets specified in K8sRunLa``uncher , instead of overwriting them.local_file_manager no longer uses the current directory as the default base_dir , instead defaulting to LOCAL_ARTIFACT_STORAGE/storage/file_manager. If you wish, you can configure LOCAL_ARTIFACT_STORAGE in your dagster.yaml file.Following the recent change to add strict Content-Security-Policy directives to Dagit, the CSP began to block the iframe used to render ipynb notebook files. This has been fixed, and these iframes should now render correctly.
Fixed an error where large files would fail to upload when using the s3_pickle_io_manager for intermediate storage.
Fixed an issue where Kubernetes environment variables defined in pipeline tags were not being applied properly to Kubernetes jobs.
Fixed tick preview in the Recent live tick timeline view for Sensors.
Added more descriptive error messages for invalid sensor evaluation functions.
dagit will now write to a temp directory in the current working directory when launched with the env var DAGSTER_HOME not set. This should resolve issues where the event log was not keeping up to date when observing runs progress live in dagit with no DAGSTER_HOME
Fixed an issue where retrying from a failed run sometimes failed if the pipeline was changed after the failure.
Fixed an issue with default config on to_job that would result in an error when using an enum config schema within a job.
A-Za-z0-9_.EventMetadata.python_artifact.Nothing published for this version
Fixed tick display in the sensor/schedule timeline view in Dagit.
dagster sensor list and dagster schedule list CLI commands to include schedules and sensors that have never been turned on.execute_in_process where providing default executor config to a job would cause config errors.ops config entry in place of solids would cause a config error.adls2_io_managerModeDefinition now validates the keys of resource_defs at definition time.Failure exceptions no longer bypass the RetryPolicy if one is set.serviceAccount.name to the user deployment Helm subchart and schema, thanks @jrouly!EcsRunLauncher will now exponentially backoff certain requests for up to a minute while waiting for ECS to reach a consistent state.launch CLI, and other modes of external execution, whereas before, memoization was only available via execute_pipeline and the execute CLI.version argument on the decorator:from dagster import root_input_manager
@root_input_manager(version="foo")
def my_root_manager(_):
pass
versioned_fs_io_manager now defaults to using the storage directory of the instance as a base directory.GraphDefinition.to_job now accepts a tags dictionary with non-string values - which will be serialized to JSON. This makes job tags work similarly to pipeline tags and solid tags.Nothing published for this version
[helm] The compute log manager now defaults to a NoOpComputeLogManager. It did not make sense to default to the LocalComputeLogManager as pipeline run
NoOpComputeLogManager. It did not make sense to default to the LocalComputeLogManager as pipeline runs are executed in ephemeral jobs, so logs could not be retrieved once these jobs were cleaned up. To have compute logs in a Kubernetes environment, users should configure a compute log manager that uses a cloud provider.dbt_pipeline to the hacker news example repo, which demonstrates how to run a dbt project within a Dagster pipeline.k8s_job_executor to match the configuration set in the K8sRunLauncher.DAGSTER_GRPC_MAX_RX_BYTES environment variable.dagster instance migrate when the asset catalog contains wiped assets.--models, --select, or --exclude flags while configuring the dbt_cli_resource, it will no longer attempt to supply these flags to commands that don’t accept them.yield_result wrote output value to the same file path if output names are the same for different solids.ops can now be used as a config entry in place of solids.EcsRunLauncher more resilient to ECS’ eventual consistency model.Nothing published for this version
Nothing published for this version
The Dagit web app now has a strict Content Security Policy.
PipelineRunStatus.solid on build_hook_context. This allows you to access the hook_context.solid parameter.dagster’s dependency on docstring-parser has been loosened.@pipeline now pulls its description from the doc string on the decorated function if it is provided.dagster new-project now no longer targets a non-existent mode.@repository functions.GraphDefinition.to_job now supports the description argument.AmazonECS_FullAccess policy. Now, the attached roles has been more narrowly scoped to only allow the daemon and dagit tasks to interact with the ECS actions required by the EcsRunLauncher.Error: Got unexpected extra arguments. Now, it ignores the entrypoint and launches succeed.Nothing published for this version
Nothing published for this version
Improved Asset catalog load times in Dagit, for Dagster instances that have fully migrated using dagster instance migrate.
dagster instance migrate.ScheduleDefinition constructor to instantiate a schedule definition, if a schedule name is not provided, the name of the schedule will now default to the pipeline name, plus “_schedule”, instead of raising an error.description and solid_retry_policy were getting dropped when using a solid_hook decorator on a pipeline definition (#4355).Nothing published for this version
Fixes implementation issues in @pipeline_failure_sensor that prevented them from working.
@pipeline_failure_sensor that prevented them from working.Nothing published for this version
Your coding agent can read these notes before it upgrades. Set up the MCP server →