NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
PyPI
A Dagster integration for celery-k8s-executor
Last release 9 days ago
21 Sep 2026
Ships on a steady schedule
a new release about every 2 weeks
Nearly every release is documented
notes for 59 of the last 60 stable releases
4 versions withdrawn
withdrawn after publishing
6 years old
581 releases · first in 2020
One column per quarter.
…factory function. Deprecated pipeline_name argument in favor of job_name in all the APIs.
k8s_job_executor is no longer experimental, and is recommended for production workloads. This executor runs each op in a separate Kubernetes job. We recommend this executor for Dagster jobs that require greater isolation than the multiprocess executor can provide within a single Kubernetes pod. The celery_k8s_job_executor will still be supported, but is recommended only for use cases where Celery is required (The most common example is to offer step concurrency limits using multiple Celery queues). Otherwise, the k8s_job_executor is the best way to get Kubernetes job isolation.make_dagster_job_from_airflow_dag factory function. Deprecated pipeline_name argument in favor of job_name in all the APIs.chardet library that was required due to an incompatibility with an old version of the aiohttp library, which has since been fixed.ins argument of the op decorator.slack_on_run_failure_sensor now says “Job” instead of “Pipeline” in its default message.DagsterTypeCheckDidNotPass error when a Dagster Type contained a List inside a Tuple (thanks @jan-eat!)dagstermill, dagster-pandas, dagster-airflow, etc)execute_in_process, when job does not have a top-level output.Nothing published for this version
[dagster-k8s] Fixed a bug that caused retries to occur twice with the k8s_job_executor
k8s_job_executorNothing published for this version
Updated dagstermill to better support job/op/graph changes by adding a define_dagstermill_op factory function. Also updated documentation and examples
define_dagstermill_op factory function. Also updated documentation and examples to reflect these changes.from dagster import get_dagster_logger)dbt_cli_resource, dbt_rpc_resource, and dbt_rpc_sync_resource) now support the dbt ls command: context.resources.dbt.ls().ins and outs properties to OpDefinition.resources_config argument on build_solid_context. The config argument has been renamed to solid_config.run_worker and step_worker, respectively.execute_in_processExecutor.execute(...) has changed from pipeline_context to plan_contextdagster, rather than repeating the job name. Thanks @skirino![dagster-docker] Added a new docker_executor which executes steps in separate Docker containers.
The dagster-daemon process can now detect hanging runs and restart crashed run workers. Currently
only supported for jobs using the docker_executor and k8s_job_executor. Enable this feature in your dagster.yaml with:
run_monitoring:
enabled: true
Documentation coming soon. Reach out in the #dagster-support Slack channel if you are interested in using this feature.
dagster-aws, dagster-github, and dagster-slack to reference job/op/graph APIs.Nothing published for this version
Nothing published for this version
All dbt resources (dbt_cli_resource, dbt_rpc_resource, and dbt_rpc_sync_resource) now support the ls command.
ls command.Nothing published for this version
Intermediate storages, which were deprecated in 0.10.0, have now been removed. Refer to the “Deprecation: Intermediate Storage” section of the 0.10.0…
execution:
celery-k8s:
Nothing published for this version
You can now configure credentials for the GCSComputeLogManager using a string or environment variable instead of passing a path to a credentials file.
GCSComputeLogManager using a string or environment variable instead of passing a path to a credentials file. Thanks @silentsokolov!dagster instance migrate would run out of memory when migrating over long run histories.Nothing published for this version
Updated click version, thanks @ashwin153!
dagster_aws.s3.sensor.get_s3_keys that would return no keys if an invalid s3 key was providedmy_log.info("foo %s", "bar") would cause errors in some scenarios.build_assets_job. The asset graph shows each node in the job’s graph with metadata about the asset it corresponds to - including asset materializations. It also contains links to upstream jobs that produce assets consumed by the job, as well as downstream jobs that consume assets produced by the job.load_assets_from_dbt_project and load_assets_from_dbt_project that would cause runs to fail if no runtime_metadata_fn argument were supplied.@asset not to infer the type of inputs and outputs from type annotations of the decorated function.@asset now accepts a compute_kind argument. You can supply values like “spark”, “pandas”, or “dbt”, and see them represented as a badge on the asset in the Dagit asset graph.Nothing published for this version
Nothing published for this version
Nothing published for this version
Note: This is a breaking change for anyone using the experimental VersionStrategy API. Instead of directly being passed solid_def and resource_def, yo…
Changed VersionStrategy.get_solid_version and VersionStrategy.get_resource_version to take in a SolidVersionContext and ResourceVersionContext, respectively. This gives VersionStrategy access to the config (in addition to the definition object) when determining the code version for memoization. (Thanks @RBrossard!).
Note: This is a breaking change for anyone using the experimental VersionStrategy API. Instead of directly being passed solid_def and resource_def, you should access them off of the context object using context.solid_def and context.resource_def respectively.
Nothing published for this version
[dagster-msteams] Introduced a new integration with Microsoft Teams, which includes a connection resource and support for sending messages to Microsof
emr_pyspark_step_launcher to fail when stderr included non-Log4J-formatted lines.applyPerUniqueValue config on the QueuedRunCoordinator to fail Helm schema validation.@asset decorator and build_assets_job APIs to construct asset-based jobs, along with Dagit support.load_assets_from_dbt_project and load_assets_from_dbt_manifest, which enable constructing asset-based jobs from DBT models.Nothing published for this version
We've deprecated output_notebook argument in define_dagstermill_solid in favor of output_notebook_name.
[dagstermill] You can now have more precise IO control over the output notebooks by specifying output_notebook_name in define_dagstermill_solid and providing your own IO manager via "output_notebook_io_manager" resource key.
output_notebook argument in define_dagstermill_solid in favor of output_notebook_name.local_output_notebook_io_manager is provided for handling local output notebook materialization.Dagit fonts have been updated.
context.log.info("foo %s", "bar") would not get formatted as expected.QueuedRunCoordinator’s tag_concurrency_limits to not be respected in some casestags argument of the @graph decorator or GraphDefinition constructor. These tags will be set on any runs of jobs are built from invoking to_job on the graph.k8s_job_executor or celery_k8s_job_executor. Use the key image inside the container_config block of the k8s solid tag.jobs argument. Each RunRequest emitted from a multi-job sensor’s evaluation function must specify a job_name.Nothing published for this version
dagster-slack has migrated off of deprecated `slackclient` (deprecated) and now uses [slack_sdk](https://slack.dev/python-slack-sdk/v3-migration/).
KubernetesRunLauncher image pull policy is now configurable in a separate field (thanks @yamrzou!).dagster-github package is now usable for GitHub Enterprise users (thanks @metinsenturk!) A hostname can now be provided via config to the dagster-github resource with the key github_hostname:execute_pipeline(
github_pipeline, {'resources': {'github': {'config': {
"github_app_id": os.getenv('GITHUB_APP_ID'),
"github_app_private_rsa_key": os.getenv('GITHUB_PRIVATE_KEY'),
"github_installation_id": os.getenv('GITHUB_INSTALLATION_ID'),
"github_hostname": os.getenv('GITHUB_HOSTNAME'),
}}}}
)
pipeline_failure_sensor and run_status_sensor queries. To take advantage of these performance gains, run a schema migration with the CLI command: dagster instance migrate.DockerRunLauncher would raise an exception when no networks were specified in its configuration.dagster-slack has migrated off of deprecated slackclient (deprecated) and now uses [slack_sdk](https://slack.dev/python-slack-sdk/v3-migration/).OpDefinition, the replacement for SolidDefinition which is the type produced by the @op decorator, is now part of the public API.daily_partitioned_config, hourly_partitioned_config, weekly_partitioned_config, and monthly_partitioned_config now accept an end_offset parameter, which allows extending the set of partitions so that the last partition ends after the current time.Nothing published for this version
Nothing published for this version
A service account can now be specified via Kubernetes tag configuration (thanks @skirino) !
Previously in Dagit, when a repository location had an error when reloaded, the user could end up on an empty page with no context about the error. Now, we immediately show a dialog with the error and stack trace, with a button to try reloading the location again when the error is fixed.
Dagster is now compatible with Python’s logging module. In your config YAML file, you can configure log handlers and formatters that apply to the entire Dagster instance. Configuration instructions and examples detailed in the docs: https://docs.dagster.io/concepts/logging/python-logging
[helm] The timeout of database statements sent to the Dagster instance can now be configured using .dagit.dbStatementTimeout.
The QueuedRunCoordinator now supports setting separate limits for each unique value with a certain key. In the below example, 5 runs with the tag (backfill: first) could run concurrently with 5 other runs with the tag (backfill: second).
run_coordinator:
module: dagster.core.run_coordinator
class: QueuedRunCoordinator
config:
tag_concurrency_limits:
- key: backfill
value:
applyLimitPerUniqueValue: True
limit: 5
.migrate.enabled=True.Nothing published for this version
Nothing published for this version
Nothing published for this version
Dagit’s dependency on graphql-ws is now pinned to < 0.4.0 to avoid a breaking change in its latest release. We expect to remove this dependency entire…
Added instance on RunStatusSensorContext for accessing the Dagster Instance from within the
run status sensors.
The inputs of a Dagstermill solid now are loaded the same way all other inputs are loaded in the framework. This allows rerunning output notebooks with properly loaded inputs outside Dagster context. Previously, the IO handling depended on temporary marshal directory.
Previously, the Dagit CLI could not target a bare graph in a file, like so:
from dagster import op, graph
@op
def my_op():
pass
@graph
def my_graph():
my_op()
This has been remedied. Now, a file foo.py containing just a graph can be targeted by the dagit
CLI: dagit -f foo.py.
When a solid, pipeline, schedule, etc. description or event metadata entry contains a markdown-formatted table, that table is now rendered in Dagit with better spacing between elements.
The hacker-news example now includes instructions on how to deploy the repository in a Kubernetes cluster using the Dagster Helm chart.
[dagster-dbt] The dbt_cli_resource now supports the dbt source snapshot-freshness command
(thanks @emilyhawkins-drizly!)
[helm] Labels are now configurable on user code deployments.
EventMetadata.asset and EventMetadata.pipeline_run in
AssetMaterialization metadata. (Thanks @ymrzkrrs and @drewsonne!)fs_io_manager, which allows
data to be passed out of the Jupyter process boundary.objects.inv is available at http://docs.dagster.io/objects.inv for other projects to link.execute_solid has been removed from the testing (https://docs.dagster.io/concepts/testing)
section. Direct invocation is recommended for testing solids.gcsfs as a dependency.create_databricks_job_solid now includes an example of how to use it.Nothing published for this version
In Dagit, the repository locations list has been moved from the Instance Status page to the Workspace page. When repository location errors are presen
context.log.info() and other similar functions now fully respect the python logging API. Concretely, log statements of the form context.log.error(“something %s happened!”, “bad”) will now work as expected, and you are allowed to add things to the “extra” field to be consumed by downstream loggers: context.log.info("foo", extra={"some":"metadata"}).config_from_files, config_from_pkg_resources, and config_from_yaml_strings have been added for constructing run config from yaml files and strings.DockerRunLauncher can now be configured to launch runs that are connected to more than one network, by configuring the networks key.env_from and volume_mounts are now properly applied to the corresponding Kubernetes run worker and job pods.end_mlflow_run_on_pipeline_finished hook now no longer errors whenever invoked.context.log calls are now not allowed. context.log.info("msg", foo="hi") should be rewritten as context.log.info("msg", extra={"foo":"hi"}).AssetMaterialization. Previously, it would still yield an AssetMaterialization where the path is a temp file path that won't exist after the notebook execution.InputContext and OutputContext now each has an asset_key that returns the asset key that was provided to the corresponding InputDefinition or OutputDefinition.Nothing published for this version
[dagster-dbt] Added a new synchronous RPC dbt resource (dbt_rpc_sync_resource), which allows you to programmatically send dbt commands to an RPC serve
dbt_rpc_sync_resource), which allows you to programmatically send dbt commands to an RPC server, returning only when the command completes (as opposed to returning as soon as the command has been sent).k8s_job_executor now adds to the secrets specified in K8sRunLa``uncher , instead of overwriting them.local_file_manager no longer uses the current directory as the default base_dir , instead defaulting to LOCAL_ARTIFACT_STORAGE/storage/file_manager. If you wish, you can configure LOCAL_ARTIFACT_STORAGE in your dagster.yaml file.Following the recent change to add strict Content-Security-Policy directives to Dagit, the CSP began to block the iframe used to render ipynb notebook files. This has been fixed, and these iframes should now render correctly.
Fixed an error where large files would fail to upload when using the s3_pickle_io_manager for intermediate storage.
Fixed an issue where Kubernetes environment variables defined in pipeline tags were not being applied properly to Kubernetes jobs.
Fixed tick preview in the Recent live tick timeline view for Sensors.
Added more descriptive error messages for invalid sensor evaluation functions.
dagit will now write to a temp directory in the current working directory when launched with the env var DAGSTER_HOME not set. This should resolve issues where the event log was not keeping up to date when observing runs progress live in dagit with no DAGSTER_HOME
Fixed an issue where retrying from a failed run sometimes failed if the pipeline was changed after the failure.
Fixed an issue with default config on to_job that would result in an error when using an enum config schema within a job.
A-Za-z0-9_.EventMetadata.python_artifact.Nothing published for this version
Fixed tick display in the sensor/schedule timeline view in Dagit.
dagster sensor list and dagster schedule list CLI commands to include schedules and sensors that have never been turned on.execute_in_process where providing default executor config to a job would cause config errors.ops config entry in place of solids would cause a config error.adls2_io_managerModeDefinition now validates the keys of resource_defs at definition time.Failure exceptions no longer bypass the RetryPolicy if one is set.serviceAccount.name to the user deployment Helm subchart and schema, thanks @jrouly!EcsRunLauncher will now exponentially backoff certain requests for up to a minute while waiting for ECS to reach a consistent state.launch CLI, and other modes of external execution, whereas before, memoization was only available via execute_pipeline and the execute CLI.version argument on the decorator:from dagster import root_input_manager
@root_input_manager(version="foo")
def my_root_manager(_):
pass
versioned_fs_io_manager now defaults to using the storage directory of the instance as a base directory.GraphDefinition.to_job now accepts a tags dictionary with non-string values - which will be serialized to JSON. This makes job tags work similarly to pipeline tags and solid tags.Nothing published for this version
[helm] The compute log manager now defaults to a NoOpComputeLogManager. It did not make sense to default to the LocalComputeLogManager as pipeline run
NoOpComputeLogManager. It did not make sense to default to the LocalComputeLogManager as pipeline runs are executed in ephemeral jobs, so logs could not be retrieved once these jobs were cleaned up. To have compute logs in a Kubernetes environment, users should configure a compute log manager that uses a cloud provider.dbt_pipeline to the hacker news example repo, which demonstrates how to run a dbt project within a Dagster pipeline.k8s_job_executor to match the configuration set in the K8sRunLauncher.DAGSTER_GRPC_MAX_RX_BYTES environment variable.dagster instance migrate when the asset catalog contains wiped assets.--models, --select, or --exclude flags while configuring the dbt_cli_resource, it will no longer attempt to supply these flags to commands that don’t accept them.yield_result wrote output value to the same file path if output names are the same for different solids.ops can now be used as a config entry in place of solids.EcsRunLauncher more resilient to ECS’ eventual consistency model.Nothing published for this version
Nothing published for this version
The Dagit web app now has a strict Content Security Policy.
PipelineRunStatus.solid on build_hook_context. This allows you to access the hook_context.solid parameter.dagster’s dependency on docstring-parser has been loosened.@pipeline now pulls its description from the doc string on the decorated function if it is provided.dagster new-project now no longer targets a non-existent mode.@repository functions.GraphDefinition.to_job now supports the description argument.AmazonECS_FullAccess policy. Now, the attached roles has been more narrowly scoped to only allow the daemon and dagit tasks to interact with the ECS actions required by the EcsRunLauncher.Error: Got unexpected extra arguments. Now, it ignores the entrypoint and launches succeed.Nothing published for this version
Improved Asset catalog load times in Dagit, for Dagster instances that have fully migrated using dagster instance migrate.
dagster instance migrate.ScheduleDefinition constructor to instantiate a schedule definition, if a schedule name is not provided, the name of the schedule will now default to the pipeline name, plus “_schedule”, instead of raising an error.description and solid_retry_policy were getting dropped when using a solid_hook decorator on a pipeline definition (#4355).Nothing published for this version
Fixes implementation issues in @pipeline_failure_sensor that prevented them from working.
@pipeline_failure_sensor that prevented them from working.Nothing published for this version
The deprecated SystemCronScheduler and K8sScheduler schedulers have been removed. All schedules are now executed using the dagster-daemon proess. See…
With the new first-class Pipeline Failure sensors, you can now write sensors to perform arbitrary actions when pipelines in your repo fail using @pipeline_failure_sensor. Out-of-the-box sensors are provided to send emails using make_email_on_pipeline_failure_sensor and slack messages using make_slack_on_pipeline_failure_sensor.
See the Pipeline Failure Sensor docs to learn more.
New first-class Asset sensors help you define sensors that launch pipeline runs or notify appropriate stakeholders when specific asset keys are materialized. This pattern also enables Dagster to infer cross-pipeline dependency links. Check out the docs here!
Solid-level retries: A new retry_policy argument to the @solid decorator allows you to easily and flexibly control how specific solids in your pipelines will be retried if they fail by setting a RetryPolicy.
Writing tests in Dagster is now even easier, using the new suite of direct invocation apis. Solids, resources, hooks, loggers, sensors, and schedules can all be invoked directly to test their behavior. For example, if you have some solid my_solid that you'd like to test on an input, you can now write assert my_solid(1, "foo") == "bar" (rather than explicitly calling execute_solid()).
[Experimental] A new set of experimental core APIs. Among many benefits, these changes unify concepts such as Presets and Partition sets, make it easier to reuse common resources within an environment, make it possible to construct test-specific resources outside of your pipeline definition, and more. These changes are significant and impactful, so we encourage you to try them out and let us know how they feel! You can learn more about the specifics here
[Experimental] There’s a new reference deployment for running Dagster on AWS ECS and a new EcsRunLauncher that launches each pipeline run in its own ECS Task.
[Experimental] There’s a new k8s_job_executor (https://docs.dagster.io/_apidocs/libraries/dagster-k8s#dagster_k8s.k8s_job_executor)which executes each solid of your pipeline in a separate Kubernetes job. This addition means that you can now choose at runtime (https://docs.dagster.io/deployment/guides/kubernetes/deploying-with-helm#executor) between single pod and multi-pod isolation for solids in your run. Previously this was only configurable for the entire deployment- you could either use the K8sRunLauncher with the default executors (in process and multiprocess) for low isolation, or you could use the CeleryK8sRunLauncher with the celery_k8s_job_executor for pod-level isolation. Now, your instance can be configured with the K8sRunLauncher and you can choose between the default executors or the k8s_job_executor.
Using the @schedule, @resource, or @sensor decorator no longer requires a context parameter. If you are not using the context parameter in these, you can now do this:
@schedule(cron_schedule="* * * * *", pipeline_name="my_pipeline")
def my_schedule():
return {}
@resource
def my_resource():
return "foo"
@sensor(pipeline_name="my_pipeline")
def my_sensor():
return RunRequest(run_config={})
Dynamic mapping and collect features are no longer marked “experimental”. DynamicOutputDefinition and DynamicOutput can now be imported directly from dagster.
Added repository_name property on SensorEvaluationContext, which is name of the repository that the sensor belongs to.
get_mapping_key is now available on SolidExecutionContext , allowing for discerning which downstream branch of a DynamicOutput you are in.
When viewing a run in Dagit, you can now download its debug file directly from the run view. This can be loaded into dagit-debug.
[dagster-dbt] A new dbt_cli_resource simplifies the process of working with dbt projects in your pipelines, and allows for a wide range of potential uses. Check out the integration guide for examples!
k8s_job_executor that caused solid tag user defined Kubernetes config to not be applied to the Kubernetes jobs.The deprecated SystemCronScheduler and K8sScheduler schedulers have been removed. All schedules are now executed using the dagster-daemon proess. See the deployment docs for more information about how to use the dagster-daemon process to run your schedules.
If you have written a custom run launcher, the arguments to the launch_run function have changed in order to enable faster run launches. launch_run now takes in a LaunchRunContext object. Additionally, run launchers should now obtain the PipelinePythonOrigin to pass as an argument to dagster api execute_run. See the implementation of DockerRunLauncher for an example of the new way to write run launchers.
[helm] .Values.dagsterDaemon.queuedRunCoordinator has had its schema altered. It is now referenced at .Values.dagsterDaemon.runCoordinator.
Previously, if you set up your run coordinator configuration in the following manner:
dagsterDaemon:
queuedRunCoordinator:
enabled: true
module: dagster.core.run_coordinator
class: QueuedRunCoordinator
config:
max_concurrent_runs: 25
tag_concurrency_limits: []
dequeue_interval_seconds: 30
It is now configured like:
dagsterDaemon:
runCoordinator:
enabled: true
type: QueuedRunCoordinator
config:
queuedRunCoordinator:
maxConcurrentRuns: 25
tagConcurrencyLimits: []
dequeueIntervalSeconds: 30
The method events_for_asset_key on DagsterInstance has been deprecated and will now issue a warning. This method was previously used in our asset sensor example code. This can be replaced by calls using the new DagsterInstance API get_event_records. The example code in our sensor documentation has been updated to use our new APIs as well.
Nothing published for this version
In Dagit, a new page has been added for user settings, including feature flags and timezone preferences. It can be accessed via the gear icon in the t
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Your coding agent can read these notes before it upgrades. Set up the MCP server →