NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
PyPI · #1318 most downloaded on PyPI
Dagster is an orchestration platform for the development, production, and observation of data assets.
Last release 3 days ago
01 Oct 2026
Ships on a steady schedule
a new release about every 2 weeks
Most releases are documented
notes for 53 of the last 60 stable releases
6 versions withdrawn
withdrawn after publishing
8 years old
745 releases · first in 2018
Fixed an error when trying to directly invoke a run status sensor when passing resources.
EnvVars used in Airbyte or Fivetran resources would show up as their processed values in the launchpad when loading assets from a live Fivetran or Airbyte instance.The OpExecutionContext and AssetExecutionContext now have a partition_keys property
OpExecutionContext and AssetExecutionContext now have a partition_keys propertydbt-core==1.7.* is now supported.build_schedule_from_partitioned_job was used with a job with multi-partitioned assets and the partitions_def argument wasn’t provided to define_asset_job.config.dagster-embedded-elt where sling’s updated_at parameter was set to the incorrect typeAssetCheckKeyPipesSubprocessClient now inherits the environment variables of the parent process in the launched subprocess.One column per quarter.
[dagster-k8s] The PipesK8sClient will now attempt to load the appropriate kubernetes config, and exposes arguments for controlling this process.
PipesK8sClient will now attempt to load the appropriate kubernetes config, and exposes arguments for controlling this process.load_asset_checks_from_modules functions for loading asset checks in tandem with load_assets_from_modules.dagster-dbt project scaffold now uses ~/.dbt/profiles.yml if a profiles.yml is not present in the dbt project directory.@dbt_assets now support PartitionMapping using DagsterDbtTranslator.get_partition_mapping.@dbt_assets. To enable this, add the following metadata to your dbt model’s metadata in your dbt project:meta:
dagster:
has_self_dependency: True
pydantic<2.0.0 but having pydantic-core installed.LastPartitionMapping dependency would raise an error. This has been fixed.instance is not available to load partitions error was raised.build_asset_with_blocking_check1.5.0 where instances that haven’t been migrated to the latest schema hit an error upon run deletion.exclude that didn’t match any dbt nodes was used in @dbt_assets, an error would be raised. The error is now properly handled.DbtCliResource.cli(...) in an op, AssetMaterialization's instead of Output are now emitted.report_asset_observation REST endpoint for reporting runless events is now available.[helm] The pipelineRun configuration in the Helm chart is now deprecated. The same config can be set under dagster-user-deployments
report_asset_observation REST API endpoint for runless external asset observation eventsDAGSTER_K8S_PG_PASSWORD_SECRET and DAGSTER_K8S_INSTANCE_CONFIG_MAP will no longer be set in all pods.build_pyspark_zip now takes an exclude parameter that can be used to customize the set of excluded files.source_key_prefix argument to load_assets_from_current_module and load_assets_from_package_name was ignoreddagster_embedded_elt where the mode parameter was not being passed to Sling, and only one asset could be created at a timepipelineRun configuration in the Helm chart is now deprecated. The same config can be set under dagster-user-deploymentssetup_for_execution and teardown_after_execution calls to the inner IOManagers of the BranchingIOManager - thank you @danielgafni!S3FakeResource.upload_fileobj() signature is now consistent with boto3 S3.Client.upload_fileobj() - thank you @jeanineharb!dbt_assets now have an optional name parameter - thank you @AlexanderVR!Added a report_asset_check REST API endpoint for runless external asset check evaluation events. This is available in cloud as well.
report_asset_check REST API endpoint for runless external asset check evaluation events. This is available in cloud as well.config argument is now supported on @graph_multi_assetDbtCliResource.AssetExecutionContext type annotation for the context parameter in @asset_check functions.profiles_dir=None into DbtCliResource would cause incorrect validation.DuckDBResource and DuckDBIOManager accept a connection_config configuration that will be passed as config to the DuckDB connection. Thanks @xjhc!Alert policies can now be set on assets + asset checks (currently experimental). Check out the alerting docs for more information.
--live-data-poll-rate that allows configuring how often the UI polls for new asset data when viewing the asset graph, asset catalog, or overview assets page. It defaults to 2000 ms.report_asset_materialization REST API endpoint for creating external asset materialization events. This is available in cloud as well.@dbt_assets decorator now accepts a backfill_policy argument, for controlling how the assets are backfilled.@dbt_assets decorator now accepts a op_tags argument, for passing tags to the op underlying the produced AssetsDefinition.get_materialize_result & get_asset_check_result to PipesClientCompletedInvocationacryl-datahub pin in the dagster-datahub package has been removed.PipesDatabricksClient now performs stdout/stderr forwarding from the Databricks master node to Dagster.dagster-dbt-cloud CLI.K8sRunLauncher. See the docs for more information.Previously, the asset backfill page would display negative counts if failed partitions were manually re-executed. This has been fixed.
Fixed an issue where the run list dialog for viewing the runs occupying global op concurrency slots did not expand to fit the content size.
Fixed an issue where selecting a partition would clear the launchpad and typing in the launchpad would clear the partition selection
Fixed various issues with the asset-graph displaying the wrong graph
The IO manager’s handle_output method is no longer invoked when observing an observable source asset.
[ui] Fixed an issue where the run config dialog could not be scrolled.
[pipes] Fixed an issue in the PipesDockerClient with parsing logs fetched via the docker client.
[external assets] Fixed an issue in external_assets_from_specs where providing multiple specs would error
[external assets] Correct copy in tooltip to explain why Materialize button is disabled on an external asset.
dagster-pipes version used in the external process.is_dagster_pipes_process has been removed from the dagster-pipes package.Previously, asset backfills targeting assets with multi-run backfill policies would raise a "did not submit all run requests" error. This has been fix
[dagster-dbt] Enabling asset checks using dbt project metadata has been deprecated.
EnvVar utility will now raise an exception if it is used outside of the context of a Dagster resource or config class. The get_value() utility will retrieve the value outside of this context.DbtCliResource, an explicit target_path can now be specified.DagsterDbtTranslator and DagsterDbtTranslatorSettings: see the docs for more information.Improved ergonomics for execution dependencies in assets - We introduced a set of APIs to simplify working with Dagster that don't use the I/O manager
Improved ergonomics for execution dependencies in assets - We introduced a set of APIs to simplify working with Dagster that don't use the I/O manager system for handling data between assets. I/O manager workflows will not be affected.
AssetDep type allows you to specify upstream dependencies with partition mappings when using the deps parameter of @asset and AssetSpec.MaterializeResult can be optionally returned from an asset to report metadata about the asset when the asset handles any storage requirements within the function body and does not use an I/O manager.AssetSpec has been added as a new way to declare the assets produced by @multi_asset. When using , the does not need to return any values to be stored by the I/O manager. Instead, the should handle any storage requirements in the body of the function.Asset checks (experimental) - You can now define, execute, and monitor data quality checks in Dagster [docs].
@asset_check decorator, as well as the check_specs argument to @asset and @multi_asset enable defining asset checks.Auto materialize customization (experimental) - AutoMaterializePolicies can now be customized [docs].
AutoMaterializeRules which determine if an asset should be materialized or skipped.PipesSubprocessClient, PipesDocketClient, PipesK8sClient, PipesDatabricksClientopen_pipes_session. One can augment existing invocations rather than replacing them wholesale.AssetExecutionContext is now a subclass of OpExecutionContext, not a type alias. The codedef my_helper_function(context: AssetExecutionContext):
...
@op
def my_op(context: OpExecutionContext):
my_helper_function(context)
will cause type checking errors. To migrate, update type hints to respect the new subclassing.
AssetExecutionContext cannot be used as the type annotation for @ops run in @jobs. To migrate, update the type hint in @op to OpExecutionContext. @ops that are used in @graph_assets may still use the AssetExecutionContext type hint.# old
@op
def my_op(context: AssetExecutionContext):
...
# correct
@op
def my_op(context: OpExecutionContext):
...
backfill_policy=BackfillPolicy.single_run() to your assets.has_dynamic_partition implementation has been optimized. Thanks @edvardlindelof!stream_to_asset_map argument to build_airbyte_assets to support the Airbyte prefix setting with special characters. Thanks @chollinger93!DatabricksPysparkStepLauncher fails to get logs when job_run doesn’t have cluster_id at root level. Thanks @PadenZach!dagster_cloud.dagster_insights module that contains utilities for capturing and submitting external metrics about data operations to Dagster Cloud via an api. Dagster Cloud Insights is a soon-to-be released feature that shows improves visibility into usage and cost metrics such as run duration and Snowflake credits in the Cloud UI.[dagster-dbt] DbtCliResource now enforces that the current installed version of dbt-core is at least version 1.4.0.
DbtCliResource now enforces that the current installed version of dbt-core is at least version 1.4.0.DbtCliResource now properly respects DBT_TARGET_PATH if it is set by the user. Artifacts from dbt invocations using DbtCliResource will now be placed in unique subdirectories of DBT_TARGET_PATH.partition_time_window attribute on OpExecutionContext and AssetExecutionContext now returns the time range, instead of raising an error.job_name property on the result object of build_hook_context.AssetSpec has been added as a new way to declare the assets produced by @multi_asset.AssetDep type allows you to specify upstream dependencies with partition mappings when using the deps parameter of @asset and AssetSpec.report_asset_check method added to ExtContext.yield from to forward reported materializations and asset check results to Dagster. Results reported from ext that are not yielded will raise an error.os.getenv() versus Dagster’s EnvVar.Deprecated ExpectationResult. This will be made irrelevant by upcoming data quality features.
ExpectationResult. This will be made irrelevant by upcoming data quality features.SensorResult.The deps parameter for @asset and @multi_asset now supports directly passing @multi_asset definitions. If an @multi_asset is passed to deps, dependenc
deps parameter for @asset and @multi_asset now supports directly passing @multi_asset definitions. If an @multi_asset is passed to deps, dependencies will be created on every asset produced by the @multi_asset.dagster instance migrate --bigint-migration.DbtCliResource now validates at definition time that its project_dir and profiles_dir arguments are directories that respectively contain a dbt_project.yml and profiles.yml.policy_id for new clusters when using the databricks_pyspark_step_launcher (thanks @zyd14!)dagster-webserver command was not indicating which port it was using in the command-line output.MaterializeResult has been added as a new return type to be used in @asset / @multi_asset materialization functionsAdded a new tooltip to asset runs to either view the asset list or lineage
dagster-ext module along with subprocess, docker, databricks, and k8s pod integrations are now available. Read more at https://github.com/dagster-io/dagster/discussions/16319. Note that the module is temporarily being published to PyPI under dagster-ext-process, but is available in python as import dagster_ext.@asset_check or AssetChecksDefinition.check_specs argument to @graph_multi_asset@graph_asset that would raise an error about nonexistant checksOpExecutionContext.add_output_metadata can now be called multiple times per output.
OpExecutionContext.add_output_metadata can now be called multiple times per output.@janosroden !)29 2 *) no longer cause exceptions in the UI or daemon.observable_source_assets with different partition definitions existed in the same code location, runs targeting those assets could fail to launch. This has been fixed.start_date of their underlying PartitionsDefinition could result in runs being launched for partitions that no longer existed. This has been fixed.@ldnicolasmay!)DagsterDbtTranslator did not properly invoke get_auto_materialize_policy and get_freshness_policy for load_assets_from_dbt_project.@graph_asset support. This can be used to implement blocking checks, by raising an exception if the check fails.@multi_asset subsetting, so only checks which target assets in the subset will execute.AssetCheckSpecs will now cause an error at definition time if they target an asset other than the one they’re defined on.run_monitoring > free_slots_after_run_end_secondsNothing published for this version
Nothing published for this version
The context object now has an asset_key property to get the AssetKey of the current asset.
context object now has an asset_key property to get the AssetKey of the current asset.dagster dev and dagster-daemon run commands now include a --log-level argument that allows you to customize the logger level threshold.AirbyteResource now includes a poll_interval key that allows you to configure how often it checks an Airbyte sync’s status.DatabricksStepLauncher to fail. Thanks @zyd14!workspace and volumes init scripts in the databricks client. Thanks @zyd14!Asset checks are now displayed in the asset graph and sidebar.
[Breaking] Asset check severity is now set at runtime on AssetCheckResult instead of in the @asset_check definition. Now you can define one check that either errors or warns depending on your check logic. ERROR severity no longer causes the run to fail. We plan to reintroduce this functionality with a different API.
[Breaking] @asset_check now requires the asset= argument, even if the asset is passed as an input to the decorated function. Example:
@asset_check(asset=my_asset)
def my_check(my_asset) -> AssetCheckResult:
...
[Breaking] AssetCheckSpec now takes asset= instead of asset_key=, and can accept either a key or an asset definition.
[Bugfix] Asset checks now work on assets with key_prefix set.
[Bugfix] Execution failure asset checks are now displayed correctly on the checks tab.
DbtCliResource in custom asset/op to API docs.Nothing published for this version
Dagster code servers now wait to shut down until any calls that they are running have finished, preventing them from stopping while in the middle of e
dagster execute job cli now accepts —-op-selection (thanks @silent-lad!)AssetsDefinition instances.@graph and @job now work again, fixing a regression introduced in 1.4.5.ins argument to graph_asset is now respected correctly.dagster dev failed on startup when the DAGSTER_GRPC_PORT `environment variable was set in the environment.deps arguments for an asset can now be specified as an iterable instead of a sequence, allowing for sets to be passed.securityContext setting now applies correctly to all init containers (thanks @maowerner!)AutoMaterializeRule.skip_on_not_all_parents_updated that enforces that an asset can only be materialized if all parents have been materialized since the asset's last materialization.AutoMaterializeRule.skip_on_parent_missing –which is already part of the behavior of the default auto-materialize policy.dagster-dbt project scaffold.[dagster-webserver] Fixed an issue that broke loading static files on Windows.
Fixed an issue that caused some missing icons in the UI.
The ConfigurablePickledObjecS3IOManager will continue to be available but is considered deprecated in favor of S3PickleIOManager. There is no change i…
@partitioned_config decorator has been added for defined configuration for partitioned jobs. Thanks @danielgafni!ConfigurablePickledObjectS3IOManager has been renamed S3PickleIOManager for simplicity. The ConfigurablePickledObjecS3IOManager will continue to be available but is considered deprecated in favor of S3PickleIOManager. There is no change in the functionality of the I/O manager.ConfigurablePickledObjectADLS2IOManager has been renamed ADLS2PickleIOManager for simplicity. The ConfigurablePickledObjectADLS2IOManager will continue to be available but is considered deprecated in favor of ADLS2PickleIOManager. There is no change in the functionality of the I/O manager.DbtCliResource, the exception message now includes a link to the dbt.log produced. This log file can be inspected for debugging.ConfigurablePickledObjectGCSIOManager has been renamed GCSPickleIOManager for simplicity. The ConfigurablePickledObjecGCSIOManager will continue to be available but is considered deprecated in favor of GCSPickleIOManager. There is no change in the functionality of the I/O manager.DagsterInvariantViolationError when executing a multi-asset where both assets have self-dependencies on earlier partitions.--debug flag raised an exception in the Dagster framework.stderr which aligns with Python’s logging defaults.BackfillPolicy, thanks @ruizh22!timeout argument when constructing a DagsterGraphQLClient.DbtCliResource and DbtCliResource.cli(...).InputContext and OutputContext have been fixed. Thanks @Sergey Mezentsev!Added a respect_materialization_data_versions option to auto materialization. It can enabled in dagster.yaml with
Added a respect_materialization_data_versions option to auto materialization. It can enabled in dagster.yaml with
auto_materialize:
respect_materialization_data_versions: True
This flag may be changed or removed in the near future.
ops or assets with multiple outputs that are all required and return type None/ Nothing will interpret an explicitly or implicitly returned value None
None/ Nothing will interpret an explicitly or implicitly returned value None to indicate that all outputs were successful.skip_reason argument to the constructor of SensorResult now accepts a string in addition to a SkipReason.step_k8s_config field to k8s_job_executor that allows you to customize the raw Kubernetes config for each step in a job. See the docs for more information.dagster-dbt project scaffold now creates the scaffold in multiple files:
constants.py contains a reference to your manifest and dbt project directoryassets.py contains your initial dbt assets definitionsdefinitions.py contains the code to load your asset definitions into the Dagster UIschedules.py contains an optional schedule to add for your dbt assetsget_auto_materialize_policy and get_freshness_policy to DagsterDbtTranslator.load_assets_from_fivetran_instance.SSHResource would warn when allow_host_key_change was set. Now known hosts are always loaded from the system hosts file, and the allow_host_key_change parameter is ignored.@graph_multi_asset now has an API docs entry.GCSComputeLogManager example in the Dagster Instance reference is now correct.@dbt_assets for the following use-cases:
--vars@graph_asset now takes a config parameter equivalent to the parameter on @graph.
@graph_asset now takes a config parameter equivalent to the parameter on @graph.dynamic_partitions_store argument to DynamicPartitionsDefinition for multi-partition run properly with dynamic partitions (Thanks @elzzz!).partitionsByAssets`` to backfillParams`` for ranged partition backfill (Thanks @ruizh22!).dbt-core==1.6 has been added.DbtCliResource now supports configuring profiles_dir.restart_policy on k8s_job_op (Thanks @Taadas!).authenticator to SnowflakePandasIOManager, which allows specifying the authentication mechanism to use (Thanks @pengw0048!).AutoMaterializePolicy with assets that had at least one source asset parent and at least one non-source asset parent. This has been fixed.AutoMaterializePolicy to a time-partitioned asset downstream of an unpartitioned asset, the latest partition would only ever be materialized a single time, rather than updating in response to any parent updates. This has been fixed.StaticPartitionsDefinition containing many thousands of partitions could take a significant amount of time.Unexpected exception error would be raised when scaffolding a pull request on a repository with no profiles.yml. This behavior has been updated to raise a more descriptive error message on the repo selection page.agentReplicas config setting on the helm chart has been renamed to isolatedAgents. In order to use this config setting, your user code dagster version needs to be 1.4.3 or greater.[ui] When viewing a run for auto-materialized assets, show a tag with information about the assets that were materialized.
dagster-dbt project scaffold didn’t create a project directory with all the scaffolded files.SpecificPartitionsPartitionMapping with auto-materialization.max_materializations_per_minute on an AutoMaterializePolicy to a non-positive number. This will now result in an error.upath_io_manager from @harrylojames; thank you!dagster project scaffold and the new dagster-dbt APIs.[dagster-dbt] When invokingdagster-dbt project scaffold on a dbt project directory, if a profiles.yml exists in the root of the directory, its content
dagster-dbt project scaffold on a dbt project directory, if a profiles.yml exists in the root of the directory, its contents are used to add dbt adapter packages to the scaffolded setup.py.max_concurrent field has been changed from 0 to None to more clearly signal its intent. A value of 0 is still interpreted as the sentinel value which dynamically allocates max_concurrent based on detected CPU count.Definitions, so that jobs are able to override the IO manager used.EnvVars in a FivetranResource would not be evaluated when loading assets from the Fivetran instance.EnvVars in an AirbyteResource would not be evaluated when loading assets from the Airbyte resource.DbtCliResource, DbtCliInvocation, @dbt_assets, DagsterDbtTranslator, dagster-dbt project scaffoldFixes a bug in dagster-dbt that was preventing it from correctly materializing subselections of dbt asset.
dagster-dbt that was preventing it from correctly materializing subselections of dbt asset.Fixes a bug in dagster-dbt that was preventing it efficiently loading dbt projects from a manifest.
dagster-dbt that was preventing it efficiently loading dbt projects from a manifest.non_argument_deps → deps – We’ve deprecated the non_argument_deps parameter of @asset and @multi_asset in favor of a new deps parameter. The new param…
AutoMaterializePolicy. It’s located under Assets → Select an asset with an AutoMaterializePolicy → Auto-materialize history tab.non_argument_deps parameter of @asset and @multi_asset in favor of a new deps parameter. The new parameter makes it clear that this is a first-class way of defining dependencies, makes code more concise, and accepts AssetsDefinition and SourceAsset objects, in addition to the strs and AssetKeys that the previous parameter accepted.DynamicPartitionsDefinition and SensorResult are no longer marked experimental.@observable_source_asset decorator now accepts an auto_observe_interval_minutes parameter. If the asset daemon is turned on, then the observation function will automatically be run at this interval. Downstream assets with eager auto-materialize policies will automatically run if the observation function indicates that the source asset has changed. [docs]dagit package is deprecated in favor of the dagster-webserver package.@dbt_assets decorator allows much more control over how Dagster runs your dbt project. [docs]dagster-dbt project scaffold command line interface makes it easy to create files and directories for a Dagster project that wraps an existing dbt project.get_asset_key_for_model and get_asset_key_for_source utilities make it easy to specify dependencies between upstream dbt assets and downstream non-dbt assets. And you can now more easily specify dependencies between dbt models and upstream non-dbt assets by specifying Dagster asset keys in the dbt metadata for dbt sources.non_argument_deps parameter of @asset and @multi_asset in favor of a new deps parameter. The new parameter makes it clear that this is a first-class way of defining dependencies, makes code more concise, and accepts AssetsDefinition and SourceAsset objects, in addition to the strs and AssetKeys that the previous parameter accepted.UPathIOManager can now be extended to load multiple partitions asynchronously (Thanks Daniel Gafni!).dagster-user-deployments.deployments.[...].readinessProbe.enabled=false.non_argument_deps in favor of deps, build_airbyte_assets now accepts a deps parameter.non_argument_deps in favor of deps, define_dagstermill_asset now accepts a deps parameter.StaticPartitionsDefinition will now raise an error.AutoMaterializePolicy's to not materialize missing assets.--path-prefix.build_asset_reconciliation_sensor (Experimental) has been removed. It was deprecated in 1.3 in favor of AutoMaterializePolicy.asset_key(s) properties on AssetIn and AssetDefinition have been removed in favor of key(s). These APIs were deprecated in 1.0.root_input_manager and RootInputManagerDefinition have been removed in favor of input_manager and InputManagerDefinition. These APIs were deprecated in 1.0.event_metadata_fn parameter on create_dagster_pandas_dataframe_type has been removed in favor of metadata_fn.@dbt_assets and DbtCliResource. See the migration guide for details.
dbt-rpc has been removed.DbtCloudResourceV2 has been removed.DbtCli has been renamed to DbtCliResource. Previously, DbtCliResource was a class alias for DbtCliClientResource.load_assets_from_dbt_project and load_assets_from_dbt_manifest now default to use_build=True.load_assets_from_dbt_project and load_assets_from_dbt_manifest has changed. Rather than assigning a group name using the model’s subdirectory, a group name will be assigned using the dbt model’s dbt group.node_info_to_definition_metadata_fn for load_assets_from_dbt_project and load_assets_from_dbt_manifest now overrides metadata instead of adding to it.load_assets_from_dbt_project and load_assets_from_dbt_manifest now must be specified using keyword arguments.DbtCliResource with load_assets_from_dbt_project and load_assets_from_dbt_manifest, stdout logs from the dbt process will now appear in the compute logs instead of the event logs.dagit python package is deprecated and will be removed in 2.0 in favor of dagster-webserver. See the migration guide for details.dagit → dagsterWebserveringress.dagit → ingress.dagsterWebserveringress.readOnlyDagit → ingress.readOnlyDagsterWebservertag:DescribeResources. Without this policy, the ECS Agent will log a deprecation warning and fall back to its old behavior (listing all ECS services in the cluster and then listing each service's tags).DbtCliClientResource, dbt_cli_resource and DbtCliOutput are now being deprecated in favor of DbtCliResource.load_assets_from_dbt_project and load_assets_from_dbt_manifest are now deprecated in favor of other options. See the migration for details.[dbt] DbtCliResource has been deprecated in favor of DbtCli.
DynamicPartitionsDefinition and SensorResult are no longer marked experimentalDagsterInstance now has a get_status_by_partition method, which returns the status of each partition for a given asset. Thanks renzhe-brian!DagsterInstance now has a get_latest_materialization_code_versions method, which returns the code version of the latest materialization for each of the provided (non-partitioned) assets.build_asset_context has been added as an asset focused replacement for build_op_context.build_op_context now accepts a partition_key_range parameter.AssetSelection.upstream_source_assets method allows selecting source assets upstream of the current selection.AssetSelection.key_prefixes and AssetSelection.groups now accept an optional include_sources parameter.DbtCli resource is no longer marked experimental.global_config parameter of the DbtCli resource has been renamed to global_config_flagsload_assets_from_dbt_project and load_assets_from_dbt_manifest now work with the DbtCli resource.manifest argument of the @dbt_assets decorator now additionally can accept a Path argument representing a path to the manifest file or dictionary argument representing the raw manifest blob.DbtCli.cli from inside a @dbt_assets-decorated function, you no longer need to supply the manifest argument as long as you provide the context argument.DbtManifest object can now generate schedules using dbt selection syntax.dbt_manifest.build_schedule(
job_name="materialize_dbt_models",
cron_schedule="0 0 * * *",
dbt_select="fqn:*"
)
DbtCli.cli and the underlying command fails, an exception will now be raised. To suppress the exception, run the DbtCli.cli(..., raise_on_error=False).OutputContext when using with_attributes or AssetsDefinition.from_graph.max_materializations_per_minute parameter, those older partitions would not be properly discarded from consideration on subsequent ticks. This has been fixed.from __future__ import annotations) would cause errors in most cases when used with Dagster definitions. This has been fixed for the vast majority of cases.AssetExecutionContext has returned to being a type alias for OpExecutionContext.time_window_partition_scope_minutes parameter of the AutoMaterializePolicy class has been removed. Instead, max_materializations_per_minute should be used to limit the number of runs that may be kicked off for a partitioned asset.DbtCliResource has been deprecated in favor of DbtCli.dagit has been deprecated in favor of a new package dagster-webserver.OpExecutionContext.asset_partition_key_range has been deprecated in favor of partition_key_range.databricks_pyspark_step_launcher will no longer error when executing steps that target a single partition of a DynamicPartitionsDefinition (thanks @weberdavid!).@observable_source_asset decorator now accepts an auto_observe_interval_minutes parameter. If the asset daemon is turned on, then the observation function will automatically be run at this interval.DbtCliTask has been renamed to DbtCliInvocationget_asset_key_by_output_name and get_node_info_by_output_name methods of DbtManifest have been renamed toget_asset_key_for_output_name and get_node_info_for_output_name, respectively.DagsterInstance, *MetadataValue, DagsterType, and others.dagster-pandera now has an API docs page.Nothing published for this version
Nothing published for this version
Nothing published for this version
Fixes a bug in dagster project from-example that was preventing it from downloading examples correctly.
dagster project from-example that was preventing it from downloading examples correctly.[dagster-dbt] The use of the state_path parameter with DbtManifestAssetSelection has been deprecated, and will be removed in the next minor release.
--name argument is now optional when running dagster project from-example.@asset(key=...).AssetKey now has a with_prefix method.AutoMaterializePolicys with large numbers of partitions.dagster instance migrate now prints information about changes to the instance database schema.dagster-cloud-agent helm chart now supports setting K8s labels on the agent deployment.[ui] Fixed an issue that prevented filtering by date on the job-specific runs tab.
[ui] “F” key with modifiers (alt, ctrl, cmd, shift) no longer toggles the filter menu on pages that support filtering.
[ui] Fix empty states on Runs table view for individual jobs, to provide links to materialize an asset or launch a run for the specific job, instead of linking to global pages.
[ui] When a run is launched from the Launchpad editor while an editor hint popover is open, the popover remained on the page even after navigation. This has been fixed.
[ui] Fixed an issue where clicking on the zoom controls on a DAG view would close the right detail panel for selected nodes.
[ui] Fixed an issue shift-selecting assets with multi-component asset keys.
[ui] Fixed an issue with the truncation of the asset stale causes popover.
When using a TimeWindowPartitionMapping with a start_offset or end_offset specified, requesting the downstream partitions of a given upstream partition would yield incorrect results. This has been fixed.
When using AutoMaterializePolicys with observable source assets, in rare cases, a second run could be launched in response to the same version being observed twice. This has been fixed.
When passing in hook_defs to define_asset_job, if any of those hooks had required resource keys, a missing resource error would surface when the hook was executed. This has been fixed.
Fixed a typo in a documentation URL in dagster-duckdb-polars tests. The URL now works correctly.
DbtManifest to fetch asset keys of sources and models: DbtManifest.get_asset_key_for_model, DbtManifest.get_asset_key_for_source. These methods are utilities for defining python assets as dependencies of dbt assets via @asset(key=manifest.get_asset_key_for_model(...).state_path parameter with DbtManifestAssetSelection has been deprecated, and will be removed in the next minor release.grpcio package (for dagster) has been removed.PartitionMapping have been removed. Defining custom partition mappings has been unsupported since 1.1.7.build_airbyte_assets. Thanks @guy-rvvup!Assets with lazy auto-materialize policies are no longer auto-materialized if they are missing but don’t need to be materialized in order to help down
meta.dagster.asset_key. This field takes in a list of strings that are used as the components of the generated AssetKey.version: 2
models:
- name: users
config:
meta:
dagster:
asset_key: ["my", "custom", "asset_key"]
meta.dagster.group. This field takes in a string that is used as the Dagster group for the generated software-defined asset corresponding to the dbt model.version: 2
models:
- name: users
config:
meta:
dagster:
group: "my_group"
dagster-msteams and dagster-mlflow packages could be installed with incompatible versions of the dagster package due to a missing pin.dagster-daemon run command sometimes kept code server subprocesses open longer than it needed to, making the process use more memory.@observable_source_assets with AutoMaterializePolicies, it was possible for downstream assets to get “stuck”, not getting materialized when other upstream assets changed, or for multiple down materializations to be kicked off in response to the same version being observed multiple times. This has been fixed.@dbt_assets
project_dir and target_path in DbtCliTask are converted from type str to type pathlib.Path.stdout.Dagster is now compatible with a breaking change introduced in croniter==1.4.0. Users of earlier versions of Dagster can pin croniter<1.4.
dagster field under +meta configuration. The following are equivalent:Before:
version: 2
models:
- name: users
config:
dagster_freshness_policy:
maximum_lag_minutes: 60
cron_schedule: '0 9 * * *'
dagster_auto_materialize_policy:
type: 'lazy'
After:
version: 2
models:
- name: users
config:
meta:
dagster:
freshness_policy:
maximum_lag_minutes: 60
cron_schedule: '0 9 * * *'
auto_materialize_policy:
type: 'lazy'
Added support for Pythonic Config classes to the @configured API, which makes reusing op and asset definitions easier:
class GreetingConfig(Config):
message: str
@op
def greeting_op(config: GreetingConfig):
print(config.message)
class HelloConfig(Config):
name: str
@configured(greeting_op)
def hello_op(config: HelloConfig):
return GreetingConfig(message=f"Hello, {config.name}!")
Added AssetExecutionContext to replace OpExecutionContext as the context object passed in to @asset functions.
TimeWindowPartitionMapping now contains an allow_nonexistent_upstream_partitions argument that, when set to True, allows a downstream partition subset to have nonexistent upstream parents.
Unpinned the alembic dependency in the dagster package.
[ui] A new “Assets” tab is available from the Overview page.
[ui] The Backfills table now includes links to the assets that were targeted by the backfill.
croniter==1.4.0. Users of earlier versions of Dagster can pin croniter<1.4.@run_failure_sensor.py.typed was missing in the dagster-graphql package. Thanks @Tanguy-LeFloch!AutoMaterializePolicys will now be cleared after 1 week.@dbt_assets:
profile and target can now be customized on the DbtCli resource.partial_parse.msgpack is detected in the target directory of your dbt project, it is now copied into the target directories created by DbtCli to take advantage of partial parsing.@dbt_assets can now be customized by overriding DbtManifest.node_info_to_metadata.AssetMaterializations.serverK8sConfig.containerConfig.name did not actually change the container name.Fixed an issue in the 1.3.8 release where the Dagster Cloud agent would sometimes fail to start up with an import error.
1.3.8 release where the Dagster Cloud agent would sometimes fail to start up with an import error.Nothing published for this version
Multipartitioned assets with one time dimension can now depend on earlier partitions of themselves.
define_asset_job now accepts a hooks argument.sqlalchemy==2.xadditionalInstanceConfig key that allows you to supply additional configuration to the Dagster instance.EcsRunLauncher now uses a different task definition family for each job, instead of registering a new task definition revision each time a different job is launched.EcsRunLauncher now includes a run_ecs_tags config key that lets you configure tags on the launched ECS task for each run.SkipReason, the SkipReason would be ignored. This has been fixed.dagster code-server start CLI, instead of only reloading your code when you initiate a reload from the Dagster UI.metadata parameter to define_asset_job (Thanks Elliot2718!)poll_sync_run to handle the “preparing” status from the Census API (Thanks ldnicolasmay!)@observable_source_asset-decorated functions can now return a DataVersionsByPartition to record versions for partitions.@dbt_assets
DbtCliTask's created by invoking DbtCli.cli(...) now have a method .is_successful(), which returns a boolean representing whether the underlying CLI process executed the dbt command successfully.@dbt_assets can now be customized by overriding DbtManifest.node_info_to_description.@dbt_assets.server_ecs_tags and run_ecs_tags that apply to each service or task created by the agent. See the docs for more information.instance.get_run_partition_data in Dagster Cloud.Adding a .env file in the working directory when running dagster dev can now be used for Dagster system variables like DAGSTER_HOME or environment var
.env file in the working directory when running dagster dev can now be used for Dagster system variables like DAGSTER_HOME or environment variables referenced in your dagster.yaml file using an env: key. Previously, setting a .env file only worked for environment variables referenced in your Dagster code.submit_job_execution can now take in a RunConfig object. Previously, it could only take a Python dictionary with the run configuration.jobs argument to Definitions./instance/runs path instead of the new /runs.create_databricks_run_now_op, thanks @srggrs!Added an experimental dagster code-server start command that can be used to launch a code server, much like dagster api grpc. Unlike dagster api grpc,
dagster code-server start command that can be used to launch a code server, much like dagster api grpc. Unlike dagster api grpc, however, dagster code-server start runs the code in a subprocess, so it can reload code from the Dagster UI without needing to restart the command. This can be useful for jobs that load code from some external source and may want to reload job definitions without restarting the process.sensors.num_submit_workers key to dagster.yaml that can be used to decrease latency when a sensor emits multiple run requests within a single tick. See the docs for more information.k8s_job_executor can now be used to launch each step of a job in its own Kubernetes, pod, even if the Dagster deployment is not using the K8sRunLauncher to launch each run in its own Kubernetes pod.io_manager_def param on an asset.maxResumeRunAttempts to null in the helm chart would cause it to be set to a default value of 3 instead of disabling run retries.k8s_job_executor would sometimes fail with a 409 Conflict error after retrying the creation of a Kubernetes pod for a step, due to the job having already been created during a previous attempt despite raising an error.op_name was passed to load_assets_from_dbt_manifest, and a select parameter was specified, a suffix would be appended to the desired op name.dagit would lead to JavaScript bundle loading errors.DbtCli, in dagster_dbt.cli. This new resource only support dbt-core>=1.4.0.@dbt_assets in dagster_dbt.asset_decorator that allows you to specify a compute function for a selected set of dbt assets that loaded as an AssetsDefinition.A new max_materializations_per_minute parameter (with a default of 1) to AutoMaterializationPolicy.eager() and AutoMaterializationPolicy.lazy() allows
max_materializations_per_minute parameter (with a default of 1) to AutoMaterializationPolicy.eager() and AutoMaterializationPolicy.lazy() allows you to set bounds on the volume of work that may be automatically kicked off for each asset. To restore the previous behavior, you can explicitly set this limit to None.DailyPartitionsDefinition, HourlyPartitionsDefinition, WeeklyPartitionsDefinition, and MonthlyPartitionsDefinition now support and end_date attribute.AutoMaterializePolicys or build_asset_reconciliation_sensor, a single new data version from an observable source asset could trigger multiple runs of the downstream assets. This has been fixed.EnvVar and IntEnvVar within raw run config - although they just returned the name of the env var rather than retrieve its value. This has been fixed to error directly.* was being interpreted as glob pattern, rather than as a dbt selector argument. We now explicitly set the default selection pattern as fqn:*.dagster-cloud serverless deploy did not create a unique image tag if the --image tag was not specified.op_name on load_assets_from_dbt_project and load_assets_from_dbt_manifest (thanks @wkeifenheim!)ExecuteJobResult was renamed to JobExecutionResult (ExecuteJobResult remains a deprecated alias)
TimeWindowPartitionMapping objects are now current-time aware. Subsequently, only upstream/downstream partitions existent at the current time are returned.ExecuteJobResult was renamed to JobExecutionResult (ExecuteJobResult remains a deprecated alias)AssetSelection.key_prefixes method allows matching asset keys starting with a provided prefix.ConfigurablePickledObjectADLS2IOManager that uses pythonic configDataProcResource follows the Pythonic resource system. The existing dataproc_resource remains supported.SnowflakeIOManager now supports private_keys that have been base64 encoded to avoid issues with newlines in the private key. Non-base64 encoded keys are still supported. See the SnowflakeIOManager documentation for more information on base64 encoded private keys.kvs, instance_info, and daemon_hearbeats table for existing Postgres storage instances that were migrating from before 1.2.2. This should unblock users relying on the existence of a primary key constraint for replication.SensorResult evaluation where multipartitioned run requests containing a dynamic partition added in a dynamic partitions request object would raise an invalid partition key error.TimeWindowPartitionMapping.ExecuteJobResult is now a deprecated alias for the new name, JobExecutionResult.airbyte_resource to load_assets_from_connections , you may now provide an instance of the AirbyteResource class, rather than just airbyte_resource.configured(...) (thanks @joel-olazagasti!)dagster-celery CLI accepted an inconsistent configuration format - it now matches the same format as the celery_executor. Thanks @boenshao!ecs_timeout in your ECS user code launcher config to extend how long the ECS agent polls for new code servers to start. Extending this timeout is useful if your code server takes an unusually long time to start up - for example, because it uses a very large image.load_assets_from_package_module and the other core load_assets_from_ methods now accept a source_key_prefix argument, which allows applying a key pref
load_assets_from_package_module and the other core load_assets_from_ methods now accept a source_key_prefix argument, which allows applying a key prefix to all the source assets that are loaded.
OpExecutionContext now has an asset_partitions_time_window_for_input method.
RunFailureSensorContext now has a get_step_failure_events method.
The Pythonic resource system now supports a set of lifecycle hooks which can be used to manage setup and teardown:
class MyAPIClientResource(ConfigurableResource):
api_key: str
_internal_client: MyAPIClient = PrivateAttr()
def setup_for_execution(self, context):
self._internal_client = MyAPIClient(self.api_key)
def get_all_items(self):
return self._internal_client.items.get()
Added support for specifying input and output config on ConfigurableIOManager.
QueuedRunCoordinator and SubmitRunContext are now exposed as public dagster exports.
[ui] Downstream cross-location dependencies of all source assets are now visible on the asset graph. Previously these dependencies were only displayed if the source asset was defined as a regular asset.
[ui] A new filtering experience is available on the Runs page after enabling feature flag “Experimental Runs table view with filtering”.
[dagster-aws] Allow the S3 compute log manager to specify a show_url_only: true config option, which will display a URL to the S3 file in dagit, instead of the contents of the log file.
[dagster-aws] PickledObjectS3IOManager now fully supports loading partitioned inputs.
[dagster-azure] PickedObjectADLS2IOManager now fully supports loading partitioned inputs.
[dagster-gcp] New GCSResource and ConfigurablePickledObjectGCSIOManager follow the Pythonic resource system. The existing gcs_resource and gcs_pickle_io_manager remain supported.
[dagster-gcp] New BigQueryResource follows the Pythonic resource system. The existing bigquery_resource remains supported.
[dagster-gcp] PickledObjectGCSIOManager now fully supports loading partitioned inputs.
[dagster-postgres] The event watching implementation has been moved from listen/notify based to the polling watcher used by MySQL and SQLite.
[dagster-slack] Add monitor_all_repositories to make_slack_on_run_failure_sensor, thanks @danielgafni!
[dagster-snowflake] New SnowflakeResource follows the Pythonic resource system. The existing snowflake_resource remains supported.
AssetMetadataValue.value that would cause an infinite recursion error.volumes and volumeMounts values have been added to the agent helm chart.load_assets_from_airbyte_instance and load_assets_from_airbyte_project now take a connection_to_auto_materialize_policy_fn for setting AutoMaterializePolicys on Airbyte assetsAdded performance improvements for yielding time-partitioned run requests.
NumReplicas parameter on the agent template in CloudFormation, or the dagsterCloudAgent.replicas field in Helm.enableZeroDowntimeDeploys parameter to true in the CloudFormation stack for your agent.AssetsDefinition.from_graph, as well as the@graph_asset and @graph_multi_asset decorators now support specifying AutoMaterializePolicys.async def ops/assets no longer prematurely finalize async generators during execution.build_asset_reconciliation_sensor) could incorrectly launch new runs for partitions that already had an in-progress run. This has been fixed.run_request_for_partition now throws an error. Instead, users should yield directly instantiated run requests via RunRequest(partition_key=...).graph_asset and graph_multi_asset now support specifying resource_defs directly (thanks @kmontag42)!node_info_to_auto_materialize_policy_fn param added to load_assets_from_dbt_* functions. (thanks @askvinni)!partition_key field to RunStatusSensorContext (thanks @pdstrnadJC)!latest_materialization_records_by_key method.allPartitions argument is provided.Performance improvements when evaluating time-partitioned run requests within sensors and schedules.
[previously deprecated, 0.15.0] metadata_entries arguments to event constructors have been removed. While MetadataEntry still exists and will only be…
build_asset_reconciliation_sensors , AutoMaterializePolicys work across code locations, as well as allow you to customize the conditions under which each asset is auto-materialized. [docs]FreshnessPolicys, Dagster’s UI now marks them as “overdue” instead of “late”.AthenaClientResource, ECRPublicResource, RedshiftClientResource, S3Resource, S3FileManagerResource, ConfigurablePickledObjectS3IOManager, SecretsManagerResource follow Pythonic resource system. The existing APIs remain supported.DatadogResource follows Pythonic resource system. The existing datadog_resource remains supported.GEContextResource follows Pythonic resource system. The existing ge_context_resource remains supported.GithubResource follows Pythonic resource system. The existing github_resource remains supported.MSTeamsResource follows Pythonic resource system. The existing msteams_resource remains supported.SlackResource follows Pythonic resource system. The existing slack_resource remains supported.pdb.set_trace no longer worked when running Dagster locally using dagster dev or dagit.@asset or Out caused an error to be thrown.pipelines or jobs could lead to blank pages.Definitions are now automatically bound to jobs. This will only result in a change in behavior if you a) have a job with no "io_manager" defined in its resource_defs and b) have supplied an IOManager with key "io_manager" to the resource_defs argument of your Definitions. Prior to 1.3.0, this would result in the job using the default filesystem-based IOManager for the key "io_manager". In 1.3.0, this will result in the "io_manager" supplied to your Definitions being used instead. The BindResourcesToJobs wrapper, introduced in 1.2 to simulate this behavior, no longer has any effect.CeleryK8sRunLauncher is now the same namespace as the Helm chart, instead of the default namespace. To restore the previous behavior, you can set the celeryK8sRunLauncher.jobNamespace field to the string default.SnowflakePandasIOManager has historically converted all timestamp data to strings before storing it in Snowflake. Now, it will instead ensure that timestamp data has a timezone, and if not, attach the UTC timezone. This allows the timestamp data to be stored as timestamps in Snowflake. If you have been storing timestamp data using the SnowflakePandasIOManager you can set the store_timestamps_as_strings=True configuration to continue storing timestamps as strings. For more information, and instructions for migrating Snowflake tables to use timestamp types, see the Migration Guide.Changes to experimental APIs
RunConfig to many APIs which previously would only accept a config dictionary.ResourceDefinition.execution config when constructing a RunConfig object.minutes_late and previous_minutes_late properties on the experimental FreshnesPolicySensorContext have been renamed to minutes_overdue and previous_minutes_overdue, respectively.Removal of deprecated APIs
metadata_entries arguments to event constructors have been removed. While MetadataEntry still exists and will only be removed in 2.0, it is no longer passable to any Dagster public API — users should always pass a dictionary of metadata values instead.latest_materialization_records_by_key function.BigQueryPandasIOManager and BigQueryPySparkIOManager.SnowflakePandasIOManager and SnowflakePySparkIOManager.Resource access (via both required_resource_keys and Pythonic resources) are now supported in observable source assets.
required_resource_keys and Pythonic resources) are now supported in observable source assets.EcsRunLauncher now allows you to customize volumes and mount points for the launched ECS task. See the API docs for more information.DuckDBPandasIOManager and DuckDBPySparkIOManager follow Pythonic resource system. The existing duckdb_pandas_io_manager and duckdb_pyspark_io_manager remain supported.BigQueryPandasIOManager and BigQueryPySparkIOManager follow Pythonic resource system. The existing bigquery_pandas_io_manager and bigquery_pyspark_io_manager remain supported.gcp_crentials , a temporary file to store the credentials will be created and the GOOGLE_APPLICATION_CREDENTIALS environment variable will be set to the temporary file. When the BigQuery resource is garbage collected, the environment variable will be unset and the temporary file deleted.SnowflakePandasIOManager and SnowflakePySparkIOManager follow Pythonic resource system. The existing snowflake_pandas_io_manager and snowflake_pyspark_io_manager remain supported.dagster dev would periodically emit a harmless but annoying warning every few minutes about a gRPC server being shut down.RunRequest(partition_key=...) object.PartitionsDefinition.run_request_for_partition would incorrectly raise an error for a job with a DynamicPartitionsDefinition that was defined with a function.define_asset_job would raise an error.__ASSET_JOB has no op named ... errors when using automatic run retries.EcsRunLauncher would fail to launch runs if the use_current_ecs_task_config field was set to False but no task_definition field was set.EcsRunLauncher now allows you to set a capacity provider strategy and customize the ephemeral storage used for launched ECS tasks. See the docs for details. Thanks AranVinkItility!AssetsDefinition.from_op. Thanks @tghanken for the fix!minimum_interval_seconds parameter to enable customizing the evaluation interval on the slack run failure sensor, thanks @ldnicolasmay!Resource annotation for Pythonic resource inputs has been renamed to ResourceParam in preparation for the release of the feature in 1.3.The run_request_for_partition method on JobDefinition and UnresolvedAssetJobDefinition is now deprecated and will be removed in 2.0.0. Instead, direct…
materialize and materialize_to_memory now both accept a selection argument that allows specifying a subset of assets to materialize.MultiPartitionsDefinition is no longer marked experimental.MultiPartitionsDefinitions with a time dimension.load_assets_from_package_module and similar methods now accept a freshness_policy, which will be applied to all loaded assets.build_sensor_context and build_multi_asset_sensor_context can now take a Definitions object in place of a RepositoryDefinitions3_resource now accepts use_ssl and verify configurations.selection argument on define_asset_job.MultiPartitionsDefinition.MultiPartitionsMapping (experimental) is loaded.TimeWindowPartitionMappings to not be bubbled up to the UI.pdb stopped working when using dagster dev.AssetMaterialization objects with a null AssetKey.TimeWindowPartitionsDefinition with a non-standard cron schedule, and also provided a minute_of_hour or similar argument in build_schedule_from_partitioned_job. Dagster would silently create the wrong cron expression. It now raises an error.PartitionsDefinition. These partitions are now ignored.PersistentAirflowDatabase where versions of airflow from 2.0.0 till 2.3.0 would not use the correct connection environment variable name.poll_sync_run function ofdagster-census that prevented polling from working correctly (thanks @ldincolasmay!)run_request_for_partition method on JobDefinition and UnresolvedAssetJobDefinition is now deprecated and will be removed in 2.0.0. Instead, directly instantiate a run request with a partition key via RunRequest(partition_key=...).Nothing published for this version
The environment_vars argument to ScheduleDefinition is deprecated (the argument is currently non-functional; environment variables no longer need to b…
AssetsDefinition.to_source_assets to method convert a set of assets to SourceAsset objects.MultiPartitionsDefinitions.mlflow pin from the dagster-mlflow package.When using build_asset_reconciliation_sensor, in some cases duplicate runs could be produced for the same partition of an asset. This has been fixed.
When using Pythonic configuration for resources, aliased field names would cause an error. This has been fixed.
Fixed an issue where context.asset_partitions_time_window_for_output threw an error when an asset was directly invoked with build_op_context.
[dagster-dbt] In some cases, use of ephemeral dbt models could cause the dagster representation of the dbt dependency graph to become incorrect. This has been fixed.
[celery-k8s] Fixed a bug that caused JSON deserialization errors when an Op or Asset emitted JSON that doesn't represent a DagsterEvent.
Fixed an issue where launching a large backfill while running dagster dev would sometimes fail with a connection error after running for a few minutes.
Fixed an issue where dagster dev would sometimes hang when running Dagster code that attempted to read in input via stdin.
Fixed an issue where runs that take a long time to import code would sometimes continue running even after they were stopped by run monitoring for taking too long to start.
Fixed an issue where AssetSelection.groups() would simultaneously select both source and regular assets and consequently raise an error.
Fixed an issue where BindResourcesToJobs would raise errors encapsulating jobs which had config specified at definition-time.
Fixed Pythonic config objects erroring when omitting optional values rather than specifying None.
Fixed Pythonic config and resources not supporting Enum values.
DagsterInstance.local_temp and DagsterInstance.ephemeral now use object instance scoped local artifact storage temporary directories instead of a shared process scoped one, removing a class of thread safety errors that could manifest on initialization.
Improved direct invocation behavior for ops and assets which specify resource dependencies as parameters, for instance:
class MyResource(ConfigurableResource):
pass
@op
def my_op(x: int, y: int, my_resource: MyResource) -> int:
return x + y
my_op(4, 5, my_resource=MyResource())
[dagster-azure] Fixed an issue with an AttributeError being thrown when using the async DefaultAzureCredential (thanks @mpicard)
[ui] Fixed an issue introduced in 1.2.3 in which no log levels were selected by default when viewing Run logs, which made it appear as if there were no logs at all.
environment_vars argument to ScheduleDefinition is deprecated (the argument is currently non-functional; environment variables no longer need to be whitelisted for schedules)[CHANGES.md](http://CHANGES.md) (thanks @fridiculous)--path-prefix can now be supplied via Helm chart (thanks @mpicard)https://github.com/dagster-io/dagster/compare/1.2.3...1.2.4
<details><summary>See All Contributors</summary>
[refactor] delete Materialization (#13030) by @smackeseyAdd in progress subsets to the partition cache (#13045) by @johannkmAdd multipartitioned assets with dynamic dimension to toys (#13061) by @clairelin135add docs example for multi-asset conditional materialization (#13054) by @sryzaAdd docs for source asset observation jobs/schedules (#13062) by @smackeseyRevert "Add in progress subsets to the partition cache (#13045)" by @johannkm[asset-reconciliation][bug] Fix issue where overly-aggressive runs would be kicked off. (#13069) by @OwenKepharttweaks to cross-repo-assets toy (#12973) by @sryzaShow unauthorized error graphql error message (#13064) by @salazarm[dagster-io/ui] Make Suggest component a bit more flexible (#13056) by @hellendagFix disabled state for launchpad button submenu (#13078) by @salazarmRe-enable in progress subsets in the partition cache (#13082) by @johannkmAdd dynamic partitions name resolver to dimension type (#13070) by @clairelin135asset sensor test docs (#13065) by @prhaFix issue where backfill fails when gRPC server is replaced mid-backfill (#13085) by @gibsondanUse instance from sensor/schedule context to instantiate resources, delay until accessed (#13041) by @benpankowAdd materializing subset to asset gql (#13046) by @johannkmfeat(helm): add path-prefix to dagit command (#13080) by @mpicardUse dynamic partition definition name for dimension of multipartition definition (#13090) by @salazarm[typing/static] Fix @repository decorator typing (#12295) by @smackesey[instance] make local artifact directory scheme thread safe (#13043) by @alangenfeld1.2.3 changelog (#13094) by @jamiedemariaEnsure pyright venvs use statically legible editable installs (#13089) by @smackesey[docs] - Remove finished code from dbt tutorial template (#13091) by @erinkcochran87[ui] Upgrade react-markdown (#13092) by @hellendagFix submitting backfills synchronously from graphql (#13093) by @gibsondanTest get and set serialized_in_progress_partition_subset (#13063) by @johannkmExitStack.pop_all -> close (#13050) by @alangenfeldAutomation: versioned docs for 1.2.3 by @elementl-devtoolsFraser/rework readme (#12565) by @frasermarlow[ui] Use DefaultLogLevels when there is no level state stored (#13109) by @hellendagSet stdin to DEVNULL when opening dagster subprocesses (#13099) by @gibsondanadd a vercel github action to build docs/storybook previews (#13052) by @prha[refactor] Remove frozen{list,dict,tags} classes (#12293) by @smackeseyDocs for setting up Gitlab CI, branch deployment guide (#12998) by @prhaAdd assets def to op context (#13088) by @clairelin135make AssetsDefinition.to_source_assets public (#13073) by @sryza[docs] New ML pipeline guide PR (#13100) by @odette-elementl[freshness-refactor][3/n] Update methods on the CachingDataTimeResolver to work with scalar data time (#12906) by @OwenKephartFixing refs to images in the README (#13126) by @tacastillorestrict vercel builds based on paths (#13129) by @prhafix missing snapshots (#13134) by @OwenKephartTelemetry for dynamic partitions (#12605) by @clairelin135fix missing snapshots (again) (#13136) by @OwenKephart[freshness-refactor][4/n] Simplify scheduling algorithm (#13019) by @OwenKephartDeprecate environment_vars argument to ScheduleDefinition, @schedule (#13044) by @smackesey[refactor] delete hourly/daily/weekly/monthly schedule decorators, PartitionScheduleDefinition, build_schedule_from_partition (#13006) by @smackesey[refactor] simplify dependency dict typing (#12521) by @smackeseycelery-k8s executor: handle stdout that's valid json but not a dagster event (#13143) by @johannkmfix typo in CHANGES.md (#13140) by @fridiculousFix branch deployment docs (#13131) by @dpeng817Update multiple agents docs (#13135) by @dpeng817add toy for eager asset reconciliation (#13066) by @sryzatrying again by moving the images to the same directory as the readme (#13127) by @tacastillo[tech][templates] moving the scaffold project's asset loader outside of the defs (#13103) by @tacastillo[docs][tutorial-revamp] Adding a section for scheduling to the tutorial (#13101) by @tacastilloreplacing corrupt .png image (#13157) by @frasermarlow[dagster-dbt] Fix bug when calculating transitive dependencies (#13128) by @OwenKephartdeploy storybook to prod when landing pushes on master (#13159) by @prha[dagster-azure] fix: AttributeError: 'coroutine' object has no attribute 'token' (#13110) by @mpicardAdd support for more asset tags (#13153) by @braunjj[caching-refactor] Remove use of get_and_update_asset_status_cache. (#13151) by @OwenKephart[asset-reconciliation][perf] Cache common properties on the TimeWindowPartitionsDefinition (#12981) by @OwenKephartMultiPartitionMapping (#12950) by @clairelin135[asset-reconciliation] Fix issue with duplicate runs for partitions with in-progress materializations (#13130) by @OwenKephartupdate telemetry documentation links (#13176) by @Abbe98[structured config] respect field aliases for resources (#13177) by @jamiedemaria[refactor] Remove PartitionSetDefinition (#13145) by @smackeseyTableMetadataValue toy (#13124) by @smackeseyfix repository decorator typing (#13180) by @smackeseyadd storybook for @core in addition to @ui (#13184) by @prhafeat(dbt): follow dbt Core's version support constraints (#13189) by @rexledesmaFix case where a pipeline load takes long enough that run monitoring kills it (#13156) by @gibsondanAllow newlines in error messages in Dagit (#13193) by @gibsondanMake sure we always wait for all grpc server processes to spin down in tests (#13146) by @gibsondanunpin mlflow from dagster-mlflow (#13194) by @gibsondanGreatly reduce the number of gRPC calls when doing large asset backfills (#13086) by @gibsondandon't run mlflow tests on python 3.11 (#13197) by @gibsondan[pythonic config] Treat Optional as both not-required and Noneable (#12975) by @benpankow[pythonic config] Add support for enums (#12979) by @benpankow[pythonic resources] Make direct invocation of ops/assets w/ resources easier (#13002) by @benpankow[bugfix] make AssetSelection.groups() resolve to only regular assets (#13196) by @smackesey[fix] Fix behavior of BindResourcesToJobs when jobs have config specified (#13057) by @benpankow[serdes] remove extraneous @cached_method use (#13206) by @alangenfeld1.2.4 changelog (#13226) by @smackesey1.2.4 by @elementl-devtools[refactor] delete Materialization (#13030) by @smackeseyAdd in progress subsets to the partition cache (#13045) by @johannkmAdd multipartitioned assets with dynamic dimension to toys (#13061) by @clairelin135add docs example for multi-asset conditional materialization (#13054) by @sryzaAdd docs for source asset observation jobs/schedules (#13062) by @smackeseyRevert "Add in progress subsets to the partition cache (#13045)" by @johannkm[asset-reconciliation][bug] Fix issue where overly-aggressive runs would be kicked off. (#13069) by @OwenKepharttweaks to cross-repo-assets toy (#12973) by @sryzaShow unauthorized error graphql error message (#13064) by @salazarm[dagster-io/ui] Make Suggest component a bit more flexible (#13056) by @hellendagFix disabled state for launchpad button submenu (#13078) by @salazarmRe-enable in progress subsets in the partition cache (#13082) by @johannkmAdd dynamic partitions name resolver to dimension type (#13070) by @clairelin135asset sensor test docs (#13065) by @prhaFix issue where backfill fails when gRPC server is replaced mid-backfill (#13085) by @gibsondanUse instance from sensor/schedule context to instantiate resources, delay until accessed (#13041) by @benpankowAdd materializing subset to asset gql (#13046) by @johannkmfeat(helm): add path-prefix to dagit command (#13080) by @mpicardUse dynamic partition definition name for dimension of multipartition definition (#13090) by @salazarm[typing/static] Fix @repository decorator typing (#12295) by @smackesey[instance] make local artifact directory scheme thread safe (#13043) by @alangenfeld1.2.3 changelog (#13094) by @jamiedemariaEnsure pyright venvs use statically legible editable installs (#13089) by @smackesey[docs] - Remove finished code from dbt tutorial template (#13091) by @erinkcochran87[ui] Upgrade react-markdown (#13092) by @hellendagFix submitting backfills synchronously from graphql (#13093) by @gibsondanTest get and set serialized_in_progress_partition_subset (#13063) by @johannkmExitStack.pop_all -> close (#13050) by @alangenfeldAutomation: versioned docs for 1.2.3 by @elementl-devtoolsFraser/rework readme (#12565) by @frasermarlow[ui] Use DefaultLogLevels when there is no level state stored (#13109) by @hellendagSet stdin to DEVNULL when opening dagster subprocesses (#13099) by @gibsondanadd a vercel github action to build docs/storybook previews (#13052) by @prha[refactor] Remove frozen{list,dict,tags} classes (#12293) by @smackeseyDocs for setting up Gitlab CI, branch deployment guide (#12998) by @prhaAdd assets def to op context (#13088) by @clairelin135make AssetsDefinition.to_source_assets public (#13073) by @sryza[docs] New ML pipeline guide PR (#13100) by @odette-elementl[freshness-refactor][3/n] Update methods on the CachingDataTimeResolver to work with scalar data time (#12906) by @OwenKephartFixing refs to images in the README (#13126) by @tacastillorestrict vercel builds based on paths (#13129) by @prhafix missing snapshots (#13134) by @OwenKephartTelemetry for dynamic partitions (#12605) by @clairelin135fix missing snapshots (again) (#13136) by @OwenKephart[freshness-refactor][4/n] Simplify scheduling algorithm (#13019) by @OwenKephartDeprecate environment_vars argument to ScheduleDefinition, @schedule (#13044) by @smackesey[refactor] delete hourly/daily/weekly/monthly schedule decorators, PartitionScheduleDefinition, build_schedule_from_partition (#13006) by @smackesey[refactor] simplify dependency dict typing (#12521) by @smackeseycelery-k8s executor: handle stdout that's valid json but not a dagster event (#13143) by @johannkmfix typo in CHANGES.md (#13140) by @fridiculousFix branch deployment docs (#13131) by @dpeng817Update multiple agents docs (#13135) by @dpeng817add toy for eager asset reconciliation (#13066) by @sryzatrying again by moving the images to the same directory as the readme (#13127) by @tacastillo[tech][templates] moving the scaffold project's asset loader outside of the defs (#13103) by @tacastillo[docs][tutorial-revamp] Adding a section for scheduling to the tutorial (#13101) by @tacastilloreplacing corrupt .png image (#13157) by @frasermarlow[dagster-dbt] Fix bug when calculating transitive dependencies (#13128) by @OwenKephartdeploy storybook to prod when landing pushes on master (#13159) by @prha[dagster-azure] fix: AttributeError: 'coroutine' object has no attribute 'token' (#13110) by @mpicardAdd support for more asset tags (#13153) by @braunjj[caching-refactor] Remove use of get_and_update_asset_status_cache. (#13151) by @OwenKephart[asset-reconciliation][perf] Cache common properties on the TimeWindowPartitionsDefinition (#12981) by @OwenKephartMultiPartitionMapping (#12950) by @clairelin135[asset-reconciliation] Fix issue with duplicate runs for partitions with in-progress materializations (#13130) by @OwenKephartupdate telemetry documentation links (#13176) by @Abbe98[structured config] respect field aliases for resources (#13177) by @jamiedemaria[refactor] Remove PartitionSetDefinition (#13145) by @smackeseyTableMetadataValue toy (#13124) by @smackeseyfix repository decorator typing (#13180) by @smackeseyadd storybook for @core in addition to @ui (#13184) by @prhafeat(dbt): follow dbt Core's version support constraints (#13189) by @rexledesmaFix case where a pipeline load takes long enough that run monitoring kills it (#13156) by @gibsondanAllow newlines in error messages in Dagit (#13193) by @gibsondanMake sure we always wait for all grpc server processes to spin down in tests (#13146) by @gibsondanunpin mlflow from dagster-mlflow (#13194) by @gibsondanGreatly reduce the number of gRPC calls when doing large asset backfills (#13086) by @gibsondandon't run mlflow tests on python 3.11 (#13197) by @gibsondan[pythonic config] Treat Optional as both not-required and Noneable (#12975) by @benpankow[pythonic config] Add support for enums (#12979) by @benpankow[pythonic resources] Make direct invocation of ops/assets w/ resources easier (#13002) by @benpankow[bugfix] make AssetSelection.groups() resolve to only regular assets (#13196) by @smackesey[fix] Fix behavior of BindResourcesToJobs when jobs have config specified (#13057) by @benpankow[serdes] remove extraneous @cached_method use (#13206) by @alangenfeld1.2.4 changelog (#13226) by @smackesey1.2.4 by @elementl-devtools</details>
Jobs defined via define_asset_job now auto-infer their partitions definitions if not explicitly defined.
define_asset_job now auto-infer their partitions definitions if not explicitly defined.define_asset_job. This allows putting them on a schedule/sensor.instance property to the HookContext object that is passed into Op Hook functions, which can be used to access the current DagsterInstance object for the hook.create_table_schema_metadata_from_dataframe function to generate a TableSchemaMetadataValue from a Pandas DataFrame. Thanks @AndyBys!dag_run configuration on the integration’s database resources.dagster asset wipe-partitions-status-cache to see the partitions displayed.1.1.18, users with a gRPC server that could not access the Dagster instance on user code deployments would see an error when launching backfills as the instance could not instantiate. This has been fixed.build_asset_reconciliation_sensor from kicking off runs of downstream assets. This has been fixed.build_asset_reconciliation_sensor is now much more performant in cases where unpartitioned assets are upstream or downstream of static-partitioned assets with a large number of partitions.jobNamespace field in the Dagster Helm chart when using the CeleryK8sRunLauncher.en-US browser locale. This has been fixed.bfc222c - [refactor] Remove symbols deprecated until 1.2 (#12360) by @smackesey
Dagster is now tested on Python 3.11.
Users can now opt in to have resources provided to Definitions bind to their jobs. Opt in by wrapping your job definitions in BindResourcesToJobs. This will become the default behavior in the future.
@op(required_resource_keys={"foo")
def my_op(context)
print(context.foo)
@job
def my_job():
my_op()
defs = Definitions(
jobs=BindResourcesToJobs([my_job])
resources={"foo": foo_resource}
Added dagster asset list and dagster asset materialize commands to Dagster’s command line interface, for listing and materializing software-defined assets.
build_schedule_from_partitioned_job now accepts jobs partitioned with a MultiPartitionsDefinition that have a time-partitioned dimension.
Added SpecificPartitionsPartitionMapping, which allows an asset, or all partitions of an asset, to depend on a specific subset of the partitions in an upstream asset.
load_asset_value now supports SourceAssets.
[ui] Ctrl+K has been added as a keyboard shortcut to open global search.
[ui] In the run logs table, the timestamp column has been moved to the far left, which will hopefully allow for better visual alignment with op names and tags.
[dagster-dbt] A new node_info_to_definition_metadata_fn to load_assets_from_dbt_project and load_assets_from_dbt_manifest allows custom metadata to be attached to the asset definitions generated from these methods.
[dagster-celery-k8s] The Kubernetes namespace that runs using the CeleryK8sRunLauncher are launched in can now be configured by setting the jobNamespace field in the Dagster Helm chart under celeryK8sRunLauncherConfig.
[dagster-gcp] The BigQuery I/O manager now accepts timeout configuration. Currently, this configuration will only be applied when working with Pandas DataFrames, and will set the number of seconds to wait for a request before using a retry.
[dagster-gcp] [dagster-snowflake] [dagster-duckdb] The BigQuery, Snowflake, and DuckDB I/O managers now support self-dependent assets. When a partitioned asset depends on a prior partition of itself, the I/O managers will now load that partition as a DataFrame. For the first partition in the dependency sequence, an empty DataFrame will be returned.
[dagster-k8s] k8s_job_op now supports running Kubernetes jobs with more than one pod (Thanks @Taadas).
AssetValueLoader.load_asset_value() didn’t load transitive resource dependencies correctly.RunConfig object with optional config arguments would lead to an error.ScheduleEvaluationContext.scheduled_execution_time to not be Optional.OpExecutionContext.partition_time_window ****(thanks @elben10).InputContext.upstream_output.log is no longer None when loading a source asset.async def ops are now invoked with asyncio.run.TimeWindowPartitionDefinition now throws an error at definition time when passed an invalid cron schedule instead of at runtime.cron_schedule_timezone inside of the config for a dbt model would not result in that property being set on the generated FreshnessPolicy. This has been fixed.GCSComputeLogManager when the session does not have permissions to generate signed urls.validate_run_config no longer accepts pipeline_def or mode arguments. These arguments refer to legacy concepts that were removed in Dagster 1.0, and since then there have been no valid values for them.Added experimental support for resource requirements in sensors and schedules. Resources can be specified using required_resource_keys and accessed through the context or specified as parameters:
@sensor(job=my_job, required_resource_keys={"my_resource"})
def my_sensor(context):
files_to_process = context.my_resource.get_files()
...
@sensor(job=my_job)
def my_sensor(context, my_resource: MyResource):
files_to_process = my_resource.get_files()
...
https://github.com/dagster-io/dagster/compare/1.2.1...1.2.2
<details><summary>See All Contributors</summary>
[dagster-gcp-pandas] add timeout config (#12637) by @jamiedemaria[docs][tutorial-revamp] Basic tutorial revamp parts 1 through 4 (#12509) by @tacastillofeat(dbt): add support for --debug (#12722) by @rexledesmaDefault useful Dagster helm chart features to on (#12737) by @gibsondan[For 1.2] Allow both protobuf 3 and 4 in dagster (#12466) by @gibsondan[toys repo] Export partitioned assets toys (#12733) by @salazarm[pythonic config] Allow using 'resource_defs' with resource args in assets (#12679) by @benpankow[refactor] Remove symbols deprecated until 1.2 (#12360) by @smackesey[dagster-snowflake] fix inconsistencies in snowflake resource (#12633) by @jamiedemaria[ui] Update permissions for launching jobs and materializing assets (#12681) by @hellendag[For 1.2] Change default run monitoring settings (#11512) by @gibsondan[For 1.2] Don't include run/job tags in k8s_job_ops k8s config computations (#12345) by @gibsondanRemove Partitioned Schedules from docs, fix dagit staleStatusCauses mocks (#12742) by @smackesey[Sensor Testing] Add "Test again" button (#12735) by @salazarm[1.2.0] [refactor] Delete DagsterTypeMaterializer (#12516) by @smackeseychange DynamicPartitionsDefinition.__repr__ (#12754) by @sryzaRecommended Project Structure Guide (#12656) by @odette-elementl[pythonic config] Add test showcasing use of Pydantic validators (#12536) by @benpankowfix serialization of TableRecord (#10731) by @sryza[dagster-gcp-pandas] revert flaky test (#12748) by @jamiedemaria[typing/static] PartitionsDefinition covariant type var (#12284) by @smackesey[rename] LogicalVersion -> DataVersion (#12500) by @smackesey[rename] LogicalVersion -> DataVersion in gql/dagit (#12501) by @smackesey[rename] Rename LogicalVersion -> DataVersion in docs (#12503) by @smackesey[dagster-airflow] add make_persistent_airflow_db_resource (#12305) by @Ramshackle-Jamathon[ui] Remove usePermissionsDEPRECATED (#12771) by @hellendag[Asset Details Page] Auto-select partition in materialization dialog. (#12734) by @salazarmfeat(databricks)!: remove create_databricks_job_op (#12600) by @rexledesmaMake permissions_for_location require keyword args (#12774) by @gibsondanRemove DynamicPartitionsDefinitions API methods (#12744) by @clairelin135Set default memory/CPU for new ECS tasks based on runtime platform (#12767) by @gibsondancron_schedule validation for time window partitions (#12761) by @prhaupdate some guide titles and descriptions (#12775) by @sryzaadd primary keys to all run tables (#12711) by @prhadocument PartitionMappings in partitions concepts page (#12768) by @sryzamake sure kv/daemon_heartbeat queries explicitly enumerate columns (#12789) by @prhaupdate API docs for freshness policies (#12788) by @OwenKephart[apidoc] asset_selection repository -> Definitions (#11303) by @yuhanFix typing on mem_io_manager (#12791) by @dpeng817[docs] add note about biquery temp tables (#12747) by @jamiedemariaRemove deprecated MetadataEntry constructors, update entry_data internal refs (#12724) by @smackesey[dagster-airflow] persistent db docs (#12485) by @Ramshackle-Jamathon[pythonic resources][fix] Fix inheriting attributes when extending Pythonic resources (#12781) by @benpankowMetadataEntryUnion (#12725) by @smackesey[docs] - Update RBAC docs for Cloud (#12752) by @erinkcochran87[dagit] Update the asset partitions / events view after run failures (#12798) by @bengotow[dagit] Asset node rendering tweaks (#12764, #12765) (#12794) by @bengotowReflect dynamically-added partitions in asset sidebar immediately (#12778) by @salazarm[Create Partition Dialog] Autofocus input (#12777) by @salazarmRemove PartitionMetadataEntry (#12726) by @smackeseyMaintain a separate cursor in the asset partition cache pointing back to the earliest in progress run (#12782) by @johannkm[ui] Run logs: make Timestamp the first column (#12805) by @hellendag[snowflake-pyspark] refactor spark session creation in tests (#12738) by @jamiedemaria[db io managers] refactor tests to test table creation (#12739) by @jamiedemaria[dagit] Fix yaml editor help context when indented on a new line (#12537) (#12804) by @bengotowmake context.upstream_output.log work for source assets (#12787) by @sryza[typing/runtime] serdes (#12523) by @smackesey[dagit] Show “Observe” on source asset details pages, not “Materialize" (#12803) by @bengotow[ui] Disable "Open in launchpad" based on permissions (#12817) by @hellendagAdd a way to cause partition health data to refetch without passing around refetch callbacks (#12822) by @salazarm[ui] Fix space oddities on Schedules overview table (#12810) by @hellendag[ESLint] Update missing graphql variable type rule to check mutations and subscriptions too (#12779) by @salazarmTest get and set new asset cache fields (#12820) by @johannkm[duckdb-polars] fix readme (#12499) by @jamiedemaria[freshness-refactor][1/n] Separate part of CachingInstanceQueryer into CachingDataTimeResolver (#12809) by @OwenKephart[ui] Affordance for copying asset key (#12829) by @hellendag[fix][dagit] Allow resources page to scroll (#12827) by @benpankow[async op] use asyncio.run (#12785) by @alangenfeld[dagster-airflow] remove double encoding of timezones (#12811) by @Ramshackle-Jamathon[ui] Support Ctrl+K to trigger search (#12840) by @hellendagVirtualize TagSelector component (#12841) by @salazarm[Launchpad] Allow adding new dynamic partition from launchpad UI (#12757) by @salazarm[tox] download latest pip (#12821) by @alangenfeld[dagster-snowflake] fix permissions issue when creating schemas (#12802) by @jamiedemaria[pythonic config] Special rendering for env vars in UI (#12446) by @benpankownothing input toy (#12849) by @sryzalatestRunForPartition gql (#12845) by @johannkm[dagster-dbt] add support for node_info_to_definition_metadata_fn (#12831) by @OwenKephartRemove nothing types from asset graph (#12848) by @salazarm[TagSelector] Allow filtering tags in dropdown (#12844) by @salazarmAdd a tooltip on asset nodes in the graph (#12853) by @salazarm[pythonic resources] Keep track of utilized env vars in external repo data (#12557) by @benpankowChangelog 1.2.0 (#12851) by @dpeng817add a local download url fallback for GCS (#12815) by @prha[freshness-refactor][2/n] Reorganize/Rename/Remove/Document CachingInstanceQueryer methods (#12813) by @OwenKephartfix type annotation for partition_time_window (#12850) by @sryzaRevert "fix type annotation for partition_time_window" (#12860) by @sryzaAutomation: versioned docs for 1.2.0 by @elementl-devtoolsAdd volatility property to CachingStaleStatusResolver (#12855) by @smackeseyUpdate CHANGES.md (#12864) by @Ramshackle-Jamathon[docs] - Add doc checklist to PR template (#12861) by @erinkcochran87Fix partition_time_window type error (#12838) by @elben10Add 1.2.0 changes to migration.md (#12866) by @clairelin135Fix config w/ dynamic partitions error (#12710) by @clairelin135[1.2] make ScheduleEvaluationContext.scheduled_execution_time non-optional (#12623) by @sryzaasset materialize CLI (#12691) by @sryza[refactor] serdes pack/deserialize APIs (#12524) by @smackesey[refactor] MayHaveInstanceWeakRef error on access non-existent _instance (#12553) by @smackeseyasset list command (#12790) by @sryzafixups to config_schedule.py (#12874) by @sryzaFix some upcoming lint errors in docstrings (#12704) by @charliermarsh[db io managers] support self dependent assets in db io managers (#12700) by @jamiedemariaFix markdown links in changelog (#12877) by @johannkmFix dagster-k8s job execution when a job has more than 1 pod (#12731) by @Taadastake grpcio-health-checking pins out of backcompat suite (#12876) by @gibsondanFix ruff (#12879) by @gibsondanneed to explicitly return a known existing column for updates (#12883) by @prhaFix docs build (#12884) by @smackeseyadd 1.2.1 changelog (#12887) by @prhaadd SpecificPartitionsPartitionMapping (#12878) by @sryzaAutomation: versioned docs for 1.2.1 by @elementl-devtools[docs] - Apply style guide to Asset versioning and caching guide (#12895) by @erinkcochran87[dagster-ui] Sync the search filter to the query string on most pages (#12899) by @bengotowon asset backfills, propagate backfill tags to runs (#12886) by @sryzaSupport SourceAssets in load_asset_value (#12859) by @sryzaFix transitive resource deps for asset value loader (#12872) by @dpeng817[typing/static] telemetry (#12714) by @smackeseyadd docs page on asset selection syntax (#12871) by @sryzaAdd dropdown for materializing using launchpad (#12852) by @salazarm[ui] Refactor permissions objects (#12903) by @hellendagavoid custom PartitionMapping on IO manager concept page (#12902) by @sryza[bk] Add changelog check step (#12865) by @benpankowtutorial intro tweaks (#12780) by @sryzaMove docs style guidelines from pull request template to contribution guide (#12875) by @sryzaremove support for legacy APIs from validate_run_config (#12881) by @sryzamake reconciliation perf test names easier to scan, and turn them all on (#12908) by @sryzaMake fetch_flattened_time_window_ranges support arbitrary statuses (#12863) by @johannkmBuild schedule from multipartitioned job (#12907) by @clairelin135[ui] Clean up temporary permission code (#12923) by @hellendag[op] error on multiple values for argument (#12473) by @alangenfeldtest assets in dynamic re-execution (#12651) by @alangenfeldSupport requiring resources in sensors, experimentally (#12401) by @benpankowSupport requiring resources in schedules, experimentally (#12697) by @benpankow[pythonic config] support Pydantic constrained types (#12703) by @benpankow[docs] Add env var docs to Pythonic config experimental docs (#12740) by @benpankow[pythonic resources] Surface utilized env vars in GraphQL (#12558) by @benpankow[dagit] Allow the lineage tab to appear before the asset definition has loaded (#12191) (#12814) by @bengotowSupport defaulting to selected partitions for Multi-partitioned Assets (#12856) by @salazarmfix execution plan step output collisions (#12799) by @alangenfeld[pythonic config] Fix using RunConfig with optional config fields (#12933) by @benpankow[Partitions] Use OrdinalPartitionSelector for STATIC partitioned assets (#12919) by @salazarmfix ruff (#12936) by @alangenfeldUnpin grpcio on dagster for python 3.11 (#12917) by @gibsondanAllow setting the job_namespace for the celery k8s run launcher (#12911) by @gibsondan[dagster-dbt] add cron_schedule_timezone to set of properties parsed from dbt config (#12904) by @OwenKephartRedesign partition backfill modal (#12928) by @salazarmFix error reporter infinite loop (#12941) by @salazarmAdd Python 3.11 to dagster buildkite (#12723) by @gibsondanBind top-level resources to jobs at Definitions-time (#12430) by @benpankow[asset-reconciliation] Do not reconcile partitions involved in active asset backfills (#12926) by @OwenKephart[asset-reconciliation] Fix behavior that could cause overly-aggressive updates (#12940) by @OwenKephartimprove execution plan snapshot use in execute_run (#12660) by @alangenfeldopen backfill options section by default (#12957) by @salazarm[Partition Backfill Dialog] Update backfill options spacing (#12959) by @salazarmResolve transitive resource deps for schedules and sensors (#12946) by @benpankowchangelog 1.2.2 (#12960) by @yuhanFix pandera tests (#12947) by @smackesey[dagit] Use pure-asset backfills instead of hidden asset job backfills (#12948) by @bengotow1.2.2 by @elementl-devtools</details>
Fixed a bug with postgres storage where daemon heartbeats were failing on instances that had not been migrated with dagster instance migrate after upg
dagster instance migrate after upgrading to 1.2.0.The previous create_databricks_job_op op factory is now deprecated.
dagster dev command that can be used to run both Dagit and the Dagster daemon in the same process during local development. [docs]DynamicPartitionsDefinition allows partitioning assets dynamically - you can add and remove partitions without reloading your definitions (experimental). [docs]TimeWindowPartitionMapping now accepts start_offset and end_offset arguments that allow specifying that time partitions depend on earlier or later time partitions of upstream assets [docs].dagster-wandb with Weights & Biases allows you to orchestrate your MLOps pipelines and maintain ML assets with Dagster. [docs]dagster-snowflake-pyspark allows you to store and load PySpark DataFrames as Snowflake tables using the snowflake_pyspark_io_manager. [docs]dagster-gcp-pandas and dagster-gcp-pyspark allow you to store and load Pandas and PySpark DataFrames as BigQuery tables using the bigquery_pandas_io_manager and bigquery_pyspark_io_manager. [docs]dagster-airflow integration library was bumped to 1.x.x, with that major bump the library has been refocused on enabling migration from Airflow to Dagster. Refer to the docs for an in-depth migration guide.create_databricks_run_now_op), as well as submitting one-off Databricks jobs (create_databricks_submit_run_op).create_databricks_job_op op factory is now deprecated.Stay tuned, as this is only the first part of the overhaul. We’ll be adding more chapters - including automating materializations, using resources, using I/O managers, and more - in the next few weeks.
@graph_asset and @graph_multi_asset.project_fully_featured example now uses the built in DuckDB and Snowflake I/O managers.make_persistent_airflow_db_resource this enables support for Airflow features like pools and cross-dagrun state sharing. In particular retry-from-failure now works for jobs generated from Airflow DAGs.BigQueryPandasTypeHandler now uses google.bigquery.Client methods load_table_from_dataframe and query rather than the pandas_gbq library to store and fetch DataFrames.args instead of both command and args for user code deployments, allowing to include a custom ENTRYPOINT in your the Dockerfile that loads your code.protobuf<4 pin in Dagster has been removed. Installing either protobuf 3 or protobuf 4 will both work with Dagster.@graph_asset and @graph_multi_asset now support passing metadata (thanks @askvinni)!@graph_asset and @graph_multi_asset to be ignored.TableRecord.@multi_asset and other functions would register as type errors for mypy and other static analyzers.start_date would not be converted correctly causing runs to fail.dagster instance migrate:
id column to the kvs, daemon_heartbeats and instance_info tables, enforcing that all tables have a primary key.The minimum grpcio version supported by Dagster has been increased to 1.44.0 so that Dagster can support both protobuf 3 and protobuf 4. Similarly, the minimum protobuf version supported by Dagster has been increased to 3.20.0. We are working closely with the gRPC team on resolving the upstream issues keeping the upper-bound grpcio pin in place in Dagster, and hope to be able to remove it very soon.
Prior to 0.9.19, asset keys were serialized in a legacy format. This release removes support for querying asset events serialized with this legacy format. Contact #dagster-support for tooling to migrate legacy events to the supported version. Users who began using assets after 0.9.19 will not be affected by this change.
[dagster-snowflake] The execute_queryand execute_queries methods of the SnowflakeResource now have consistent behavior based on the values of the fetch_results and use_pandas_result parameters. If fetch_results is True, the standard Snowflake result will be returned. If fetch_results and use_pandas_result are True, a pandas DataFrame will be returned. If fetch_results is False and use_pandas_result is True, an error will be raised. If both are False, no result will be returned.
[dagster-snowflake] The execute_queries command now returns a list of DataFrames when use_pandas_result is True, rather than appending the results of each query to a single DataFrame.
[dagster-shell] The default behavior of the execute and execute_shell_command functions is now to include any environment variables in the calling op. To restore the previous behavior, you can pass in env={} to these functions.
[dagster-k8s] Several Dagster features that were previously disabled by default in the Dagster Helm chart are now enabled by default. These features are:
dagster/max_retries tag set. You can configure a global number of retries in the Helm chart by setting run_retries.max_retries to a value greater than the default of 0.Each of these features can be disabled in the Helm chart to restore the previous behavior.
[dagster-k8s] The experimental k8s_job_op op and execute_k8s_job functions no longer automatically include configuration from a dagster-k8s/config tag on the Dagster job in the launched Kubernetes job. To include raw Kubernetes configuration in a k8s_job_op, you can set the container_config, pod_template_spec_metadata, pod_spec_config, or job_metadata config fields on the k8s_job_op (or arguments to the execute_k8s_job function).
[dagster-databricks] The integration has now been refactored to support the official Databricks API.
create_databricks_job_op is now deprecated. To submit one-off runs of Databricks tasks, you must now use the create_databricks_submit_run_op.databricks_client resource must now begin with https://.LogicalVersion has been renamed to DataVersion and LogicalVersionProvenance has been renamed to DataProvenance.DynamicPartitionsDefinition to add, remove, and check for existence of partitions have been removed. Refer to documentation for updated API methods.MetadataEntry have been removed.DagsterTypeMaterializer, DagsterTypeMaterializerContext, and @dagster_type_materializer have been removed.PartitionScheduleDefinition has been removed.RunRecord.pipeline_run has been removed (use RunRecord.dagster_run).DependencyDefinition.solid has been removed (use DependencyDefinition.node).pipeline_run argument to build_resources has been removed (use dagster_run)iteritems usage was removed and changed to the recommended items within dagster-snowflake-pandas (thanks @sethkimmel3)!@asset_graph decorator (thanks @simonvanderveldt)!DataVersions can now be returned on OutputOpExecutionContext.get_asset_provenancePicking a github issue was not correctly rendering, this has been fixed (thanks @Sedosa)!Further performance improvements for build_asset_reconciliation_sensor.
build_asset_reconciliation_sensor.TimeWindowPartitionMappings with offsets. (Thanks Sean Han!)runtimePlatform field for the task definitions of the runs that it launches, allowing it to launch runs using Windows Docker images.create_databricks_run_now_op), as well as submitting one-off Databricks jobs (create_databricks_submit_run_op). See the new Databricks guide for more details.DuckDBPolarsTypeHandler for use with build_duckdb_io_manager, which allows loading / storing Polars DataFrames from/to DuckDB. (Thanks Pezhman Zarabadi-Poor!)bigquery_pyspark_io_manager.DataFrame.AssetSelection which matched no assets was passed into define_asset_job, the resulting job would target all assets in the repository. This has been fixed.build_schedule_from_partitioned_job.fs_io_manager to store multipartitioned materializations in directory levels by dimension. This resolves a bug on windows where multipartitioned materializations could not be stored with the fs_io_manager.context.partition_key would raise an error when executing on a partition range within a single run via Dagit.Definitions reloaded messages on fresh page loads.location information.ProgrammingError was imported from the wrong library, this has been fixed. Thanks @herbert-allium!Output objects rather than using Dagster’s auto-generated versions.get_asset_provenance method on OpExecutionContext allows fetching logical version provenance for an arbitrary asset key.examples/assets_dynamic_partitions folderhttps://github.com/dagster-io/dagster/compare/1.1.20...1.1.21
<details><summary>See All Contributors</summary>
dagster-census api docs (#12413) by @yuhangraph_asset and graph_multi_asset decorators (#10152) by @sryza[dagster-snowflake-pyspark] fix bug loading partitions (#12472) by @jamiedemariaadd graphql fields for querying run tags (#12409) by @prhaAdd stale status causes (#11953) by @smackeseyfix (#12477) by @salazarmUpdate Contributing doc with instructions for ruff/pyright (#12481) by @smackesey[bigquery] mark bigquery io manager experimental (#12479) by @jamiedemariaadd partial tag autocomplete for run filter input (#12410) by @prhaSupport env valueFrom in Helm chart (#12425) by @johannkmUpdate GQL to expose StaleStatus and StaleStatusCause (#11952) by @smackeseyremove timestamp comparisons of code location entries to reduce OSS dagit replica spam (#12407) by @prhaFix state status logical version test (#12484) by @smackeseyfix ruff (#12486) by @alangenfelduse opt_nullable_mapping for dagster library versions (#12487) by @alangenfeldclarify error when op is missing argument for In (#12456) by @sryzaRemove existing RunConfig class (#12488) by @benpankow[pythonic resources] Clean up initialization of env vars, treat resource objects as immutable (#12445) by @benpankow[structured config] Add support for Selectors w/ pydantic discriminated unions (#11280) by @benpankowAllow setting logical version inside op (#12189) by @smackeseyReplace usages of nslookupwithnc for user deployments (#11033) by @michaeljguarinoRevert "Replace usages of nslookupwithnc for user deployments (#11033)" by @johannkm[pythonic resources] Last set of class renames (#12490) by @benpankow[dagster-azure] Add support for DefaultAzureCredential for adls2_resource (#11309) by @mpicardAdd example of customizing task role and execution role arn to the ECS agent docs (#12491) by @gibsondanadd dagster-duckdb-polars library (#12197) by @pzarabadip[draft][pythonic config][docs] Introduce intro to Resources doc utilizing Pythonic resources (#12260) by @benpankow[pythonic config] Add structured RunConfig object for specifying runtime, job config (#11965) by @benpankow[draft][pythonic config][docs] Introduce intro to Config doc utilizing Pythonic config (#12349) by @benpankow1.1.20 changelog (#12506) by @benpankowrefactor(databricks): lift polling methods up to the client (#12382) by @rexledesma[fix] fix sphinx airflow version parsing (#12507) by @benpankowAutomation: versioned docs for 1.1.20 by @elementl-devtools[dagit] Expose range-based asset health, use it for partition status rendering (#12302) by @bengotow[dagit] Use range-based asset health for asset partitions / job partitions pages (#12434) by @bengotowFix 1.1.20 changelog codeblock (#12525) by @johannkm[dagit] Delete generated GraphQL types before regenerating (#12518) by @hellendag[refactor] Delete build_solid_context (#12513) by @smackesey[typing/static] serdes (#12522) by @smackesey[refactor] make ResolvedRunConfig.to_dict use "ops" (#12514) by @smackeseyImprove multipartition performance for get_partition (#12431) by @clairelin135feat(databricks): override user agent in resource (#12526) by @rexledesmaBump typing-extensions dep to >=4.4.0 (#12529) by @smackesey[docs] Snowflake reference page fixes (#12455) by @jamiedemaria[bugfix] Make assets downstream of partitions never stale in dagit (#12528) by @smackeseyEnable internal testing for writing asset cached status data (#12497) by @clairelin135[typing] fix typing in daemon tests (#12475) by @dpeng817dynamic partitions toy (#12533) by @sryzapare down multi-partition runtime type checking in upath IO manager (#12508) by @sryza[dagit] Upgrade to Jest 29, allow more time for coverage collection (#12534) by @bengotowAdd description to invariant check (#12496) by @CodeMySky[fix] Correctly resolve asset jobs with empty selections (#12531) by @OwenKephart[dagit] Support backfills on partition-mapped asset selections (#12458) by @bengotow[dagit] Remove unnecessary usage of <TestProvider> (#12519) by @bengotowFix multipartitions w/ fs_io_manager on windows (#12414) by @clairelin135Fix master (#12550) by @clairelin135Booleans that check permissions for specific objects and automatically incorporate per-location checks (#12548) by @gibsondandocs(dagster-dbt): add example to retrieve asset keys from dbt selection (#12323) by @rexledesmaFix IO manager doc snippets (#12545) by @clairelin135fix setup.cfg in create package (#12559) by @jamiedemariaFetch multipartition key from context methods (#12512) by @clairelin135Include partition in asset materialization planned event (#12333) by @johannkmhandle multiple non-time-window partitions in db io managers (#12517) by @sryzaAdd runtime_platform to EcsRunLauncher (#12566) by @gibsondanTag Selector Component (#12564) by @salazarmStorybook for LaunchAssetChoosePartitionsDialog (#12567) by @salazarmUpdate setting-up-alerts.mdx (#12582) by @johannkmMove useViewport to uiformcore (#12578) by @salazarm[ui] Always show "Reload all" button (#12588) by @hellendagBump minimist from 1.2.5 to 1.2.8 in /js_modules/dagit (#12595) by @dependabot[bot]Expose partition definition name (#12585) by @salazarmadd dynamic partitions example (#12494) by @sryza[dagit] Do not capitalize tag keys in the Dagit op metadata panel (#12591) by @bengotow[graphql] empty numPartitions and partitionNames for cross-partitioning backfills (#12462) by @sryza[dagit] Use last path component of linked assets on the asset graph + add tooltips (#12590) by @bengotow[dagster-snowflake] Fix ProgrammingError import in snowflake_io_manager (#12576) by @herbert-alliumremove dagster-cloud dep from dynamic partitions example (#12603) by @sryzafix future ticks for partitioned schedules (#12601) by @sryzafix loading partitioned source assets in non-partitioned non-asset jobs (#12586) by @sryzaUpdate stale reasons to remove UNKNOWN, add MISSING (#12584) by @smackeseyGQL Add dynamic partition mutation (#12562) by @clairelin135Add asset provenance access to OpExecutionContext (#12467) by @smackeseyfeat(databricks): add basic op implementations (#12492) by @rexledesmadocs(databricks): add guide for Databricks integration (#12326) by @rexledesmamigrate some tests away from AssetGroup (#12606) by @sryzadivest permissions for editing sensor from updating sensor cursor (#12589) by @dpeng817[dagit] Fix tests pinned to February (#12616) by @bengotowMove tests, fixtures, and storybooks into folders (#12614) by @salazarmAdd root stale causes for assets (#12619) by @smackeseydocs for graphs that depend on assets (#12597) by @sryza[UI] Dynamic partition creation and selection (#12615) by @salazarm[asset-reconciliation][perf] Do not get/set known_used_data in asset reconciliation loop (#12433) by @OwenKephart[dagster-gcp-pyspark] Add BigQuery PySpark type handler (#12398) by @jamiedemariaAutomating pipeline guide (#12547) by @odette-elementlCapture db timeout errors for tag key queries (#12596) by @prhaGet partitions with planned but not completed materializations, the event log query (#12404) by @johannkmAdd failures to partition status cache (#12599) by @johannkm[bigquery-pandas] pipe gcp location through to type handler (#12587) by @jamiedemaria[docs, bigquery] tutorial and reference guide (#12452) by @jamiedemariamake freshness policies work with graph_asset and graph_multi_asset (#12630) by @sryzaRevert "Add failures to partition status cache (#12599)" by @johannkmRevert "Get partitions with planned but not completed materializations, the event log query (#12404)" by @johannkmModify get_first_partition_window to account for offset (#12504) by @clairelin135[dagit] Switch to staleStatus for stale tags, display causes in tooltips (#12611) by @bengotowChange format of staleness root causes and remove old resolvers (#12628) by @smackeseyHelm: Support valueFrom in user deployments (#12644) by @johannkmUpdate asset versioning guide (#12659) by @smackeseyupdate CHANGES.md for 1.1.21 (#12662) by @sryzadocs(databricks): update guide based on feedback (#12669) by @rexledesma1.1.21 by @elementl-devtools</details>
ad6f84f - document missing breaking change in 1.1.19 changelog (#12424) by @sryza
The new @graph_asset and @graph_multi_asset decorators make it more ergonomic to define graph-backed assets.
Dagster will auto-infer dependency relationships between single-dimensionally partitioned assets and multipartitioned assets, when the single-dimensional partitions definition is a dimension of the MultiPartitionsDefinition.
A new Test sensor / Test schedule button that allows you to perform a dry-run of your sensor / schedule. Check out the docs on this functionality here for sensors and here for schedules.
[dagit] Added (back) tag autocompletion in the runs filter, now with improved query performance.
[dagit] The Dagster libraries and their versions that were used when loading definitions can now be viewed in the actions menu for each code location.
New bigquery_pandas_io_manager can store and load Pandas dataframes in BigQuery.
[dagster-snowflake, dagster-duckdb] SnowflakeIOManagers and DuckDBIOManagers can now default to loading inputs as a specified type if a type annotation does not exist for the input.
[dagster-dbt] Added the ability to use the “state:” selector
[dagster-k8s] The Helm chart now supports the full kubernetes env var spec for Dagit and the Daemon. E.g.
dagit:
env:
- name: “FOO”
valueFrom:
fieldRef:
fieldPath: metadata.uid
ruff, pyright).https://github.com/dagster-io/dagster/compare/1.1.19...1.1.20
<details><summary>See All Contributors</summary>
disable check_same_thread on in-memory sqlite storage (#12229) by @alangenfeld[direct invoke] yield implicit Nothing Output (#12309) by @alangenfeldFix multipartitions run length encoding error (#12329) by @clairelin135[freshness-policies] Allow setting freshness policies when using graph-backed assets (#12357) by @OwenKephartAdd skip reason to asset sensor (#12343) by @OwenKephartFix partitions backfill deserialization error (#12238) by @clairelin135Move CachingRepositoryData.from_list and from_dict into standalone function (#12321) by @schrocknrefactor(databricks): divest from databricks_api in favor of databricks-cli (#12153) by @rexledesma nullsafe array index access (#12362) by @salazarmChange schedule button text (#12361) by @dpeng817[dagit] add full serialized error to graphql errors (#12228) by @alangenfeld[dagster-pandas][dagster-pandera] assign a typing_type for generated pandas dataframe DagsterTypes (#12363) by @OwenKephart[typing/static] Execution API types (#12330) by @smackesey[refactor] DependencyDefinition renames (#12338) by @smackesey[refactor] execute_step renames (#12354) by @smackesey[typing/runtime] Standardize StepInputSource.load_input_object (#12342) by @smackesey[refactor] Delete NodeInput.solid_name (#12339) by @smackesey[refactor] NodeDefiniton.iterate_solid_defs -> iterate_op_defs (#12336) by @smackesey[refactor] GraphDefinition method renames (#12335) by @smackesey[2/n][structured config] Enable struct config resources, IO managers to depend on other resources (#11645) by @benpankow[refactor] local var/private arg solid -> node (#12337) by @smackeseyDagsterLibraryRegistry (#12266) by @alangenfeld[refactor] misc core solid -> node renames (#12368) by @smackesey[refactor] dagster._core.definitions.solid_container -> node_container (#12369) by @smackesey[refactor] Assorted local var solid -> node (#12370) by @smackeseyFix intermittent dynamic partitions table SQLite concurrency error (#12367) by @clairelin135change storage signature for run tags (#12348) by @prha1.1.19 Changelog (#12378) by @OwenKephartguide to how assets relate to ops and graphs (#12204) by @sryza[pythonic config] Rename pythonic config classes (#12235) by @benpankowupdates tests to handle new kubernetes resources field (#12395) by @alangenfeld[structured config] Migrate resources from project-fully-featured to struct config resources (#11785) by @benpankowrefactor(databricks): add types to databricks.py (#12364) by @rexledesmarefactor(databricks): consolidate types (#12366) by @rexledesmaadd dagster_libraries to ListRepositoriesResponse (#12267) by @alangenfeld[graphql] add RepositoryLocation.dagsterLibraryVersions (#12268) by @alangenfeld[dagit] add dagster libraries menu to code location row (#12315) by @alangenfeld1.1.19 changelog: reorder code block (#12402) by @yuhanrefactor(databricks): use databricks_cli's raw api client (#12377) by @rexledesma[docs] [snowflake] Add partitions to snowflake guide (#12231) by @jamiedemariaAdd valid start time check to materialized time partitions subsets (#12403) by @clairelin135Add api docs for some PartitionsDefinition and PartitionMapping classes (#12365) by @sryzaAdd text to timestamp dropdown (#12379) by @dpeng817[refactor] IExecutionStep.solid_handle -> node_handle (#12371) by @smackesey[asset-reconciliation] Factor in more run statuses (#12412) by @OwenKephart[refactor] *ExecutionContext.solid_config -> op_config (#12372) by @smackesey[refactor] ResolvedRunConfig.solids -> ops (#12373) by @smackeseyAutomation: versioned docs for 1.1.19 by @elementl-devtools[refactor] assorted Dagstermill renames (#12380) by @smackesey[refactor] Context solid renames (#12374) by @smackeseylambda_solid -> solid (#10816) by @smackesey[refactor] Assorted pipeline_run -> dagster_run (#12383) by @smackeseyMake a script to template out new dagster packages (#12389) by @jamiedemaria[library template] add registry call (#12418) by @alangenfeld[db io managers] connection refactor (#12258) by @jamiedemariaCode location alerting docs (#12411) by @dpeng817[refactor] remove @solid decorator (#10952) by @smackesey[refactor] Delete PipelineRunsFilter (#12384) by @smackesey[db io managers] add default_load_type (#12356) by @jamiedemaria[refactor] Delete RunRecord.pipeline_run (#12385) by @smackesey[refactor] pipeline_run_from_storage -> dagster_run_from_storage (#12386) by @smackesey[Docs RFC] Dynamic Partitions (#12227) by @clairelin135Auto infer multipartition <-> single dimension mapping (#12400) by @clairelin135[refactor] execution pipeline_run -> dagster_run (#12388) by @smackesey[test-api-update] execution_tests/dynamic_tests (#12427) by @smackeseyConsider the run worker unhealthy is the job has no active pods but the run is in a non-terminal state (#11510) by @gibsondanfix: only inspect schema when we may create tables (#12269) by @plaflammeMore helpful asset key mismatch errors (#12008) by @benpankowAdd docs for testing schedules/sensors via UI (#12381) by @dpeng817document missing breaking change in 1.1.19 changelog (#12424) by @sryzaBigQuery IO manager (#11425) by @jamiedemaria[dagster-gcp-pandas] API docs fix (#12450) by @jamiedemaria[graphql] launch backfills over assets with different partitionings, if all roots have same partitioning (#11827) by @sryzaFix resolution error with input manager key and custom dagster type (#12449) by @clairelin135[dagster-dbt] Add ability to use the "state:" selector (#12432) by @OwenKephartRevert "More helpful asset key mismatch errors (#12008)" (#12459) by @benpankow[dagit] Add missing React keys to prevent new warning toasts (#12210) by @bengotow[CustomConfirmationDialog] Allow overriding the button text (#12444) by @salazarmadd another todo to create_dagster_package (#12453) by @jamiedemaria[dagster-gcp-pandas] register library in init (#12469) by @jamiedemaria[bugfix] fix projected logical version resolution for asset downstream of self-dep (#12443) by @smackeseyfix: resolve correct legacy arguments for emr pyspark step launcher (#12419) by @rexledesma[dagster-dbt] Add missing files for tests (#12471) by @OwenKephartAdd task_role_arn and execution_role_arn to EcsContainerContext (#12358) by @gibsondandagster-census api docs (#12413) by @yuhangraph_asset and graph_multi_asset decorators (#10152) by @sryza[dagster-snowflake-pyspark] fix bug loading partitions (#12472) by @jamiedemariaAdd stale status causes (#11953) by @smackeseyUpdate Contributing doc with instructions for ruff/pyright (#12481) by @smackesey[bigquery] mark bigquery io manager experimental (#12479) by @jamiedemariafix (#12477) by @salazarmadd graphql fields for querying run tags (#12409) by @prhaadd partial tag autocomplete for run filter input (#12410) by @prhaUpdate GQL to expose StaleStatus and StaleStatusCause (#11952) by @smackeseySupport env valueFrom in Helm chart (#12425) by @johannkmFix state status logical version test (#12484) by @smackeseyuse opt_nullable_mapping for dagster library versions (#12487) by @alangenfeldfix ruff (#12486) by @alangenfeld[pythonic resources] Clean up initialization of env vars, treat resource objects as immutable (#12445) by @benpankow[structured config] Add support for Selectors w/ pydantic discriminated unions (#11280) by @benpankow[pythonic resources] Last set of class renames (#12490) by @benpankowAllow setting logical version inside op (#12189) by @smackeseyAdd example of customizing task role and execution role arn to the ECS agent docs (#12491) by @gibsondanRemove existing RunConfig class (#12488) by @benpankow[pythonic config] Add structured RunConfig object for specifying runtime, job config (#11965) by @benpankow[draft][pythonic config][docs] Introduce intro to Resources doc utilizing Pythonic resources (#12260) by @benpankow[draft][pythonic config][docs] Introduce intro to Config doc utilizing Pythonic config (#12349) by @benpankow1.1.20 changelog (#12506) by @benpankow[fix] fix sphinx airflow version parsing (#12507) by @benpankow1.1.20 by @elementl-devtools</details>
@multi_asset no longer accepts Out values in the dictionary passed to its outs argument. This was experimental and deprecated. Instead, use AssetOut.
FreshnessPolicy object now supports a cron_schedule_timezone argument.AssetsDefinition.from_graph now supports a freshness_policies_by_output_name parameter.@asset_sensor will now display an informative SkipReason when no new materializations have been created since the last sensor tick.AssetsDefinition now has a to_source_asset method, which returns a representation of this asset as a SourceAsset.from dagster import asset, job, op
@asset
def emails_to_send():
...
@op
def send_emails(emails) -> None:
...
@job
def send_emails_job():
send_emails(emails_to_send.to_source_asset())
--dagit-host/-h argument to the dagster dev command to allow customization of the host where Dagit runs.input_manager_key to an asset’s input, incorrect resource config could be used when loading that input. This has been fixed.runK8sConfig in the Dagster Helm chart would not pass configuration through to pods launched using the k8s_job_executor.execute_k8s_job op downstream of a dynamic output would result in k8s jobs with duplicate names being created. This has been fixed.asset_key, asset_partitions, and asset_partitions_defs arguments on Out.@multi_asset no longer accepts Out values in the dictionary passed to its outs argument. This was experimental and deprecated. Instead, use AssetOut.top_level_resources argument to the repository decorator has been renamed to _top_level_resources to emphasize that it should not be set manually.load_asset_values now accepts resource configuration (thanks @Nintorac!)UPathIOManager, paths with the "." character in them would be incorrectly truncated, which could result in multiple distinct objects being written to the same path. This has been fixed. (Thanks @spenczar!)https://github.com/dagster-io/dagster/compare/1.1.18...1.1.19
<details><summary>See All Contributors</summary>
feat(dagster-dbt): support dbt-core 1.4.x (#11902) by @rexledesmaFixes/improvements for pyright script (#12175) by @smackeseynux examples in oss 1/: add dagster_cloud.yaml to prep cloud nux onboarding (#12172) by @yuhan[dagster-dbt] [1/2] Add streaming entrypoint for dbt cli execution (#12086) by @OwenKephartFix import (#12185) by @OwenKephart(tick-testing 3/6) type annotations to schedules gql (#12057) by @dpeng817(tick-testing 4/6) mutation to dry-run sensor (#11616) by @dpeng817[db io managers] support static partitions (#12129) by @jamiedemariaPass correct config to io manager when using input_manager_key (#12053) by @jamiedemaria[dagster-dbt] [2/2] Add ability to stream events while executing dbt assets (#12100) by @OwenKephartAgent downtime alert docs (#12186) by @johannkm(tick-testing 5/6) Add SensorType (#12021) by @dpeng817(tick-testing 6/6) Add dry run mutations for schedules (#11869) by @dpeng817[dagit] Add some icons for Cloud (#12187) by @hellendag[1/n][structured config] Add ability to runtime-configure struct-config resources (#11773) by @benpankowfix pyright (#12193) by @jamiedemaria[dagit] Storybooks for asset partition and event details + dayJS fix (#12178) by @bengotow[dagster-wandb] Integration with Weights & Biases (#10470) by @chrishisteStore the instance ref on the grpc server class if it's present, so that grpc api calls can use it if it's there (#12194) by @gibsondanW&B integration follow up 2/ update example to defs (#12059) by @yuhannux examples in oss 2/: add required env vars in README's front matter (#12154) by @yuhanSensor testing UI (#12148) by @salazarmvalidate time partition keys when adding to a TimeWindowPartitionsSubset (#12195) by @OwenKephartMake Partition GRPC calls with instance_ref args optional (#12196) by @clairelin135Add public api doc for create_repository_using_definitions_args, update typehints, and cleanup Definitions docstring (#12176) by @schrockn[dagster-airflow] [docs] migration guide updates/considerations (#12198) by @Ramshackle-Jamathon[dagster-airflow] refactor airflow_db resources (#12202) by @Ramshackle-JamathonFix alert docs (#12208) by @johannkmTemporarily disable pyright in BK (#12212) by @smackeseyRename "Loading multiple repositories" section in workspace files docs (#12201) by @gibsondanMake top_level_resources argument _top_level_resources (#12211) by @schrocknRevert "nux examples in oss 2/: add required env vars in README's front matter" (#12199) by @yuhanPyright config/script fixes and improvements (#12206) by @smackeseyAdd guide for observable source assets and versioning (#12118) by @schrockn1.1.18 Changelog (#12217) by @smackeseyFix dagster-wandb placeholder (#12222) by @smackeseyAdd "--dagit-host" arg to dagster dev (#12220) by @gibsondandiff against origin/master in quick_pyright (#12200) by @sryzaRe-enable pyright in BK (#12214) by @smackeseyRemove restart-on-failure from local docker agent guide (#12224) by @gibsondan[dagit] Storybooks for MetadataEntry rendering (#12177) by @bengotowAdd airbyte guide to experimental guide index (#12207) by @smackeseyAutomation: versioned docs for 1.1.18 by @elementl-devtoolsSchedules Testing UI (#12160) by @salazarmKeep dots in paths in UPathIOManager (#12174) by @spenczarremove Out.asset_key (#12221) by @sryza[structured config] Fix env vars not working with direct Resource instantiation (#11468) by @benpankowallow using SourceAssets to satisfy node inputs in non-asset jobs (#12091) by @sryzaAdd observe function (source asset analogue to materialize) (#11996) by @smackeseytreat pyright warnings as errors for buildkite purposes (#12239) by @gibsondanUpdate serverless docs with new run isolation default (#12225) by @johannkmdocs(dbt-cloud): add instructions to cache dbt Cloud compilation (#11793) by @rexledesmaAllow dagster api grpc --max-workers (#12246) by @smackeseyAllow Ursula to test dynamic partition writes (#12215) by @clairelin135[docs] fix dbt tutorial error (#12255) by @jamiedemaria[docs] - Correct name of freshness sensor context object (#12250) by @erinkcochran87document system tags on metadata/tags page (#12256) by @sryza[dagster-airflow] remove ref to cli (#12262) by @Ramshackle-JamathonRevamp intro of partitions concepts page (#12257) by @sryzaRemove @experimental from build_assets_job and build_source_asset_observation_job (#12252) by @smackesey[fix] Surface the correct description for multi-assets when a description is provided for the op (#12271) by @OwenKephartremove legacy APIs from dagster-k8s tests (#12020) by @sryzaUse Definitions in the body of materialize functions (and other public-facing execution functions) (#12038) by @schrockn[typing/static] assets (#12003) by @smackesey[typing/runtime] Make GraphDefinition.create_adjacency_lists internally public (#12291) by @smackesey[typing/runtime] add AssetsDefinition.partition_mappings (#12290) by @smackeseyin docs navigation, make partitions independent from schedules (#12264) by @sryza[typing/static] storage (#12283) by @smackesey[api docs] Show PartitionsDefinition doc strings on docs site (#12265) by @jamiedemaria[db io manager] Support MultiPartitions (#12165) by @jamiedemariaload asset value config (#10991) by @Nintoracremove some handling for solid in composition.py (#12278) by @sryzaAdd shortcuts for '--dagit-host' and '--dagit-port' to dagster dev (#12301) by @gibsondan[typing/runtime] [gql] eliminate resolver **kwargs (2) (#12241) by @smackesey[typing/runtime] [gql] eliminate resolver **kwargs (4) (#12243) by @smackesey[typing/runtime] [gql] eliminate resolver **kwargs (5) (#12244) by @smackesey[Runs Table] Open link in new tab (#12307) by @salazarm[typing/runtime] [gql] eliminate resolver **kwargs (3) (#12242) by @smackeseyadd types and break up big functions in dagstermill impl (#12274) by @sryza[typing/runtime] [gql] eliminate resolver **kwargs (1) (#11724) by @smackeseyadd cloud to scaffold (#12306) by @slopp[typing/runtime] Remove unnecessary isinstance checks (#12287) by @smackesey[typing/runtime] Make AssetKey.to_string always return AssetKey (#12285) by @smackesey[typing/runtime] Refactor cached_method (#12004) by @smackeseyRemove mypy from tox files (#12112) by @smackesey[typing/runtime] Asset graph comparisons (#12286) by @smackeseyrefactor(databricks): remove legacy Dagster definitions (#12150) by @rexledesmadocs: update scaffold to use dagster dev (#12297) by @rexledesma[db io managers] create schema if not exist (#11764) by @jamiedemariaAssetsDefinition.to_source_asset (#12203) by @sryzain tags doc, fix schedule tag and add sensor tag (#12313) by @sryzaSimplify buildkite-build-test-project-image (#12029) by @gibsondanMove repository_definition into subpackage (#12318) by @schrocknBreak up repository_definition subpackage into multiple files (#12319) by @schrockn[docs] - Build on local Dagster guide (#12168) by @erinkcochran87[freshness-policies] add a cron_schedule_timezone argument to the FreshnessPolicy class (#12263) by @OwenKephart[typing/runtime] Refactor subset selection tree (#12289) by @smackeseyFix unimported symbol BK (#12327) by @smackeseynux examples in oss 3/: sync dagster-io/quickstart-*/setup.py to example/* (#12234) by @yuhan[dagster-k8s] In the execute_k8s_job op, use step key to generate the k8s job name (#12344) by @OwenKephartInclude instance-level / code-location-level runK8sConfig in step pods (#12308) by @gibsondanremove unused lazy-repository docs snippet (#12277) by @sryza[asset-reconciliation][perf] Perf regression tests (#12230) by @OwenKepharts/_CacheingDefinitionIndex/CacheingDefinitionIndex/g (#12320) by @schrockn[typing/runtime] Massage set flattening (#12288) by @smackesey[asset-reconciliation][perf] Better caching of most recent materializations (#12237) by @OwenKephart[db io managers] Dynamic Partition tests (#12216) by @jamiedemariabump limit (#12355) by @OwenKephartFix multipartitions run length encoding error (#12329) by @clairelin135[freshness-policies] Allow setting freshness policies when using graph-backed assets (#12357) by @OwenKephart nullsafe array index access (#12362) by @salazarmChange schedule button text (#12361) by @dpeng817[dagster-pandas][dagster-pandera] assign a typing_type for generated pandas dataframe DagsterTypes (#12363) by @OwenKephartFix partitions backfill deserialization error (#12238) by @clairelin1351.1.19 Changelog (#12378) by @OwenKephartguide to how assets relate to ops and graphs (#12204) by @sryzaupdates tests to handle new kubernetes resources field (#12395) by @alangenfeld1.1.19 changelog: reorder code block (#12402) by @yuhan1.1.19 by @elementl-devtools</details>
Your coding agent can read these notes before it upgrades. Set up the MCP server →