dagster-webserver

Bugfixes

• Previously, asset backfills targeting assets with multi-run backfill policies would raise a "did not submit all run requests" error. This has been fixed.

Dagster Cloud

• The experimental dagster-insights package has receieved some API surface area updates and bugfixes.

New

• Dagster now automatically infers a dependency relationship between a time-partitioned asset and a multi-partitioned asset with a time dimension. Previously, this was only inferred when the time dimension was the same in each asset.
• The EnvVar utility will now raise an exception if it is used outside of the context of a Dagster resource or config class. The get_value() utility will retrieve the value outside of this context.
• [ui] The runs page now displays a “terminate all” button at the top, to bulk terminate in-progress runs.
• [ui] Asset Graph - Various performance improvements that make navigating large asset graphs smooth
• [ui] Asset Graph - The graph now only fetches data for assets within the viewport solving timeout issues with large asset graphs
• [ui] Asset Graph Sidebar - The sidebar now shows asset status
• [dagster-dbt] When executing dbt invocations using DbtCliResource, an explicit target_path can now be specified.
• [dagster-dbt] Asset checks can now be enabled by using DagsterDbtTranslator and DagsterDbtTranslatorSettings: see the docs for more information.
• [dagster-embedded-elt] Dagster library for embedded ELT

Bugfixes

• [ui] Fixed various issues on the asset details page where partition names would overflow outside their containers
• [ui] Backfill notification - Fixed an issue where the backfill link didn’t take the —path-prefix option into account
• [ui] Fixed an issue where the instance configuration yaml would persist rendering even after navigating away from the page.
• [ui] Fixed issues where config yaml displays could not be scrolled.
• [dagster-webserver] Fixed a performance issue that caused the UI to load slowly

Deprecations

• [dagster-dbt] Enabling asset checks using dbt project metadata has been deprecated.

Major Changes since 1.4.0 (core) / 0.20.0 (libraries)

Core

• Improved ergonomics for execution dependencies in assets  - We introduced a set of APIs to simplify working with Dagster that don't use the I/O manager system for handling data between assets. I/O manager workflows will not be affected.

  • AssetDep type allows you to specify upstream dependencies with partition mappings when using the deps parameter of @asset and AssetSpec.
  • MaterializeResult can be optionally returned from an asset to report metadata about the asset when the asset handles any storage requirements within the function body and does not use an I/O manager.
  • AssetSpec has been added as a new way to declare the assets produced by @multi_asset. When using , the does not need to return any values to be stored by the I/O manager. Instead, the should handle any storage requirements in the body of the function.

• Asset checks (experimental) - You can now define, execute, and monitor data quality checks in Dagster [docs].

  • The @asset_check decorator, as well as the check_specs argument to @asset and @multi_asset enable defining asset checks.
  • Materializing assets from the UI will default to executing their asset checks. You can also execute individual checks.
  • When viewing an asset in the asset graph or the asset details page, you can see whether its checks have passed, failed, or haven’t run successfully.

• Auto materialize customization (experimental) - AutoMaterializePolicies can now be customized [docs].

  • All policies are composed of a set of AutoMaterializeRules which determine if an asset should be materialized or skipped.
  • To modify the default behavior, rules can be added to or removed from a policy to change the conditions under which assets will be materialized.

dagster-pipes

• Dagster pipes is a new library that implements a protocol for launching compute into external execution environments and consuming streaming logs and Dagster metadata from those environments. See https://github.com/dagster-io/dagster/discussions/16319 for more details on the motivation and vision behind Pipes.
• Out-the-box integrations
  • Clients: local subprocess, Docker containers, Kubernetes, and Databricks
    • PipesSubprocessClient, PipesDocketClient, PipesK8sClient, PipesDatabricksClient
  • Transport: Unix pipes, Filesystem, s3, dbfs
  • Languages: Python
• Dagster pipes is composable with existing launching infrastructure via open_pipes_session. One can augment existing invocations rather than replacing them wholesale.

Since 1.4.17 (core) / 0.20.17 (libraries)

New

• [ui] Global Asset Graph performance improvement - the first time you load the graph it will be cached to disk and any subsequent load of the graph should load instantly.

Bugfixes

• Fixed a bug where deleted runs could retain instance-wide op concurrency slots.

Breaking Changes

• AssetExecutionContext is now a subclass of OpExecutionContext, not a type alias. The code

def my_helper_function(context: AssetExecutionContext):
    ...

@op
def my_op(context: OpExecutionContext):
    my_helper_function(context)

will cause type checking errors. To migrate, update type hints to respect the new subclassing.

• AssetExecutionContext cannot be used as the type annotation for @ops run in @jobs. To migrate, update the type hint in @op to OpExecutionContext. @ops that are used in @graph_assets may still use the AssetExecutionContext type hint.

# old
@op
def my_op(context: AssetExecutionContext):
    ...

# correct
@op
def my_op(context: OpExecutionContext):
    ...

• [ui] We have removed the option to launch an asset backfill as a single run. To achieve this behavior, add backfill_policy=BackfillPolicy.single_run() to your assets.

Community Contributions

• has_dynamic_partition implementation has been optimized. Thanks @edvardlindelof!
• [dagster-airbyte] Added an optional stream_to_asset_map argument to build_airbyte_assets to support the Airbyte prefix setting with special characters. Thanks @chollinger93!
• [dagster-k8s] Moved “labels” to a lower precedence. Thanks @jrouly!
• [dagster-k8s] Improved handling of failed jobs. Thanks @Milias!
• [dagster-databricks] Fixed an issue where DatabricksPysparkStepLauncher fails to get logs when job_run doesn’t have cluster_id at root level. Thanks @PadenZach!
• Docs type fix from @sethusabarish, thank you!

Documentation

• Our Partitions documentation has gotten a facelift! We’ve split the original page into several smaller pages, as follows:
  • Partitions - An overview of what a partition is, benefits, and how to use it
  • Partitioning assets - Details about partitioning assets
  • Partitioning ops - Details about partitioning ops
  • Testing partitions - As described

Dagster Cloud

• New dagster-insights sub-module - We have released an experimental dagster_cloud.dagster_insights module that contains utilities for capturing and submitting external metrics about data operations to Dagster Cloud via an api. Dagster Cloud Insights is a soon-to-be released feature that shows improves visibility into usage and cost metrics such as run duration and Snowflake credits in the Cloud UI.

New

• [dagster-dbt] DbtCliResource now enforces that the current installed version of dbt-core is at least version 1.4.0.
• [dagster-dbt] DbtCliResource now properly respects DBT_TARGET_PATH if it is set by the user. Artifacts from dbt invocations using DbtCliResource will now be placed in unique subdirectories of DBT_TARGET_PATH.

Bugfixes

• When executing a backfill that targets a range of time partitions in a single run, the partition_time_window attribute on OpExecutionContext and AssetExecutionContext now returns the time range, instead of raising an error.
• Fixed an issue where the asset backfill page raised a GraphQL error for backfills that targeted different partitions per-asset.
• Fixed job_name property on the result object of build_hook_context.

Experimental

• AssetSpec has been added as a new way to declare the assets produced by @multi_asset.
• AssetDep type allows you to specify upstream dependencies with partition mappings when using the deps parameter of @asset and AssetSpec.
• [dagster-ext] report_asset_check method added to ExtContext.
• [dagster-ext] ext clients now must use yield from to forward reported materializations and asset check results to Dagster. Results reported from ext that are not yielded will raise an error.

Documentation

• The Dagster UI documentation got an overhaul! We’ve updated all our screenshots and added a number of previously undocumented pages/features, including:
  • The Overview page, aka the Factory Floor
  • Job run compute logs
  • Global asset lineage
  • Overview > Resources
• The Resources documentation has been updated to include additional context about using resources, as well as when to use os.getenv() versus Dagster’s EnvVar.
• Information about custom loggers has been moved from the Loggers documentation to its own page, Custom loggers.

New

• [ui] When using the search input within Overview pages, if the viewer’s code locations have not yet fully loaded into the app, a loading spinner will now appear to indicate that search results are pending.

Bugfixes

• Fixed an asset backfill bug that caused occasionally caused duplicate runs to be kicked off in response to manual runs upstream.
• Fixed an issue where launching a run from the Launchpad that included many assets would sometimes raise an exception when trying to create the tags for the run.
• [ui] Fixed a bug where clicking to view a job from a run could lead to an empty page in situations where the viewer’s code locations had not yet loaded in the app.

Deprecations

• Deprecated ExpectationResult. This will be made irrelevant by upcoming data quality features.

Community Contributions

• Enabled chunked backfill runs to target more than one asset, thanks @ruizh22!

Experimental

• Users can now emit arbitrary asset materializations, observations, and asset check evaluations from sensors via SensorResult.

New

• The deps parameter for @asset and @multi_asset now supports directly passing @multi_asset definitions. If an @multi_asset is passed to deps, dependencies will be created on every asset produced by the @multi_asset.
• Added an optional data migration to convert storage ids to use 64-bit integers instead of 32-bit integers. This will incur some downtime, but may be required for instances that are handling a large number of events. This migration can be invoked using dagster instance migrate --bigint-migration.
• [ui] Dagster now allows you to run asset checks individually.
• [ui] The run list and run details page now show the asset checks targeted by each run.
• [ui] In the runs list, runs launched by schedules or sensors will now have tags that link directly to those schedules or sensors.
• [ui] Clicking the "N assets" tag on a run allows you to navigate to the filtered asset graph as well as view the full list of asset keys.
• [ui] Schedules, sensors, and observable source assets now appear on the resource “Uses” page.
• [dagster-dbt] The DbtCliResource now validates at definition time that its project_dir and profiles_dir arguments are directories that respectively contain a dbt_project.yml and profiles.yml.
• [dagster-databricks] You can now configure a policy_id for new clusters when using the databricks_pyspark_step_launcher (thanks @zyd14!)
• [ui] Added an experimental sidebar to the Asset lineage graph to aid in navigating large graphs. You can enable this feature under user settings.

Bugfixes

• Fixed an issue where the dagster-webserver command was not indicating which port it was using in the command-line output.
• Fixed an issue with the quickstart_gcp example wasn’t setting GCP credentials properly when setting up its IOManager.
• Fixed an issue where the process output for Dagster run and step containers would repeat each log message twice in JSON format when the process finished.
• [ui] Fixed an issue where the config editor failed to load when materializing certain assets.
• [auto-materialize] Previously, rematerializing an old partition of an asset which depended on a prior partition of itself would result in a chain of materializations to propagate that change all the way through to the most recent partition of this asset. To prevent these “slow-motion backfills”, this behavior has been updated such that these updates are no longer propagated.

Experimental

• MaterializeResult has been added as a new return type to be used in @asset / @multi_asset materialization functions
• [ui] The auto-materialize page now properly indicates that the feature is experimental and links to our documentation.

Documentation

• The Concepts category page got a small facelift, to bring it line with how the side navigation is organized.

Dagster Cloud

• Previously, when importing a dbt project in Cloud, naming the code location “dagster” would cause build failures. This is now disabled and an error is now surfaced.

New

• Added a new tooltip to asset runs to either view the asset list or lineage

Bugfixes

• [ui] Fixed an issue where re-executing a run from a particular run's page wouldn’t navigate to the newly created run

Experimental

• [dagster-ext] An initial version of the dagster-ext module along with subprocess, docker, databricks, and k8s pod integrations are now available. Read more at https://github.com/dagster-io/dagster/discussions/16319. Note that the module is temporarily being published to PyPI under dagster-ext-process, but is available in python as import dagster_ext.
• [asset checks] Added an ‘execute’ button to run checks without materializing the asset. Currently this is only supported for checks defined with @asset_check or AssetChecksDefinition.
• [asset checks] Added check_specs argument to @graph_multi_asset
• [asset checks] Fixed a bug with checks on @graph_asset that would raise an error about nonexistant checks

New

• OpExecutionContext.add_output_metadata can now be called multiple times per output.

Bugfixes

• The double evaluation of log messages in sensor logging has been fixed (thanks @janosroden !)
• Cron schedules targeting leap day (ending with 29 2 *) no longer cause exceptions in the UI or daemon.
• Previously, if multiple partitioned observable_source_assets with different partition definitions existed in the same code location, runs targeting those assets could fail to launch. This has been fixed.
• When using AutoMaterializePolicies with assets that depended on prior partitions of themselves, updating the start_date of their underlying PartitionsDefinition could result in runs being launched for partitions that no longer existed. This has been fixed.
• Fixed an issue where auto-materilization could sometimes produce duplicate runs if there was an error in the middle of an auto-materialization tick.
• [dagster-census] A recent change to the Census API broke compatibility with this integration. This has been fixed (thanks @ldnicolasmay!)
• [dagster-dbt] Fixed an issue where DagsterDbtTranslator did not properly invoke get_auto_materialize_policy and get_freshness_policy for load_assets_from_dbt_project.
• [ui] Fixed a number of interaction bugs with the Launchpad config editor, including issues with newlines and multiple cursors.
• [ui] Asset keys and partitions presented in the asset checks UI are sorted to avoid flickering.
• [ui] Backfill actions (terminate backfill runs, cancel backfill submission) are now available from an actions menu on the asset backfill details page.

Community Contributions

• Typo fix in run monitoring docs (thanks c0dk)!
• Grammar fixes in testing docs (thanks sonnyarora)!
• Typo fix in contribution docs (thanks tab1tha)!

Experimental

• [dagster-dbt][asset checks] Added support to model dbt tests as Dagster asset checks.
• [asset checks] Added @graph_asset support. This can be used to implement blocking checks, by raising an exception if the check fails.
• [asset checks] Fixed @multi_asset subsetting, so only checks which target assets in the subset will execute.
• [asset checks] AssetCheckSpecs will now cause an error at definition time if they target an asset other than the one they’re defined on.
• [asset checks] The status of asset checks now appears in the asset graph and asset graph sidebar.

Dagster Cloud

• [Experimental] Added support for freeing global op concurrency slots after runs have finished, using the deployment setting: run_monitoring > free_slots_after_run_end_seconds

New

• The context object now has an asset_key property to get the AssetKey of the current asset.
• Performance improvements to the auto-materialize daemon when running on large asset graphs.
• The dagster dev and dagster-daemon run commands now include a --log-level argument that allows you to customize the logger level threshold.
• [dagster-airbyte] AirbyteResource now includes a poll_interval key that allows you to configure how often it checks an Airbyte sync’s status.

Bugfixes

• Fixed an issue where the dagster scheduler would sometimes raise an error if a schedule set its cron_schedule to a list of strings and also had its default status set to AUTOMATICALLY_RUNNING.
• Fixed an issue where the auto-materialize daemon would sometimes raise a RecursionError when processing asset graphs with long upstream dependency chains.
• [ui] Fixed an issue where the Raw Compute Logs dropdown on the Run page sometimes didn’t show the current step name or properly account for retried steps.

Community Contributions

• [dagster-databricks] Fixed a regression causing DatabricksStepLauncher to fail. Thanks @zyd14!
• Fixed an issue where Dagster raised an exception when combining observable source assets with multiple partitions definitions. Thanks @aroig!
• [dagster-databricks] Added support for client authentication with OAuth. Thanks @zyd14!
• [dagster-databricks] Added support for workspace and volumes init scripts in the databricks client. Thanks @zyd14!
• Fixed a missing import in our docs. Thanks @C0DK!

Experimental

• Asset checks are now displayed in the asset graph and sidebar.

• [Breaking] Asset check severity is now set at runtime on AssetCheckResult instead of in the @asset_check definition. Now you can define one check that either errors or warns depending on your check logic. ERROR severity no longer causes the run to fail. We plan to reintroduce this functionality with a different API.

• [Breaking] @asset_check now requires the asset= argument, even if the asset is passed as an input to the decorated function. Example:

  @asset_check(asset=my_asset)
  def my_check(my_asset) -> AssetCheckResult:
      ...
  

• [Breaking] AssetCheckSpec now takes asset= instead of asset_key=, and can accept either a key or an asset definition.

• [Bugfix] Asset checks now work on assets with key_prefix set.

• [Bugfix] Execution failure asset checks are now displayed correctly on the checks tab.

Documentation

• [dagster-dbt] Added example of invoking DbtCliResource in custom asset/op to API docs.
• [dagster-dbt] Added reference to explain how a dbt manifest can be created at run time or build time.
• [dagster-dbt] Added reference to outline the steps required to deploy a Dagster and dbt project in CI/CD.
• Miscellaneous fixes to broken links and typos.

New

• Dagster code servers now wait to shut down until any calls that they are running have finished, preventing them from stopping while in the middle of executing sensor ticks or other long-running operations.
• The dagster execute job cli now accepts —-op-selection (thanks @silent-lad!)
• [ui] Option (Alt) + R now reloads all code locations (OSS only)

Bugfixes

• Adds a check to validate partition mappings when directly constructing AssetsDefinition instances.
• Assets invoked in composition functions like @graph and @job now work again, fixing a regression introduced in 1.4.5.
• Fixed an issue where a race condition with parallel runs materializing the same asset could cause a run to hang.
• Fixed an issue where including a resource in both a schedule and a job raised a “Cannot specify resource requirements” exception when the definitions were loaded.
• The ins argument to graph_asset is now respected correctly.
• Fixed an issue where the daemon process could sometimes stop with a heartbeat failure when the first sensor it ran took a long time to execute.
• Fixed an issue where dagster dev failed on startup when the DAGSTER_GRPC_PORT `environment variable was set in the environment.
• deps arguments for an asset can now be specified as an iterable instead of a sequence, allowing for sets to be passed.
• [dagster-aws] Fixed a bug where the S3PickleIOManager didn’t correctly handle missing partitions when allow_missing_partitions was set. Thanks @o-sirawat!
• [dagster-k8s] in the helm chart, the daemon securityContext setting now applies correctly to all init containers (thanks @maowerner!)

Community Contributions

• [dagster-databricks] Migrated to use new official databricks Python SDK. Thanks @judahrand!

Experimental

• New APIs for defining and executing checks on software-defined assets. These APIs are very early and subject to change. The corresponding UI has limited functionality. Docs
• Adds a new auto-materialize skip rule AutoMaterializeRule.skip_on_not_all_parents_updated that enforces that an asset can only be materialized if all parents have been materialized since the asset's last materialization.
• Exposed an auto-materialize skip rule – AutoMaterializeRule.skip_on_parent_missing –which is already part of the behavior of the default auto-materialize policy.
• Auto-materialize evaluation history will now be stored for 1 month, instead of 1 week.
• The auto-materialize asset daemon now includes more logs about what it’s doing for each asset in each tick in the Dagster Daemon process output.

Documentation

• [dagster-dbt] Added reference docs for dagster-dbt project scaffold.

Dagster Cloud

• Fixed an issue where the Docker agent would sometimes fail to load code locations with long names with a hostname connection error.

Bugfixes

• [dagster-webserver] Fixed an issue that broke loading static files on Windows.

Bugfixes

• Fixed an issue that caused some missing icons in the UI.

New

• A new @partitioned_config decorator has been added for defined configuration for partitioned jobs. Thanks @danielgafni!
• [dagster-aws] The ConfigurablePickledObjectS3IOManager has been renamed S3PickleIOManager for simplicity. The ConfigurablePickledObjecS3IOManager will continue to be available but is considered deprecated in favor of S3PickleIOManager. There is no change in the functionality of the I/O manager.
• [dagster-azure] The ConfigurablePickledObjectADLS2IOManager has been renamed ADLS2PickleIOManager for simplicity. The ConfigurablePickledObjectADLS2IOManager will continue to be available but is considered deprecated in favor of ADLS2PickleIOManager. There is no change in the functionality of the I/O manager.
• [dagster-dbt] When an exception is raised when invoking a dbt command using DbtCliResource, the exception message now includes a link to the dbt.log produced. This log file can be inspected for debugging.
• [dagster-gcp] The ConfigurablePickledObjectGCSIOManager has been renamed GCSPickleIOManager for simplicity. The ConfigurablePickledObjecGCSIOManager will continue to be available but is considered deprecated in favor of GCSPickleIOManager. There is no change in the functionality of the I/O manager.

Bugfixes

• Fixed a bug that caused a DagsterInvariantViolationError when executing a multi-asset where both assets have self-dependencies on earlier partitions.
• Fixed an asset backfill issue where some runs continue to be submitted after a backfill is requested for cancellation.
• [dagster-dbt] Fixed an issue where using the --debug flag raised an exception in the Dagster framework.
• [ui] “Launched run” and “Launched backfill” toasts in the Dagster UI behave the same way. To open in a new tab, hold the cmd/ctrl key when clicking “View”
• [ui] When opening step compute logs, the view defaults to stderr which aligns with Python’s logging defaults.
• [ui] When viewing a global asset graph with more than 100 assets, the “choose a subset to display” prompt is correctly aligned to the query input.

Community Contributions

• Fix for loading assets with a BackfillPolicy, thanks @ruizh22!

Experimental

• [dagster-graphql] The Dagster GraphQL Python client now includes a default timeout of 300 seconds for each query, to ensure that GraphQL requests don’t hang and never return a response. If you are running a query that is expected to take longer than 300 seconds, you can set the timeout argument when constructing a DagsterGraphQLClient.
• [ui] We are continuing to improve the new horizontal rendering of the asset graph, which you can enable in Settings. This release increases spacing between nodes and improves the traceability of arrows on the graph.

Documentation

• Several Pythonic resources and I/O managers now have API docs entries.
• Updated the tutorial’s example project and content to be more explicit about resources.
• [dagster-dbt] Added API docs examples for DbtCliResource and DbtCliResource.cli(...).
• Some code samples in API docs for InputContext and OutputContext have been fixed. Thanks @Sergey Mezentsev!

Dagster Cloud

• When setting up a new organization by importing a dbt project, using GitLab is now supported.

Experimental

• Added a respect_materialization_data_versions option to auto materialization. It can enabled in dagster.yaml with

  auto_materialize:
      respect_materialization_data_versions: True
  

  This flag may be changed or removed in the near future.

New

• ops or assets with multiple outputs that are all required and return type None/ Nothing will interpret an explicitly or implicitly returned value None to indicate that all outputs were successful.
• The skip_reason argument to the constructor of SensorResult now accepts a string in addition to a SkipReason.
• [dagster-k8s] Added a step_k8s_config field to k8s_job_executor that allows you to customize the raw Kubernetes config for each step in a job. See the docs for more information.
• [dagster-k8s] Launched run pods now have an additional code location label.
• [dagster-ui] The runs table now lets you toggle which tags are always visible.
• [dagster-dbt] dagster-dbt project scaffold now creates the scaffold in multiple files:
  • constants.py contains a reference to your manifest and dbt project directory
  • assets.py contains your initial dbt assets definitions
  • definitions.py contains the code to load your asset definitions into the Dagster UI
  • schedules.py contains an optional schedule to add for your dbt assets
• [dagster-dbt] Added new methods get_auto_materialize_policy and get_freshness_policy to DagsterDbtTranslator.
• [dagster-fivertran] Sync options can now be passed to load_assets_from_fivetran_instance.
• [dagster-wandb] W&B IO Manager now handles partitions natively. (Thanks @chrishiste!)

Bugfixes

• Previously, canceling large asset backfills would cause the daemon to time out and display a “not running” error. This has been fixed.
• [dagster-ssh] Previously the SSHResource would warn when allow_host_key_change was set. Now known hosts are always loaded from the system hosts file, and the allow_host_key_change parameter is ignored.
• Previously, when using AutoMaterializePolicies, partitioned assets downstream of partitioned observable source assets could be materialized before their parent partitions were observed. This has been fixed.

Documentation

• @graph_multi_asset now has an API docs entry.
• The GCSComputeLogManager example in the Dagster Instance reference is now correct.
• Several outdated K8s documentation links have been removed from the Customizing your Kubernetes deployment guide.
• Added callouts to the GitHub and GitLab Branch Deployment guides specifying that some steps are optional for Serverless users.
• The “Graphs” page under the “Concepts” section has been renamed to “Op Graphs” and moved inside under the “Ops” heading.
• [dagster-dbt] Added API examples for @dbt_assets for the following use-cases:
  • Running dbt commands with flags
  • Running dbt commands with --vars
  • Running multiple dbt commands
  • Retrieving dbt artifacts after running a dbt command
  • Invoking other Dagster resouces alongside dbt
  • Defining and accessing Dagster config alongside dbt

Dagster Cloud

• The viewer role now has permission to edit their own user tokens.

New

• @graph_asset now takes a config parameter equivalent to the parameter on @graph.
• Added an optional dynamic_partitions_store argument to DynamicPartitionsDefinition for multi-partition run properly with dynamic partitions (Thanks @elzzz!).
• [dagster-grpahql] Added partitionsByAssets`` to backfillParams`` for ranged partition backfill (Thanks @ruizh22!).
• [dagster-dbt] Support for dbt-core==1.6 has been added.
• [dagster-dbt] DbtCliResource now supports configuring profiles_dir.
• [dagster-k8s] Allow specifying restart_policy on k8s_job_op (Thanks @Taadas!).
• [dagster-snowflake] Added authenticator to SnowflakePandasIOManager, which allows specifying the authentication mechanism to use (Thanks @pengw0048!).

Bugfixes

• In some situations, multiple materializations of the same asset could be kicked off when using a lazy AutoMaterializePolicy with assets that had at least one source asset parent and at least one non-source asset parent. This has been fixed.
• After applying an eager AutoMaterializePolicy to a time-partitioned asset downstream of an unpartitioned asset, the latest partition would only ever be materialized a single time, rather than updating in response to any parent updates. This has been fixed.
• Fixed an issue that would cause the creation of a StaticPartitionsDefinition containing many thousands of partitions could take a significant amount of time.
• The run coordinator daemon now uses a fresh request context on each iteration, fixing an issue where stale grpc server references could be used in certain high volume conditions.
• Automatically generated data versions for partitioned assets now correctly reflect the data versions of upstream partitions. Previously, they were computed using the data versions from the most recent materializations of upstream assets regardless of partition.
• [dagster-airbyte] Previously, attempting to load assets from an Airbyte instance in which some of the tables had hyphens in their name would result in an error. This has been fixed.
• [dagster-dbt] Previously, attempting to load assets from a dbt project in which some of the models had hyphens in their name would result in an error. This has been fixed.
• [dagstermill] Fixed a bug where known state for executing dagstermill ops was not correctly passed in (Thanks @motuzov!).

Documentation

• Added the starter project’s template for Dagster University.
• Fixed an incorrect method name in DagsterDbtTranslator Docs (Thanks @akan72!).

Dagster Cloud

• When importing a dbt project on the Dagster Cloud setup page, an Unexpected exception error would be raised when scaffolding a pull request on a repository with no profiles.yml. This behavior has been updated to raise a more descriptive error message on the repo selection page.
• The running multiple agents guide has been revamped to discuss running agent replicas and zero-downtime deployment of the agent.
• The agentReplicas config setting on the helm chart has been renamed to isolatedAgents. In order to use this config setting, your user code dagster version needs to be 1.4.3 or greater.

New

• [ui] When viewing a run for auto-materialized assets, show a tag with information about the assets that were materialized.
• [ui] In the Auto-materialize History view, when one or more of an asset’s parents have been updated, the set of updated parents will be viewable.
• [ui] Link to the auto-materialized history for an asset from the asset DAG view.
• [ui] For runs that were the result of auto-observation, show a tag for this in the Runs list view.
• Added warnings for storage incompatibility with the experimental global op concurrency.

Bugfixes

• [dagster-dbt] Fixed an issue where dagster-dbt project scaffold didn’t create a project directory with all the scaffolded files.
• Fixed an issue which could cause errors when using the SpecificPartitionsPartitionMapping with auto-materialization.

Breaking Change

• Previously, it was possible to set max_materializations_per_minute on an AutoMaterializePolicy to a non-positive number. This will now result in an error.

Community Contributions

• Fix for loading multipartitions paths in upath_io_manager from @harrylojames; thank you!
• Docs typo fix from @C0DK; thank you!

Documentation

• Revamped the dagster-dbt tutorial to take advantage of dagster project scaffold and the new dagster-dbt APIs.

New

• [dagster-dbt] When invokingdagster-dbt project scaffold on a dbt project directory, if a profiles.yml exists in the root of the directory, its contents are used to add dbt adapter packages to the scaffolded setup.py.
• The default sentinel value for the multiprocessing executor’s max_concurrent field has been changed from 0 to None to more clearly signal its intent. A value of 0 is still interpreted as the sentinel value which dynamically allocates max_concurrent based on detected CPU count.

Bugfixes

• IO managers defined on jobs will now be properly merged with resources defined in Definitions, so that jobs are able to override the IO manager used.
• [dagster-fivetran] Fixed an issue where EnvVars in a FivetranResource would not be evaluated when loading assets from the Fivetran instance.
• [dagster-airbyte] Fixed an issue where EnvVars in an AirbyteResource would not be evaluated when loading assets from the Airbyte resource.

Documentation

• [dagster-dbt] Added API docs for DbtCliResource, DbtCliInvocation, @dbt_assets, DagsterDbtTranslator, dagster-dbt project scaffold
• [dagster-dbt] Expanded references for new APIs:
  • Added documentation to customize asset definition attributes for dbt assets
  • Added documentation to define upstream and downstream dependencies to dbt assets
  • Added documentation to define schedules for dbt assets

Bugfixes

• Fixes a bug in dagster-dbt that was preventing it from correctly materializing subselections of dbt asset.

Bugfixes

• Fixes a bug in dagster-dbt that was preventing it efficiently loading dbt projects from a manifest.

Major Changes since 1.3.0 (core) / 0.19.0 (libraries)

Core

• Auto-materialize history – We’ve added a UI that tracks why assets were or were not materialized according to theirAutoMaterializePolicy. It’s located under Assets → Select an asset with an AutoMaterializePolicy → Auto-materialize history tab.
• Auto-materialize performance – We’ve made significant performance improvements to the Asset Daemon, allowing it to keep up with asset graphs containing thousands of assets and assets with a large history of previously-materialized partitions.
• Asset backfill cancellation — Asset backfills can now be canceled, bring them to parity with job backfills. When an asset backfill is requested for cancellation, the daemon cancels runs until all runs are terminated, then marks the backfill as “canceled”.
• non_argument_deps → deps – We’ve deprecated the non_argument_deps parameter of @asset and @multi_asset in favor of a new deps parameter. The new parameter makes it clear that this is a first-class way of defining dependencies, makes code more concise, and accepts AssetsDefinition and SourceAsset objects, in addition to the strs and AssetKeys that the previous parameter accepted.
• Group-level asset status UI – the new Assets Overview dashboard, located underneath the Activity tab of the Overview page, shows the status all the assets in your deployment, rolled up by group.
• Op concurrency (experimental) — We’ve added a feature that allows limiting the number of concurrently executing ops across runs. [docs]
• DynamicPartitionsDefinition and SensorResult are no longer marked experimental.
• Automatically observe source assets, without defining jobs (experimental) – The @observable_source_asset decorator now accepts an auto_observe_interval_minutes parameter. If the asset daemon is turned on, then the observation function will automatically be run at this interval. Downstream assets with eager auto-materialize policies will automatically run if the observation function indicates that the source asset has changed. [docs]
• Dagit → Dagster UI – To reduce the number of Dagster-specific terms that new users need to learn when learning Dagster, “Dagit” has been renamed to the “The Dagster UI”. The dagit package is deprecated in favor of the dagster-webserver package.
• Default config in the Launchpad - When you open the launchpad to kick off a job or asset materialization, Dagster will now automatically populate the default values for each field.

dagster-dbt

• The new @dbt_assets decorator allows much more control over how Dagster runs your dbt project. [docs]
• The new dagster-dbt project scaffold command line interface makes it easy to create files and directories for a Dagster project that wraps an existing dbt project.
• Improved APIs for defining asset dependencies – The new get_asset_key_for_model and get_asset_key_for_source utilities make it easy to specify dependencies between upstream dbt assets and downstream non-dbt assets. And you can now more easily specify dependencies between dbt models and upstream non-dbt assets by specifying Dagster asset keys in the dbt metadata for dbt sources.

Since 1.3.14 (core) / 0.19.14 (libraries)

New

• The published Dagster Docker images now use Python 3.10, instead of 3.7.
• We’ve deprecated the non_argument_deps parameter of @asset and @multi_asset in favor of a new deps parameter. The new parameter makes it clear that this is a first-class way of defining dependencies, makes code more concise, and accepts AssetsDefinition and SourceAsset objects, in addition to the strs and AssetKeys that the previous parameter accepted.
• The UPathIOManager can now be extended to load multiple partitions asynchronously (Thanks Daniel Gafni!).
• By default, Dagster will now automatically load default config values into the launchpad. This behavior can be disabled in the user settings page.
• [dagster-k8s] The Helm chart now sets readiness probes on user code deployment servers by default. These can be disabled with dagster-user-deployments.deployments.[...].readinessProbe.enabled=false.
• [dagster-airbyte] In line with the deprecation of non_argument_deps in favor of deps, build_airbyte_assets now accepts a deps parameter.
• [dagstermill] In line with the deprecation of non_argument_deps in favor of deps, define_dagstermill_asset now accepts a deps parameter.

Bugfixes

• Duplicate partition keys passed to StaticPartitionsDefinition will now raise an error.
• Fixed a bug that caused lazy AutoMaterializePolicy's to not materialize missing assets.
• [ui] Fixed an issue where global search and large DAGs were broken when using --path-prefix.
• Schedule and sensor run submissions are now kept up to date with the current workspace, fixing an issue where a stale reference to a server would be used in some conditions.

Breaking Changes

• Support for Python 3.7 has been dropped.
• build_asset_reconciliation_sensor (Experimental) has been removed. It was deprecated in 1.3 in favor of AutoMaterializePolicy.
• asset_key(s) properties on AssetIn and AssetDefinition have been removed in favor of key(s). These APIs were deprecated in 1.0.
• root_input_manager and RootInputManagerDefinition have been removed in favor of input_manager and InputManagerDefinition. These APIs were deprecated in 1.0.
• [dagster-pandas] The event_metadata_fn parameter on create_dagster_pandas_dataframe_type has been removed in favor of metadata_fn.
• [dagster-dbt] The library has been substantially revamped to support the new @dbt_assets and DbtCliResource. See the migration guide for details.
  • Group names for dbt assets are now taken from a dbt model's group. Before, group names were determined using the model's subdirectory path.
  • Support for dbt-rpc has been removed.
  • The class alias DbtCloudResourceV2 has been removed.
  • DbtCli has been renamed to DbtCliResource. Previously, DbtCliResource was a class alias for DbtCliClientResource.
  • load_assets_from_dbt_project and load_assets_from_dbt_manifest now default to use_build=True.
  • The default assignment of groups to dbt models loaded from load_assets_from_dbt_project and load_assets_from_dbt_manifest has changed. Rather than assigning a group name using the model’s subdirectory, a group name will be assigned using the dbt model’s dbt group.
  • The argument node_info_to_definition_metadata_fn for load_assets_from_dbt_project and load_assets_from_dbt_manifest now overrides metadata instead of adding to it.
  • The arguments for load_assets_from_dbt_project and load_assets_from_dbt_manifest now must be specified using keyword arguments.
  • When using the new DbtCliResource with load_assets_from_dbt_project and load_assets_from_dbt_manifest, stdout logs from the dbt process will now appear in the compute logs instead of the event logs.

Deprecations

• The dagit python package is deprecated and will be removed in 2.0 in favor of dagster-webserver. See the migration guide for details.
• The following fields containing “dagit” in the Dagster helm chart schema have been deprecated in favor of “dagsterWebserver” equivalents (see migration guide for details):
  • dagit → dagsterWebserver
  • ingress.dagit → ingress.dagsterWebserver
  • ingress.readOnlyDagit → ingress.readOnlyDagsterWebserver
• [Dagster Cloud ECS Agent] We've introduced performance improvements that rely on the AWS Resource Groups Tagging API. To enable, grant your agent's IAM policy permission to tag:DescribeResources. Without this policy, the ECS Agent will log a deprecation warning and fall back to its old behavior (listing all ECS services in the cluster and then listing each service's tags).
• DbtCliClientResource, dbt_cli_resource and DbtCliOutput are now being deprecated in favor of DbtCliResource.
• A number of arguments on load_assets_from_dbt_project and load_assets_from_dbt_manifest are now deprecated in favor of other options. See the migration for details.

Community Contributions

• Docs typo fix from @chodera, thank you!
• Run request docstring fix from @Jinior, thank you!

Documentation

• All public methods in the Dagster API now have docstrings.
• The entirety of the documentation has been updated to now refer to the “Dagster webserver” or “Dagster UI” where “Dagit” was previously used for both entities.

New

• DynamicPartitionsDefinition and SensorResult are no longer marked experimental
• DagsterInstance now has a get_status_by_partition method, which returns the status of each partition for a given asset. Thanks renzhe-brian!
• DagsterInstance now has a get_latest_materialization_code_versions method, which returns the code version of the latest materialization for each of the provided (non-partitioned) assets.
• The error message for when an asset illegally depends on itself is now more informative.
• Further performance improvements for the Asset Daemon.
• Performance improvements in the asset graph view for large asset graphs.
• Pandas 2.x is now supported in all dagster packages.
• build_asset_context has been added as an asset focused replacement for build_op_context.
• build_op_context now accepts a partition_key_range parameter.
• New AssetSelection.upstream_source_assets method allows selecting source assets upstream of the current selection.
• AssetSelection.key_prefixes and AssetSelection.groups now accept an optional include_sources parameter.
• The AutoMaterialize evaluations UI now provides more details about partitions and waiting on upstream assets.
• [dbt] The DbtCli resource is no longer marked experimental.
• [dbt] The global_config parameter of the DbtCli resource has been renamed to global_config_flags
• [dbt] load_assets_from_dbt_project and load_assets_from_dbt_manifest now work with the DbtCli resource.
• [dbt] The manifest argument of the @dbt_assets decorator now additionally can accept a Path argument representing a path to the manifest file or dictionary argument representing the raw manifest blob.
• [dbt] When invoking DbtCli.cli from inside a @dbt_assets-decorated function, you no longer need to supply the manifest argument as long as you provide the context argument.
• [dbt] The DbtManifest object can now generate schedules using dbt selection syntax.

dbt_manifest.build_schedule(
      job_name="materialize_dbt_models",
      cron_schedule="0 0 * * *",
      dbt_select="fqn:*"
  )

• [dbt] When invoking DbtCli.cli and the underlying command fails, an exception will now be raised. To suppress the exception, run the DbtCli.cli(..., raise_on_error=False).
• [ui] You can now alphabetically sort your partitions on the asset partitions page
• [ui] A button in the “Run is materializing this asset” and “Run failed to materialize this asset” banners provides direct access to the relevant run logs

Bugfixes

• Fixed a bug that caused asset metadata to not be available available on the OutputContext when using with_attributes or AssetsDefinition.from_graph.
• Previously, if a partitioned asset at the root of the graph had more missing partitions than its AutoMaterializePolicy’s max_materializations_per_minute parameter, those older partitions would not be properly discarded from consideration on subsequent ticks. This has been fixed.
• Fixed a bug that caused AutoMaterializePolicy.lazy() to not materialize missing assets that were downstream of assets without an AutoMaterializePolicy.
• In rare cases, the AssetDaemon could hit an exception when using a combination of freshness policies and observable source assets. This has been fixed.
• Previously, string type annotations (most commonly via modules containing from __future__ import annotations) would cause errors in most cases when used with Dagster definitions. This has been fixed for the vast majority of cases.
• AssetExecutionContext has returned to being a type alias for OpExecutionContext.
• [ui] Date filtering on the runs page now takes your timezone into consideration
• [ui] Fixed a bug where selecting partitions in the launchpad dialog cleared out your configuration
• [ui] In the run Gantt chart, executed steps that follow skipped steps no longer render off the far right of the visualization.
• [ui] Cancelling a running backfill no longer makes canceled partitions un-selectable on the job partitions page and backfill modal, and cancellation is shown in gray instead of red.

Breaking Changes

• [experimental] The internal time_window_partition_scope_minutes parameter of the AutoMaterializePolicy class has been removed. Instead, max_materializations_per_minute should be used to limit the number of runs that may be kicked off for a partitioned asset.

Deprecations

• [dbt] DbtCliResource has been deprecated in favor of DbtCli.
• The python package dagit has been deprecated in favor of a new package dagster-webserver.
• OpExecutionContext.asset_partition_key_range has been deprecated in favor of partition_key_range.

Community Contributions

• The databricks_pyspark_step_launcher will no longer error when executing steps that target a single partition of a DynamicPartitionsDefinition (thanks @weberdavid!).
• Increased timeout on readinessProbe for example user code images, which prevents breakages in certain scenarios (thanks @leehuwuj)!
• Avoid creation of erroneous local directories by GCS IO manager (thanks @peterjclaw)!
• Fixed typo in intro docs (thanks @adeboyed)!
• Fix typo in bigquery docs (thanks @nigelainscoe)!
• Fix typing on run tag validation (thanks @yuvalgimmunai)!
• Allow passing repositoryCredentials arn as config to ecs run launcher (thanks @armandobelardo)!

Experimental

• The @observable_source_asset decorator now accepts an auto_observe_interval_minutes parameter. If the asset daemon is turned on, then the observation function will automatically be run at this interval.
• [dbt] DbtCliTask has been renamed to DbtCliInvocation
• [dbt] The get_asset_key_by_output_name and get_node_info_by_output_name methods of DbtManifest have been renamed toget_asset_key_for_output_name and get_node_info_for_output_name, respectively.
• [ui] A new feature flag allows you to switch Asset DAG rendering to a tighter horizontal layout, which may be preferable in some scenarios

Documentation

• Many public methods that were missing in the API docs are now documented. Updated classes include DagsterInstance, *MetadataValue, DagsterType, and others.
• dagster-pandera now has an API docs page.
• Deprecated methods in the API docs now are marked with a special badge.