dagster-cloud

New

• [kubernetes] securityContext can now be set in the dagsterCloudAgent section of the Helm chart.
• [kubernetes] The agent Helm chart now includes a serverK8sConfig and runK8sConfig key that allows you to specify additional Kubernetes config that will be applied to each pod spun up by the agent to run Dagster code. Code locations can also be configured with a server_k8s_config or run_k8s_config dictionary with additional Kubernetes config in the pods that are spun up by the agent for that code location. See the Kubernetes agent configuration reference for more information.
• [ecs] The ECS agent can now be configured with a server_resources and/or run_resources dictionary that will specify CPU and memory values for each task that is spun up by the agent to run Dagster code. Code locations can also be configured with a server_resources and/or run_resources dictionary that applies to each task spun up by the agent for that code location. See the ECS agent configuration reference for more information.
• The Dagster Cloud agent will now re-upload information about each code location to Dagster Cloud from every time it starts up. Previously, the agent would only upload changes to Dagster Cloud when a code location was updated, meaning it was possible for the agent to become out of sync with what was shown in Dagit when the agent restarted.
• The dagster-cloud serverless deploy-python-executable command now supports a --build-in-linux-docker flag that builds the dependencies within a local Linux Docker container. This enables deploying source-only dependencies (sdists) from non Linux environment.
• When the Dagster Cloud agent stops heartbeating (for example, when it is being upgraded), dequeueing runs will pause until the agent is available again.
• Restored some metadata to the Code Locations tab in Dagster Cloud, including image, python file, module name, and commit hash.
• Added an --asset-key argument to the dagster-cloud job launch CLI command that allows the job launch to only materialize one or more specific assets from the job.
• max_concurrent_dequeueconfig has been added to the run_queue section of deployment config to allow slowing the rate at which queued runs are launched.

Bugfixes

• Fixed an issue where the kubernetes agent was sometimes unable to move runs into a failed state when a run worker crashed or was interrupted by the Kubernetes cluster.
• A regression in retry handling for HTTP 429 responses released in 1.1.7 has been resolved.
• Cases where network errors were incorrectly reporting that they had tried and exhausted retries have been corrected.
• Fixed an issue where when adding or updating an environment variable, the change sometimes wasn’t reflected in branch deployments until they were redeployed.

New

• Non-isolated runs in Dagster Cloud Serverless now default to running at most 2 ops in parallel at once, to reduce the default memory usage of these runs. This number can be increased from the launchpad by configuring the execution key, for example:

execution:
  config:
    multiprocess:
      max_concurrent: 4

• Run dequeue operations can now happen concurrently, improving the throughput of starting new runs.

Bugfixes

• Fixed an issue where specifying a dictionary of proxies in the dagster_cloud_api.proxies key in an agent’s dagster.yaml file raised an error when proxies were also being set using environment variables.

New

• Dagster Cloud Serverless can now deploy changes to your code using PEX files instead of building a new Docker image on each change, resulting in much faster code updates.

  • To update your existing GitHub workflows to use the PEX based fast deploys:
    1. Replace the YAML files in your .github/workflows directory with updated YAML files found in our quickstart repository.
    2. Update the new YAML files and set DAGSTER_CLOUD_URL to the value in your original YAML files.

• The dagster-cloud serverless command now supports two new sub commands for fast deploys using PEX files:

  1. dagster-cloud serverless deploy-python-executable can be used instead of dagster-cloud serverless deploy to use the fast deploys mechanism. The existing deploy command is unchanged.
  2. dagster-cloud serverless upload-base-image can be used to upload a custom base image used to run code deployed using the above deploy-python-executable command. Using custom base images is optional.

  More details can be found in our docs.

• Runs that are launched from the Dagit UI in Dagster Cloud serverless can now be configured as either non-isolated or isolated. Non-isolated runs are for iterating quickly and trade off isolation for speed. Isolated runs are for production and compute heavy Assets/Jobs. For more information see the docs.

• Email alerts from Dagster Cloud now include the name of the deployment in the email subject.

New

• A handful of changes have been made to URLs in Dagit:
  • The /instance URL path prefix has been removed. E.g. /instance/runs can now be found at /runs.
  • The /workspace URL path prefix has been changed to /locations. E.g. the URL for job my_job in repository foo@bar can now be found at /locations/foo@bar/jobs/my_job.
• In Dagit, the “Workspace” navigation item in the top nav has been moved to be a tab under the “Deployment” section of the app, and is renamed to “Definitions”.

Dependency Changes

• We’ve upgraded our dagster/dagster-cloud-agent Docker image’s base from python:3.8.12-slim to python:3.8.15-slim to patch some vulnerabilities in the old base image.

New

• A new “Environment Variables” section in the Deployment tab allows you to set environment variables that will be included whenever your Dagster code runs. See the docs for more information. In order for environment variables to be included, your code must be using dagster version 1.0.17 or higher, and if you’re using a Hybrid agent, it must also be at dagster version 1.0.17 or higher.

Bugfixes

• Fixed an issue where stack traces from errors that occurred within user code were sometimes not displayed in Dagit.
• Fixed an issue where failure during automatic retries before run creation would result in alerts not being sent. Now, if a failure occurs during the automatic retry of a run before the next run has been created, a specific alert will be sent.

Documentation

• Updated Environment variables and secrets documentation to include info about using the new “Environment Variables” section in the Deployment tab
• Added a dedicated guide for setting environment variables using a Hybrid agent’s configuration

Documentation

• Added documentation for using environment variables in Dagster Cloud.
• Added a configuration reference for the Docker agent.
• Added example configuration for environment variables and secrets to the Kubernetes and Amazon ECS agent configuration references.

New

• [dagit] The new Overview and Workspace pages have been enabled for all users, after being gated with a feature flag for the last several releases. These changes include design updates, virtualized tables, and more performant querying.
• When running a Dagster Cloud agent, the Python logging configuration of the agent process can now be configured by passing in a --agent-logging-config-path option to the dagster-cloud agent run command. This config can also be set in the Dagster Cloud Helm chart using the loggingConfig key.
• Added a dagster-cloud settings saml remove-identity-provider-metadata command that can be used to reset the SAML configuration for an organization.
• You can now re-execute a run from failure in a Serverless deployment in Dagster Cloud without needing to define an IO manager

Bugfixes

• Fixed the createdBefore filter on the run GraphQL endpoint.

New

• The default Docker image for Serverless no longer runs pip install --upgrade pip.

Experimental

• Non-isolated runs: Previously all runs launched in an isolated run environment. Opting in to this new feature adds a default option to the Launchpad to execute without isolation. The run will start faster, but with fewer compute resources. This is great for testing, but not recommended for compute or memory intensive production jobs. This will only apply to runs launched via the UI. Scheduled runs won't be affected. To enable:
  • Upgrade your agent to at least 1.0.15
  • In the UI, click on profile icon → User Settings → Enable non-isolated runs

New

• The Amazon ECS agent will now re-use existing task definitions for the services that it spins up for each code location when the configuration for the service’s task definition has not changed. Previously, it would register a new task definition every time the service was re-deployed.

Bugfixes

• Fixed an issue where Serverless images would error on the absence of a script dagster_cloud_pre_install.sh inside of the source folder.

New

• The ECS Agent now shares a single task definition for every code location (instead of one task definition for the code location and another for any runs originating from that code location)
• dagster-cloud serverless deploy now blocks until the code location loads instead of returning immediately. This brings its behavior in line with other commands in the CLI and makes it safer to chain together multiple commands in a row (like launching a job after you deploy) without first needing to check if the deploy was successful.
• The deferred job snapshots option for workspace updates is now defaulted to on.

Bugfixes

• [dagit] For users with the “New workspace page” feature flag enabled, code locations could not be added from the Deployment page. This has been fixed.

New

• The ECS agent can now run in ECS clusters using EC2 capacity providers. See the ECS agent docs for more information.
• The CloudFormation template for the ECS agent now configures multi-AZ support for the containers it launches.
• The default amount of time that a Dagster Cloud agent waits for user code to import being timing out has been increased from 60 seconds to 180 seconds, to avoid false positives when loading code with heavy imports or large numbers of assets.
• Added a deployment setting to configure the default role for SAML-provisioned users.

Bugfixes

• The dagster-cloud CLI no longer lists branch deployments in the list of default deployments during setup.

Bugfixes

• The GitHub setup flow for Serverless will now display an error if Actions builds are blocked from starting due to GitHub permissions.

New

• When alerts are configured in Dagster Cloud on a job that also has automatic retries , the alert will now only fire if the final retry fails.
• Performance improvements to the Overview page.
• Job snapshot uploads now use a thread pool for parallelism.

Bugfixes

• Fixed an issue where links to the compute logs in Dagit would sometimes fail to load.
• Improved user-facing error messages for Serverless GitHub setup.

New

• Performance improvements for the Partitions page.
• A new scheme for workspaces updates has been added that improves performance in deployments with a large number of jobs. This is currently opt-in via the defer_job_snapshots setting in user_code_launcher config in dagster.yaml, or via deferJobSnapshots in the workspace section of the Dagster Cloud helm chart.

Bugfixes

• Fixed an issue where assets sometimes didn’t appear in the Asset Catalog while in Folder view.
• Fixed an issue where the Create Token button would erroneously appear when revoking other users’ tokens.

New

• [Kubernetes] Custom Kubernetes schedulers can now be applied to pods launched by the Dagster Cloud agent. To change the scheduler for all pods created by the agent, set workspace.schedulerName in the agent Helm chart. To set it for only the pods created as part of a particular code location, set the following configuration for the code location on the Workspace tab:

  container_context:
    k8s:
      scheduler_name: your-custom-schedule-name
  

  To change it for the agent, you'll still need to set dagsterCloudAgent.schedulerName in the agent Helm chart.

Bugfixes

• Fixed an issue where the “Latest run” column on the Instance Status page sometimes displayed an older run instead of the most recent run.
• Added a pin to grpcio to address an issue with the recent 0.48.1 grpcio release that was sometimes causing Dagster code servers to hang.

New

• [Kubernetes] The Agent Helm chart now supports dagsterCloudAgent.schedulerName and workspace.schedulerName for specifying a custom Kubernetes scheduler for the Agent pod and workspace pods, respectively.
• [Kubernetes] The Agent Helm chart now supports pointing an agent at a list of deployments using the dagsterCloud.deployments field.

Bugfixes

• Fixed an issue where GitHub Actions runs would trigger in an incorrect order during Serverless setup.

Bugfixes

• Fixed an issue where login emails would not work for certain orgs

New

• The top navigation of the Dagster Cloud UI now has a help menu with links to documentation and support.
• Added more user feedback to GitHub setup in Serverless new user experience.
• The Dagster Cloud usage page is now clarified to be in UTC time.
• Values have been added to the user-cloud helm chart for setting TTL (time to live) on user code servers managed by the agent.

Bugfixes

• The dagster-cloud serverless command is no longer erroneously hidden from the help command list.
• Fixed an issue where inspecting asset groups in Code Previews did not lead to the correct link.

New

• Introduced various UI improvements to the new organization onboarding flow.

New

• The dagster-cloud workspace CLI now has a configurable Agent heartbeat timeout. The CLI will logspew every few seconds while waiting for the Agent to sync a code location.
• When a code location is added or updated, a notification will appear in Dagit, and the Workspace tab will automatically refresh.
• Added additional retries when the ECS agent encounters a problem spinning up a new service.

Bugfixes

• Trying to add a code location with an empty name now throws an error message.
• Fixed an issue where the alert policy UI would not allow creating an alert policy with no tags, which targets all jobs.

Documentation

• The docs have moved! All Dagster docs now live at docs.dagster.io. Click here to check out the new and improved Cloud docs.

Documentation

• The Dagster Cloud docs now live alongside all the other Dagster docs! Check them out by nagivating to Deployment > Cloud.

New

• Performance improvements when the Dagster Cloud agent is deploying more than one code location at the same time.
• The default timeout for runs to start has been increased from 5 minutes to 10 minutes. We had observed that ECS tasks often would exceed timeout when pulling a large image.

New

• Branch Deployments are now ordered by their most recent commit time on the Deployments tab.
• You can now access the name of the current Dagster Cloud deployment within your Dagster code by checking the DAGSTER_CLOUD_DEPLOYMENT_NAME environment variable. You can use this to change the behavior of your code depending on the deployment in which it is running.

Bugfixes

• Fixed an issue where launching a run with just whitespace in the Launchpad would create an error.
• Fixed an issue where creating a code location that started with a numeric character failed to load when using the Kubernetes agent.

New

• Improved query performance for loading the Asset Graph in Dagit.

• The ECS agent is now more resilient to ECS’s eventual consistency model. In your dagster.yaml, you can configure how long ECS polls against AWS ECS API endpoints by setting:

  user_code_launcher:
    module: dagster_cloud.workspace.ecs
      class: EcsUserCodeLauncher
      config:
        timeout: 180 # default: 300
  

  And you can configure how long ECS will retry a failed AWS ECS API endpoint by setting:

  user_code_launcher:
    module: dagster_cloud.workspace.ecs
      class: EcsUserCodeLauncher
      config:
        grace_period: 20 # default: 10
  

• The Dagster Cloud API may start returning HTTP response code 429 when it receives an unusually large number of requests in a short period of time. The HTTP response will include a Retry-After header indicating the number of seconds to wait before retrying the request.

Bugfixes

• Fixed an issue where the Kubernetes agent didn’t work correctly when setting container_context.namespace to run code locations in different Kubernetes namespaces.

New

• The dagster-cloud CLI now supports adding a single code location from a file containing a list of locations by specifying the --location-name parameter.

New

• User code servers now support a configurable time-to-live (TTL). The agent will spin down any user code servers that haven’t served requests recently and will spin them back up the next time they’re needed. You can use this TTL to save compute cost because your user code servers will spend less time sitting idle. You can configure TTL in your agent’s dagster.yaml:

user_code_launcher:
  config:
    server_ttl:
      enabled: true
      ttl_seconds: 7200 # 2 hours

• When viewing agent and user tokens under the settings page, tokens are now sorted by latest creation date.

Bugfixes

• When configuring alert policies, all inputs are now validated before submission.
• When copying an agent related error message, the original error and stack trace are now included.

New

• Added Terms of Service to Dagster Cloud.
• Organization admins can now be managed from the Cloud permissions page. For more information, see the Cloud RBAC docs.

Bugfixes

• Fixed an issue where an empty list of email addresses could be configured on an email alert policy.
• When an agent is accidentally configured with a user token rather than an agent token, an informative exception is now displayed.
• Fixed an issue where jobs using memoization sometimes did not launch in Dagster Cloud.

New

• The new features announced in the Dagster 0.15.0 release are also available to Dagster Cloud users. In particular, software-defined assets are now marked fully stable and are ready for prime time - we recommend using them whenever your goal using Dagster is to build and maintain data assets.
• The ECS agent CloudFormation template is now versioned, allowing you to fix a specific version of the agent.
• You can now configure job-level retries to re-execute the full run rather than just failed Ops by setting the dagster/retry_strategy tag to ALL_STEPS. See the docs for more information.

New

• The Kubernetes agent and ECS agent can now include arbitrary environment variables in the containers that they create to run Dagster code. You can configure environment variables by setting env_vars in the dagster.yaml file for the agent, which will apply them to all code locations. You can also set env_vars on individual code locations. For example, to configure a code location to always set FOO_ENV_VAR to bar_value when creating Kubernetes pods, you can set this config in the Workspace tab or via the Dagster Cloud GitHub action:

location_name:my_location
image: my_repo:my_tag
code_source:
  package_name: my_package
container_context:
  k8s:
    env_vars:
      - FOO_ENV_VAR=bar_value
      - WILL_LOAD_ENV_VAR_FROM_PROCESS_IF_VALUE_NOT_SET

• Introduced a checklist to guide new organizations in setting up their Cloud deployment.

• Users with Editor or Admin permissions can now edit deployment settings from the Dagster Cloud UI, by clicking on the gear icon next to the deployment name in the deployment switcher in the upper-right corner of the screen.

Bugfixes

• Fixed a bug where renaming an alert created a new alert.
• Attempting to create a deployment with a duplicate name now produces a clear error.

New

• Added a new dagster-cloud-cli package. You can install this package if you want to interact with the CLI without also installing many of the additional packages that dagster-cloud depends on, simplifying installation on M1 Macs.
• Deployment settings can now be configured using a YAML editor in the Dagster Cloud UI.

Bugfixes

• Fixed an issue where a user could set their run coordinator’s max_concurrent_runs to be negative.
• [dagit] Alerts can now be configured on a deployment with no jobs.

New

• The Backfill page in Dagit now loads much faster when there are backfills with large numbers of runs.

Bugfixes

• Added a pin to version 3 of the protobuf library in Dagster Cloud, due to a breaking change in version 4 that was causing an error on startup.
• Fixed an issue where configuration errors while launching a run sometimes didn’t display a useful explanation for why the configuration was invalid.
• Fixed an issue where Dagster Cloud would sometimes fail to fire failure alerts due to external rate limits.