apache-airflow-providers-google

Features

• Add autodetect arg in BQCreateExternalTable Operator (#22710)

• Add links for BigQuery Data Transfer (#22280)

• Modify transfer operators to handle more data (#22495)

• Create Endpoint and Model Service, Batch Prediction and Hyperparameter Tuning Jobs operators for Vertex AI service (#22088)

• PostgresToGoogleCloudStorageOperator - BigQuery schema type for time zone naive fields (#22536)

• Update secrets backends to use get_conn_value instead of get_conn_uri (#22348)

Bug Fixes

• Fix the docstrings (#22497)

• Fix 'download_media' url in 'GoogleDisplayVideo360SDFtoGCSOperator' (#22479)

• Fix to 'CloudBuildRunBuildTriggerOperator' fails to find build id. (#22419)

• Fail ''LocalFilesystemToGCSOperator'' if src does not exist (#22772)

• Remove coerce_datetime usage from GCSTimeSpanFileTransformOperator (#22501)

Misc

• Refactor: BigQuery to GCS Operator (#22506)

• Remove references to deprecated operators/params in PubSub operators (#22519)

• New design of system tests (#22311)

Features

• Support Uploading Bigger Files to Google Drive (#22179)

• Change the default 'chunk_size' to a clear representation & add documentation (#22222)

• Add guide for DataprocInstantiateInlineWorkflowTemplateOperator (#22062)

• Allow for uploading metadata with GCS Hook Upload (#22058)

• Add Dataplex operators (#20377)

Misc

• Add support for ARM platform (#22127)

• Add Trove classifiers in PyPI (Framework :: Apache Airflow :: Provider)

• Use yaml safe load (#22091)

Features

• Add Looker PDT operators (#20882)

• Add autodetect arg to external table creation in GCSToBigQueryOperator (#21944)

• Add Dataproc assets/links (#21756)

• Add Auto ML operators for Vertex AI service (#21470)

• Add GoogleCalendarToGCSOperator (#20769)

• Make project_id argument optional in all dataproc operators (#21866)

• Allow templates in more DataprocUpdateClusterOperator fields (#21865)

• Dataflow Assets (#21639)

• Extract ClientInfo to module level (#21554)

• Datafusion assets (#21518)

• Dataproc metastore assets (#21267)

• Normalize *_conn_id parameters in BigQuery sensors (#21430)

Bug Fixes

• Fix bigquery_dts parameter docstring typo (#21786)

• Fixed PostgresToGCSOperator fail on empty resultset for use_server_side_cursor=True (#21307)

• Fix multi query scenario in bigquery example DAG (#21575)

Misc

• Support for Python 3.10

• Unpin 'google-cloud-memcache' (#21912)

• Unpin ''pandas-gbq'' and remove unused code (#21915)

• Suppress hook warnings from the Bigquery transfers (#20119)

Features

• Add hook for integrating with Google Calendar (#20542)

• Add encoding parameter to 'GCSToLocalFilesystemOperator' to fix #20901 (#20919)

• batch as templated field in DataprocCreateBatchOperator (#20905)

• Make timeout Optional for wait_for_operation (#20981)

• Add more SQL template fields renderers (#21237)

• Create CustomJob and Datasets operators for Vertex AI service (#21253)

• Support to upload file to Google Shared Drive (#21319)

• (providers_google) add a location check in bigquery (#19571)

• Add support for BeamGoPipelineOperator (#20386)

• Google Cloud Composer opearators (#21251)

• Enable asynchronous job submission in BigQuery hook (#21385)

• Optionally raise an error if source file does not exist in GCSToGCSOperator (#21391)

Bug Fixes

• Cloudsql import links fix. (#21199)

• Fix BigQueryDataTransferServiceHook.get_transfer_run() request parameter (#21293)

• :bug: (BigQueryHook) fix compatibility with sqlalchemy engine (#19508)

Misc

• Refactor operator links to not create ad hoc TaskInstances (#21285)

Features

• Add optional location to bigquery data transfer service (#15088) (#20221)

• Add Google Cloud Tasks how-to documentation (#20145)

• Added example DAG for MSSQL to Google Cloud Storage (GCS) (#19873)

• Support regional GKE cluster (#18966)

• Delete pods by default in KubernetesPodOperator (#20575)

Bug Fixes

• Fixes docstring for PubSubCreateSubscriptionOperator (#20237)

• Fix missing get_backup method for Dataproc Metastore (#20326)

• BigQueryHook fix typo in run_load doc string (#19924)

• Fix passing the gzip compression parameter on sftp_to_gcs. (#20553)

• switch to follow_redirects on httpx.get call in CloudSQL provider (#20239)

• avoid deprecation warnings in BigQuery transfer operators (#20502)

• Change download_video parameter to resourceName (#20528)

• Fix big query to mssql/mysql transfer issues (#20001)

• Fix setting of project ID in ''provide_authorized_gcloud'' (#20428)

Misc

• Move source_objects datatype check out of GCSToBigQueryOperator.__init__ (#20347)

• Organize S3 Classes in Amazon Provider (#20167)

• Providers facebook hook multiple account (#19377)

• Remove deprecated method call (blob.download_as_string) (#20091)

• Remove deprecated template_fields from GoogleDriveToGCSOperator (#19991)

Note! optional features of the apache-airflow-providers-facebook and apache-airflow-providers-amazon require newer versions of the providers (as specified in the dependencies)

Features

• Added wait mechanizm to the DataprocJobSensor to avoid 509 errors when Job is not available (#19740)

• Add support in GCP connection for reading key from Secret Manager (#19164)

• Add dataproc metastore operators (#18945)

• Add support of 'path' parameter for GCloud Storage Transfer Service operators (#17446)

• Move 'bucket_name' validation out of '__init__' in Google Marketing Platform operators (#19383)

• Create dataproc serverless spark batches operator (#19248)

• updates pipeline_timeout CloudDataFusionStartPipelineOperator (#18773)

• Support impersonation_chain parameter in the GKEStartPodOperator (#19518)

Bug Fixes

• Fix badly merged impersonation in GKEPodOperator (#19696)

Features

• Add value to 'namespaceId' of query (#19163)

• Add pre-commit hook for common misspelling check in files (#18964)

• Support query timeout as an argument in CassandraToGCSOperator (#18927)

• Update BigQueryCreateExternalTableOperator doc and parameters (#18676)

• Replacing non-attribute template_fields for BigQueryToMsSqlOperator (#19052)

• Upgrade the Dataproc package to 3.0.0 and migrate from v1beta2 to v1 api (#18879)

• Use google cloud credentials when executing beam command in subprocess (#18992)

• Replace default api_version of FacebookAdsReportToGcsOperator (#18996)

• Dataflow Operators - use project and location from job in on_kill method. (#18699)

Bug Fixes

• Fix hard-coded /tmp directory in CloudSQL Hook (#19229)

• Fix bug in Dataflow hook when no jobs are returned (#18981)

• Fix BigQueryToMsSqlOperator documentation (#18995)

• Move validation of templated input params to run after the context init (#19048)

• Google provider catch invalid secret name (#18790)

Breaking changes

• Migrate Google Cloud Build from Discovery API to Python SDK (#18184)

Features

• Add index to the dataset name to have separate dataset for each example DAG (#18459)

• Add missing __init__.py files for some test packages (#18142)

• Add possibility to run DAGs from system tests and see DAGs logs (#17868)

• Rename AzureDataLakeStorage to ADLS (#18493)

• Make next_dagrun_info take a data interval (#18088)

• Use parameters instead of params (#18143)

• New google operator: SQLToGoogleSheetsOperator (#17887)

Bug Fixes

• Fix part of Google system tests (#18494)

• Fix kubernetes engine system test (#18548)

• Fix BigQuery system test (#18373)

• Fix error when create external table using table resource (#17998)

• Fix ''BigQuery'' data extraction in ''BigQueryToMySqlOperator'' (#18073)

• Fix providers tests in main branch with eager upgrades (#18040)

• fix(CloudSqlProxyRunner): don't query connections from Airflow DB (#18006)

• Remove check for at least one schema in GCSToBigquery (#18150)

• deduplicate running jobs on BigQueryInsertJobOperator (#17496)

Features

• Add error check for config_file parameter in GKEStartPodOperator (#17700)

• Gcp ai hyperparameter tuning (#17790)

• Allow omission of 'initial_node_count' if 'node_pools' is specified (#17820)

• [Airflow 13779] use provided parameters in the wait_for_pipeline_state hook (#17137)

• Enable specifying dictionary paths in 'template_fields_renderers' (#17321)

• Don't cache Google Secret Manager client (#17539)

• [AIRFLOW-9300] Add DatafusionPipelineStateSensor and aync option to the CloudDataFusionStartPipelineOperator (#17787)

Bug Fixes

• GCP Secret Manager error handling for missing credentials (#17264)

Misc

• Optimise connection importing for Airflow 2.2.0

• Adds secrets backend/logging/auth information to provider yaml (#17625)

Breaking changes

• Updated GoogleAdsHook to support newer API versions after google deprecated v5. Google Ads v8 is the new default API. (#17111)

• Google Ads Hook: Support newer versions of the google-ads library (#17160)

Warning

The underlying google-ads library had breaking changes.

Previously the google ads library returned data as native protobuf messages. Now it returns data as proto-plus objects that behave more like conventional Python objects.

To preserve compatibility the hook's search() converts the data back to native protobuf before returning it. Your existing operators should work as before, but due to the urgency of the v5 API being deprecated it was not tested too thoroughly. Therefore you should carefully evaluate your operator and hook functionality with this new version.

In order to use the API's new proto-plus format, you can use the search_proto_plus() method.

For more information, please consult google-ads migration document:

Features

• Standardise dataproc location param to region (#16034)

• Adding custom Salesforce connection type + SalesforceToS3Operator updates (#17162)

Bug Fixes

• Update alias for field_mask in Google Memmcache (#16975)

• fix: dataprocpysparkjob project_id as self.project_id (#17075)

• Fix GCStoGCS operator with replace diabled and existing destination object (#16991)

Breaking changes

• Auto-apply apply_default decorator (#15667)

Warning

Due to apply_default decorator removal, this version of the provider requires Airflow 2.1.0+. If your Airflow version is < 2.1.0, and you want to install this provider version, first upgrade Airflow to at least version 2.1.0. Otherwise your Airflow package version will be upgraded automatically and you will have to manually run airflow upgrade db to complete the migration.

• Move plyvel to google provider extra (#15812)

• Fixes AzureFileShare connection extras (#16388)

Features

• Add extra links for google dataproc (#10343)

• add oracle connection link (#15632)

• pass wait_for_done parameter down to _DataflowJobsController (#15541)

• Use api version only in GoogleAdsHook not operators (#15266)

• Implement BigQuery Table Schema Update Operator (#15367)

• Add BigQueryToMsSqlOperator (#15422)

Bug Fixes

• Fix: GCS To BigQuery source_object (#16160)

• Fix: Unnecessary downloads in ``GCSToLocalFilesystemOperator (#16171)``

• Fix bigquery type error when export format is parquet (#16027)

• Fix argument ordering and type of bucket and object (#15738)

• Fix sql_to_gcs docstring lint error (#15730)

• fix: ensure datetime-related values fully compatible with MySQL and BigQuery (#15026)

• Fix deprecation warnings location in google provider (#16403)

We are proud to announce the General Availability of Apache Airflow 3.0 — the most significant release in the project's history. This version introduces a service-oriented architecture, a stable DAG authoring interface, expanded support for event-driven and ML workflows, and a fully modernized UI built on React. Airflow 3.0 reflects years of community investment and lays the foundation for the next era of scalable, modular orchestration.

Highlights

• Service-Oriented Architecture: A new Task Execution API and airflow api-server enable task execution in remote environments with improved isolation and flexibility (AIP-72).

• Edge Executor: A new executor that supports distributed, event-driven, and edge-compute workflows (AIP-69), now generally available.

• Stable Authoring Interface: DAG authors should now use the new airflow.sdk namespace to import core DAG constructs like @dag, @task, and DAG.

• Scheduler-Managed Backfills: Backfills are now scheduled and tracked like regular DAG runs, with native UI and API support (AIP-78).

• DAG Versioning: Airflow now tracks structural changes to DAGs over time, enabling inspection of historical DAG definitions via the UI and API (AIP-66).

• Asset-Based Scheduling: The dataset model has been renamed and redesigned as assets, with a new @asset decorator and cleaner event-driven DAG definition (AIP-74, AIP-75).

• Support for ML and AI Workflows: DAGs can now run with logical_date=None, enabling use cases such as model inference, hyperparameter tuning, and non-interval workflows (AIP-83).

• Removal of Legacy Features: SLAs, SubDAGs, DAG and Xcom pickling, and several internal context variables have been removed. Use the upgrade tools to detect deprecated usage.

• Split CLI and API Changes: The CLI has been split into airflow and airflowctl (AIP-81), and REST API now defaults to logical_date=None when triggering a new DAG run.

• Modern React UI: A complete UI overhaul built on React and FastAPI includes version-aware views, backfill management, and improved DAG and task introspection (AIP-38, AIP-84).

• Migration Tooling: Use ruff and airflow config update to validate DAGs and configurations. Upgrade requires Airflow 2.7 or later and Python 3.9–3.12.

Significant Changes

Airflow 3.0 introduces the most significant set of changes since the 2.0 release, including architectural shifts, new execution models, and improvements to DAG authoring and scheduling.

Task Execution API & Task SDK (AIP-72)

Airflow now supports a service-oriented architecture, enabling tasks to be executed remotely via a new Task Execution API. This API decouples task execution from the scheduler and introduces a stable contract for running tasks outside of Airflow's traditional runtime environment.

To support this, Airflow introduces the Task SDK — a lightweight runtime environment for running Airflow tasks in external systems such as containers, edge environments, or other runtimes. This lays the groundwork for language-agnostic task execution and brings improved isolation, portability, and extensibility to Airflow-based workflows.

Airflow 3.0 also introduces a new airflow.sdk namespace that exposes the core authoring interfaces for defining DAGs and tasks. DAG authors should now import objects like DAG, @dag, and @task from airflow.sdk rather than internal modules. This new namespace provides a stable, forward-compatible interface for DAG authoring across future versions of Airflow.

Edge Executor (AIP-69)

Airflow 3.0 introduces the Edge Executor as a generally available feature, enabling execution of tasks in distributed or remote compute environments. Designed for event-driven and edge-compute use cases, the Edge Executor integrates with the Task Execution API to support task orchestration beyond the traditional Airflow runtime. This advancement facilitates hybrid and cross-environment orchestration patterns, allowing task workers to operate closer to data or application layers.

Scheduler-Managed Backfills (AIP-78)

Backfills are now fully managed by the scheduler, rather than being launched as separate command-line jobs. This change unifies backfill logic with regular DAG execution and ensures that backfill runs follow the same scheduling, versioning, and observability models as other DAG runs.

Airflow 3.0 also introduces native UI and REST API support for initiating and monitoring backfills, making them more accessible and easier to integrate into automated workflows. These improvements lay the foundation for smarter, safer historical reprocessing — now available directly through the Airflow UI and API.

DAG Versioning (AIP-66)

Airflow 3.0 introduces native DAG versioning. DAG structure changes (e.g., renamed tasks, dependency shifts) are now tracked directly in the metadata database. This allows users to inspect historical DAG structures through the UI and API, and lays the foundation for safer backfills, improved observability, and runtime-determined DAG logic.

Note: DAG bundles are not initialized in the triggerer. In practice, this means that triggers cannot come from a DAG bundle. This is because the triggerer does not deal with changes in trigger code over time, as everything happens in the main process. Triggers can come from anywhere else on sys.path instead.

React UI Rewrite (AIP-38, AIP-84)

Airflow 3.0 ships with a completely redesigned user interface built on React and FastAPI. This modern architecture improves responsiveness, enables more consistent navigation across views, and unlocks new UI capabilities — including support for DAG versioning, asset-centric DAG definitions, and more intuitive filtering and search.

The new UI replaces the legacy Flask-based frontend and introduces a foundation for future extensibility and community contributions.

Asset-Based Scheduling & Terminology Alignment (AIP-74, AIP-75)

The concept of Datasets has been renamed to Assets, unifying terminology with common practices in the modern data ecosystem. The internal model has also been reworked to better support future features like asset partitions and validations.

The @asset decorator and related changes to the DAG parser enable clearer, asset-centric DAG definitions, allowing Airflow to more naturally support event-driven and data-aware scheduling patterns.

This renaming impacts modules, classes, functions, configuration keys, and internal models. Key changes include:

• Dataset → Asset

• DatasetEvent → AssetEvent

• DatasetAlias → AssetAlias

• airflow.datasets.* → airflow.sdk.*

• airflow.timetables.simple.DatasetTriggeredTimetable → airflow.timetables.simple.AssetTriggeredTimetable

• airflow.timetables.datasets.DatasetOrTimeSchedule → airflow.timetables.assets.AssetOrTimeSchedule

• airflow.listeners.spec.dataset.on_dataset_created → airflow.listeners.spec.asset.on_asset_created

• airflow.listeners.spec.dataset.on_dataset_changed → airflow.listeners.spec.asset.on_asset_changed

• core.dataset_manager_class → core.asset_manager_class

• core.dataset_manager_kwargs → core.asset_manager_kwargs

Unified Scheduling Field

Airflow 3.0 removes the legacy schedule_interval and timetable parameters. DAGs must now use the unified schedule field for all time- and event-based scheduling logic. This simplifies DAG definition and improves consistency across scheduling paradigms.

Updated Scheduling Defaults

Airflow 3.0 changes the default behavior for new DAGs by setting catchup_by_default = False in the configuration file. This means DAGs that do not explicitly set catchup=... will no longer backfill missed intervals by default. This change reduces confusion for new users and better reflects the growing use of on-demand and event-driven workflows.

The default DAG schedule has been changed to None from @once.

Restricted Metadata Database Access

Task code can no longer directly access the metadata database. Interactions with DAG state, task history, or DAG runs must be performed via the Airflow REST API or exposed context. This change improves architectural separation and enables remote execution.

Future Logical Dates No Longer Supported

Airflow no longer supports triggering DAG runs with a logical date in the future. This change aligns with the logical execution model and removes ambiguity in backfills and event-driven DAGs. Use logical_date=None to trigger runs with the current timestamp.

Context Behavior for Asset and Manually Triggered DAGs

For DAG runs triggered by an Asset event or through the REST API without specifying a logical_date, Airflow now sets logical_date=None by default. These DAG runs do not have a data interval, and attempting to access data_interval_start, data_interval_end, or logical_date from the task context will raise a KeyError.

DAG authors should use dag_run.logical_date and perform appropriate checks or fallbacks if supporting multiple trigger types. This change improves consistency with event-driven semantics but may require updates to existing DAGs that assume these values are always present.

Improved Callback Behavior

Airflow 3.0 refines task callback behavior to improve clarity and consistency. In particular, on_success_callback is no longer executed when a task is marked as SKIPPED, aligning it more closely with expected semantics.

Updated Default Configuration

Several default configuration values have been updated in Airflow 3.0 to better reflect modern usage patterns and simplify onboarding:

• catchup_by_default is now set to False by default. DAGs will not automatically backfill unless explicitly configured to do so.

• create_cron_data_intervals is now set to False by default. As a result, cron expressions will be interpreted using the CronTriggerTimetable instead of the legacy CronDataIntervalTimetable. This only affects DAGs that pass a bare cron string to schedule=; DAGs that pass an explicit timetable instance are unaffected. If you rely on the data interval semantics (data_interval_start / data_interval_end, or templated values like ds / ts derived from logical_date), set create_cron_data_intervals=True explicitly before the upgrade. Flipping the value later, after Airflow 3 DAG runs already exist, will skip one scheduled run on each affected DAG to avoid colliding with the previous run's logical_date.

• SimpleAuthManager is now the default auth_manager. To continue using Flask AppBuilder-based authentication, install the apache-airflow-providers-fab provider and explicitly set auth_manager = airflow.providers.fab.auth_manager.FabAuthManager.

These changes represent the most significant evolution of the Airflow platform since the release of 2.0 — setting the stage for more scalable, event-driven, and language-agnostic orchestration in the years ahead.

Executor & Scheduler Updates

Airflow 3.0 introduces several important improvements and behavior changes in how DAGs and tasks are scheduled, prioritized, and executed.

Standalone DAG Processor Required

Airflow 3.0 now requires the standalone DAG processor to parse DAGs. This dedicated process improves scheduler performance, isolation, and observability. It also simplifies architecture by clearly separating DAG parsing from scheduling logic. This change may affect custom deployments that previously used embedded DAG parsing.

Priority Weight Capped by Pool Slots

The priority_weight value on a task is now capped by the number of available pool slots. This ensures that resource availability remains the primary constraint in task execution order, preventing high-priority tasks from starving others when resource contention exists.

Teardown Task Handling During DAG Termination

Teardown tasks will now be executed even when a DAG run is terminated early. This ensures that cleanup logic is respected, improving reliability for workflows that use teardown tasks to manage ephemeral infrastructure, temporary files, or downstream notifications.

Improved Scheduler Fault Tolerance

Scheduler components now use run_with_db_retries to handle transient database issues more gracefully. This enhances Airflow's fault tolerance in high-volume environments and reduces the likelihood of scheduler restarts due to temporary database connection problems.

Mapped Task Stats Accuracy

Airflow 3.0 fixes a bug that caused incorrect task statistics to be reported for dynamic task mapping. Stats now accurately reflect the number of mapped task instances and their statuses, improving observability and debugging for dynamic workflows.

SequentialExecutor has been removed

SequentialExecutor was primarily used for local testing but is now redundant, as LocalExecutor supports SQLite with WAL mode and provides better performance with parallel execution. Users should switch to LocalExecutor or CeleryExecutor as alternatives.

DAG Authoring Enhancements

Airflow 3.0 includes several changes that improve consistency, clarity, and long-term stability for DAG authors.

New Stable DAG Authoring Interface: airflow.sdk

Airflow 3.0 introduces a new, stable public API for DAG authoring under the airflow.sdk namespace, available via the apache-airflow-task-sdk package.

The goal of this change is to decouple DAG authoring from Airflow internals (Scheduler, API Server, etc.), providing a forward-compatible, stable interface for writing and maintaining DAGs across Airflow versions.

DAG authors should now import core constructs from airflow.sdk rather than internal modules.

Key Imports from airflow.sdk:

• Classes:

  • Asset

  • BaseNotifier

  • BaseOperator

  • BaseOperatorLink

  • BaseSensorOperator

  • Connection

  • Context

  • DAG

  • EdgeModifier

  • Label

  • ObjectStoragePath

  • Param

  • TaskGroup

  • Variable

• Decorators and Functions:

  • @asset

  • @dag

  • @setup

  • @task

  • @task_group

  • @teardown

  • chain

  • chain_linear

  • cross_downstream

  • get_current_context

  • get_parsing_context

For an exhaustive list of available classes, decorators, and functions, check airflow.sdk.__all__.

All DAGs should update imports to use airflow.sdk instead of referencing internal Airflow modules directly. Legacy import paths (e.g., airflow.models.dag.DAG, airflow.decorator.task) are deprecated and will be removed in a future Airflow version. Some additional utilities and helper functions that DAGs sometimes use from airflow.utils.* and others will be progressively migrated to the Task SDK in future minor releases.

These future changes aim to complete the decoupling of DAG authoring constructs from internal Airflow services. DAG authors should expect continued improvements to airflow.sdk with no backwards-incompatible changes to existing constructs.

For example, update:

# Old (Airflow 2.x)
from airflow.models import DAG
from airflow.decorators import task

# New (Airflow 3.x)
from airflow.sdk import DAG, task

Renamed Parameter: fail_stop → fail_fast

The DAG argument fail_stop has been renamed to fail_fast for improved clarity. This parameter controls whether a DAG run should immediately stop execution when a task fails. DAG authors should update any code referencing fail_stop to use the new name.

Context Cleanup and Parameter Removal

Several legacy context variables have been removed or may no longer be available in certain types of DAG runs, including:

• conf

• execution_date

• dag_run.external_trigger

In asset-triggered and manually triggered DAG runs with logical_date=None, data interval fields such as data_interval_start and data_interval_end may not be present in the task context. DAG authors should use explicit references such as dag_run.logical_date and conditionally check for the presence of interval-related fields where applicable.

Task Context Utilities Moved

Internal task context functions such as get_parsing_context have been moved to a more appropriate location (e.g., airflow.models.taskcontext). DAG authors using these utilities directly should update import paths accordingly.

Trigger Rule Restrictions

The TriggerRule.ALWAYS rule can no longer be used with teardown tasks or tasks that are expected to honor upstream dependency semantics. DAG authors should ensure that teardown logic is defined with the appropriate trigger rules for consistent task resolution behavior.

Asset Aliases for Reusability

A new utility function, create_asset_aliases(), allows DAG authors to define reusable aliases for frequently referenced Assets. This improves modularity and reuse across DAG files and is particularly helpful for teams adopting asset-centric DAGs.

Operator Links interface changed

The Operator Extra links, which can be defined either via plugins or custom operators now do not execute any user code in the Airflow UI, but instead push the "full" links to XCom backend and the link is fetched from the XCom backend when viewing task details, for example from grid view.

Example for users with custom links class:

@attr.s(auto_attribs=True)
class CustomBaseIndexOpLink(BaseOperatorLink):
    """Custom Operator Link for Google BigQuery Console."""

    index: int = attr.ib()

    @property
    def name(self) -> str:
        return f"BigQuery Console #{self.index + 1}"

    @property
    def xcom_key(self) -> str:
        return f"bigquery_{self.index + 1}"

    def get_link(self, operator, *, ti_key):
        search_queries = XCom.get_one(
            task_id=ti_key.task_id, dag_id=ti_key.dag_id, run_id=ti_key.run_id, key="search_query"
        )
        return f"https://console.cloud.google.com/bigquery?j={search_query}"

The link has an xcom_key defined, which is how it will be stored in the XCOM backend, with key as xcom_key and value as the entire link, this case: https://console.cloud.google.com/bigquery?j=search

Plugins no longer support adding executors, operators & hooks

Operator (including Sensors), Executors & Hooks can no longer be registered or imported via Airflow's plugin mechanism. These types of classes are just treated as plain Python classes by Airflow, so there is no need to register them with Airflow. They can be imported directly from their respective provider packages.

Before:

from airflow.hooks.my_plugin import MyHook

You should instead import it as:

from my_plugin import MyHook

Support for ML & AI Use Cases (AIP-83)

Airflow 3.0 expands the types of DAGs that can be expressed by removing the constraint that each DAG run must correspond to a unique data interval. This change, introduced in AIP-83, enables support for workflows that don't operate on a fixed schedule — such as model training, hyperparameter tuning, and inference tasks.

These ML- and AI-oriented DAGs often run ad hoc, are triggered by external systems, or need to execute multiple times with different parameters over the same dataset. By allowing multiple DAG runs with logical_date=None, Airflow now supports these scenarios natively without requiring workarounds.

Config & Interface Changes

Airflow 3.0 introduces several configuration and interface updates that improve consistency, clarify ownership of core utilities, and remove legacy behaviors that were no longer aligned with modern usage patterns.

Default Value Handling

Airflow no longer silently updates configuration options that retain deprecated default values. Users are now required to explicitly set any config values that differ from the current defaults. This change improves transparency and prevents unintentional behavior changes during upgrades.

Refactored Config Defaults

Several configuration defaults have changed in Airflow 3.0 to better reflect modern usage patterns:

• The default value of catchup_by_default is now False. DAGs will not backfill missed intervals unless explicitly configured to do so.

• The default value of create_cron_data_intervals is now False. Cron expressions are now interpreted using the CronTriggerTimetable instead of the legacy CronDataIntervalTimetable. This change simplifies interval logic and aligns with the future direction of Airflow's scheduling system. Set this flag explicitly before upgrading from Airflow 2 if you rely on data interval semantics; flipping it later (after Airflow 3 DAG runs exist) will skip one scheduled run per affected DAG.

Refactored Internal Utilities

Several core components have been moved to more intuitive or stable locations:

• The SecretsMasker class has been relocated to airflow.sdk.execution_time.secrets_masker.

• The ObjectStoragePath utility previously located under airflow.io is now available via airflow.sdk.

These changes simplify imports and reflect broader efforts to stabilize utility interfaces across the Airflow codebase.

Improved inlet_events, outlet_events, and triggering_asset_events

Asset event mappings in the task context are improved to better support asset use cases, including new features introduced in AIP-74.

Events of an asset or asset alias are now accessed directly by a concrete object to avoid ambiguity. Using a str to access events is no longer supported. Use an Asset or AssetAlias object, or Asset.ref to refer to an entity explicitly instead, such as:

outlet_events[Asset.ref(name="myasset")]  # Get events for asset named "myasset".
outlet_events[AssetAlias(name="myalias")]  # Get events for asset alias named "myalias".

Alternatively, two helpers for_asset and for_asset_alias are added as shortcuts:

outlet_events.for_asset(name="myasset")  # Get events for asset named "myasset".
outlet_events.for_asset_alias(name="myalias")  # Get events for asset alias named "myalias".

The internal representation of asset event triggers now also includes an explicit uri field, simplifying traceability and aligning with the broader asset-aware execution model introduced in Airflow 3.0. DAG authors interacting directly with inlet_events may need to update logic that assumes the previous structure.

Behaviour change in xcom_pull

In Airflow 2, the xcom_pull() method allowed pulling XComs by key without specifying task_ids, despite the fact that the underlying DB model defines task_id as part of the XCom primary key. This created ambiguity: if two tasks pushed XComs with the same key, xcom_pull() would pull whichever one happened to be first, leading to unpredictable behavior.

Airflow 3 resolves this inconsistency by requiring task_ids when pulling by key. This change aligns with the task-scoped nature of XComs as defined by the schema, ensuring predictable and consistent behavior.

DAG Authors should update their dags to use task_ids if their dags used xcom_pull without task_ids such as:

kwargs["ti"].xcom_pull(key="key")

Should be updated to:

kwargs["ti"].xcom_pull(task_ids="task1", key="key")

Removed Configuration Keys

As part of the deprecation cleanup, several legacy configuration options have been removed. These include:

• [scheduler] allow_trigger_in_future

• [scheduler] use_job_schedule

• [scheduler] use_local_tz

• [scheduler] processor_poll_interval

• [logging] dag_processor_manager_log_location

• [logging] dag_processor_manager_log_stdout

• [logging] log_processor_filename_template

All the webserver configurations have also been removed since API server now replaces webserver, so the configurations like below have no effect:

• [webserver] allow_raw_html_descriptions

• [webserver] cookie_samesite

• [webserver] error_logfile

• [webserver] access_logformat

• [webserver] web_server_master_timeout

• etc

Several configuration options previously located under the [webserver] section have been moved to the new ``[api]`` section. The following configuration keys have been moved:

• [webserver] web_server_host → [api] host

• [webserver] web_server_port → [api] port

• [webserver] workers → [api] workers

• [webserver] web_server_worker_timeout → [api] worker_timeout

• [webserver] web_server_ssl_cert → [api] ssl_cert

• [webserver] web_server_ssl_key → [api] ssl_key

• [webserver] access_logfile → [api] access_logfile

The following DAG parsing configuration options were moved to the new ``[dag_processor]`` section:

• [core] dag_file_processor_timeout → [dag_processor] dag_file_processor_timeout

• [scheduler] parsing_processes → [dag_processor] parsing_processes

• [scheduler] file_parsing_sort_mode → [dag_processor] file_parsing_sort_mode

• [scheduler] max_callbacks_per_loop → [dag_processor] max_callbacks_per_loop

• [scheduler] min_file_process_interval → [dag_processor] min_file_process_interval

• [scheduler] stale_dag_threshold → [dag_processor] stale_dag_threshold

• [scheduler] print_stats_interval → [dag_processor] print_stats_interval

Users should review their airflow.cfg files or use the airflow config lint command to identify outdated or removed options.

Upgrade Tooling

Airflow 3.0 includes improved support for upgrade validation. Use the following tools to proactively catch incompatible configs or deprecated usage patterns:

• airflow config lint: Identifies removed or invalid config keys

• ruff check --select AIR30 --preview: Flags removed interfaces and common migration issues

CLI & API Changes

Airflow 3.0 introduces changes to both the CLI and REST API interfaces to better align with service-oriented deployments and event-driven workflows.

Split CLI Architecture (AIP-81)

The Airflow CLI has been split into two distinct interfaces:

• The core airflow CLI now handles only local functionality (e.g., airflow tasks test, airflow dags list).

• Remote functionality, including triggering DAGs or managing connections in service-mode environments, is now handled by a separate CLI called airflowctl, distributed via the apache-airflow-client package.

This change improves security and modularity for deployments that use Airflow in a distributed or API-first context.

REST API v2 replaces v1

The legacy REST API v1, previously built with Connexion and Marshmallow, has been replaced by a modern FastAPI-based REST API v2.

This new implementation improves performance, aligns more closely with web standards, and provides a consistent developer experience across the API and UI.

Key changes include stricter validation (422 errors instead of 400), the removal of the execution_date parameter in favor of logical_date, and more consistent query parameter handling.

The v2 API is now the stable, fully supported interface for programmatic access to Airflow, and also powers the new UI - achieving full feature parity between the UI and API.

For details, see the Airflow REST API v2 documentation.

REST API: DAG Trigger Behavior Updated

The behavior of the POST /dags/{dag_id}/dagRuns endpoint has changed. If a logical_date is not explicitly provided when triggering a DAG via the REST API, it now defaults to None.

This aligns with event-driven DAGs and manual runs in Airflow 3.0, but may break backward compatibility with scripts or tools that previously relied on Airflow auto-generating a timestamped logical_date.

Removed CLI Flags and Commands

Several deprecated CLI arguments and commands that were marked for removal in earlier versions have now been cleaned up in Airflow 3.0. Run airflow --help to review the current set of available commands and arguments.

• Deprecated --ignore-depends-on-past cli option is replaced by --depends-on-past ignore.

• --tree flag for airflow tasks list command is removed. The format of the output with that flag can be expensive to generate and extremely large, depending on the DAG. airflow dag show is a better way to visualize the relationship of tasks in a DAG.

• Changing dag_id from flag (-d, --dag-id) to a positional argument in the dags list-runs CLI command.

• The airflow db init and airflow db upgrade commands have been removed. Use airflow db migrate instead to initialize or migrate the metadata database. If you would like to create default connections use airflow connections create-default-connections.

• airflow api-server has replaced airflow webserver cli command.

Provider Refactor & Standardization

Airflow 3.0 completes the migration of several core operators, sensors, hooks, and triggers into the new apache-airflow-providers-standard package. This package now includes commonly used components such as:

• PythonOperator, BashOperator

• ExternalTaskSensor, FileSensor

• ShortCircuitOperator, LatestOnlyOperator

• SubprocessHook, FilesystemHook

• DateTimeTrigger, TimeDeltaTrigger, FileTrigger

These operators, sensors, hooks, and triggers were previously bundled inside airflow-core but are now treated as provider-managed components to improve modularity, testability, and lifecycle independence.

This change enables more consistent versioning across providers and prepares Airflow for a future where all integrations — including "standard" ones — follow the same interface model.

To maintain compatibility with existing DAGs, the apache-airflow-providers-standard package is installable on both Airflow 2.x and 3.x. Users upgrading from Airflow 2.x are encouraged to begin updating import paths and testing provider installation in advance of the upgrade.

Legacy imports such as airflow.operators.python.PythonOperator are deprecated and will be removed soon. They should be replaced with:

from airflow.providers.standard.operators.python import PythonOperator

The SimpleHttpOperator has been migrated to apache-airflow-providers-http and renamed to HttpOperator

UI & Usability Improvements

Airflow 3.0 introduces a modernized user experience that complements the new React-based UI architecture (see Significant Changes). Several areas of the interface have been enhanced to improve visibility, consistency, and navigability.

New Home Page

The Airflow Home page now provides a high-level operational overview of your environment. It includes health checks for core components (Scheduler, Triggerer, DAG Processor), summary stats for DAG and task instance states, and a real-time feed of asset-triggered events. This view helps users quickly identify pipeline health, recent activity, and potential failures.

Unified DAG List View

The DAG List page has been refreshed with a cleaner layout and improved responsiveness. Users can browse DAGs by name, tags, or owners. While full-text search has not yet been integrated, filters and navigation have been refined for clarity in large deployments.

Version-Aware Graph and Grid Views

The Graph and Grid views now display task information in the context of the DAG version that was used at runtime. This improves traceability for DAGs that evolve over time and provides more accurate debugging of historical runs.

Expanded DAG Graph Visualization

The Graph view now supports visualizing the full chain of asset and task dependencies, including assets consumed or produced across DAG boundaries. This allows users to inspect upstream and downstream lineage in a unified view, making it easier to trace data flows, debug triggering behavior, and understand conditional dependencies between assets and tasks.

DAG Code View

The "Code" tab now displays the exact DAG source as parsed by the scheduler for the selected DAG version. This allows users to inspect the precise code that was executed, even for historical runs, and helps debug issues related to versioned DAG changes.

Improved Task Log Access

Task log access has been streamlined across views. Logs are now easier to access from both the Grid and Task Instance pages, with cleaner formatting and reduced visual noise.

Enhanced Asset and Backfill Views

New UI components support asset-centric DAGs and backfill workflows:

• Asset definitions are now visible from the DAG details page, allowing users to inspect upstream and downstream asset relationships.

• Backfills can be triggered and monitored directly from the UI, including support for scheduler-managed backfills introduced in Airflow 3.0.

These improvements make Airflow more accessible to operators, data engineers, and stakeholders working across both time-based and event-driven workflows.

Deprecations & Removals

A number of deprecated features, modules, and interfaces have been removed in Airflow 3.0, completing long-standing migrations and cleanups.

Users are encouraged to review the following removals to ensure compatibility:

• SubDag support has been removed entirely, including the SubDagOperator, related CLI and API interfaces. TaskGroups are now the recommended alternative for nested DAG structures.

• SLAs have been removed: The legacy SLA feature, including SLA callbacks and metrics, has been removed. A more flexible replacement mechanism, DeadlineAlerts, is planned for a future version of Airflow. Users who relied on SLA-based notifications should consider implementing custom alerting using task-level success/failure hooks or external monitoring integrations.

• Pickling support has been removed: All legacy features related to DAG pickling have been fully removed. This includes the PickleDag CLI/API, as well as implicit behaviors around store_serialized_dags = False. DAGs must now be serialized using the JSON-based serialization system. Ensure any custom Python objects used in DAGs are JSON-serializable.

• Context parameter cleanup: Several previously available context variables have been removed from the task execution context, including conf, execution_date, and dag_run.external_trigger. These values are either no longer applicable or have been renamed (e.g., use dag_run.logical_date instead of execution_date). DAG authors should ensure that templated fields and Python callables do not reference these deprecated keys.

• Deprecated core imports have been fully removed. Any use of airflow.operators.*, airflow.hooks.*, or similar legacy import paths should be updated to import from their respective providers.

• Configuration cleanup: Several legacy config options have been removed, including:

  • scheduler.allow_trigger_in_future: DAG runs can no longer be triggered with a future logical date. Use logical_date=None instead.

  • scheduler.use_job_schedule and scheduler.use_local_tz have also been removed. These options were deprecated and no longer had any effect.

• Deprecated utility methods such as those in airflow.utils.helpers, airflow.utils.process_utils, and airflow.utils.timezone have been removed. Equivalent functionality can now be found in the standard Python library or Airflow provider modules.

• Removal of deprecated CLI flags and behavior: Several CLI entrypoints and arguments that were marked for removal in earlier versions have been cleaned up.

To assist with the upgrade, tools like ruff (e.g., rule AIR302) and airflow config lint can help identify obsolete imports and configuration keys. These utilities are recommended for locating and resolving common incompatibilities during migration. Please see Upgrade Guide for more information.

Summary of Removed Features

The following table summarizes user-facing features removed in 3.0 and their recommended replacements. Not all of these are called out individually above.

Feature	Replacement / Notes
SubDagOperator / SubDAGs	Use TaskGroups
SLA callbacks / metrics	Deadline Alerts (planned post-3.0)
DAG Pickling	Use JSON serialization; pickling is no longer supported
Xcom Pickling	Use custom Xcom backend; pickling is no longer supported
execution_date context var	Use dag_run.logical_date
conf and dag_run.external_trigger	Removed from context; use DAG params or dag_run APIs
Core EmailOperator	Use EmailOperator from the smtp provider
none_failed_or_skipped rule	Use none_failed_min_one_success
dummy trigger rule	Use always
fail_stop argument	Use fail_fast
store_serialized_dags=False	DAGs are always serialized; config has no effect
Deprecated core imports	Import from appropriate provider package
SequentialExecutor & DebugExecutor	Use LocalExecutor for testing
.airflowignore regex	Uses glob syntax by default

Migration Tooling & Upgrade Process

Airflow 3 was designed with migration in mind. Many Airflow 2 DAGs will work without changes, especially if deprecation warnings were addressed in earlier releases. To support the upgrade, Airflow 3 includes validation tools such as ruff and airflow config update, as well as a simplified startup model.

For a step-by-step upgrade process, see the Upgrade Guide.

Minimum Supported Versions

To upgrade to Airflow 3.0, you must be running Airflow 2.7 or later.

Airflow 3.0 supports the following Python versions:

• Python 3.9

• Python 3.10

• Python 3.11

• Python 3.12

Earlier versions of Airflow or Python are not supported due to architectural changes and updated dependency requirements.

DAG Compatibility Checks

Airflow now includes a Ruff-based linter with custom rules to detect DAG patterns and interfaces that are no longer compatible with Airflow 3.0. These checks are packaged under the AIR30x rule series. Example usage:

ruff check dags/ --select AIR301  --preview
ruff check dags/ --select AIR301 --fix  --preview

These checks can automatically fix many common issues such as renamed arguments, removed imports, or legacy context variable usage.

Configuration Migration

Airflow 3.0 introduces a new utility to validate and upgrade your Airflow configuration file:

airflow config update
airflow config update --fix

This utility detects removed or deprecated configuration options and, if desired, updates them in-place.

Additional validation is available via:

airflow config lint

This command surfaces obsolete configuration keys and helps align your environment with Airflow 3.0 requirements.

Metadata Database Upgrade

As with previous major releases, the Airflow 3.0 upgrade includes schema changes to the metadata database. Before upgrading, it is strongly recommended that you back up your database and optionally run:

airflow db clean

to remove old task instance, log, or XCom data. To apply the new schema:

airflow db migrate

Startup Behavior Changes

Airflow components are now started explicitly. For example:

airflow api-server        # Replaces airflow webserver
airflow dag-processor     # Required in all environments

These changes reflect Airflow's new service-oriented architecture.

Resources

• Upgrade Guide

• Airflow AIPs

Airflow 3.0 represents more than a year of collaboration across hundreds of contributors and dozens of organizations. We thank everyone who helped shape this release through design discussions, code contributions, testing, documentation, and community feedback. For full details, migration guidance, and upgrade best practices, refer to the official Upgrade Guide and join the conversation on the Airflow dev and user mailing lists.

Release Date: 2021-05-06

Breaking changes

Change in AutoMLPredictOperator

The params parameter in airflow.providers.google.cloud.operators.automl.AutoMLPredictOperator class was renamed operation_params because it conflicted with a param parameter in the BaseOperator class.

Integration with the apache.beam provider

In 3.0.0 version of the provider we’ve changed the way of integrating with the apache.beam provider. The previous versions of both providers caused conflicts when trying to install them together using PIP > 20.2.4. The conflict is not detected by PIP 20.2.4 and below but it was there and the version of Google BigQuery python client was not matching on both sides. As the result, when both apache.beam and google provider were installed, some features of the BigQuery operators might not work properly. This was cause by apache-beam client not yet supporting the new google python clients when apache-beam[gcp] extra was used. The apache-beam[gcp] extra is used by Dataflow operators and while they might work with the newer version of the Google BigQuery python client, it is not guaranteed.

This version introduces additional extra requirement for the apache.beam extra of the google provider and symmetrically the additional requirement for the google extra of the apache.beam provider. Both google and apache.beam provider do not use those extras by default, but you can specify them when installing the providers. The consequence of that is that some functionality of the Dataflow operators might not be available.

Unfortunately the only complete solution to the problem is for the apache.beam to migrate to the new (>=2.0.0) Google Python clients.

This is the extra for the google provider:

extras_require = ( { # ... "apache.beam" : [ "apache-airflow-providers-apache-beam" , "apache-beam[gcp]" ], # ... }, )

And likewise this is the extra for the apache.beam provider:

extras_require = ({ "google" : [ "apache-airflow-providers-google" , "apache-beam[gcp]" ]},)

You can still run this with PIP version <= 20.2.4 and go back to the previous behaviour:

pip install apache-airflow-providers-google [ apache.beam ]

or

pip install apache-airflow-providers-apache-beam [ google ]

But be aware that some BigQuery operators functionality might not be available in this case.

Features

[Airflow-15245] - passing custom image family name to the DataProcClusterCreateoperator (#15250)

Bug Fixes

Bugfix: Fix rendering of ''object_name'' in ''GCSToLocalFilesystemOperator'' (#15487)

Fix typo in DataprocCreateClusterOperator (#15462)

Fixes wrongly specified path for leveldb hook (#15453)

Release Date: 2021-04-13

Features

Adds 'Trino' provider (with lower memory footprint for tests) (#15187)

update remaining old import paths of operators (#15127)

Override project in dataprocSubmitJobOperator (#14981)

GCS to BigQuery Transfer Operator with Labels and Description parameter (#14881)

Add GCS timespan transform operator (#13996)

Add job labels to bigquery check operators. (#14685)

Use libyaml C library when available. (#14577)

Add Google leveldb hook and operator (#13109) (#14105)

Bug fixes

Google Dataflow Hook to handle no Job Type (#14914)

Release Date: 2021-02-08

Breaking changes

Updated google-cloud-* libraries

This release of the provider package contains third-party library updates, which may require updating your DAG files or custom hooks and operators, if you were using objects from those libraries. Updating of these libraries is necessary to be able to use new features made available by new versions of the libraries and to obtain bug fixes that are only available for new versions of the library.

Details are covered in the UPDATING.md files for each library, but there are some details that you should pay attention to.

Library name

Previous constraints

Current constraints

Upgrade Documentation

google-cloud-automl

=0.4.0,<2.0.0

=2.1.0,<3.0.0

Upgrading google-cloud-automl

google-cloud-bigquery-datatransfer

=0.4.0,<2.0.0

=3.0.0,<4.0.0

Upgrading google-cloud-bigquery-datatransfer

google-cloud-datacatalog

=0.5.0,<0.8

=3.0.0,<4.0.0

Upgrading google-cloud-datacatalog

google-cloud-dataproc

=1.0.1,<2.0.0

=2.2.0,<3.0.0

Upgrading google-cloud-dataproc

google-cloud-kms

=1.2.1,<2.0.0

=2.0.0,<3.0.0

Upgrading google-cloud-kms

google-cloud-logging

=1.14.0,<2.0.0

=2.0.0,<3.0.0

Upgrading google-cloud-logging

google-cloud-monitoring

=0.34.0,<2.0.0

=2.0.0,<3.0.0

Upgrading google-cloud-monitoring

google-cloud-os-login

=1.0.0,<2.0.0

=2.0.0,<3.0.0

Upgrading google-cloud-os-login

google-cloud-pubsub

=1.0.0,<2.0.0

=2.0.0,<3.0.0

Upgrading google-cloud-pubsub

google-cloud-tasks

=1.2.1,<2.0.0

=2.0.0,<3.0.0

Upgrading google-cloud-task

The field names use the snake_case convention

If your DAG uses an object from the above mentioned libraries passed by XCom, it is necessary to update the naming convention of the fields that are read. Previously, the fields used the CamelSnake convention, now the snake_case convention is used.

Before:

set_acl_permission = GCSBucketCreateAclEntryOperator ( task_id = "gcs-set-acl-permission" , bucket = BUCKET_NAME , entity = "user-{{ task_instance.xcom_pull('get-instance')['persistenceIamIdentity'].split(':', 2)[1] }}" , role = "OWNER" , )

After:

set_acl_permission = GCSBucketCreateAclEntryOperator ( task_id = "gcs-set-acl-permission" , bucket = BUCKET_NAME , entity = "user-{{ task_instance.xcom_pull('get-instance')['persistence_iam_identity']" ".split(':', 2)[1] }}" , role = "OWNER" , )

Features

Add Apache Beam operators (#12814)

Add Google Cloud Workflows Operators (#13366)

Replace 'google_cloud_storage_conn_id' by 'gcp_conn_id' when using 'GCSHook' (#13851)

Add How To Guide for Dataflow (#13461)

Generalize MLEngineStartTrainingJobOperator to custom images (#13318)

Add Parquet data type to BaseSQLToGCSOperator (#13359)

Add DataprocCreateWorkflowTemplateOperator (#13338)

Add OracleToGCS Transfer (#13246)

Add timeout option to gcs hook methods. (#13156)

Add regional support to dataproc workflow template operators (#12907)

Add project_id to client inside BigQuery hook update_table method (#13018)

Bug fixes

Fix four bugs in StackdriverTaskHandler (#13784)

Decode Remote Google Logs (#13115)

Fix and improve GCP BigTable hook and system test (#13896)

updated Google DV360 Hook to fix SDF issue (#13703)

Fix insert_all method of BigQueryHook to support tables without schema (#13138)

Fix Google BigQueryHook method get_schema() (#13136)

Fix Data Catalog operators (#13096)

Breaking changes

Updated google-cloud-* libraries

This release of the provider package contains third-party library updates, which may require updating your DAG files or custom hooks and operators, if you were using objects from those libraries. Updating of these libraries is necessary to be able to use new features made available by new versions of the libraries and to obtain bug fixes that are only available for new versions of the library.

Details are covered in the UPDATING.md files for each library, but there are some details that you should pay attention to.

Library name	Previous constraints	Current constraints	Upgrade Documentation
google-cloud-automl	>=0.4.0,<2.0.0	>=2.1.0,<3.0.0	Upgrading google-cloud-automl
google-cloud-bigquery-datatransfer	>=0.4.0,<2.0.0	>=3.0.0,<4.0.0	Upgrading google-cloud-bigquery-datatransfer
google-cloud-datacatalog	>=0.5.0,<0.8	>=3.0.0,<4.0.0	Upgrading google-cloud-datacatalog
google-cloud-dataproc	>=1.0.1,<2.0.0	>=2.2.0,<3.0.0	Upgrading google-cloud-dataproc
google-cloud-kms	>=1.2.1,<2.0.0	>=2.0.0,<3.0.0	Upgrading google-cloud-kms
google-cloud-logging	>=1.14.0,<2.0.0	>=2.0.0,<3.0.0	Upgrading google-cloud-logging
google-cloud-monitoring	>=0.34.0,<2.0.0	>=2.0.0,<3.0.0	Upgrading google-cloud-monitoring
google-cloud-os-login	>=1.0.0,<2.0.0	>=2.0.0,<3.0.0	Upgrading google-cloud-os-login
google-cloud-pubsub	>=1.0.0,<2.0.0	>=2.0.0,<3.0.0	Upgrading google-cloud-pubsub
google-cloud-tasks	>=1.2.1,<2.0.0	>=2.0.0,<3.0.0	Upgrading google-cloud-task

The field names use the snake_case convention

If your DAG uses an object from the above mentioned libraries passed by XCom, it is necessary to update the naming convention of the fields that are read. Previously, the fields used the CamelSnake convention, now the snake_case convention is used.

Before:

set_acl_permission = GCSBucketCreateAclEntryOperator(
    task_id="gcs-set-acl-permission",
    bucket=BUCKET_NAME,
    entity="user-{{ task_instance.xcom_pull('get-instance')['persistenceIamIdentity'].split(':', 2)[1] }}",
    role="OWNER",
)

After:

set_acl_permission = GCSBucketCreateAclEntryOperator(
    task_id="gcs-set-acl-permission",
    bucket=BUCKET_NAME,
    entity="user-{{ task_instance.xcom_pull('get-instance')['persistence_iam_identity']"
    ".split(':', 2)[1] }}",
    role="OWNER",
)

Features

• Add Apache Beam operators (#12814)

• Add Google Cloud Workflows Operators (#13366)

• Replace 'google_cloud_storage_conn_id' by 'gcp_conn_id' when using 'GCSHook' (#13851)

• Add How To Guide for Dataflow (#13461)

• Generalize MLEngineStartTrainingJobOperator to custom images (#13318)

• Add Parquet data type to BaseSQLToGCSOperator (#13359)

• Add DataprocCreateWorkflowTemplateOperator (#13338)

• Add OracleToGCS Transfer (#13246)

• Add timeout option to gcs hook methods. (#13156)

• Add regional support to dataproc workflow template operators (#12907)

• Add project_id to client inside BigQuery hook update_table method (#13018)

Bug fixes

• Fix four bugs in StackdriverTaskHandler (#13784)

• Decode Remote Google Logs (#13115)

• Fix and improve GCP BigTable hook and system test (#13896)

• updated Google DV360 Hook to fix SDF issue (#13703)

• Fix insert_all method of BigQueryHook to support tables without schema (#13138)

• Fix Google BigQueryHook method get_schema() (#13136)

• Fix Data Catalog operators (#13096)

Initial version of the provider.