apache-airflow-providers-imap

📦 PyPI: https://pypi.org/project/apache-airflow/3.2.0/ 📚 Docs: https://airflow.apache.org/docs/apache-airflow/3.2.0/ 🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow/3.2.0/release_notes.html 🐳 Docker Image: "docker pull apache/airflow:3.2.0" 🚏 Constraints: https://github.com/apache/airflow/tree/constraints-3.2.0

## Significant Changes

### Asset Partitioning

The headline feature of Airflow 3.2.0 is asset partitioning — a major evolution of data-aware scheduling. Instead of triggering Dags based on an entire asset, you can now schedule downstream processing based on specific partitions of data. Only the relevant slice of data triggers downstream work, making pipeline orchestration far more efficient and precise.

This matters when working with partitioned data lakes — date-partitioned S3 paths, Hive table partitions, BigQuery table partitions, or any other partitioned data store. Previously, any update to an asset triggered all downstream Dags regardless of which partition changed. Now only the right work gets triggered at the right time.

For detailed usage instructions, see /authoring-and-scheduling/assets.

### Multi-Team Deployments

Airflow 3.2 introduces multi-team support, allowing organizations to run multiple isolated teams within a single Airflow deployment. Each team can have its own Dags, connections, variables, pools, and executors— enabling true resource and permission isolation without requiring separate Airflow instances per team.

This is particularly valuable for platform teams that serve multiple data engineering or data science teams from shared infrastructure, while maintaining strong boundaries between teams' resources and access.

For detailed usage instructions, see /core-concepts/multi-team.

Warning

Multi-Team Deployments are experimental in 3.2.0 and may change in future versions based on user feedback.

### Synchronous callback support for Deadline Alerts

Deadline Alerts now support synchronous callbacks via SyncCallback in addition to the existing asynchronous AsyncCallback. Synchronous callbacks are executed by the executor (rather than the triggerer), and can optionally target a specific executor via the executor parameter.

A Dag can also define multiple Deadline Alerts by passing a list to the deadline parameter, and each alert can use either callback type.

Warning

Deadline Alerts are experimental in 3.2.0 and may change in future versions based on user feedback. Synchronous deadline callbacks (SyncCallback) do not currently support Connections stored in the Airflow metadata database.

For detailed usage instructions, see /howto/deadline-alerts.

### UI Enhancements & Performance

• Grid View Virtualization: The Grid view now uses virtualization -- only visible rows are rendered to the DOM. This dramatically improves performance when viewing Dags with large numbers of task runs, reducing render time and memory usage for complex Dags. (#60241)

• XCom Management in the UI: You can now add, edit, and delete XCom values directly from the Airflow UI. This makes it much easier to debug and manage XCom state during development and day-to-day operations without needing CLI commands. (#58921)

• HITL Detail History: The Human-in-the-Loop approval interface now includes a full history view, letting operators and reviewers see the complete audit trail of approvals and rejections for any task. (#56760, #55952)

• Gantt Chart Improvements:

  • All task tries displayed: Gantt chart now shows every attempt, not just the latest

  • Task display names in Gantt: task_display_name shown for better readability (#61438)

  • ISO dates in Gantt: Cross-browser consistent date format (#61250)

  • Fixed null datetime crash: Gantt chart no longer crashes on tasks with null datetime fields

### New --only-idle flag for the scheduler CLI

The airflow scheduler command has a new --only-idle flag that only counts runs when the scheduler is idle. This helps users run the scheduler once and process all triggered Dags and queued tasks. It requires and complements the --num-runs flag so one can set a small value instead of guessing how many iterations the scheduler needs.

### Replace per-run TI summary requests with a single NDJSON stream

The grid, graph, gantt, and task-detail views now fetch task-instance summaries through a single streaming HTTP request (GET /ui/grid/ti_summaries/{dag_id}?run_ids=...) instead of one request per run. The server emits one JSON line per run as soon as that run's task instances are ready, so columns appear progressively rather than all at once.

What changed:

• GET /ui/grid/ti_summaries/{dag_id}?run_ids=... is now the sole endpoint for TI summaries, returning an application/x-ndjson stream where each line is a serialized GridTISummaries object for one run.

• The old single-run endpoint GET /ui/grid/ti_summaries/{dag_id}/{run_id} has been removed.

• The serialized Dag structure is loaded once and shared across all runs that share the same dag_version_id, avoiding redundant deserialization.

• All UI views (grid, graph, gantt, task instance, mapped task instance, group task instance) use the stream endpoint, passing one or more run_ids.

### Structured JSON logging for all API server output

The new json_logs option under the [logging] section makes Airflow produce all its output as newline-delimited JSON (structured logs) instead of human-readable formatted logs. This covers the API server (gunicorn/uvicorn), including access logs, warnings, and unhandled exceptions.

Not all components support this yet — notably airflow celery worker but any non-JSON output when json_logs is enabled will be treated as a bug. (#63365)

### Remove legacy OTel Trace metaclass and shared tracer wrappers

The interfaces and functions located in airflow.traces were internal code that provided a standard way to manage spans in internal Airflow code. They were not intended as user-facing code and were never documented. They are no longer needed so we remove them in 3.2. (#63452)

### Move task-level exception imports into the Task SDK

Airflow now sources task-facing exceptions (AirflowSkipException, TaskDeferred, etc.) from airflow.sdk.exceptions. airflow.exceptions still exposes the same exceptions, but they are proxies that emit DeprecatedImportWarning so Dag authors can migrate before the shim is removed.

What changed:

• Runtime code now consistently raises the SDK versions of task-level exceptions.

• The Task SDK redefines these classes so workers no longer depend on airflow-core at runtime.

• airflow.providers.common.compat.sdk centralizes compatibility imports for providers.

Behaviour changes:

• Sensors and other helpers that validate user input now raise ValueError (instead of AirflowException) when poke_interval/ timeout arguments are invalid.

• Importing deprecated exception names from airflow.exceptions logs a warning directing users to the SDK import path.

Exceptions now provided by ``airflow.sdk.exceptions``:

• AirflowException and AirflowNotFoundException

• AirflowRescheduleException and AirflowSensorTimeout

• AirflowSkipException, AirflowFailException, AirflowTaskTimeout, AirflowTaskTerminated

• TaskDeferred, TaskDeferralTimeout, TaskDeferralError

• DagRunTriggerException and DownstreamTasksSkipped

• AirflowDagCycleException and AirflowInactiveAssetInInletOrOutletException

• ParamValidationError, DuplicateTaskIdFound, TaskAlreadyInTaskGroup, TaskNotFound, XComNotFound

• AirflowOptionalProviderFeatureException

Backward compatibility:

• Existing Dags/operators that still import from airflow.exceptions continue to work, though they log warnings.

• Providers can rely on airflow.providers.common.compat.sdk to keep one import path that works across supported Airflow versions.

Migration:

• Update custom operators, sensors, and extensions to import exception classes from airflow.sdk.exceptions (or from the provider compat shim).

• Adjust custom validation code to expect ValueError for invalid sensor arguments if it previously caught AirflowException.

### Support numeric multiplier values for retry_exponential_backoff parameter

The retry_exponential_backoff parameter now accepts numeric values to specify custom exponential backoff multipliers for task retries. Previously, this parameter only accepted boolean values (True or False), with True using a hardcoded multiplier of 2.0.

New behavior:

• Numeric values (e.g., 2.0, 3.5) directly specify the exponential backoff multiplier

• retry_exponential_backoff=2.0 doubles the delay between each retry attempt

• retry_exponential_backoff=0 or False disables exponential backoff (uses fixed retry_delay)

Backwards compatibility:

Existing Dags using boolean values continue to work:

• retry_exponential_backoff=True → converted to 2.0 (maintains original behavior)

• retry_exponential_backoff=False → converted to 0.0 (no exponential backoff)

API changes:

The REST API schema for retry_exponential_backoff has changed from type: boolean to type: number. API clients must use numeric values (boolean values will be rejected).

Migration:

While boolean values in Python Dags are automatically converted for backwards compatibility, we recommend updating to explicit numeric values for clarity:

• Change retry_exponential_backoff=True → retry_exponential_backoff=2.0

• Change retry_exponential_backoff=False → retry_exponential_backoff=0

### Move serialization/deserialization (serde) logic into Task SDK

Airflow now sources serde logic from airflow.sdk.serde instead of airflow.serialization.serde. Serializer modules have moved from airflow.serialization.serializers.* to airflow.sdk.serde.serializers.*. The old import paths still work but emit DeprecatedImportWarning to guide migration. The backward compatibility layer will be removed in Airflow 4.

What changed:

• Serialization/deserialization code moved from airflow-core to task-sdk package

• Serializer modules moved from airflow.serialization.serializers.* to airflow.sdk.serde.serializers.*

• New serializers should be added to airflow.sdk.serde.serializers.* namespace

Code interface changes:

• Import serializers from airflow.sdk.serde.serializers.* instead of airflow.serialization.serializers.*

• Import serialization functions from airflow.sdk.serde instead of airflow.serialization.serde

Backward compatibility:

• Existing serializers importing from airflow.serialization.serializers.* continue to work with deprecation warnings

• All existing serializers (builtin, datetime, pandas, numpy, etc.) are available at the new location

Migration:

• For existing custom serializers: Update imports to use airflow.sdk.serde.serializers.*

• For new serializers: Add them to airflow.sdk.serde.serializers.* namespace (e.g., create task-sdk/src/airflow/sdk/serde/serializers/your_serializer.py)

### Methods removed from PriorityWeightStrategy

On (experimental) class PriorityWeightStrategy, functions serialize() and deserialize() were never used anywhere, and have been removed. They should not be relied on in user code. (#59780)

### Methods removed from TaskInstance

On class TaskInstance, functions run(), render_templates(), get_template_context(), and private members related to them have been removed. The class has been considered internal since 3.0, and should not be relied on in user code. (#59780, #59835)

### Modify the information returned by DagBag

New behavior:

• DagBag now uses Path.relative_to for consistent cross-platform behavior.

• FileLoadStat now has two additional nullable fields: bundle_path and bundle_name.

Backward compatibility:

FileLoadStat will no longer produce paths beginning with / with the meaning of "relative to the dags folder". This is a breaking change for any custom code that performs string-based path manipulations relying on this behavior. Users are advised to update such code to use pathlib.Path. (#59785)

### Remove --conn-id option from airflow connections list

The redundant --conn-id option has been removed from the airflow connections list CLI command. Use airflow connections get instead. (#59855)

### Add operator-level render_template_as_native_obj override

Operators can now override the Dag-level render_template_as_native_obj setting, enabling fine-grained control over whether templates are rendered as native Python types or strings on a per-task basis. Set render_template_as_native_obj=True or False on any operator to override the Dag setting, or leave as None (default) to inherit from the Dag.

### Add gunicorn support for API server with zero-downtime worker recycling

The API server now supports gunicorn as an alternative server with rolling worker restarts to prevent memory accumulation in long-running processes.

Key Benefits:

• Rolling worker restarts: New workers spawn and pass health checks before old workers are killed, ensuring zero downtime during worker recycling.

• Memory sharing: Gunicorn uses preload + fork, so workers share memory via copy-on-write. This significantly reduces total memory usage compared to uvicorn's multiprocess mode where each worker loads everything independently.

• Correct FIFO signal handling: Gunicorn's SIGTTOU kills the oldest worker (FIFO), not the newest (LIFO), which is correct for rolling restarts.

Configuration:

[api]
# Use gunicorn instead of uvicorn
server_type = gunicorn

# Enable rolling worker restarts every 12 hours
worker_refresh_interval = 43200

# Restart workers one at a time
worker_refresh_batch_size = 1

Or via environment variables:

export AIRFLOW__API__SERVER_TYPE=gunicorn
export AIRFLOW__API__WORKER_REFRESH_INTERVAL=43200

Requirements:

Install the gunicorn extra: pip install 'apache-airflow-core[gunicorn]'

Note on uvicorn (default):

The default uvicorn mode does not support rolling worker restarts because:

1. With workers=1, there is no master process to send signals to

2. uvicorn's SIGTTOU kills the newest worker (LIFO), defeating rolling restart purposes

3. Each uvicorn worker loads everything independently with no memory sharing

If you need worker recycling or memory-efficient multi-worker deployment, use gunicorn. (#60921)

### Improved performance of rendered task instance fields cleanup for Dags with many mapped tasks (~42x faster)

The config max_num_rendered_ti_fields_per_task is renamed to num_dag_runs_to_retain_rendered_fields (old name still works with deprecation warning).

Retention is now based on the N most recent dag runs rather than N most recent task executions, which may result in fewer records retained for conditional/sparse tasks. (#60951)

### AuthManager Backfill permissions are now handled by the requires_access_dag on the DagAccessEntity.Run

is_authorized_backfill of the BaseAuthManager interface has been removed. Core will no longer call this method and their provider counterpart implementation will be marked as deprecated. Permissions for backfill operations are now checked against the DagAccessEntity.Run permission using the existing requires_access_dag decorator. In other words, if a user has permission to run a Dag, they can perform backfill operations on it.

Please update your security policies to ensure that users who need to perform backfill operations have the appropriate DagAccessEntity.Run permissions. (Users having the Backfill permissions without having the DagRun ones will no longer be able to perform backfill operations without any update)

### Python 3.14 support added

Airflow 3.2.0 adds support for Python 3.14. (#63787)

### Reduce API server memory by eliminating SerializedDAG loads on task start

The API server no longer loads the full SerializedDAG when starting tasks, significantly reducing memory usage. (#60803)

### Remove MySQL client from container images

MySQL client support has been removed from official Airflow container images. MySQL users building on official images must install the client themselves. (#57146)

### Add support for async callables in PythonOperator

The PythonOperator parameter python_callable now also supports async callables in Airflow 3.2, allowing users to run async def functions without manually managing an event loop. (#60268)

### Make start_date optional for @continuous schedule

The schedule="@continuous" parameter now works without requiring a start_date, and any Dags with this schedule will begin running immediately when unpaused. (#61405)

## New Features

• Add FIPS support by making Python LTO configurable via PYTHON_LTO build argument (#58337)

• Add support for task queue-based Trigger assignment to specific Triggerer hosts via the new --queues CLI option for the trigger command (#59239)

• Add --show-values and --hide-sensitive flags to CLI connections list and variables list to hide sensitive values by default (#62344)

• Add support for setting individual secrets backend kwargs via AIRFLOW__SECRETS__BACKEND_KWARG__<KEY> environment variables (#63312)

• Add only_new parameter to Dag clear to only clear newly added task instances (#59764)

• Add log_timestamp_format config option for customizing component log timestamps (#63321)

• Add --action-on-existing-key option to pools import and connections import CLI commands (#62702)

• Add back --use-migration-files flag for airflow db init (#62234)

• Add AllowedKeyMapper for partition key validation in asset partitioning (#61931)

• Add ChainMapper for chaining multiple partition mappers (#64094)

• Add cryptographic signature verification for Python source packages in Docker builds (#63345)

• Add Human-in-the-Loop (HITL) Review system for AgenticOperator (#63081)

• Add @task.stub decorator to allow tasks in other languages to be defined in Dags (#56055)

• Add support for creating connections using URI in SDK (#62211)

• Add note support to TriggerDagRunOperator (#60810)

• Add allowed_run_types to whitelist specific Dag run types (#61833)

• Add OR operator support in API search parameters (#60008)

• Add API filtering for Dags by timetable type (#58852)

• Add wildcard support for dag_id and dag_run_id in bulk task instance endpoint (#57441)

• Add operator_name_pattern, pool_pattern, queue_pattern as task instance search filters (#57571)

• Add update_mask support for bulk PATCH APIs (#54597)

• Add asset event emission listener event (#61718)

• Add source parameter to Param (#58615)

• Add lazy filtering for inlet events by time range, ordering, and limit (#54891)

• Add ability to get previous TaskInstance on RuntimeTaskInstance (#59712)

• Add required context messages to all DagRun state change notifications (#56272)

• Add max_trigger_to_select_per_loop config for Triggerer HA setup (#58803)

• Add uvicorn_logging_level config option to control API server access logs (#56062)

• Add correlation-id support to Execution API for request tracing (#57458)

• Add executor.running_dags gauge metric to expose count of running Dags (#52815)

• Add submodules support to GitDagBundle (#59911)

• Add HTTP URL authentication support to GitHook for Dag bundles (#58194)

• Add stream method to RemoteIO for ObjectStorage (#54813)

• Add CLI hot-reload support via --dev flag (#57741)

• Add auth list-envs command to list CLI environments and auth status (#61426)

• Add Dag bundles to airflow info command output (#59124)

• Add new arguments to db_clean to explicitly include or exclude Dags (#56663)

• UI: Add Jobs page to the Airflow UI (#61512)

• UI: Add version change indicators for Dag and bundle versions in Grid view (#53216)

• UI: Add segmented state bar for collapsed task groups and mapped tasks (#61854)

• UI: Add date range filter for Dag executions (#60772)

• UI: Add "Select Recent Configurations" to trigger form, restoring Airflow 2 functionality (#56406)

• UI: Add copy button to logs (#61185)

• UI: Add filename display to Dag Code tab for easier file identification (#60759)

• UI: Add Dag run state filter to grid view options (#55898)

• UI: Add task upstream/downstream filter to Graph and Grid views (#57237)

• UI: Add filters to Task Instances tab (#56920)

• UI: Add display of active Dag runs count in header with auto-refresh (#58332)

• UI: Add Dag ID pattern search to Dag Runs and Task Instances pages (#55691)

• UI: Add delete button for Dag runs in more options menu (#55696)

• UI: Add depth filter to TaskStreamFilter (#60549)

• UI: Add theme config support (#58411)

• UI: Add support for globalCss in custom themes (#61161)

• UI: Add display of logged-in user in settings button (#58981)

• UI: Add tooltip for explaining task filter traversal (#61401)

• UI: Add self-service JWT token generation for API and CLI access (#63195)

• UI: Add bulk operations for edge workers page (#64033)

• UI: Add real-time concurrency control for edge workers (#63142)

• UI: Add run_after date filter on Dag runs page (#62797)

• UI: Add bundle version filter on Dag runs page (#62810)

• UI: Add icon support for theme customization (#62172)

• UI: Add Monaco editor for all JSON editing fields (#62708)

• UI: Add run type legend tooltip to grid view (#62946)

• UI: Allow customizing gray, black, and white color tokens in AIRFLOW__API__THEME in addition to brand (#64232)

## Bug Fixes

• Fix sensitive configuration values not being masked in public config APIs; treat the deprecated non-sensitive-only value as True (#59880)

• Fix InvalidStatsNameException for pool names with invalid characters by auto-normalizing them when emitting metrics (#59938)

• Fix JWT tokens appearing in task logs by excluding the token field from workload object representations (#62964)

• Fix security iframe navigation when AIRFLOW__API__BASE_URL basename is configured (#63141)

• Fix grid view URL for dynamic task groups producing 404 by not appending /mapped to group URLs (#63205)

• Fix ti_skip_downstream overwriting RUNNING tasks to SKIPPED in HA deployments (#63266)

• Fix duplicate task execution when running multiple schedulers (#60330)

• Fix callback starvation across Dag bundles (#63795)

• Fix @task decorator failing for tasks that return falsy values like 0 or empty string (#63788)

• Fix LatestOnlyOperator not working when direct upstream of a dynamically mapped task (#62287)

• Fix inconsistent XCom return type in mapped task groups with dynamic mapping (#59104)

• Fix task group lookup using wrong Dag version for historical runs, causing 404 errors in grid view (#63360)

• Fix import errors when updating Dags in other bundles (#63615)

• Fix DagRun span emission crash when context_carrier is None (#64087)

• Fix false error logs for partitioned timetables when next_dagrun fields are None (#63962)

• Fix timetable serialization error when decoding relativedelta (#61671)

• Fix task_instance_mutation_hook receiving run_id=None during TaskInstance creation (#63049)

• Fix scheduler crash on None dag_version access (#62225)

• Fix MetastoreBackend.expunge_all() corrupting shared session state (#63080)

• Fix triggerer logger file descriptor closed prematurely when trigger is removed (#62103)

• Fix airflowignore negation pattern handling for directory-only patterns (#62860)

• Fix false warnings for TYPE_CHECKING-only forward references in TaskFlow decorators (#63053)

• Fix structlog JSON serialization crash on non-serializable objects (#62656)

• Fix backward compatibility for deadline alert serialization (#63701)

• Fix queued_tasks type mismatch in hybrid executors (CeleryKubernetesExecutor, LocalKubernetesExecutor) (#63744)

• Fix Celery tasks not being registered at worker startup (#63110)

• Fix asset partition detection incorrectly identifying Dags as partitioned (#62864)

• Fix pathlib.Path objects incorrectly resolved by Jinja templater in Task SDK (#63306)

• Fix state mismatch in Kubernetes executor after pod completion (#63061)

• Fix make_partial_model for API Pydantic models (#63716)

• Fix WTForms validator compatibility in connection form (#63823)

• Fix _execution_api_server_url() ignoring configured value and falling back to edge config (#63192)

• Fix DetachedInstanceError for airflow tasks render command (#63916)

• Fix scheduler isolating per-dag-run failures to prevent a single DagRun crashing all scheduling (#62893)

• Fix task argument order in @task definition causing Dag parsing errors (#62174)

• Fix limit parameter not sent in execute_list server requests (#63048)

• Fix circular import from airflow.configuration causing ImportError on Python 3.14 (#63787)

• Fix map_index range validation in CLI commands (#62626)

• Fix nullable ORM fields by restoring correct defaults and dropping unreleased corrective migration (#63899)

• Fix race condition in auth manager initialization on concurrent requests (#62431)

• Fix FabAuthManager race condition on startup with multiple workers (#62737)

• Fix FabAuthManager race condition when workers concurrently create permissions, roles, and resources (#63842)

• Fix JWTValidator not handling GUESS algorithm with JWKS (#63115)

• Fix FabAuthManager first idle MySQL disconnect in token auth (#62919)

• Fix JWTBearerTIPathDep import errors in Human-In-The-Loop routes (#63277)

• Fix 403 from roles endpoint despite admin rights in FAB provider (#64097)

• Fix task log filters not working in full-screen mode (#62747)

• Fix duplicate log reads when resuming from log_pos (#63531)

• Fix 404 errors from worker log server for historical retry attempts now handled gracefully (#62475)

• Fix Elasticsearch/OpenSearch logging exception details missing in task log tab (#63739)

• Fix task-level audit logs missing success/running events (#61932)

• Fix null dag_run_conf causing serialization error in BackfillResponse (#63259)

• Fix CLI asset materialization using wrong Dag run type (#63815)

• Fix migration 0094 performance: use SQL instead of Python deserialization (#63628)

• Fix migration reliability: replace savepoints with per-Dag transactions (#63591)

• Fix slow downgrade performance by adding index to deadline.callback_id (#63612)

• Fix MySQL reserved keyword interval causing query failures in deadline_alert (#63494)

• Fix MySQL serialize_dag query failure during deadline migration (#63804)

• Fix SQLite downgrade failures caused by FK constraints during batch table recreation (#63437)

• Fix migration 0096 downgrade failing when team table has existing rows (#63449)

• Fix missing warning about hardcoded 24h visibility_timeout that kills long-running Celery tasks (#62869)

• Fix scheduler memory issue by removing eager loading of all task instances (#60956)

• Fix MySQL sort buffer overflow in deadline alert migration (#61806)

• Fix failing to manually trigger a Dag with CronPartitionedTimetable (#62441)

• Fix race condition in AssetModel when updating asset partition DagRun — adds mutex lock (#59183)

• Fix FAB auth_manager load_user causing PendingRollbackError (#61943)

• Fix N+1 query: add joinedload for asset in dags_needing_dagruns() (#60957)

• Fix Dag Processor health check threshold matching SchedulerJob/TriggererJob pattern (#58704)

• Fix NotMapped exception when clearing task instances with downstream/upstream (#58922)

• Fix missing asset events for partitioned DagRun (#61433)

• Fix missing partition_key filter in PALK when creating DagRun (#61831)

• Fix Dag params API contract broken by earlier change (#56831)

• Fix OAuth session race condition causing false 401 errors during login (#61287)

• Fix ObjectStoragePath to exclude conn_id from storage options passed to fsspec (#62701)

• Fix unable to import list value for Variable (#61508)

• Fix plugin registration returning early on duplicate names (#60498)

• Fix circular import when using XComObjectStorageBackend (#55805)

• Fix deadline alert hashing bug (#61702)

• Fix task SDK to read default_email_on_failure/default_email_on_retry from config (#59912)

• Fix Celery worker crash on macOS due to non-serializable local function (#62655)

• Fix Redis import race condition in Celery executor (#61362)

• Fix incorrect state query parameter for task instances in Dashboard (#59086)

• Fix TaskInstance.get_dagrun returning None in task_instance_mutation_hook (#60726)

• Fix Simple Auth Manager login showing cryptic error on failed authentication (#64303)

• Fix dag_display_name property bypass for DagStats query (#64256)

• Fix TaskAlreadyRunningError not raised when starting an already-running task instance (#60855)

• Fix Teardown tasks not waiting for all in-scope tasks to complete (#64181)

• Fix enable_swagger_ui config not respected in API server (#64376)

• Fix: add check for xcom permission when result is specified for DagRun wait endpoint (#64415)

• Fix conf.has_option not respects default provider metadata (#64209)

• Fix teardown scope causing unnecessary database writes during task scheduling (#64558)

• Fix live task log output not visible in stdout when using Elasticsearch log forwarding (#64067)

• Fix TaskInstance crash when refreshing task weight for non-serialized operators (#64557)

• Fix Variables secrets backend conflict check exiting early when multiple backends are configured (#64062)

• UI: Fix Dag run accessor key on clear task instance page (#64072)

• UI: Fix searchable dropdown not working for Dag params enum fields (#63895)

• UI: Fix newline rendering in Dag warning alert (#63588)

• UI: Fix XCom edit modal value not repopulating on reopen (#62798)

• UI: Fix task duration tooltip not displaying correctly (#63639)

• UI: Fix elapsed time not showing for running tasks (#63619)

• UI: Fix RenderedJsonField collapse behavior (#63831)

• UI: Fix RenderedJsonField not displaying in table cells (#63245)

• UI: Fix full-screen log dropdown z-index after Chakra upgrade (#63816)

• UI: Fix asset materialization run type display (#63819)

• UI: Fix pools with unlimited (-1) slots not rendering correctly (#62831)

• UI: Fix DurationChart labels and disable animation flicker during auto-refresh (#62835)

• UI: Fix 403 error not shown when unauthorized user re-parses Dag (#61560)

• UI: Fix logical date filter on /dagruns page not working (#62848)

• UI: Fix inflated total_received count in partitioned Dag runs view (#62786)

• UI: Fix edge executor navigation when behind reverse proxy with subpath (#63777)

• UI: Fix queries not invalidated on Dag run add/delete (#64269)

• UI: Fix RenderedJsonField flickering when collapsed (#64261)

• UI: Fix Docs menu REST API link visibility when API docs are disabled (#64359)

• UI: Fix TISummaries not refreshing when gridRuns are invalidated (#64113)

• UI: Fix guard against null/undefined dates in Gantt chart to prevent RangeError (#64031)

• UI: Block polling requests to endpoints that returned 403 Forbidden (#64333)

• UI: Fix Gantt view still visible when time range is outside DagRun window (#64179)

• UI: Fix Human-in-the-Loop (HITL) operator options not displaying when exactly 4 choices are configured (#64453)

## Miscellaneous

• Deprecate api.page_size config in favor of api.fallback_page_limit (#61067)

• Improve Dag callback relevancy by passing a context-relevant task instance based on the Dag's final state instead of an arbitrary lexicographical selection (#61274)

• Optimize get_dag_runs API endpoint performance (#63940)

• Improve historical metrics endpoint performance (#63526)

• Add TTL cache with single-flight deduplication to Keycloak filter_authorized_dag_ids (#63184)

• Reduce Celery worker memory usage with gc.freeze (#62212)

• Eliminate duplicate JOINs in get_task_instances endpoint (#62910)

• Replace large IN clause in asset queries with CTE and JOIN for better SQL performance (#62114)

• Add row lock to prevent race conditions during asset-triggered DagRun creation (#60773)

• Add ConnectionResponse serializer safeguard to prevent accidental sensitive field exposure (#63883)

• Add missing dag_id filter on DagRun task instances API query (#62750)

• Add missing HTTP timeout to FAB JWKS fetching (#63058)

• Add additional permission check in asset materialization endpoint (#63338)

• Filter backfills list by readable Dags (authorization enforcement) (#63003)

• Hide SQL statements in exception details when expose_stacktrace is disabled (#63028)

• Use default max depth to redact Variable values in API responses (#63480)

• Validate update_mask fields in PATCH API endpoints against Pydantic models (#62657)

• Align key/id path validation for variables and connections in Execution API (#63897)

• Add order_by parameter to GET /permissions endpoint for pagination consistency (#63418)

• Implement truncation logic for rendered template values (#61878)

• Add BaseXcom to airflow.sdk public exports (#63116)

• Make TaskSDK conf respect default config from provider metadata (#62696)

• Add OTel trace import shim via airflow.sdk.observability.trace (#63554)

• Improve 3.2.0 deadline migration performance (#63920)

• Improve 3.2.0 downgrade migration for external_executor_id on PostgreSQL (#63625)

• Skip backfilling old DagRun.created_at during migration for faster upgrades (#63825)

• Add INFO-level logging to asset scheduling path (#63958)

• Improve log file template for ExecuteCallback by including dag_id and run_id (#62616)

• Improve Dag processor timeout logging clarity (#62328)

• Deprecate get_connection_form_widgets and get_ui_field_behaviour hook methods (#63711)

• Add missing deprecation warnings for [workers] config section (#63659)

• Expose TaskInstance API for external task management (#61568)

• Remove deprecated airflow.datasets, airflow.timetables.datasets, and airflow.utils.dag_parsing_context modules (#62927)

• Remove PyOpenSSL from core dependencies (#63869)

• Optimize fail-fast check to avoid loading SerializedDAG (#56694)

• Improve performance of task queue processing by switching from pop(0) to popleft() (#61376)

• Optimize K8s API usage for watching pod events, fixing hanging communication (#59080)

• Remove N+1 database queries for team names (#61471)

• Improve XCom value handling in extra links API (#61641)

• Remove .git folder from versions in GitDagBundle to reduce storage size (#57069)

• Deprecate subprocess exec utils from airflow.utils.process_utils (#57193)

• Improve error handling in edge worker on 405 responses (#60425)

• Improve deferrable KubernetesPodOperator handling of deleted pods between polls (#56976)

• Improve event log entries when a pod fails for K8s executor (#60800)

• Refactor XCom API to use shared serialization constants (#64148)

• Improve temporal mapper to be timezone aware for asset partitioning (#62709)

• Improve dag version inflation checker logic and fix false-positive detection (#61345)

• Rename ToXXXMapper to StartOfXXXMapper in partition-mapper for clarity (#64160)

• Run DB check only for core components in prod entrypoint (#63413)

• Fix partitioned asset events incorrectly triggering non-partition-aware Dags (#63848)

• Improve partitioned DagRun sorting by partition_date (#62866)

• Allow gray, black, and white color tokens in AIRFLOW__API__THEME config (#64232)

• Add parent task spans and nest worker/trigger spans for improved observability (#63839)

• UI: Enhance code view to support search and diff (#55467)

• UI: Improve UX for adding custom DeadlineReferences (#57222)

• UI: Enhance FilterBar with DateRangeFilter for compact UI (#56173)

• UI: Move deadline alerts into their own table for UI integration (#58248)

• UI: Persist tag filter selection in Dag grid view (#63273)

• UI: Show HITL review tab only for review-enabled task instances (#63477)

• UI: Updated button styles for adding Connections, Variables, and Pools (#62607)

• UI: Add clear permission toast for 403 errors on user actions (#61588)

## Doc Only Changes

• Add documentation marking pre/post-execute task hooks as GA (no longer experimental) (#59656)

• Add RedisTaskHandler configuration example (#63898)

• Add documentation explaining difference between deferred vs async operators (#63500)

• Add auth manager section in multi-team documentation (#63208)

• Add documentation about shared libraries in _shared folders (#63468)

• Clarify plugin folder module registration in modules_management docs (#63634)

• Clarify max_active_tasks Dag parameter documentation (#63217)

• Clarify HLL in extraction precedence docs (#63723)

• Clarify Ubuntu/Debian venv requirement in quick start guide (#63244)

• Fix Git connection docs to match actual GitHook parameters (#63265)

• Mention Python 3.14 support in docs (#63950)

• Add Dag documentation for example_bash_decorator (#62948)

• Add Russian translation for UI (#63450)

• Add Hungarian translation (#62925)

• Complete Traditional Chinese translations (#62652)

• Add asset partition documentation (#63262)

• Add guide for dag version inflation and its checker (#64100)

📦 PyPI: https://pypi.org/project/apache-airflow/3.1.1/ 📚 Docs: https://airflow.apache.org/docs/apache-airflow/3.1.1/ 🛠️ Release Notes: https://airflow.apache.org/docs/apache-airflow/3.1.1/release_notes.html 🪶 Sources: https://airflow.apache.org/docs/apache-airflow/3.1.1/installation/installing-from-sources.html 🐳 Docker Image: "docker pull apache/airflow:3.1.1" 🚏 Constraints: https://github.com/apache/airflow/tree/constraints-3.1.1

Bug Fixes

• Fix execution failures with NULL dag_run.conf during upgrades from earlier versions (#56729)
• Fix memory leak in remote logging connection cache (#56695)
• Fix DAG processor crash with pre-import module optimization enabled (#56779)
• Fix scheduler crash with email notifications (#56431)
• Fix scheduler crash during 3.0 to 3.1 migration when retry_delay is None (#56236)
• Fix task retries executing wrong method after deferred state (#56737)
• Fix retry callbacks not executing for externally killed tasks (#56607)
• Fix custom timetable generate_run_id not called for manual triggers (#56699)
• Fix KeyError when accessing retry_delay on MappedOperator without explicit value (#56605)
• Fix task-sdk connection error handling to match airflow-core behavior (#56653)
• Fix topological sort for Grid View (#56963)
• Fix get_ti_count and get_task_states access in callback requests (#56860)
• Fix Connection or Variable access in Server context (#56602)
• Fix .airflowignore order precedence (#56832)
• Fix migration errors for Pydantic 2.12.0 compatibility (#56581)
• Fix: Correctly parse JSON for --dag_run_conf in airflow dags backfill CLI (#56599)
• UI: Fix note modal does not change markdown text after change (#56092)
• UI: Fix Grid for cleared runs when tasks were removed (#56297)
• UI: Fix log text selection contrast in light mode (#56893)
• UI: Fix Advanced Search button overlap in DAG List View (#56777)
• UI: Fix view for many DAG tags (#55604)
• UI: Fix asset name text overflow in DAGs list view (#55914)
• UI: Fix auto refresh when only 1 dag run is running (#56649)
• UI: Fix UI keeps poking pools API when no permission (#56626)
• UI: Fix multi-line drag selection in task log view (#56300)
• UI: Fix task named 'root' causes blue screen on hover (#56926)
• UI: Fix cron expression display for Day-of-Month and Day-of-Week conflicts (#56255)
• UI: Fix Grid view performance issues with SerializedDagModel query optimization (#56938)
• Fix: Gracefully handle FastAPI plugins with empty url_prefix (#55262)
• Fix: Allow mapped tasks to accept zero-length inputs on rerun (#56162)
• Fix: Enable API to clear task instances by specifying map indexes (#56945)
• Fix: Add max_retry_delay to MappedOperator model (#56951)
• Fix: Use name passed to @asset decorator when fetching the asset (#56611)
• UI: Add English as a fallback locale (#56934)

Miscellaneous

• Add Greek UI translation (#56724)
• Add Thai UI translation (#56946)
• Add Polish translations (#56825)
• Close German translation gaps for full UI translation (#56981)
• Fix Hebrew typo in translations (#56168)
• Improve DAG and task missing error handling in callbacks (#56725)
• Improve UI retry strategy on client errors (#56638)
• Improve get dag grid structure endpoint speed (#56937)
• Optimize grid structure query with DISTINCT for dag_version_id lookup (#56565)
• Add configurable timeout for Execution API requests (#56969)
• Prevent unnecessary kubernetes client imports in workers (#56692)
• Lazy import PodGenerator for deserialization (#56733)
• Serialize pydantic models in json mode for JSON serialization compatibility (#56939)
• Update authentication to handle JWT token in backend (#56677)
• Update bulk API permission check to handle action_on_existence (#56672)
• Migrate CreateAssetEventsBody to Pydantic v2 ConfigDict (#56772)
• Restore timetable active_runs_limit check (#56922)
• Add is_favorite to UI dags list (#56341)
• Add executor, hostname, and queue columns to TaskInstances page (#55922)
• Add resize function for DAG Documentation (#56344)
• Add optional pending dag runs check to auto refresh (#56648)
• Add auto refresh to backfill banner (#56774)
• UI: Add Expand/Collapse all to XComs page (#56285)
• UI: Update recent runs bar chart and improve responsiveness (#56314)
• UI: Update duration format to show milliseconds (#56961)
• UI: Modify min width for task names in grid view (#56952)
• UI: Use Task Display Name in Graph if existing (#56511)
• UI: Use Task Display Name in Grid if existing (#56410)
• UI: Use TI duration from database instead of UI calculated (#56329)
• UI: Make DAG Run ID visible in DAG Header Card (#56409)
• UI: Modify calendar cell colors (#56161)
• UI: Modify log highlight color (#56894)
• UI: Fix show appropriate time units in grid view (#56414)
• UI: Reduce default columns of DAG Run and Task Instance lists (#55968)
• UI: Add expand and collapse functionality for task groups (#56334)
• UI: Avoid using rem for icons for Safari compatibility (#56304)
• UI: Add ANSI support to log viewer (#56721)
• UI: Support Dynamic UI Alerts (#56259)
• UI: Disable Gantt view by default (#56242)
• UI: Use welcome on dashboard instead of airflow (#56074)
• UI: Improve clipboard button visibility with hover effect (#56484)
• UI: Allow sub-pages in React UI plugins (#56485)
• Include task instance id in log printed by supervisor (#56383)
• Emit log stream stopped warning as ndjson (#56480)
• Detect interactive terminal to set colored logging with override env variable support (#56157)
• Add back deprecation warning for sla_miss_callback (#56127)
• Move natsort dependency to airflow-core (#56582)
• Added missing babel dependency in Task SDK (#56592)
• Remove unused dagReports API endpoint (#56621)

Doc Only Changes

• Improve API sort documentation (#56617)
• Improve API doc for ordering query param (#55988)
• Add Audit Logs detailed documentation (#56719)
• Update serializer document to reflect latest changes in codebase (#56857)
• Update ASF logos in documentation to the new Oak logo (#56601)
• Enhance triggering_asset_event retrieval documentation in DAGs (#56957)
• Remove self-reference in best practices documentation (#56111)
• Fix supported Python versions in README (#56734)

Full Changelog: https://github.com/apache/airflow/compare/3.1.0...3.1.1

We are thrilled to announce the release of Apache Airflow 3.1.0, an update that puts humans at the center of data workflows.

Read more about what 3.1.0 brings in https://airflow.apache.org/blog/airflow-3.1.0/

📦 PyPI: https://pypi.org/project/apache-airflow/3.1.0/
📚 Core Airflow Docs: https://airflow.apache.org/docs/apache-airflow/3.1.0/
📚 Task SDK Docs: https://airflow.apache.org/docs/task-sdk/1.1.0/
🛠️ Release Notes: https://airflow.apache.org/docs/apache-airflow/3.1.0/release_notes.html
🚏 Constraints: https://github.com/apache/airflow/tree/constraints-3.1.0

Apache Airflow 3.1.0 represents an extraordinary community effort, showcasing the vibrant ecosystem that drives this project forward with 163 contributors making this release possible across 1,400+ commits.

Screenshots

<img width="1600" height="775" alt="favorite" src="https://github.com/user-attachments/assets/0dec07d2-56b6-4b02-adcc-6431d89fa52c" /> <img width="1871" height="1118" alt="gantt" src="https://github.com/user-attachments/assets/3a53106f-352c-477c-9398-f2f7c5ae8b16" />

New Features

• Add Calendar and Gantt chart views to modern React UI with enhanced filtering (#54252, #51667)
• Add Python 3.13 support for Airflow runtime and dependencies (#46891)
• Add SQLAlchemy 2.0 support with various compatibility fixes for Python 3.13 (#52233, #52518, #54940)
• Add support for the psycopg3 postgres driver (#52976)
• Add ability to track & display user who triggers DAG runs (#51738, #53510, #54164, #55112)
• Add toggle for log grouping in task log viewer for better organization (#51146)
• Add tag filtering improvements with Any/All selection options (#51162)
• Add comprehensive filtering for DAG runs, task instances, and audit logs (#53652, #54210, #55082)
• Add XCom browsing with filtering and improved navigation (#54049)
• Add bulk task instance actions and deletion endpoints (#50443, #50165, #50235)
• Add DAG run deletion functionality through UI (#50368)
• Add test connection button for connection validation (#51055)
• Add hyperlink support for URLs in XCom values (#54288)
• Add pool column to task instances list and improve pool integration (#51185, #51031)
• Add drag-and-drop log grouping and improved log visualization (#51146)
• Add color support for XCom JSON display (#51323)
• Add configuration column to DAG runs page (#51270)
• Add enhanced note visibility and management in task headers (#51764, #54163)
• Introduce React plugin system (AIP-68) for modern UI extensions (#52255)
• Add support for external view plugins via iframe integration (#51003, #51889)
• Add dashboard integration capabilities for custom React apps (#54131, #54144)
• Add comprehensive plugin development tools and documentation (#53643)
• Implement complete HITL operator suite (HITLOperator, ApprovalOperator, HITLEntryOperator) for human decision workflows (#52868)
• Add HITL UI integration with role-based access and form handling (#53035)
• Add HITL API endpoints with filtering and query support (#53376, #53923)
• Add HITL utility functions for generating URLs to required actions page (#54827)
• Improve HITL user experience with bug fixes, UI enhancements, and data model consistency (#55463, #55539, #55575, #55546, #55543, #55536, #55535)
• Add ordering and filtering support for HITL details endpoints (#55217)
• Add "No Response Received" required action state (#55149)
• Add operator filter for HITL task instances (#54773)
• Implement deadline alert system for proactive DAG monitoring (AIP-86) (#53951, #53903, #53201, #55086)
• Add configurable reference points and notification callbacks (#50677, #50093)
• Add deadline calculation and tracking in DAG execution lifecycle (#51638, #50925)
• Add comprehensive UI translation support for 16 languages (#51266, #51038, #51219, #50929, #50981, #51793 and more)
• Add right-to-left (RTL) layout support for Arabic and Hebrew (#51376)
• Add language selection interface and browser preference detection (#51369)
• Add translation completeness validation and automated checks (#51166, #51131)
• Add calendar data API endpoints for DAG execution visualization (#52748)
• Add endpoint to watch DAG runs until completion (#51920, #53346)
• Add DAG run ID pattern search functionality (#52437)
• Add multi-sorting capabilities for improved data navigation (#53408)
• Add bulk connection deletion API and UI (#51201)
• Add task group detail pages across DAG runs (#50412, #50309)
• Add asset event tracking with last event timestamps (#50060, #50279)
• Add has_import_errors filter to Core API GET /dags endpoint (#54563)
• Add dag_version filter to get_dag_runs endpoint (#54882)
• Add pattern search for event log endpoint (#55114)
• Add dry_run support with consistent audit log handling (#55116)
• Add utility functions for generic filter counting (#54817)
• Add keyboard navigation for Grid view interface (#51784)
• Add improved error handling for plugin import failures (#49643)
• Add plugin validation in /plugins API with warnings for invalid plugins (#55673)
• Improve accessibility for screen readers and assistive technologies with proper language detection (#55839)
• Add enhanced variable management with upsert operations (#48547)
• Add favorites/pinning support for DAG dashboard organization (#51264)
• Add system theme support with automatic OS preference detection (#52649)
• Add hotkey shortcut to toggle between Grid and Graph views (#54667)
• Add queued DAGs filter button to DAGs page (#55052)
• Add DAG parsing duration visibility in UI (#54752)
• Add owner links support in DAG Header UI for better navigation (#50627)
• Add dag_display_name aliases for improved API consistency (#50332, #50065, #50014, #49933, #49641)
• Add enhanced search capabilities with SearchParamsKeys constants (#55218)
• Add ALL_DONE_MIN_ONE_SUCCESS trigger rule for flexible task dependencies (#53959)
• Add fail_when_dag_is_paused parameter to TriggerDagRunOperator for better control (#48214)
• Add XCom validation to prevent empty keys in XCom.set() and XCom.get() operations (#46929)
• Add collapsible plugin menu when multiple plugins are present (#55265)
• Add external view plugin categories (admin, browse, docs, user) (#52737)
• Add iframe plugins integration to DAG pages (#52795)
• Add plugin error display in UI with comprehensive error handling (#49643, #49436)
• Add collapsible failed task logs to prevent React error overflow (#54377)
• Add dynamic legend system for calendar view (#55155)
• Add React UI for Edge functionality (#53563)
• Add pending actions display to DAG UI (#55041)
• Add description field for filter parameters (#54903)
• Add Catalan language support to Airflow UI (#55013)
• Add Hungarian language support to Airflow UI (#54716)
• Add map_index validation in categorize_task_instances (#54791)
• Add Grid view UX improvements (#54846)
• Add HITL UX improvements for better user experience (#54990)
• Add async support for Notifiers (AIP-86) (#53831)
• Add filtering capabilities for tasks view (#54484)
• Add asset-based filtering support to DAG API endpoint (#54263)
• Add iframe plugins to navigation (#51706)
• Add RTL (right-to-left) layout support for Arabic and Hebrew (#51376)
• Add test connection button to UI (#51055)
• Add task instance bulk actions endpoint (#50443)
• Add connection bulk deletion functionality (#51201)
• Add pool column to task instances list (#51185)
• Add iframe_views to backend plugin support (#51003)
• Add keyboard shortcuts to clear and mark state for task instances and DAG runs (#50885)
• Add deadline relationship to DAG runs and deadline model (#50925, #50093)
• Add DAG run deletion UI (#50368)
• Add task instance deletion UI and endpoint (#50235, #50165)
• Switch all airflow logging to structlog (#52651, #55434, #55431, #55638)
• Add Filter Bar to Audit Log (#55487)
• Add Filters UI for Asset View (#54640)
• Update color palette and leverage Chakra semantic tokens (#53981, #55739)
• Improve calendar view UI with enhanced tooltips and visual fixes (#55476)

Bug Fixes

• Fix DAG list filtering to include QUEUED runs with null start_date (#52668)
• Fix XCom deletion failure for mapped task instances through bulk deletion API (#51850)
• Fix XCom deletion failure for mapped task instances (#54954)
• Fix task timeout handling within task SDK (#54089)
• Fix task instance tries API duplicate entries (#50597)
• Fix connection validation and type checking during construction (#54759)
• Fix mapped task instance index display in Task Instances tab (#55363)
• Fix Gantt chart state mismatch with Grid view (#55300)
• Fix Gantt chart status color display issues (#55296)
• Fix XCom mapping for dynamically-mapped task groups (#51556)
• Fix missing ti_successes and related metrics in Airflow 3.0 Task SDK (#55322)
• Fix bulk operation permissions for connection, pool and variable (#55278)
• Fix clearTaskInstances API: Restore include_past/future support on UI (#54416)
• Fix migration when XCom has NaN values (#53812)
• Fix HITL related UI schema generated by prek hooks (#55204)
• Fix consistent no-log handling for tasks with try_number=0 in API and UI (#55035)
• Fix timezone conversion in datetime trigger parameters (#54593)
• Fix audit log payload for DAG pause/unpause actions (#55091)
• Fix pushing None as an XCom value (#55080)
• Fix scheduler processing of cleared running tasks stuck in RESTARTING state (#55084)
• Fix XCom deletion failure for mapped task instances (#54954)
• Fix outgoing graph edges should exit opposite of incoming edges (#54789)
• Fix external links in Navigation buttons (#52220)
• Fix Error when viewing DAG details of a no longer configured bundle (#52086)
• Fix compatibility with new numpy and pandas versions (#52071)
• Fix connection recovery from URI when host has protocol (#51953)
• Fix last DAG run not showing on DAG listing (#51115)
• Fix task instance tries API returning duplicate entries (#50597)
• Fix Graph view vanishing and loading issues (#53886, #54756)
• Fix rendered template display formatting for better readability (#53657)
• Fix Grid view expand/collapse button functionality (#54257)
• Fix tooltip visibility and positioning issues (#53913)
• Fix grid keyboard navigation focus management (#54271)
• Fix plugin registration for invalid objects and middleware registration (#55264, #55399)
• Fix external links for plugins with undefined URL routes (#55221)
• Fix language display consistency and flag representation (#51560, #51177)
• Fix RTL layout rendering for Arabic and Hebrew interfaces (#51853)
• Fix graph export cropping when view is partial (#55012)
• Fix log viewer "Toggle Source" to hide only source fields, not all structured log fields (#55474)
• Output on stdout/stderr from within tasks is now filterable in the Sources list in the UI log view (#55508)
• Redact JWT tokens in task logs (#55499)
• Fix grid view to handle long task name (#55332)
• Allow slash characters in Variable keys similar to Airflow 2.x (#55324)
• Fix Grid cache invalidation for multi-run task operations (#55504)
• Fix Gantt chart rendering issues (#55554)
• Fix XCom access in DAG processor callbacks for notifiers (#55542)
• Fix alignment of arrows in RTL mode for right-to-left languages (#55619)
• Fix connection form extras not inferring correct type in UI (#55492)
• Fix incorrect log timestamps in UI when default_timezone is not UTC (#54431)
• Fix handling of priority_weight for DAG processor callbacks (#55436)
• Fix pointless requests from Gantt view when there is no Run ID (#55668)
• Ensure filename and lineno of logger calls are present in Task Logs (#55581)
• Fix DAG disappearing after callback execution in stale detection (#55698)
• Fix DB downgrade to Airflow 2 when fab tables exists (#55738)
• Fix UI stats endpoint causing dashboard loading issues (#55733)
• Fix unintended console output when DAG not found in serialized_dag table (#54972)
• Fix scheduler handling of orphaned tasks from Airflow 2 during upgrade (#55848)
• Fix logging format to respect existing configuration during upgrade to prevent unexpected log format changes (#55824)
• Fix Grid view crashes when DAG version information is missing (#55771)
• Fix compatibility for custom triggers migrating from Airflow 2.x that use synchronous connection calls (#55799)
• Fix DAG runs triggered from UI incorrectly marked as REST API triggers instead of UI triggers (#54650)
• Fix XCom API responses failing when encountering non-serializable objects by falling back to string representation (#55880)
• Fix asset queue display in UI showing incorrect timestamps for deleted queue events (#54652)
• Fix SQLite database migrations failing due to foreign key constraint handling (#55883)
• Fix DAG deserialization failure when using non-default weight_rule values like 'absolute' (#55906)
• Fix async connection retrieval in triggerer context preventing event loop blocking (#55812)
• Fix Airflow downgrade compatibility by handling serialized DAG format conversion from v3 to v2 (#55975)
• Fix 'All Log Levels' filter not working in task log viewer (#55851)
• Fix Grid view scrollbar overlapping issues on Firefox browser (#55960)
• Fix Gantt chart misalignment with Grid view layout (#55995)
• Fix Grid view task names being extremely collapsed and unreadable when displaying many DAG runs (#55997)
• Fix LocalExecutor race condition where tasks could start before database state was committed (#56010)
• Use SecretCache for connection and variable access in task sdk (#55972)

Miscellaneous

• Move secrets masker to shared distribution for better modularity (#54449)
• Move email notifications from scheduler to DAG processor for better architecture (#55238)
• Add graph UI load optimization with latest run info endpoint (#53429)
• Optimize UI bundle size by moving translations to dynamic loading (#51735)
• Relocate Task SDK components for improved separation (#55174, #54795)
• Refactor trigger rule utilities and weight rule consolidation (#54797, #53393)
• Remove deprecated Airflow 2.x modules and legacy imports (#50482)
• Clean up unused code and improve module organization (#52176, #52173, #53031)
• Add SQLAlchemy 2.0 CI support for future compatibility (#52233)
• Improve test fixtures and SDK communication testing (#54795, #50603)
• Add translation completeness linting and validation tools (#51166)
• Upgrade to latest versions of important dependencies (#55350)
• Move webserver configuration options to API section (#50693, #50656)
• Improve DAG bundle handling and versioning support (#47592)
• Add database management CLI tools for external database operations (#50657)
• Add comprehensive HITL operator documentation and examples (#54618)
• Add guards for registering middlewares from plugins (#55399)
• Optimize Gantt group expansion with de-bouncing and deferred rendering (#55334)
• Differentiate between triggers and watchers currently running for better visibility (#55376)
• Removed unused config: dag_stale_not_seen_duration (#55601, #55684)
• Update UI's query client strategy for improved performance (#55528)
• Unify datetime format across the UI for consistency (#55572)
• Mark React Apps as Experimental for Airflow 3.1 release (#55478)
• Improve OOM error messaging for clearer task failure diagnosis (#55602)
• Display responder username for better audit trail in HITL workflows (#55509)
• The constraint file do not contain developer dependencies anymore (#53631)
• Add hyperlinks to dag_id column in DAG Runs and Task Instances pages for better navigation (#55648)
• Add responsive web design (RWD) support to Grid view (#55745)

Doc Only Changes

• Add comprehensive Human-in-the-Loop operator tutorial and examples (#54618)
• Add deadline alerts configuration and usage documentation (#53727)
• Make term Dag consistent in docs task-sdk (#55100)
• Add migration guide for upgrading from legacy SLA functionality to deadline alerts (#55743)
• Add DAG bundles triggerer limitation documentation (#55232)
• Add deadline alerts usage guides and best practices (#53727)
• Remove --preview flag from ruff check instructions for Airflow 3 upgrade path (#55516)
• Add documentation for context parameter (#55377)

📣 We are proud to announce the General Availability of Apache Airflow® 3.0, the most significant release in the project’s history.

Airflow 3.0 builds on the foundation of Airflow 2 and introduces a new service-oriented architecture, a modern React-based UI, enhanced security, and a host of long-requested features such as DAG versioning, improved backfills, event-driven scheduling, and support for remote execution.

You can read more about what 3.0 brings in https://airflow.apache.org/blog/airflow-three-point-oh-is-here/.

📦 PyPI: https://pypi.org/project/apache-airflow/3.0.0/ 📚 Docs: https://airflow.apache.org/docs/apache-airflow/3.0.0 🛠️ Release Notes: https://airflow.apache.org/docs/apache-airflow/3.0.0/release_notes.html 🪶 Sources: https://airflow.apache.org/docs/apache-airflow/3.0.0/installation/installing-from-sources.html

This is the result of 300+ developers within the Airflow community working together tirelessly for many months! A huge thank you to all of them for their contributions.

Resources

• Upgrade Guide
• Airflow AIPs

Bug Fixes

• Lazy Jinja2 context (#20217)
• Exclude snowflake-sqlalchemy v1.2.5 (#20245)
• Move away from legacy importlib.resources API (#19091)
• Move setgid as the first command executed in forked task runner (#20040)
• Fix race condition when starting DagProcessorAgent (#19935)
• Limit httpx to <0.20.0 (#20218)
• Log provider import errors as debug warnings (#20172)
• Bump minimum required alembic version (#20153)
• Fix log link in gantt view (#20121)
• fixing #19028 by moving chown to use sudo (#20114)
• Lift off upper bound for MarkupSafe (#20113)
• Fix infinite recursion on redact log (#20039)
• Fix db downgrades (#19994)
• Context class handles deprecation (#19886)
• Fix possible reference to undeclared variable (#19933)
• Validate DagRun state is valid on assignment (#19898)
• Workaround occasional deadlocks with MSSQL (#19856)
• Enable task run setting to be able reinitialise (#19845)
• Fix log endpoint for same task (#19672)
• Cast macro datetime string inputs explicitly (#19592)
• Do not crash with stacktrace when task instance is missing (#19478)
• Fix log timezone in task log view (#19342) (#19401)
• Fix: Add taskgroup tooltip to graph view (#19083)
• Rename execution date in forms and tables (#19063)
• Simplify "invalid TI state" message (#19029)
• Handle case of nonexistent file when preparing file path queue (#18998)
• Do not create dagruns for DAGs with import errors (#19367)
• Fix field relabeling when switching between conn types (#19411)
• KubernetesExecutor should default to template image if used (#19484)
• Fix task instance api cannot list task instances with None state (#19487)
• Fix IntegrityError in DagFileProcessor.manage_slas (#19553)
• Declare data interval fields as serializable (#19616)
• Relax timetable class validation (#19878)
• Fix labels used to find queued KubernetesExecutor pods (#19904)
• Fix moved data migration check for MySQL when replcation is used (#19999)

Doc only changes

• Warn without tracebacks when example_dags are missing deps (#20295)
• Deferrable operators doc clarification (#20150)
• Ensure the example DAGs are all working (#19355)
• Updating core example DAGs to use TaskFlow API where applicable (#18562)
• Add xcom clearing behaviour on task retries (#19968)
• Add a short chapter focusing on adapting secret format for connections (#19859)
• Add information about supported OS-es for Apache Airflow (#19855)
• Update docs to reflect that changes to the base_log_folder require updating other configs (#19793)
• Disclaimer in KubernetesExecutor pod template docs (#19686)
• Add upgrade note on execution_date -> run_id (#19593)
• Expanding .output operator property information in TaskFlow tutorial doc (#19214)
• Add example SLA DAG (#19563)
• Add a proper example to patch DAG (#19465)
• Add DAG file processing description to Scheduler Concepts (#18954)
• Updating explicit arg example in TaskFlow API tutorial doc (#18907)
• Adds back documentation about context usage in Python/@task (#18868)
• Add release date for when an endpoint/field is added in the REST API (#19203)
• Better pod_template_file examples (#19691)
• Add decription on how you can customize image entrypoint (#18915)
• Dags-in-image pod template example should not have dag mounts (#19337)

Bug Fixes

• Fix bug when checking for existence of a Variable (#19395)
• Fix Serialization whenrelativedelta is passed as schedule_interval (#19418)
• Fix moving of dangling TaskInstance rows for SQL Server (#19425)
• Fix task instance modal in gantt view (#19258)
• Fix serialization of Params with set data type (#19267)
• Check if job object is None before calling .is_alive() (#19380)
• Task should fail immediately when pod is unprocessable (#19359)
• Fix downgrade for a DB Migration (#19390)
• Only mark SchedulerJobs as failed, not any jobs (#19375)
• Fix message on "Mark as" confirmation page (#19363)
• Bugfix: Check next run exists before reading data interval (#19307)
• Fix MySQL db migration with default encoding/collation (#19268)
• Fix hidden tooltip position (#19261)
• sqlite_default Connection has been hard-coded to /tmp, use gettempdir instead (#19255)
• Fix Toggle Wrap on DAG code page (#19211)
• Clarify "dag not found" error message in CLI (#19338)
• Add Note to SLA regarding schedule_interval (#19173)
• Use execution_date to check for existing DagRun for TriggerDagRunOperator (#18968)
• Add explicit session parameter in PoolSlotsAvailableDep (#18875)
• FAB still requires WTForms<3.0 (#19466)
• Fix missing dagruns when catchup=True (#19528)

Doc only changes

• Add missing parameter documentation for "timetable" (#19282)
• Improve Kubernetes Executor docs (#19339)
• Update image tag used in docker docs

Bug Fixes

• Fix Unexpected commit error in schedulerjob (#19213)
• Add DagRun.logical_date as a property (#19198)
• Clear ti.next_method and ti.next_kwargs on task finish (#19183)
• Faster PostgreSQL db migration to Airflow 2.2 (#19166)
• Remove incorrect type comment in Swagger2Specification._set_defaults classmethod (#19065)
• Add TriggererJob to jobs check command (#19179, #19185)
• Hide tooltip when next run is None (#19112)
• Crerate TI context with data interval compat layer (#19148)
• Fix queued dag runs changes catchup=False behaviour (#19130, #19145)
• add detailed information to logging when a dag or a task finishes. (#19097)
• Warn about unsupported Python 3.10 (#19060)
• Fix catchup by limiting queued dagrun creation using max_active_runs (#18897)
• Prevent scheduler crash when serialized dag is missing (#19113)
• Don't install SQLAlchemy/Pendulum adapters for other DBs (#18745)
• Workaround libstdcpp TLS error (#19010)
• Change ds, ts, etc. back to use logical date (#19088)
• Ensure task state doesn't change when marked as failed/success/skipped (#19095)
• Relax packaging requirement (#19087)
• Rename trigger page label to Logical Date (#19061)
• Allow Param to support a default value of None (#19034)
• Upgrade old DAG/task param format when deserializing from the DB (#18986)
• Don't bake ENV and _cmd into tmp config for non-sudo (#18772)
• CLI: Fail backfill command before loading DAGs if missing args (#18994)
• BugFix: Null execution date on insert to task_fail violating NOT NULL (#18979)
• Try to move "dangling" rows in upgradedb (#18953)
• Row lock TI query in SchedulerJob._process_executor_events (#18975)
• Sentry before send fallback (#18980)
• Fix XCom.delete error in Airflow 2.2.0 (#18956)
• Check python version before starting triggerer (#18926)

Doc only changes

• Update access control documentation for TaskInstances and DagRuns (#18644)
• Add information about keepalives for managed Postgres (#18850)
• Doc: Add Callbacks Section to Logging & Monitoring (#18842)
• Group PATCH DAGrun together with other DAGRun endpoints (#18885)

New Features

• AIP-39: Add (customizable) Timetable class to Airflow for richer scheduling behaviour (#15397, #16030, #16352, #17030, #17122, #17414, #17552, #17755, #17989, #18084, #18088, #18244, #18266, #18420, #18434, #18421, #18475, #18499, #18573, #18522, #18729, #18706, #18742, #18786, #18804)
• AIP-40: Add Deferrable "Async" Tasks (#15389, #17564, #17565, #17601, #17745, #17747, #17748, #17875, #17876, #18129, #18210, #18214, #18552, #18728, #18414)
• Add a Docker Taskflow decorator (#15330, #18739)
• Add Airflow Standalone command (#15826)
• Display alert messages on dashboard from local settings (#18284)
• Advanced Params using json-schema (#17100)
• Ability to test connections from UI or API (#15795, #18750)
• Add Next Run to UI (#17732)
• Add default weight rule configuration option (#18627)
• Add a calendar field to choose the execution date of the DAG when triggering it (#16141)
• Allow setting specific cwd for BashOperator (#17751)
• Show import errors in DAG views (#17818)
• Add pre/post execution hooks [Experimental] (#17576)
• Added table to view providers in Airflow ui under admin tab (#15385)
• Adds secrets backend/logging/auth information to provider yaml (#17625)
• Add date format filters to Jinja environment (#17451)
• Introduce RESTARTING state (#16681)
• Webserver: Unpause DAG on manual trigger (#16569)
• API endpoint to create new user (#16609)
• Add insert_args for support transfer replace (#15825)
• Add recursive flag to glob in filesystem sensor (#16894)
• Add conn to jinja template context (#16686)
• Add default_args for TaskGroup (#16557)
• Allow adding duplicate connections from UI (#15574)
• Allow specifying multiple URLs via the CORS config option (#17941)
• Implement API endpoint for DAG deletion (#17980)
• Add DAG run endpoint for marking a dagrun success or failed(#17839)
• Add support for kinit options [-f|-F] and [-a|-A] (#17816)
• Queue support for DaskExecutor using Dask Worker Resources (#16829, #18720)
• Make auto refresh interval configurable (#18107)

Improvements

• Small improvements for Airflow UI (#18715, #18795)
• Rename processor_poll_interval to scheduler_idle_sleep_time (#18704)
• Check the allowed values for the logging level (#18651)
• Fix error on triggering a dag that doesn't exist using dagrun_conf (#18655)
• Add muldelete action to TaskInstanceModelView (#18438)
• Avoid importing DAGs during clean DB installation (#18450)
• Require can_edit on DAG privileges to modify TaskInstances and DagRuns (#16634)
• Make Kubernetes job description fit on one log line (#18377)
• Always draw borders if task instance state is null or undefined (#18033)
• Inclusive Language (#18349)
• Improved log handling for zombie tasks (#18277)
• Adding Variable.update method and improving detection of variable key collisions (#18159)
• Add note about params on trigger DAG page (#18166)
• Change TaskInstance and TaskReschedule PK from execution_date to run_id (#17719)
• Adding TaskGroup support in BaseOperator.chain() (#17456)
• Advises the kernel to not cache log files generated by Airflow (#18054)
• Allow filtering DAGS by tags in the REST API (#18090)
• Optimize imports of Providers Manager (#18052)
• Adds capability of Warnings for incompatible community providers (#18020)
• Serialize the template_ext attribute to show it in UI (#17985)
• Add robots.txt and X-Robots-Tag header (#17946)
• Refactor BranchDayOfWeekOperator, DayOfWeekSensor (#17940)
• Update error message to guide the user into self-help mostly (#17929)
• Update to Celery 5 (#17397)
• Avoid redirect loop for users with no permissions (#17838)
• Improve graph view load time for dags with open groups (#17821)
• Add links to provider's documentation (#17736)
• Remove Marshmallow schema warnings (#17753)
• Rename none_failed_or_skipped by none_failed_min_one_success trigger rule (#17683)
• Remove [core] store_dag_code & use DB to get Dag Code (#16342)
• Rename task_concurrency to max_active_tis_per_dag (#17708)
• Import Hooks lazily individually in providers manager (#17682)
• Adding support for multiple task-ids in the external task sensor (#17339)
• Replace execution_date with run_id in airflow tasks run command (#16666)
• Avoid endless redirect loop when user has no roles (#17613)
• Make output from users cli command more consistent (#17642)
• Open relative extra links in place (#17477)
• Move worker_log_server_port option to the logging section (#17621)
• Use gunicorn to serve logs generated by worker (#17591)
• Improve validation of Group id (#17578)
• Simplify 404 page (#17501)
• Add XCom.clear so it's hookable in custom XCom backend (#17405)
• Add deprecation notice for SubDagOperator (#17488)
• Support DAGS folder being in different location on scheduler and runners (#16860)
• Remove /dagrun/create and disable edit form generated by F.A.B (#17376)
• Enable specifying dictionary paths in template_fields_renderers (#17321)
• error early if virtualenv is missing (#15788)
• Handle connection parameters added to Extra and custom fields (#17269)
• Fix airflow celery stop to accept the pid file. (#17278)
• Remove DAG refresh buttons (#17263)
• Deprecate dummy trigger rule in favor of always (#17144)
• Be verbose about failure to import airflow_local_settings (#17195)
• Include exit code in AirflowException str when BashOperator fails. (#17151)
• Adding EdgeModifier support for chain() (#17099)
• Only allows supported field types to be used in custom connections (#17194)
• Secrets backend failover (#16404)
• Warn on Webserver when using SQLite or SequentialExecutor (#17133)
• Extend init_containers defined in pod_override (#17537)
• Client-side filter dag dependencies (#16253)
• Improve executor validation in CLI (#17071)
• Prevent running airflow db init/upgrade migrations and setup in parallel. (#17078)
• Update chain() and cross_downstream() to support XComArgs (#16732)
• Improve graph view refresh (#16696)
• When a task instance fails with exception, log it (#16805)
• Set process title for serve-logs and LocalExecutor (#16644)
• Rename test_cycle to check_cycle (#16617)
• Set process title for Worker when using LocalExecutor (#16623)
• Add schema as DbApiHook instance attribute (#16521, #17423)
• Improve compatibility with MSSQL (#9973)
• Add transparency for unsupported connection type (#16220)
• Call resource based fab methods (#16190)
• Format more dates with timezone (#16129)
• Replace deprecated dag.sub_dag with dag.partial_subset (#16179)
• Treat AirflowSensorTimeout as immediate failure without retrying (#12058)
• Marking success/failed automatically clears failed downstream tasks (#13037)
• Add close/open indicator for import dag errors (#16073)
• Add collapsible import errors (#16072)
• Always return a response in TI's action_clear view (#15980)
• Add cli command to delete user by email (#15873)
• Use resource and action names for FAB permissions (#16410)
• Rename DAG concurrency ([core] dag_concurrency) settings for easier understanding (#16267, #18730)
• Calendar UI improvements (#16226)
• Refactor: SKIPPED should not be logged again as SUCCESS (#14822)
• Remove version limits for dnspython (#18046, #18162)
• Accept custom run ID in TriggerDagRunOperator (#18788)

Bug Fixes

• Make REST API patch user endpoint work the same way as the UI (#18757)
• Properly set start_date for cleared tasks (#18708)
• Ensure task_instance exists before running update on its state(REST API) (#18642)
• Make AirflowDateTimePickerWidget a required field (#18602)
• Retry deadlocked transactions on deleting old rendered task fields (#18616)
• Fix retry_exponential_backoff divide by zero error when retry delay is zero (#17003)
• Improve how UI handles datetimes (#18611, #18700)
• Bugfix: dag_bag.get_dag should return None, not raise exception (#18554)
• Only show the task modal if it is a valid instance (#18570)
• Fix accessing rendered {{ task.x }} attributes from within templates (#18516)
• Add missing email type of connection (#18502)
• Don't use flash for "same-page" UI messages. (#18462)
• Fix task group tooltip (#18406)
• Properly fix dagrun update state endpoint (#18370)
• Properly handle ti state difference between executor and scheduler (#17819)
• Fix stuck "queued" tasks in KubernetesExecutor (#18152)
• Don't permanently add zip DAGs to sys.path (#18384)
• Fix random deadlocks in MSSQL database (#18362)
• Deactivating DAGs which have been removed from files (#17121)
• When syncing dags to db remove dag_tag rows that are now unused (#8231)
• Graceful scheduler shutdown on error (#18092)
• Fix mini scheduler not respecting wait_for_downstream dep (#18338)
• Pass exception to run_finished_callback for Debug Executor (#17983)
• Make XCom.get_one return full, not abbreviated values (#18274)
• Use try/except when closing temporary file in task_runner (#18269)
• show next run if not none (#18273)
• Fix DB session handling in XCom.set (#18240)
• Sort adopted tasks in _check_for_stalled_adopted_tasks method (#18208)
• Fix external_executor_id not being set for manually run jobs (#17207)
• Fix deleting of zipped Dags in Serialized Dag Table (#18243)
• Return explicit error on user-add for duplicated email (#18224)
• Remove loading dots even when last run data is empty (#18230)
• Swap dag import error dropdown icons (#18207)
• Automatically create section when migrating config (#16814)
• Set encoding to utf-8 by default while reading task logs (#17965)
• Set task state to failed when pod is DELETED while running (#18095)
• Apply parent dag permissions to subdags (#18160)
• Change id collation for MySQL to case-sensitive (#18072)
• Logs task launch exception in StandardTaskRunner (#17967)
• Applied permissions to self._error_file (#15947)
• Hide variable import form if user lacks permission (#18000)
• Fix blank dag dependencies view (#17990)
• Add missing menu access for dag dependencies and configurations pages (#17450)
• Fix passing Jinja templates in DateTimeSensor (#17959)
• Fixing bug which restricted the visibility of ImportErrors (#17924)
• Fix grammar in traceback.html (#17942)
• Fix DagRunState enum query for MySQLdb driver (#17886)
• Fixed button size in "Actions" group. (#17902)
• Fix log links on graph TI modal (#17862)
• Only show import errors for DAGs a user can access (#17835)
• Show all import_errors from zip files (#17759)
• fix EXTRA_LOGGER_NAMES param and related docs (#17808)
• Use one interpreter for Airflow and gunicorn (#17805)
• Fix: Mysql 5.7 id utf8mb3 (#14535)
• Fix dag_processing.last_duration metric random holes (#17769)
• Automatically use utf8mb3_general_ci collation for MySQL (#17729)
• Fix wrong query on running tis (#17631)
• Add root to tree refresh url (#17633)
• Do not delete running DAG from the UI (#17630)
• Rescue if a DagRun's DAG was removed from db (#17544)
• fix: filter condition of TaskInstance does not work #17535 (#17548)
• Regression on pid reset to allow task start after heartbeat (#17333)
• Dont use TaskInstance in CeleryExecutor.trigger_tasks (#16248)
• Remove locks for upgrades in MSSQL (#17213)
• Create virtualenv via python call (#17156)
• Ensure a DAG is acyclic when running DAG.cli() (#17105)
• Translate non-ascii characters (#17057)
• Change the logic of None comparison in model_list template (#16893)
• Have UI and POST /task_instances_state API endpoint have same behaviour (#16539)
• ensure task is skipped if missing sla (#16719)
• Fix direct use of cached_property module (#16710)
• Fix TI success confirm page (#16650)
• Cattrs 1.7.0 released by the end of May 2021 break lineage usage (#16173)
• Fix Celery executor getting stuck randomly because of reset_signals in multiprocessing (#15989)
• Modify return value check in python virtualenv jinja template (#16049)
• Fix dag dependency search (#15924)
• Make custom JSON encoder support Decimal (#16383)
• Bugfix: Allow clearing tasks with just dag_id and empty subdir (#16513)
• Convert port value to a number before calling test connection (#16497)
• Handle missing/null serialized DAG dependencies (#16393)
• Correctly set dag.fileloc when using the @dag decorator (#16384)
• Fix TI success/failure links (#16233)
• Correctly implement autocomplete early return in airflow/www/views.py (#15940)
• Backport fix to allow pickling of Loggers to Python 3.6 (#18798)
• Fix bug that Backfill job fail to run when there are tasks run into reschedule state (#17305, #18806)

Doc only changes

• Update dagbag_size documentation (#18824)
• Update documentation about bundle extras (#18828)
• Fix wrong Postgres search_path set up instructions (#17600)
• Remove AIRFLOW_GID from Docker images (#18747)
• Improve error message for BranchPythonOperator when no task_id to follow (#18471)
• Improve guidance to users telling them what to do on import timeout (#18478)
• Explain scheduler fine-tuning better (#18356)
• Added example JSON for airflow pools import (#18376)
• Add sla_miss_callback section to the documentation (#18305)
• Explain sentry default environment variable for subprocess hook (#18346)
• Refactor installation pages (#18282)
• Improves installing from sources pages for all components (#18251)
• Doc: Use closer.lua script for downloading sources (#18179)
• Improves quick-start docker-compose warnings and documentation (#18164)
• Production-level support for MSSQL (#18382)
• Update non-working example in documentation (#18067)
• Improve the description of how to handle dynamic task generation (#17963)
• Remove default_args pattern + added get_current_context() use for Core Airflow example DAGs (#16866)
• Update max_tis_per_query to better render on the webpage (#17971)
• Adds Github Oauth example with team based authorization (#17896)
• Update docker.rst (#17882)
• Example xcom update (#17749)
• Add doc warning about connections added via envvars (#17915)
• fix wrong documents around upgrade-check.rst (#17903)
• Add Brent to Committers list (#17873)
• Improves documentation about modules management (#17757)
• Remove deprecated metrics from metrics.rst (#17772)
• Make sure "production-readiness" of docker-compose is well explained (#17731)
• Doc: Update Upgrade to v2 docs with Airflow 1.10.x EOL dates (#17710)
• Doc: Replace deprecated param from docstrings (#17709)
• docs(dagowner): describe dag owner more carefully (#17699)
• docs(impersonation): update note so avoid misintrepretation (#17701)
• Docs: Make DAG.is_active read-only in API (#17667)
• Improve cross-links to operators and hooks references (#17622)
• Update documentation regarding Python 3.9 support (#17611)
• Fix MySQL database character set instruction (#17603)
• Document overriding XCom.clear for data lifecycle management (#17589)
• Path correction in docs for airflow core (#17567)
• docs(celery): reworded, add actual multiple queues example (#17541)
• Doc: Add FAQ to speed up parsing with tons of dag files (#17519)
• Improve image building documentation for new users (#17409)
• Doc: Strip unnecessary arguments from MariaDB JIRA URL (#17296)
• Update warning about MariaDB and multiple schedulers (#17287)
• Doc: Recommend using same configs on all Airflow components (#17146)
• Move docs about masking to a new page (#17007)
• Docs: Suggest use of Env vars instead of Airflow Vars in best practises doc (#16926)
• Docs: Better description for pod_template_file (#16861)
• Add Aneesh Joseph as Airflow Committer (#16835)
• Docs: Added new pipeline example for the tutorial docs (#16548)
• Remove upstart from docs (#16672)
• Add new committers: Jed and TP (#16671)
• Docs: Fix flask-ouathlib to flask-oauthlib in Upgrading docs (#16320)
• Docs: Fix creating a connection docs (#16312)
• Docs: Fix url for Elasticsearch (#16275)
• Small improvements for README.md files (#16244)
• Fix docs for dag_concurrency (#16177)
• Check syntatic correctness for code-snippets (#16005)
• Improvements for Docker Image docs (#14843)
• Add proper link for wheel packages in docs. (#15999)
• Add Docs for default_pool slots (#15997)
• Add memory usage warning in quick-start documentation (#15967)
• Update example KubernetesExecutor git-sync pod template file (#15904)
• Docs: Fix Taskflow API docs (#16574)
• Added new pipeline example for the tutorial docs (#16084)
• Updating the DAG docstring to include render_template_as_native_obj (#16534)
• Update docs on setting up SMTP (#16523)
• Docs: Fix API verb from POST to PATCH (#16511)

Misc/Internal

• Renaming variables to be consistent with code logic (#18685)
• Simplify strings previously split across lines (#18679)
• fix exception string of BranchPythonOperator (#18623)
• Add multiple roles when creating users (#18617)
• Move FABs base Security Manager into Airflow. (#16647)
• Remove unnecessary css state colors (#18461)
• Update boto3 to <1.19 (#18389)
• Improve coverage for airflow.security.kerberos module (#18258)
• Fix kinesis test (#18337)
• Fix provider test acessing importlib-resources (#18228)
• Silence warnings in tests from using SubDagOperator (#18275)
• Fix usage of range(len()) to enumerate (#18174)
• Test coverage on the autocomplete view (#15943)
• Update version added fields in airflow/config_templates/config.yml (#18128)
• Add "packaging" to core requirements (#18122)
• Adds LoggingMixins to BaseTrigger (#18106)
• Limit colorlog version (6.x is incompatible) (#18099)
• Fix building docs in main builds (#18035)
• Remove upper-limit on tenacity (#17593)
• Remove redundant numpy dependency (#17594)
• Bump mysql-connector-python to latest version (#17596)
• Make pandas an optional core dependency (#17575)
• Add more typing to airflow.utils.helpers (#15582)
• Chore: Some code cleanup in airflow/utils/db.py (#17090)
• Refactor: Remove processor_factory from DAG processing (#16659)
• Remove AbstractDagFileProcessorProcess from dag processing (#16816)
• Update TaskGroup typing (#16811)
• Update click to 8.x (#16779)
• Remove remaining Pylint disables (#16760)
• Remove duplicated try, there is already a try in create_session (#16701)
• Removes pylint from our toolchain (#16682)
• Refactor usage of unneeded function call (#16653)
• Add type annotations to setup.py (#16658)
• Remove SQLAlchemy <1.4 constraint (#16630) (Note: our dependencies still have a requirement on <1.4)
• Refactor dag.clear method (#16086)
• Use DAG_ACTIONS constant (#16232)
• Use updated _get_all_non_dag_permissions method (#16317)
• Add updated-name wrappers for built-in FAB methods (#16077)
• Pins docutils to <0.17 until breaking behaviour is fixed (#16133)
• Remove TaskInstance.log_filepath attribute (#15217)
• Removes unnecessary function call in airflow/www/app.py (#15956)
• Move plyvel to google provider extra (#15812)
• Update permission migrations to use new naming scheme (#16400)
• Use resource and action names for FAB (#16380)
• Swap out calls to find_permission_view_menu for get_permission wrapper (#16377)
• Fix deprecated default for fab_logging_level to WARNING (#18783)
• Allow running tasks from UI when using CeleryKubernetesExecutor (#18441)

Release Date: 2022-01-06

Features

Allow setting port in IMAP Connection (#20440)

New Features """"""""""""

• Add PythonVirtualenvDecorator to Taskflow API (#14761)
• Add Taskgroup decorator (#15034)
• Create a DAG Calendar View (#15423)
• Create cross-DAG dependencies view (#13199)
• Add rest API to query for providers (#13394)
• Mask passwords and sensitive info in task logs and UI (#15599)
• Add SubprocessHook for running commands from operators (#13423)
• Add DAG Timeout in UI page "DAG Details" (#14165)
• Add WeekDayBranchOperator (#13997)
• Add JSON linter to DAG Trigger UI (#13551)
• Add DAG Description Doc to Trigger UI Page (#13365)
• Add airflow webserver URL into SLA miss email. (#13249)
• Add read only REST API endpoints for users (#14735)
• Add files to generate Airflow's Python SDK (#14739)
• Add dynamic fields to snowflake connection (#14724)
• Add read only REST API endpoint for roles and permissions (#14664)
• Add new datetime branch operator (#11964)
• Add Google leveldb hook and operator (#13109) (#14105)
• Add plugins endpoint to the REST API (#14280)
• Add worker_pod_pending_timeout support (#15263)
• Add support for labeling DAG edges (#15142)
• Add CUD REST API endpoints for Roles (#14840)
• Import connections from a file (#15177)
• A bunch of template_fields_renderers additions (#15130)
• Add REST API query sort and order to some endpoints (#14895)
• Add timezone context in new ui (#15096)
• Add query mutations to new UI (#15068)
• Add different modes to sort dag files for parsing (#15046)
• Auto refresh on Tree View (#15474)
• BashOperator to raise AirflowSkipException on exit code 99 (by default, configurable) (#13421) (#14963)
• Clear tasks by task ids in REST API (#14500)
• Support jinja2 native Python types (#14603)
• Allow celery workers without gossip or mingle modes (#13880)
• Add airflow jobs check CLI command to check health of jobs (Scheduler etc) (#14519)
• Rename DateTimeBranchOperator to BranchDateTimeOperator (#14720)

Improvements """"""""""""

• Add optional result handler callback to DbApiHook (#15581)
• Update Flask App Builder limit to recently released 3.3 (#15792)
• Prevent creating flask sessions on REST API requests (#15295)
• Sync DAG specific permissions when parsing (#15311)
• Increase maximum length of pool name on Tasks to 256 characters (#15203)
• Enforce READ COMMITTED isolation when using mysql (#15714)
• Auto-apply apply_default to subclasses of BaseOperator (#15667)
• Emit error on duplicated DAG ID (#15302)
• Update KubernetesExecutor pod templates to allow access to IAM permissions (#15669)
• More verbose logs when running airflow db check-migrations (#15662)
• When one_success mark task as failed if no success (#15467)
• Add an option to trigger a dag w/o changing conf (#15591)
• Add Airflow UI instance_name configuration option (#10162)
• Add a decorator to retry functions with DB transactions (#14109)
• Add return to PythonVirtualenvOperator's execute method (#14061)
• Add verify_ssl config for kubernetes (#13516)
• Add description about secret_key when Webserver > 1 (#15546)
• Add Traceback in LogRecord in JSONFormatter (#15414)
• Add support for arbitrary json in conn uri format (#15100)
• Adds description field in variable (#12413) (#15194)
• Add logs to show last modified in SFTP, FTP and Filesystem sensor (#15134)
• Execute on_failure_callback when SIGTERM is received (#15172)
• Allow hiding of all edges when highlighting states (#15281)
• Display explicit error in case UID has no actual username (#15212)
• Serve logs with Scheduler when using Local or Sequential Executor (#15557)
• Deactivate trigger, refresh, and delete controls on dag detail view. (#14144)
• Turn off autocomplete for connection forms (#15073)
• Increase default worker_refresh_interval to 6000 seconds (#14970)
• Only show User's local timezone if it's not UTC (#13904)
• Suppress LOG/WARNING for a few tasks CLI for better CLI experience (#14567)
• Configurable API response (CORS) headers (#13620)
• Allow viewers to see all docs links (#14197)
• Update Tree View date ticks (#14141)
• Make the tooltip to Pause / Unpause a DAG clearer (#13642)
• Warn about precedence of env var when getting variables (#13501)
• Move [celery] default_queue config to [operators] default_queue to re-use between executors (#14699)

Bug Fixes """""""""

• Fix 500 error from updateTaskInstancesState API endpoint when dry_run not passed (#15889)
• Ensure that task preceding a PythonVirtualenvOperator doesn't fail (#15822)
• Prevent mixed case env vars from crashing processes like worker (#14380)
• Fixed type annotations in DAG decorator (#15778)
• Fix on_failure_callback when task receive SIGKILL (#15537)
• Fix dags table overflow (#15660)
• Fix changing the parent dag state on subdag clear (#15562)
• Fix reading from zip package to default to text (#13962)
• Fix wrong parameter for drawDagStatsForDag in dags.html (#13884)
• Fix QueuedLocalWorker crashing with EOFError (#13215)
• Fix typo in NotPreviouslySkippedDep (#13933)
• Fix parallelism after KubeExecutor pod adoption (#15555)
• Fix kube client on mac with keepalive enabled (#15551)
• Fixes wrong limit for dask for python>3.7 (should be <3.7) (#15545)
• Fix Task Adoption in KubernetesExecutor (#14795)
• Fix timeout when using XCom with KubernetesPodOperator (#15388)
• Fix deprecated provider aliases in "extras" not working (#15465)
• Fixed default XCom deserialization. (#14827)
• Fix used_group_ids in dag.partial_subset (#13700) (#15308)
• Further fix trimmed pod_id for KubernetesPodOperator (#15445)
• Bugfix: Invalid name when trimmed pod_id ends with hyphen in KubernetesPodOperator (#15443)
• Fix incorrect slots stats when TI pool_slots > 1 (#15426)
• Fix DAG last run link (#15327)
• Fix sync-perm to work correctly when update_fab_perms = False (#14847)
• Fixes limits on Arrow for plexus test (#14781)
• Fix UI bugs in tree view (#14566)
• Fix AzureDataFactoryHook failing to instantiate its connection (#14565)
• Fix permission error on non-POSIX filesystem (#13121)
• Fix spelling in "ignorable" (#14348)
• Fix get_context_data doctest import (#14288)
• Correct typo in GCSObjectsWtihPrefixExistenceSensor (#14179)
• Fix order of failed deps (#14036)
• Fix critical CeleryKubernetesExecutor bug (#13247)
• Fix four bugs in StackdriverTaskHandler (#13784)
• func.sum may return Decimal that break rest APIs (#15585)
• Persist tags params in pagination (#15411)
• API: Raise AlreadyExists exception when the execution_date is same (#15174)
• Remove duplicate call to sync_metadata inside DagFileProcessorManager (#15121)
• Extra docker-py update to resolve docker op issues (#15731)
• Ensure executors end method is called (#14085)
• Remove user_id from API schema (#15117)
• Prevent clickable bad links on disabled pagination (#15074)
• Acquire lock on db for the time of migration (#10151)
• Skip SLA check only if SLA is None (#14064)
• Print right version in airflow info command (#14560)
• Make airflow info work with pipes (#14528)
• Rework client-side script for connection form. (#14052)
• API: Add CollectionInfo in all Collections that have total_entries (#14366)
• Fix task_instance_mutation_hook when importing airflow.models.dagrun (#15851)

Doc only changes """"""""""""""""

• Fix docstring of SqlSensor (#15466)
• Small changes on "DAGs and Tasks documentation" (#14853)
• Add note on changes to configuration options (#15696)
• Add docs to the markdownlint and yamllint config files (#15682)
• Rename old "Experimental" API to deprecated in the docs. (#15653)
• Fix documentation error in git_sync_template.yaml (#13197)
• Fix doc link permission name (#14972)
• Fix link to Helm chart docs (#14652)
• Fix docstrings for Kubernetes code (#14605)
• docs: Capitalize & minor fixes (#14283) (#14534)
• Fixed reading from zip package to default to text. (#13984)
• An initial rework of the "Concepts" docs (#15444)
• Improve docstrings for various modules (#15047)
• Add documentation on database connection URI (#14124)
• Add Helm Chart logo to docs index (#14762)
• Create a new documentation package for Helm Chart (#14643)
• Add docs about supported logging levels (#14507)
• Update docs about tableau and salesforce provider (#14495)
• Replace deprecated doc links to the correct one (#14429)
• Refactor redundant doc url logic to use utility (#14080)
• docs: NOTICE: Updated 2016-2019 to 2016-now (#14248)
• Skip DAG perm sync during parsing if possible (#15464)
• Add picture and examples for Edge Labels (#15310)
• Add example DAG & how-to guide for sqlite (#13196)
• Add links to new modules for deprecated modules (#15316)
• Add note in Updating.md about FAB data model change (#14478)

Misc/Internal """""""""""""

• Fix logging.exception redundancy (#14823)
• Bump stylelint to remove vulnerable sub-dependency (#15784)
• Add resolution to force dependencies to use patched version of lodash (#15777)
• Update croniter to 1.0.x series (#15769)
• Get rid of Airflow 1.10 in Breeze (#15712)
• Run helm chart tests in parallel (#15706)
• Bump ssri from 6.0.1 to 6.0.2 in /airflow/www (#15437)
• Remove the limit on Gunicorn dependency (#15611)
• Better "dependency already registered" warning message for tasks #14613 (#14860)
• Pin pandas-gbq to <0.15.0 (#15114)
• Use Pip 21.* to install airflow officially (#15513)
• Bump mysqlclient to support the 1.4.x and 2.x series (#14978)
• Finish refactor of DAG resource name helper (#15511)
• Refactor/Cleanup Presentation of Graph Task and Path Highlighting (#15257)
• Standardize default fab perms (#14946)
• Remove datepicker for task instance detail view (#15284)
• Turn provider's import warnings into debug logs (#14903)
• Remove left-over fields from required in provider_info schema. (#14119)
• Deprecate tableau extra (#13595)
• Use built-in cached_property on Python 3.8 where possible (#14606)
• Clean-up JS code in UI templates (#14019)
• Bump elliptic from 6.5.3 to 6.5.4 in /airflow/www (#14668)
• Switch to f-strings using flynt. (#13732)
• use jquery ready instead of vanilla js (#15258)
• Migrate task instance log (ti_log) js (#15309)
• Migrate graph js (#15307)
• Migrate dags.html javascript (#14692)
• Removes unnecessary AzureContainerInstance connection type (#15514)
• Separate Kubernetes pod_launcher from core airflow (#15165)
• update remaining old import paths of operators (#15127)
• Remove broken and undocumented "demo mode" feature (#14601)
• Simplify configuration/legibility of Webpack entries (#14551)
• remove inline tree js (#14552)
• Js linting and inline migration for simple scripts (#14215)
• Remove use of repeated constant in AirflowConfigParser (#14023)
• Deprecate email credentials from environment variables. (#13601)
• Remove unused 'context' variable in task_instance.py (#14049)
• Disable suppress_logs_and_warning in cli when debugging (#13180)

Bug Fixes

• Bugfix: Return XCom Value in the XCom Endpoint API (#13684)
• Bugfix: Import error when using custom backend and sql_alchemy_conn_secret (#13260)
• Allow PID file path to be relative when daemonize a process (scheduler, kerberos, etc) (#13232)
• Bugfix: no generic DROP CONSTRAINT in MySQL during airflow db upgrade (#13239)
• Bugfix: Sync Access Control defined in DAGs when running sync-perm (#13377)
• Stop sending Callback Requests if no callbacks are defined on DAG (#13163)
• BugFix: Dag-level Callback Requests were not run (#13651)
• Stop creating duplicate Dag File Processors (#13662)
• Filter DagRuns with Task Instances in removed State while Scheduling (#13165)
• Bump datatables.net from 1.10.21 to 1.10.22 in /airflow/www (#13143)
• Bump datatables.net JS to 1.10.23 (#13253)
• Bump dompurify from 2.0.12 to 2.2.6 in /airflow/www (#13164)
• Update minimum cattrs version (#13223)
• Remove inapplicable arg 'output' for CLI pools import/export (#13071)
• Webserver: Fix the behavior to deactivate the authentication option and add docs (#13191)
• Fix: add support for no-menu plugin views (#11742)
• Add python-daemon limit for python 3.8+ to fix daemon crash (#13540)
• Change the default celery worker_concurrency to 16 (#13612)
• Audit Log records View should not contain link if dag_id is None (#13619)
• Fix invalid continue_token for cleanup list pods (#13563)
• Switches to latest version of snowflake connector (#13654)
• Fix backfill crash on task retry or reschedule (#13712)
• Setting max_tis_per_query to 0 now correctly removes the limit (#13512)
• Fix race conditions in task callback invocations (#10917)
• Fix webserver exiting when gunicorn master crashes (#13518)(#13780)
• Fix SQL syntax to check duplicate connections (#13783)
• BaseBranchOperator will push to xcom by default (#13704) (#13763)
• Fix Deprecation for configuration.getsection (#13804)
• Fix TaskNotFound in log endpoint (#13872)
• Fix race condition when using Dynamic DAGs (#13893)
• Fix: Linux/Chrome window bouncing in Webserver
• Fix db shell for sqlite (#13907)
• Only compare updated time when Serialized DAG exists (#13899)
• Fix dag run type enum query for mysqldb driver (#13278)
• Add authentication to lineage endpoint for experimental API (#13870)
• Do not add User role perms to custom roles. (#13856)
• Do not add Website.can_read access to default roles. (#13923)
• Fix invalid value error caused by long Kubernetes pod name (#13299)
• Fix DB Migration for SQLite to upgrade to 2.0 (#13921)
• Bugfix: Manual DagRun trigger should not skip scheduled runs (#13963)
• Stop loading Extra Operator links in Scheduler (#13932)
• Added missing return parameter in read function of FileTaskHandler (#14001)
• Bugfix: Do not try to create a duplicate Dag Run in Scheduler (#13920)
• Make v1/config endpoint respect webserver expose_config setting (#14020)
• Disable row level locking for Mariadb and MySQL <8 (#14031)
• Bugfix: Fix permissions to triggering only specific DAGs (#13922)
• Fix broken SLA Mechanism (#14056)
• Bugfix: Scheduler fails if task is removed at runtime (#14057)
• Remove permissions to read Configurations for User and Viewer roles (#14067)
• Fix DB Migration from 2.0.1rc1

Improvements

• Increase the default min_file_process_interval to decrease CPU Usage (#13664)
• Dispose connections when running tasks with os.fork & CeleryExecutor (#13265)
• Make function purpose clearer in example_kubernetes_executor example dag (#13216)
• Remove unused libraries - flask-swagger, funcsigs (#13178)
• Display alternative tooltip when a Task has yet to run (no TI) (#13162)
• User werkzeug's own type conversion for request args (#13184)
• UI: Add queued_by_job_id & external_executor_id Columns to TI View (#13266)
• Make json-merge-patch an optional library and unpin it (#13175)
• Adds missing LDAP "extra" dependencies to ldap provider. (#13308)
• Refactor setup.py to better reflect changes in providers (#13314)
• Pin pyjwt and Add integration tests for Apache Pinot (#13195)
• Removes provider-imposed requirements from setup.cfg (#13409)
• Replace deprecated decorator (#13443)
• Streamline & simplify __eq__ methods in models Dag and BaseOperator (#13449)
• Additional properties should be allowed in provider schema (#13440)
• Remove unused dependency - contextdecorator (#13455)
• Remove 'typing' dependency (#13472)
• Log migrations info in consistent way (#13458)
• Unpin mysql-connector-python to allow 8.0.22 (#13370)
• Remove thrift as a core dependency (#13471)
• Add NotFound response for DELETE methods in OpenAPI YAML (#13550)
• Stop Log Spamming when [core] lazy_load_plugins is False (#13578)
• Display message and docs link when no plugins are loaded (#13599)
• Unpin restriction for colorlog dependency (#13176)
• Add missing Dag Tag for Example DAGs (#13665)
• Support tables in DAG docs (#13533)
• Add python3-openid dependency (#13714)
• Add __repr__ for Executors (#13753)
• Add description to hint if conn_type is missing (#13778)
• Upgrade azure blob to v12 (#12188)
• Add extra field to get_connnection REST endpoint (#13885)
• Make Smart Sensors DB Migration idempotent (#13892)
• Improve the error when DAG does not exist when running dag pause command (#13900)
• Update airflow_local_settings.py to fix an error message (#13927)
• Only allow passing JSON Serializable conf to TriggerDagRunOperator (#13964)
• Bugfix: Allow getting details of a DAG with null start_date (REST API) (#13959)
• Add params to the DAG details endpoint (#13790)
• Make the role assigned to anonymous users customizable (#14042)
• Retry critical methods in Scheduler loop in case of OperationalError (#14032)

Doc only changes

• Add Missing Statsd Metrics in Docs (#13708)
• Add Missing Email configs in Configuration doc (#13709)
• Add quick start for Airflow on Docker (#13660)
• Describe which Python versions are supported (#13259)
• Add note block to 2.x migration docs (#13094)
• Add documentation about webserver_config.py (#13155)
• Add missing version information to recently added configs (#13161)
• API: Use generic information in UpdateMask component (#13146)
• Add Airflow 2.0.0 to requirements table (#13140)
• Avoid confusion in doc for CeleryKubernetesExecutor (#13116)
• Update docs link in REST API spec (#13107)
• Add link to PyPI Repository to provider docs (#13064)
• Fix link to Airflow master branch documentation (#13179)
• Minor enhancements to Sensors docs (#13381)
• Use 2.0.0 in Airflow docs & Breeze (#13379)
• Improves documentation regarding providers and custom connections (#13375)(#13410)
• Fix malformed table in production-deployment.rst (#13395)
• Update celery.rst to fix broken links (#13400)
• Remove reference to scheduler run_duration param in docs (#13346)
• Set minimum SQLite version supported (#13412)
• Fix installation doc (#13462)
• Add docs about mocking variables and connections (#13502)
• Add docs about Flask CLI (#13500)
• Fix Upgrading to 2 guide to use rbac UI (#13569)
• Make docs clear that Auth can not be disabled for Stable API (#13568)
• Remove archived links from docs & add link for AIPs (#13580)
• Minor fixes in upgrading-to-2.rst (#13583)
• Fix Link in Upgrading to 2.0 guide (#13584)
• Fix heading for Mocking section in best-practices.rst (#13658)
• Add docs on how to use custom operators within plugins folder (#13186)
• Update docs to register Operator Extra Links (#13683)
• Improvements for database setup docs (#13696)
• Replace module path to Class with just Class Name (#13719)
• Update DAG Serialization docs (#13722)
• Fix link to Apache Airflow docs in webserver (#13250)
• Clarifies differences between extras and provider packages (#13810)
• Add information about all access methods to the environment (#13940)
• Docs: Fix FAQ on scheduler latency (#13969)
• Updated taskflow api doc to show dependency with sensor (#13968)
• Add deprecated config options to docs (#13883)
• Added a FAQ section to the Upgrading to 2 doc (#13979)

The full changelog is about 3,000 lines long (already excluding everything backported to 1.10), so for now I’ll simply share some of the major features in 2.0.0 compared to 1.10.14:

A new way of writing dags: the TaskFlow API (AIP-31)

(Known in 2.0.0alphas as Functional DAGs.)

DAGs are now much much nicer to author especially when using PythonOperator. Dependencies are handled more clearly and XCom is nicer to use

A quick teaser of what DAGs can now look like:

from airflow.decorators import dag, task
from airflow.utils.dates import days_ago

@dag(default_args={'owner': 'airflow'}, schedule_interval=None, start_date=days_ago(2))
def tutorial_taskflow_api_etl():
   @task
   def extract():
       return {"1001": 301.27, "1002": 433.21, "1003": 502.22}

   @task
   def transform(order_data_dict: dict) -> dict:
       total_order_value = 0

       for value in order_data_dict.values():
           total_order_value += value

       return {"total_order_value": total_order_value}

   @task()
   def load(total_order_value: float):

       print("Total order value is: %.2f" % total_order_value)

   order_data = extract()
   order_summary = transform(order_data)
   load(order_summary["total_order_value"])

tutorial_etl_dag = tutorial_taskflow_api_etl()

Fully specified REST API (AIP-32)

We now have a fully supported, no-longer-experimental API with a comprehensive OpenAPI specification

Read more here:

REST API Documentation.

Massive Scheduler performance improvements

As part of AIP-15 (Scheduler HA+performance) and other work Kamil did, we significantly improved the performance of the Airflow Scheduler. It now starts tasks much, MUCH quicker.

Over at Astronomer.io we’ve benchmarked the scheduler—it’s fast (we had to triple check the numbers as we don’t quite believe them at first!)

Scheduler is now HA compatible (AIP-15)

It’s now possible and supported to run more than a single scheduler instance. This is super useful for both resiliency (in case a scheduler goes down) and scheduling performance.

To fully use this feature you need Postgres 9.6+ or MySQL 8+ (MySQL 5, and MariaDB won’t work with more than one scheduler I’m afraid).

There’s no config or other set up required to run more than one scheduler—just start up a scheduler somewhere else (ensuring it has access to the DAG files) and it will cooperate with your existing schedulers through the database.

For more information, read the Scheduler HA documentation.

Task Groups (AIP-34)

SubDAGs were commonly used for grouping tasks in the UI, but they had many drawbacks in their execution behaviour (primarirly that they only executed a single task in parallel!) To improve this experience, we’ve introduced “Task Groups”: a method for organizing tasks which provides the same grouping behaviour as a subdag without any of the execution-time drawbacks.

SubDAGs will still work for now, but we think that any previous use of SubDAGs can now be replaced with task groups. If you find an example where this isn’t the case, please let us know by opening an issue on GitHub

For more information, check out the Task Group documentation.

Refreshed UI

We’ve given the Airflow UI a visual refresh and updated some of the styling. Check out the UI section of the docs for screenshots.

We have also added an option to auto-refresh task states in Graph View so you no longer need to continuously press the refresh button :).

Smart Sensors for reduced load from sensors (AIP-17)

If you make heavy use of sensors in your Airflow cluster, you might find that sensor execution takes up a significant proportion of your cluster even with “reschedule” mode. To improve this, we’ve added a new mode called “Smart Sensors”.

This feature is in “early-access”: it’s been well-tested by AirBnB and is “stable”/usable, but we reserve the right to make backwards-incompatible changes to it in a future release (if we have to. We’ll try very hard not to!)

Simplified KubernetesExecutor

For Airflow 2.0, we have re-architected the KubernetesExecutor in a fashion that is simultaneously faster, easier to understand, and more flexible for Airflow users. Users will now be able to access the full Kubernetes API to create a .yaml pod_template_file instead of specifying parameters in their airflow.cfg.

We have also replaced the executor_config dictionary with the pod_override parameter, which takes a Kubernetes V1Pod object for a 1:1 setting override. These changes have removed over three thousand lines of code from the KubernetesExecutor, which makes it run faster and creates fewer potential errors.

Airflow core and providers: Splitting Airflow into 60+ packages

Airflow 2.0 is not a monolithic “one to rule them all” package. We’ve split Airflow into core and 61 (for now) provider packages. Each provider package is for either a particular external service (Google, Amazon, Microsoft, Snowflake), a database (Postgres, MySQL), or a protocol (HTTP/FTP). Now you can create a custom Airflow installation from “building” blocks and choose only what you need, plus add whatever other requirements you might have. Some of the common providers are installed automatically (ftp, http, imap, sqlite) as they are commonly used. Other providers are automatically installed when you choose appropriate extras when installing Airflow.

The provider architecture should make it much easier to get a fully customized, yet consistent runtime with the right set of Python dependencies.

But that’s not all: you can write your own custom providers and add things like custom connection types, customizations of the Connection Forms, and extra links to your operators in a manageable way. You can build your own provider and install it as a Python package and have your customizations visible right in the Airflow UI.

Security

As part of Airflow 2.0 effort, there has been a conscious focus on Security and reducing areas of exposure. This is represented across different functional areas in different forms. For example, in the new REST API, all operations now require authorization. Similarly, in the configuration settings, the Fernet key is now required to be specified.

Configuration

Configuration in the form of the airflow.cfg file has been rationalized further in distinct sections, specifically around “core”. Additionally, a significant amount of configuration options have been deprecated or moved to individual component-specific configuration files, such as the pod-template-file for Kubernetes execution-related configuration.

We’ve tried to make as few breaking changes as possible and to provide deprecation path in the code, especially in the case of anything called in the DAG. That said, please read through UPDATING.md to check what might affect you. For example: We re-organized the layout of operators (they now all live under airflow.providers.*) but the old names should continue to work - you’ll just notice a lot of DeprecationWarnings that need to be fixed up.

Release Date: 2021-02-08

Updated documentation and readme files.

Release Date: 2020-12-14

Initial version of the provider.

Previous Next

Was this entry helpful?

Suggest a change on this page