Release Date: 2026-08-08

Features

OpenLineage: add execute_in_thread to emit task events without forking (#68708)

Add team_name tag to OpenLineage metrics (#69109)

Fix openlineage team name handling, add dag_version_data and TI note (#70772)

Bug Fixes

OpenLineage: reuse per-process adapter in dag-state-change pool workers (#69283)

Fix OpenLineage team_name access and other failing paths (#70626)

Misc

Bump openlineage libraries to 1.52 in openlineage provider (#70773)

Doc-only

Add OpenLineage parent info to EMR Spark steps (#70182)

Document GlueJobOperator support for OpenLineage Spark injection (#69652)

Release Date: 2026-07-10

Features

Add unified emission_policy config for OpenLineage provider (#66992)

Misc

Doc-only

Add OpenLineage Spark conf injection to DatabricksSubmitRunOperator (#67894)

Release Date: 2026-06-22

Misc

Add openlineage system tests for missing use cases (#68664)

Release Date: 2026-06-07

Features

Openlineage: Read HTTP API key auth from Airflow connection (#66342)

Bug Fixes

Self-heal ProcessPoolExecutor on BrokenProcessPool (#67400)

Misc

Remove further findings from positional session check (#67712)

Doc-only

Correct README.rst common compat version reference inconsistency (#67554)

Release Date: 2026-05-23

Features

Add standardized SQL check representation for listeners (#66849)

Bug Fixes

Cache connections in OpenLineage SQL hook lineage (#64843)

Release Date: 2026-05-11

Features

Remove the DualStatsManager and the Stats interfaces (#63932)

Add public emit_dataset_lineage and emit_query_lineage API for OpenLineage provider (#65788)

Add dynamically emitted asset events to OpenLineage (#65727)

Bug Fixes

Prevent stats from breaking executor initializer (#66351)

Manual state change should not use fork-execute model on scheduler (#65677)

Misc

Cleanup integration names for consistency (#66287)

Bump Ol client dependency to 1.46 (#66145)

Release Date: 2026-04-26

Features

Add rendered map_index template to OpenLineage events (#65529)

Bug Fixes

Fix DetachedInstanceError in OpenLineage listener by using safe_getattr (#65579)

Release Date: 2026-04-13

Features

Add OpenLineage parent and transport info injection to 'EmrServerlessStartJobOperator' (#64807)

Add OpenLineage parent info injection to GlueJobOperator (#64513)

Add dagrun partition info to OpenLineage events (#64897)

Fix OpenLineage SQL utils emitting duplicate datasets for tables in multiple schemas (#64622)

Misc

Fix example dags documentation links (#65071)

Release Date: 2026-03-28

Features

Add Deadline Alerts to OL events (#63352)

Misc

Add Python 3.14 Support (#63520)

Doc-only

Clarify HLL in extraction precedence (#63723)

Fix typos and spelling (#64139)

Release Date: 2026-03-13

Features

Add DagRun note to OL events (#62347)

Significant Changes

DeltaTriggerTimetable for trigger-based scheduling (#47074)

This change introduces DeltaTriggerTimetable, a new built-in timetable that complements the existing suite of Airflow timetables by supporting delta-based trigger schedules without relying on data intervals.

Airflow currently has two major types of timetables:

• Data interval-based (e.g., CronDataIntervalTimetable, DeltaDataIntervalTimetable)
• Trigger-based (e.g., CronTriggerTimetable)

However, there was no equivalent trigger-based option for delta intervals like timedelta(days=1). As a result, even simple schedules like schedule=timedelta(days=1) were interpreted through a data interval lens—adding unnecessary complexity for users who don't care about upstream/downstream data dependencies.

This feature is backported to Airflow 2.11.0 to help users begin transitioning before upgrading to Airflow 3.0.

• In Airflow 2.11, schedule=timedelta(...) still defaults to DeltaDataIntervalTimetable.
• A new config option [scheduler] create_delta_data_intervals (default: True) allows opting in to DeltaTriggerTimetable.
• In Airflow 3.0, this config defaults to False, meaning DeltaTriggerTimetable becomes the default for timedelta schedules.

By flipping this config in 2.11, users can preview and adopt the new scheduling behavior in advance — minimizing surprises during upgrade.

Consistent timing metrics across all backends (#39908, #43966)

Previously, Airflow reported timing metrics in milliseconds for StatsD but in seconds for other backends such as OpenTelemetry and Datadog. This inconsistency made it difficult to interpret or compare timing metrics across systems.

Airflow 2.11 introduces a new config option:

• [metrics] timer_unit_consistency (default: False in 2.11, True and dropped in Airflow 3.0).

When enabled, all timing metrics are consistently reported in milliseconds, regardless of the backend.

This setting has become mandatory and always True in Airflow 3.0 (the config will be removed), so enabling it in 2.11 allows users to migrate early and avoid surprises during upgrade.

Ease migration to Airflow 3

This release introduces several changes to help users prepare for upgrading to Airflow 3:

• All models using execution_date now also include a logical_date field. Airflow 3 drops execution_date entirely in favor of logical_date (#44283)
• Added airflow config lint and airflow config update commands in 2.11 to help audit and migrate configs for Airflow 3.0. (#45736, #50353, #46757)

Python 3.8 support removed

Support for Python 3.8 has been removed, as it has reached end-of-life. Airflow 2.11 requires Python 3.9, 3.10, 3.11, or 3.12.

New Features

• Introduce DeltaTriggerTimetable (#47074)
• Backport airflow config update and airflow config lint changes to ease migration to Airflow 3 (#45736, #50353)
• Add link to show task in a DAG in DAG Dependencies view (#47721)
• Align timers and timing metrics (ms) across all metrics loggers (#39908, #43966)

Bug Fixes

• Don't resolve path for DAGs folder (#46877)
• Fix ti.log_url timestamp format from "%Y-%m-%dT%H:%M:%S%z" to "%Y-%m-%dT%H:%M:%S.%f%z" (#50306)
• Ensure that the generated airflow.cfg contains a random fernet_key and secret_key (#47755)
• Fixed setting rendered_map_index via internal api (#49057)
• Store rendered_map_index from TaskInstancePydantic into TaskInstance (#48571)
• Allow using log_url property on TaskInstancePydantic (Internal API) (#50560)
• Fix Trigger Form with Empty Object Default (#46872)
• Fix TypeError when deserializing task with execution_timeout set to None (#46822)
• Always populate mapped tasks (#46790)
• Ensure check_query_exists returns a bool (#46707)
• UI: /xcom/list got exception when applying filter on the value column (#46053)
• Allow to set note field via the experimental internal api (#47769)

Miscellaneous

• Add logical_date to models using execution_date (#44283)
• Drop support for Python 3.8 (#49980, #50015)
• Emit warning for deprecated BaseOperatorLink.get_link signature (#46448)

Doc Only Changes

• Unquote executor airflow.cfg variable (#48084)
• Update XCom docs to show examples of pushing multiple XComs (#46284, #47068)

Significant Changes

No significant changes.

Bug Fixes

• Revert "Fix: DAGs are not marked as stale if the dags folder change" (#42220, #42217)
• Add missing open telemetry span and correct scheduled slots documentation (#41985)
• Fix require_confirmation_dag_change (#42063) (#42211)
• Only treat null/undefined as falsy when rendering XComEntry (#42199) (#42213)
• Add extra and renderedTemplates as keys to skip camelCasing (#42206) (#42208)
• Do not camelcase xcom entries (#42182) (#42187)
• Fix task_instance and dag_run links from list views (#42138) (#42143)
• Support multi-line input for Params of type string in trigger UI form (#40414) (#42139)
• Fix details tab log url detection (#42104) (#42114)
• Add new type of exception to catch timeout (#42064) (#42078)
• Rewrite how DAG to dataset / dataset alias are stored (#41987) (#42055)
• Allow dataset alias to add more than one dataset events (#42189) (#42247)

Miscellaneous

• Limit universal-pathlib below 0.2.4 as it breaks our integration (#42101)
• Auto-fix default deferrable with LibCST (#42089)
• Deprecate --tree flag for tasks list cli command (#41965)

Doc Only Changes

• Update security_model.rst to clear unauthenticated endpoints exceptions (#42085)
• Add note about dataclasses and attrs to XComs page (#42056)
• Improve docs on markdown docs in DAGs (#42013)
• Add warning that listeners can be dangerous (#41968)

Significant Changes

No significant changes.

Bug Fixes

• Handle Example dags case when checking for missing files (#41874)
• Fix logout link in "no roles" error page (#41845)
• Set end_date and duration for triggers completed with end_from_trigger as True. (#41834)
• DAGs are not marked as stale if the dags folder change (#41829)
• Fix compatibility with FAB provider versions <1.3.0 (#41809)
• Don't Fail LocalTaskJob on heartbeat (#41810)
• Remove deprecation warning for cgitb in Plugins Manager (#41793)
• Fix log for notifier(instance) without name (#41699)
• Splitting syspath preparation into stages (#41694)
• Adding url sanitization for extra links (#41680)
• Fix InletEventsAccessors type stub (#41607)
• Fix UI rendering when XCom is INT, FLOAT, BOOL or NULL (#41605)
• Fix try selector refresh (#41503)
• Incorrect try number subtraction producing invalid span id for OTEL airflow (#41535)
• Add WebEncoder for trigger page rendering to avoid render failure (#41485)
• Adding tojson filter to example_inlet_event_extra example dag (#41890)
• Add backward compatibility check for executors that don't inherit BaseExecutor (#41927)

Miscellaneous

• Bump webpack from 5.76.0 to 5.94.0 in /airflow/www (#41879)
• Adding rel property to hyperlinks in logs (#41783)
• Field Deletion Warning when editing Connections (#41504)
• Make Scarf usage reporting in major+minor versions and counters in buckets (#41900)
• Lower down universal-pathlib minimum to 0.2.2 (#41943)
• Protect against None components of universal pathlib xcom backend (#41938)

Doc Only Changes

• Remove Debian bullseye support (#41569)
• Add an example for auth with keycloak (#41791)

Significant Changes

Datasets no longer trigger inactive DAGs (#38891)

Previously, when a DAG is paused or removed, incoming dataset events would still trigger it, and the DAG would run when it is unpaused or added back in a DAG file. This has been changed; a DAG's dataset schedule can now only be satisfied by events that occur when the DAG is active. While this is a breaking change, the previous behavior is considered a bug.

The behavior of time-based scheduling is unchanged, including the timetable part of DatasetOrTimeSchedule.

try_number is no longer incremented during task execution (#39336)

Previously, the try number (try_number) was incremented at the beginning of task execution on the worker. This was problematic for many reasons. For one it meant that the try number was incremented when it was not supposed to, namely when resuming from reschedule or deferral. And it also resulted in the try number being "wrong" when the task had not yet started. The workarounds for these two issues caused a lot of confusion.

Now, instead, the try number for a task run is determined at the time the task is scheduled, and does not change in flight, and it is never decremented. So after the task runs, the observed try number remains the same as it was when the task was running; only when there is a "new try" will the try number be incremented again.

One consequence of this change is, if users were "manually" running tasks (e.g. by calling ti.run() directly, or command line airflow tasks run), try number will no longer be incremented. Airflow assumes that tasks are always run after being scheduled by the scheduler, so we do not regard this as a breaking change.

/logout endpoint in FAB Auth Manager is now CSRF protected (#40145)

The /logout endpoint's method in FAB Auth Manager has been changed from GET to POST in all existing AuthViews (AuthDBView, AuthLDAPView, AuthOAuthView, AuthOIDView, AuthRemoteUserView), and now includes CSRF protection to enhance security and prevent unauthorized logouts.

OpenTelemetry Traces for Apache Airflow (#37948).

This new feature adds capability for Apache Airflow to emit 1) airflow system traces of scheduler, triggerer, executor, processor 2) DAG run traces for deployed DAG runs in OpenTelemetry format. Previously, only metrics were supported which emitted metrics in OpenTelemetry. This new feature will add richer data for users to use OpenTelemetry standard to emit and send their trace data to OTLP compatible endpoints.

Decorator for Task Flow (@skip_if, @run_if) to make it simple to apply whether or not to skip a Task. (#41116)

This feature adds a decorator to make it simple to skip a Task.

Using Multiple Executors Concurrently (#40701)

Previously known as hybrid executors, this new feature allows Airflow to use multiple executors concurrently. DAGs, or even individual tasks, can be configured to use a specific executor that suits its needs best. A single DAG can contain tasks all using different executors. Please see the Airflow documentation for more details. Note: This feature is still experimental. See documentation on Executor <https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/index.html#using-multiple-executors-concurrently>_ for a more detailed description.

Scarf based telemetry: Does Airflow collect any telemetry data? (#39510)

Airflow integrates Scarf to collect basic usage data during operation. Deployments can opt-out of data collection by setting the [usage_data_collection]enabled option to False, or the SCARF_ANALYTICS=false environment variable. See FAQ on this <https://airflow.apache.org/docs/apache-airflow/stable/faq.html#does-airflow-collect-any-telemetry-data>_ for more information.

New Features

• AIP-61 Hybrid Execution (AIP-61)
• AIP-62 Getting Lineage from Hook Instrumentation (AIP-62)
• AIP-64 TaskInstance Try History (AIP-64)
• AIP-44 Internal API (AIP-44)
• Enable ending the task directly from the triggerer without going into the worker. (#40084)
• Extend dataset dependencies (#40868)
• Feature/add token authentication to internal api (#40899)
• Add DatasetAlias to support dynamic Dataset Event Emission and Dataset Creation (#40478)
• Add example DAGs for inlet_events (#39893)
• Implement accessors to read dataset events defined as inlet (#39367)
• Decorator for Task Flow, to make it simple to apply whether or not to skip a Task. (#41116)
• Add start execution from triggerer support to dynamic task mapping (#39912)
• Add try_number to log table (#40739)
• Added ds_format_locale method in macros which allows localizing datetime formatting using Babel (#40746)
• Add DatasetAlias to support dynamic Dataset Event Emission and Dataset Creation (#40478, #40723, #40809, #41264, #40830, #40693, #41302)
• Use sentinel to mark dag as removed on re-serialization (#39825)
• Add parameter for the last number of queries to the DB in DAG file processing stats (#40323)
• Add prototype version dark mode for Airflow UI (#39355)
• Add ability to mark some tasks as successful in dag test (#40010)
• Allow use of callable for template_fields (#37028)
• Filter running/failed and active/paused dags on the home page(#39701)
• Add metrics about task CPU and memory usage (#39650)
• UI changes for DAG Re-parsing feature (#39636)
• Add Scarf based telemetry (#39510, #41318)
• Add dag re-parsing request endpoint (#39138)
• Redirect to new DAGRun after trigger from Grid view (#39569)
• Display endDate in task instance tooltip. (#39547)
• Implement accessors to read dataset events defined as inlet (#39367, #39893)
• Add color to log lines in UI for error and warnings based on keywords (#39006)
• Add Rendered k8s pod spec tab to ti details view (#39141)
• Make audit log before/after filterable (#39120)
• Consolidate grid collapse actions to a single full screen toggle (#39070)
• Implement Metadata to emit runtime extra (#38650)
• Add executor field to the DB and parameter to the operators (#38474)
• Implement context accessor for DatasetEvent extra (#38481)
• Add dataset event info to dag graph (#41012)
• Add button to toggle datasets on/off in dag graph (#41200)
• Add run_if & skip_if decorators (#41116)
• Add dag_stats rest api endpoint (#41017)
• Add listeners for Dag import errors (#39739)
• Allowing DateTimeSensorAsync, FileSensor and TimeSensorAsync to start execution from trigger during dynamic task mapping (#41182)

Improvements

• Allow set Dag Run resource into Dag Level permission: extends Dag's access_control feature to allow Dag Run resource permissions. (#40703)
• Improve security and error handling for the internal API (#40999)
• Datasets UI Improvements (#40871)
• Change DAG Audit log tab to Event Log (#40967)
• Make standalone dag file processor works in DB isolation mode (#40916)
• Show only the source on the consumer DAG page and only triggered DAG run in the producer DAG page (#41300)
• Update metrics names to allow multiple executors to report metrics (#40778)
• Format DAG run count (#39684)
• Update styles for renderedjson component (#40964)
• Improve ATTRIBUTE_REMOVED sentinel to use class and more context (#40920)
• Make XCom display as react json (#40640)
• Replace usages of task context logger with the log table (#40867)
• Rollback for all retry exceptions (#40882) (#40883)
• Support rendering ObjectStoragePath value (#40638)
• Add try_number and map_index as params for log event endpoint (#40845)
• Rotate fernet key in batches to limit memory usage (#40786)
• Add gauge metric for 'last_num_of_db_queries' parameter (#40833)
• Set parallelism log messages to warning level for better visibility (#39298)
• Add error handling for encoding the dag runs (#40222)
• Use params instead of dag_run.conf in example DAG (#40759)
• Load Example Plugins with Example DAGs (#39999)
• Stop deferring TimeDeltaSensorAsync task when the target_dttm is in the past (#40719)
• Send important executor logs to task logs (#40468)
• Open external links in new tabs (#40635)
• Attempt to add ReactJSON view to rendered templates (#40639)
• Speeding up regex match time for custom warnings (#40513)
• Refactor DAG.dataset_triggers into the timetable class (#39321)
• add next_kwargs to StartTriggerArgs (#40376)
• Improve UI error handling (#40350)
• Remove double warning in CLI when config value is deprecated (#40319)
• Implement XComArg concat() (#40172)
• Added get_extra_dejson method with nested parameter which allows you to specify if you want the nested json as string to be also deserialized (#39811)
• Add executor field to the task instance API (#40034)
• Support checking for db path absoluteness on Windows (#40069)
• Introduce StartTriggerArgs and prevent start trigger initialization in scheduler (#39585)
• Add task documentation to details tab in grid view (#39899)
• Allow executors to be specified with only the class name of the Executor (#40131)
• Remove obsolete conditional logic related to try_number (#40104)
• Allow Task Group Ids to be passed as branches in BranchMixIn (#38883)
• Javascript connection form will apply CodeMirror to all textarea's dynamically (#39812)
• Determine needs_expansion at time of serialization (#39604)
• Add indexes on dag_id column in referencing tables to speed up deletion of dag records (#39638)
• Add task failed dependencies to details page (#38449)
• Remove webserver try_number adjustment (#39623)
• Implement slicing in lazy sequence (#39483)
• Unify lazy db sequence implementations (#39426)
• Add __getattr__ to task decorator stub (#39425)
• Allow passing labels to FAB Views registered via Plugins (#39444)
• Simpler error message when trying to offline migrate with sqlite (#39441)
• Add soft_fail to TriggerDagRunOperator (#39173)
• Rename "dataset event" in context to use "outlet" (#39397)
• Resolve RemovedIn20Warning in airflow task command (#39244)
• Determine fail_stop on client side when db isolated (#39258)
• Refactor cloudpickle support in Python operators/decorators (#39270)
• Update trigger kwargs migration to specify existing_nullable (#39361)
• Allowing tasks to start execution directly from triggerer without going to worker (#38674)
• Better db migrate error messages (#39268)
• Add stacklevel into the suppress_and_warn warning (#39263)
• Support searching by dag_display_name (#39008)
• Allow sort by on all fields in MappedInstances.tsx (#38090)
• Expose count of scheduled tasks in metrics (#38899)
• Use declarative_base from sqlalchemy.orm instead of sqlalchemy.ext.declarative (#39134)
• Add example DAG to demonstrate emitting approaches (#38821)
• Give on_task_instance_failed access to the error that caused the failure (#38155)
• Simplify dataset serialization (#38694)
• Add heartbeat recovery message to jobs (#34457)
• Remove select_column option in TaskInstance.get_task_instance (#38571)
• Don't create session in get_dag if not reading dags from database (#38553)
• Add a migration script for encrypted trigger kwargs (#38358)
• Implement render_templates on TaskInstancePydantic (#38559)
• Handle optional session in _refresh_from_db (#38572)
• Make type annotation less confusing in task_command.py (#38561)
• Use fetch_dagrun directly to avoid session creation (#38557)
• Added output_processor parameter to BashProcessor (#40843)
• Improve serialization for Database Isolation Mode (#41239)
• Only orphan non-orphaned Datasets (#40806)
• Adjust gantt width based on task history dates (#41192)
• Enable scrolling on legend with high number of elements. (#41187)

Bug Fixes

• Bugfix for get_parsing_context() when ran with LocalExecutor (#40738)
• Validating provider documentation urls before displaying in views (#40933)
• Move import to make PythonOperator working on Windows (#40424)
• Fix dataset_with_extra_from_classic_operator example DAG (#40747)
• Call listener on_task_instance_failed() after ti state is changed (#41053)
• Add never_fail in BaseSensor (#40915)
• Fix tasks API endpoint when DAG doesn't have start_date (#40878)
• Fix and adjust URL generation for UI grid and older runs (#40764)
• Rotate fernet key optimization (#40758)
• Fix class instance vs. class type in validate_database_executor_compatibility() call (#40626)
• Clean up dark mode (#40466)
• Validate expected types for args for DAG, BaseOperator and TaskGroup (#40269)
• Exponential Backoff Not Functioning in BaseSensorOperator Reschedule Mode (#39823)
• local task job: add timeout, to not kill on_task_instance_success listener prematurely (#39890)
• Move Post Execution Log Grouping behind Exception Print (#40146)
• Fix triggerer race condition in HA setting (#38666)
• Pass triggered or existing DAG Run logical date to DagStateTrigger (#39960)
• Passing external_task_group_id to WorkflowTrigger (#39617)
• ECS Executor: Set tasks to RUNNING state once active (#39212)
• Only heartbeat if necessary in backfill loop (#39399)
• Fix trigger kwarg encryption migration (#39246)
• Fix decryption of trigger kwargs when downgrading. (#38743)
• Fix wrong link in TriggeredDagRuns (#41166)
• Pass MapIndex to LogLink component for external log systems (#41125)
• Add NonCachingRotatingFileHandler for worker task (#41064)
• Add argument include_xcom in method resolve an optional value (#41062)
• Sanitizing file names in example_bash_decorator DAG (#40949)
• Show dataset aliases in dependency graphs (#41128)
• Render Dataset Conditions in DAG Graph view (#41137)
• Add task duration plot across dagruns (#40755)
• Add start execution from trigger support for existing core sensors (#41021)
• add example dag for dataset_alias (#41037)
• Add dataset alias unique constraint and remove wrong dataset alias removing logic (#41097)
• Set "has_outlet_datasets" to true if "dataset alias" exists (#41091)
• Make HookLineageCollector group datasets by (#41034)
• Enhance start_trigger_args serialization (#40993)
• Refactor BaseSensorOperator introduce skip_policy parameter (#40924)
• Fix viewing logs from triggerer when task is deferred (#41272)
• Refactor how triggered dag run url is replaced (#41259)
• Added support for additional sql alchemy session args (#41048)
• Allow empty list in TriggerDagRun failed_state (#41249)
• Clean up the exception handler when run_as_user is the airflow user (#41241)
• Collapse docs when click and folded (#41214)
• Update updated_at when saving to db as session.merge does not trigger on-update (#40782)
• Fix query count statistics when parsing DAF file (#41149)
• Method Resolution Order in operators without __init__ (#41086)
• Ensure try_number incremented for empty operator (#40426)

Miscellaneous

• Remove the Experimental flag from OTel Traces (#40874)
• Bump packaging version to 23.0 in order to fix issue with older otel (#40865)
• Simplify _auth_manager_is_authorized_map function (#40803)
• Use correct unknown executor exception in scheduler job (#40700)
• Add D1 pydocstyle rules to pyproject.toml (#40569)
• Enable enforcing pydocstyle rule D213 in ruff. (#40448, #40464)
• Update Dag.test() to run with an executor if desired (#40205)
• Update jest and babel minor versions (#40203)
• Refactor BashOperator and Bash decorator for consistency and simplicity (#39871)
• Add AirflowInternalRuntimeError for raise non catchable errors (#38778)
• ruff version bump 0.4.5 (#39849)
• Bump pytest to 8.0+ (#39450)
• Remove stale comment about TI index (#39470)
• Configure back_populates between DagScheduleDatasetReference.dag and DagModel.schedule_dataset_references (#39392)
• Remove deprecation warnings in endpoints.py (#39389)
• Fix SQLA deprecations in Airflow core (#39211)
• Use class-bound attribute directly in SA (#39198, #39195)
• Fix stacklevel for TaskContextLogger (#39142)
• Capture warnings during collect DAGs (#39109)
• Resolve B028 (no-explicit-stacklevel) in core (#39123)
• Rename model ImportError to ParseImportError for avoid shadowing with builtin exception (#39116)
• Add option to support cloudpickle in PythonVenv/External Operator (#38531)
• Suppress SubDagOperator examples warnings (#39057)
• Add log for running callback (#38892)
• Use model_dump instead of dict for serialize Pydantic V2 model (#38933)
• Widen cheat sheet column to avoid wrapping commands (#38888)
• Update hatchling to latest version (1.22.5) (#38780)
• bump uv to 0.1.29 (#38758)
• Add missing serializations found during provider tests fixing (#41252)
• Bump ws from 7.5.5 to 7.5.10 in /airflow/www (#40288)
• Improve typing for allowed/failed_states in TriggerDagRunOperator (#39855)

Doc Only Changes

• Add filesystems and dataset-uris to "how to create your own provider" page (#40801)
• Fix (TM) to (R) in Airflow repository (#40783)
• Set otel_on to True in example airflow.cfg (#40712)
• Add warning for _AIRFLOW_PATCH_GEVENT (#40677)
• Update multi-team diagram proposal after Airflow 3 discussions (#40671)
• Add stronger warning that MSSQL is not supported and no longer functional (#40565)
• Fix misleading mac menu structure in howto (#40440)
• Update k8s supported version in docs (#39878)
• Add compatibility note for Listeners (#39544)
• Update edge label image in documentation example with the new graph view (#38802)
• Update UI doc screenshots (#38680)
• Add section "Manipulating queued dataset events through REST API" (#41022)
• Add information about lack of security guarantees for docker compose (#41072)
• Add links to example dags in use params section (#41031)
• Change task_id from send_email to send_email_notification in taskflow.rst (#41060)
• Remove unnecessary nginx redirect rule from reverse proxy documentation (#38953)

Significant Changes

No significant changes.

Bug Fixes

• Fix bug that makes AirflowSecurityManagerV2 leave transactions in the idle in transaction state (#39935)
• Fix alembic auto-generation and rename mismatching constraints (#39032)
• Add the existing_nullable to the downgrade side of the migration (#39374)
• Fix Mark Instance state buttons stay disabled if user lacks permission (#37451). (#38732)
• Use SKIP LOCKED instead of NOWAIT in mini scheduler (#39745)
• Remove DAG Run Add option from FAB view (#39881)
• Add max_consecutive_failed_dag_runs in API spec (#39830)
• Fix example_branch_operator failing in python 3.12 (#39783)
• Fetch served logs also when task attempt is up for retry and no remote logs available (#39496)
• Change dataset URI validation to raise warning instead of error in Airflow 2.9 (#39670)
• Visible DAG RUN doesn't point to the same dag run id (#38365)
• Refactor SafeDogStatsdLogger to use get_validator to enable pattern matching (#39370)
• Fix custom actions in security manager has_access (#39421)
• Fix HTTP 500 Internal Server Error if DAG is triggered with bad params (#39409)
• Fix static file caching is disabled in Airflow Webserver. (#39345)
• Fix TaskHandlerWithCustomFormatter now adds prefix only once (#38502)
• Do not provide deprecated execution_date in @apply_lineage (#39327)
• Add missing conn_id to string representation of ObjectStoragePath (#39313)
• Fix sql_alchemy_engine_args config example (#38971)
• Add Cache-Control "no-store" to all dynamically generated content (#39550)

Miscellaneous

• Limit yandex provider to avoid mypy errors (#39990)
• Warn on mini scheduler failures instead of debug (#39760)
• Change type definition for provider_info_cache decorator (#39750)
• Better typing for BaseOperator defer (#39742)
• More typing in TimeSensor and TimeSensorAsync (#39696)
• Re-raise exception from strict dataset URI checks (#39719)
• Fix stacklevel for _log_state helper (#39596)
• Resolve SA warnings in migrations scripts (#39418)
• Remove unused index idx_last_scheduling_decision on dag_run table (#39275)

Doc Only Changes

• Provide extra tip on labeling DynamicTaskMapping (#39977)
• Improve visibility of links / variables / other configs in Configuration Reference (#39916)
• Remove 'legacy' definition for CronDataIntervalTimetable (#39780)
• Update plugins.rst examples to use pyproject.toml over setup.py (#39665)
• Fix nit in pg set-up doc (#39628)
• Add Matomo to Tracking User Activity docs (#39611)
• Fix Connection.get -> Connection. get_connection_from_secrets (#39560)
• Adding note for provider dependencies (#39512)
• Update docker-compose command (#39504)
• Update note about restarting triggerer process (#39436)
• Updating S3LogLink with an invalid bucket link (#39424)
• Update testing_packages.rst (#38996)
• Add multi-team diagrams (#38861)

Significant Changes

Stackdriver logging bugfix requires Google provider 10.17.0 or later (#38071)

If you use Stackdriver logging, you must use Google provider version 10.17.0 or later. Airflow 2.9.1 now passes gcp_log_name to the StackdriverTaskHandler instead of name, and this will fail on earlier provider versions.

This fixes a bug where the log name configured in [logging] remove_base_log_folder was overridden when Airflow configured logging, resulting in task logs going to the wrong destination.

Bug Fixes

• Make task log messages include run_id (#39280)
• Copy menu_item href for nav bar (#39282)
• Fix trigger kwarg encryption migration (#39246, #39361, #39374)
• Add workaround for datetime-local input in firefox (#39261)
• Add Grid button to Task Instance view (#39223)
• Get served logs when remote or executor logs not available for non-running task try (#39177)
• Fixed side effect of menu filtering causing disappearing menus (#39229)
• Use grid view for Task Instance's log_url (#39183)
• Improve task filtering UX (#39119)
• Improve rendered_template ux in react dag page (#39122)
• Graph view improvements (#38940)
• Check that the dataset<>task exists before trying to render graph (#39069)
• Hostname was "redacted", not "redact"; remove it when there is no context (#39037)
• Check whether AUTH_ROLE_PUBLIC is set in check_authentication (#39012)
• Move rendering of map_index_template so it renders for failed tasks as long as it was defined before the point of failure (#38902)
• Undeprecate BaseXCom.get_one method for now (#38991)
• Add inherit_cache attribute for CreateTableAs custom SA Clause (#38985)
• Don't wait for DagRun lock in mini scheduler (#38914)
• Fix calendar view with no DAG Run (#38964)
• Changed the background color of external task in graph (#38969)
• Fix dag run selection (#38941)
• Fix SAWarning 'Coercing Subquery object into a select() for use in IN()' (#38926)
• Fix implicit cartesian product in AirflowSecurityManagerV2 (#38913)
• Fix problem that links in legacy log view can not be clicked (#38882)
• Fix dag run link params (#38873)
• Use async db calls in WorkflowTrigger (#38689)
• Fix audit log events filter (#38719)
• Use methodtools.lru_cache instead of functools.lru_cache in class methods (#37757)
• Raise deprecated warning in airflow dags backfill only if -I / --ignore-first-depends-on-past provided (#38676)

Miscellaneous

• TriggerDagRunOperator deprecate execution_date in favor of logical_date (#39285)
• Force to use Airflow Deprecation warnings categories on @deprecated decorator (#39205)
• Add warning about run/import Airflow under the Windows (#39196)
• Update is_authorized_custom_view from auth manager to handle custom actions (#39167)
• Add in Trove classifiers Python 3.12 support (#39004)
• Use debug level for minischeduler skip (#38976)
• Bump undici from 5.28.3 to 5.28.4 in /airflow/www (#38751)

Doc Only Changes

• Fix supported k8s version in docs (#39172)
• Dynamic task mapping PythonOperator op_kwargs (#39242)
• Add link to user and role commands (#39224)
• Add k8s 1.29 to supported version in docs (#39168)
• Data aware scheduling docs edits (#38687)
• Update DagBag class docstring to include all params (#38814)
• Correcting an example taskflow example (#39015)
• Remove decorator from rendering fields example (#38827)

Significant Changes

Following Listener API methods are considered stable and can be used for production system (were experimental feature in older Airflow versions) (#36376):

Lifecycle events:

• on_starting
• before_stopping

DagRun State Change Events:

• on_dag_run_running
• on_dag_run_success
• on_dag_run_failed

TaskInstance State Change Events:

• on_task_instance_running
• on_task_instance_success
• on_task_instance_failed

Support for Microsoft SQL-Server for Airflow Meta Database has been removed (#36514)

After discussion <https://lists.apache.org/thread/r06j306hldg03g2my1pd4nyjxg78b3h4>__ and a voting process <https://lists.apache.org/thread/pgcgmhf6560k8jbsmz8nlyoxosvltph2>__, the Airflow's PMC and Committers have reached a resolution to no longer maintain MsSQL as a supported Database Backend.

As of Airflow 2.9.0 support of MsSQL has been removed for Airflow Database Backend.

A migration script which can help migrating the database before upgrading to Airflow 2.9.0 is available in airflow-mssql-migration repo on Github <https://github.com/apache/airflow-mssql-migration>_. Note that the migration script is provided without support and warranty.

This does not affect the existing provider packages (operators and hooks), DAGs can still access and process data from MsSQL.

Dataset URIs are now validated on input (#37005)

Datasets must use a URI that conform to rules laid down in AIP-60, and the value will be automatically normalized when the DAG file is parsed. See documentation on Datasets <https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/datasets.html>_ for a more detailed description on the rules.

You may need to change your Dataset identifiers if they look like a URI, but are used in a less mainstream way, such as relying on the URI's auth section, or have a case-sensitive protocol name.

The method get_permitted_menu_items in BaseAuthManager has been renamed filter_permitted_menu_items (#37627)

Add REST API actions to Audit Log events (#37734)

The Audit Log event name for REST API events will be prepended with api. or ui., depending on if it came from the Airflow UI or externally.

Official support for Python 3.12 (#38025)

There are a few caveats though:

• Pendulum2 does not support Python 3.12. For Python 3.12 you need to use Pendulum 3 <https://pendulum.eustace.io/blog/announcing-pendulum-3-0-0.html>_

• Minimum SQLAlchemy version supported when Pandas is installed for Python 3.12 is 1.4.36 released in April 2022. Airflow 2.9.0 increases the minimum supported version of SQLAlchemy to 1.4.36 for all Python versions.

Not all Providers support Python 3.12. At the initial release of Airflow 2.9.0 the following providers are released without support for Python 3.12:

• apache.beam - pending on Apache Beam support for 3.12 <https://github.com/apache/beam/issues/29149>_
• papermill - pending on Releasing Python 3.12 compatible papermill client version including this merged issue <https://github.com/nteract/papermill/pull/771>_

Prevent large string objects from being stored in the Rendered Template Fields (#38094)

There's now a limit to the length of data that can be stored in the Rendered Template Fields. The limit is set to 4096 characters. If the data exceeds this limit, it will be truncated. You can change this limit by setting the [core]max_template_field_length configuration option in your airflow config.

Change xcom table column value type to longblob for MySQL backend (#38401)

Xcom table column value type has changed from blob to longblob. This will allow you to store relatively big data in Xcom but process can take a significant amount of time if you have a lot of large data stored in Xcom.

To downgrade from revision: b4078ac230a1, ensure that you don't have Xcom values larger than 65,535 bytes. Otherwise, you'll need to clean those rows or run airflow db clean xcom to clean the Xcom table.

New Features

• Allow users to write dag_id and task_id in their national characters, added display name for dag / task (v2) (#38446)
• Prevent large objects from being stored in the RTIF (#38094)
• Use current time to calculate duration when end date is not present. (#38375)
• Add average duration mark line in task and dagrun duration charts. (#38214, #38434)
• Add button to manually create dataset events (#38305)
• Add Matomo as an option for analytics_tool. (#38221)
• Experimental: Support custom weight_rule implementation to calculate the TI priority_weight (#38222)
• Adding ability to automatically set DAG to off after X times it failed sequentially (#36935)
• Add dataset conditions to next run datasets modal (#38123)
• Add task log grouping to UI (#38021)
• Add dataset_expression to grid dag details (#38121)
• Introduce mechanism to support multiple executor configuration (#37635)
• Add color formatting for ANSI chars in logs from task executions (#37985)
• Add the dataset_expression as part of DagModel and DAGDetailSchema (#37826)
• Add TaskFail entries to Gantt chart (#37918)
• Allow longer rendered_map_index (#37798)
• Inherit the run_ordering from DatasetTriggeredTimetable for DatasetOrTimeSchedule (#37775)
• Implement AIP-60 Dataset URI formats (#37005)
• Introducing Logical Operators for dataset conditional logic (#37101)
• Add post endpoint for dataset events (#37570)
• Show custom instance names for a mapped task in UI (#36797)
• Add excluded/included events to get_event_logs api (#37641)
• Add datasets to dag graph (#37604)
• Show dataset events above task/run details in grid view (#37603)
• Introduce new config variable to control whether DAG processor outputs to stdout (#37439)
• Make Datasets hashable (#37465)
• Add conditional logic for dataset triggering (#37016)
• Implement task duration page in react. (#35863)
• Add queuedEvent endpoint to get/delete DatasetDagRunQueue (#37176)
• Support multiple XCom output in the BaseOperator (#37297)
• AIP-58: Add object storage backend for xcom (#37058)
• Introduce DatasetOrTimeSchedule (#36710)
• Add on_skipped_callback to BaseOperator (#36374)
• Allow override of hovered navbar colors (#36631)
• Create new Metrics with Tagging (#36528)
• Add support for openlineage to AFS and common.io (#36410)
• Introduce @task.bash TaskFlow decorator (#30176, #37875)
• Added functionality to automatically ingest custom airflow.cfg file upon startup (#36289)

Improvements

• More human friendly "show tables" output for db cleanup (#38654)
• Improve trigger assign_unassigned by merging alive_triggerer_ids and get_sorted_triggers queries (#38664)
• Add exclude/include events filters to audit log (#38506)
• Clean up unused triggers in a single query for all dialects except MySQL (#38663)
• Update Confirmation Logic for Config Changes on Sensitive Environments Like Production (#38299)
• Improve datasets graph UX (#38476)
• Only show latest dataset event timestamp after last run (#38340)
• Add button to clear only failed tasks in a dagrun. (#38217)
• Delete all old dag pages and redirect to grid view (#37988)
• Check task attribute before use in sentry.add_tagging() (#37143)
• Mysql change xcom value col type for MySQL backend (#38401)
• ExternalPythonOperator use version from sys.version_info (#38377)
• Replace too broad exceptions into the Core (#38344)
• Add CLI support for bulk pause and resume of DAGs (#38265)
• Implement methods on TaskInstancePydantic and DagRunPydantic (#38295, #38302, #38303, #38297)
• Made filters bar collapsible and add a full screen toggle (#38296)
• Encrypt all trigger attributes (#38233, #38358, #38743)
• Upgrade react-table package. Use with Audit Log table (#38092)
• Show if dag page filters are active (#38080)
• Add try number to mapped instance (#38097)
• Add retries to job heartbeat (#37541)
• Add REST API events to Audit Log (#37734)
• Make current working directory as templated field in BashOperator (#37968)
• Add calendar view to react (#37909)
• Add run_id column to log table (#37731)
• Add tryNumber to grid task instance tooltip (#37911)
• Session is not used in _do_render_template_fields (#37856)
• Improve MappedOperator property types (#37870)
• Remove provide_session decorator from TaskInstancePydantic methods (#37853)
• Ensure the "airflow.task" logger used for TaskInstancePydantic and TaskInstance (#37857)
• Better error message for internal api call error (#37852)
• Increase tooltip size of dag grid view (#37782) (#37805)
• Use named loggers instead of root logger (#37801)
• Add Run Duration in React (#37735)
• Avoid non-recommended usage of logging (#37792)
• Improve DateTimeTrigger typing (#37694)
• Make sure all unique run_ids render a task duration bar (#37717)
• Add Dag Audit Log to React (#37682)
• Add log event for auto pause (#38243)
• Better message for exception for templated base operator fields (#37668)
• Clean up webserver endpoints adding to audit log (#37580)
• Filter datasets graph by dag_id (#37464)
• Use new exception type inheriting BaseException for SIGTERMs (#37613)
• Refactor dataset class inheritance (#37590)
• Simplify checks for package versions (#37585)
• Filter Datasets by associated dag_ids (GET /datasets) (#37512)
• Enable "airflow tasks test" to run deferrable operator (#37542)
• Make datasets list/graph width adjustable (#37425)
• Speedup determine installed airflow version in ExternalPythonOperator (#37409)
• Add more task details from rest api (#37394)
• Add confirmation dialog box for DAG run actions (#35393)
• Added shutdown color to the STATE_COLORS (#37295)
• Remove legacy dag details page and redirect to grid (#37232)
• Order XCom entries by map index in API (#37086)
• Add data_interval_start and data_interval_end in dagrun create API endpoint (#36630)
• Making links in task logs as hyperlinks by preventing HTML injection (#36829)
• Improve ExternalTaskSensor Async Implementation (#36916)
• Make Datasets Pathlike (#36947)
• Simplify query for orphaned tasks (#36566)
• Add deferrable param in FileSensor (#36840)
• Run Trigger Page: Configurable number of recent configs (#36878)
• Merge nowait and skip_locked into with_row_locks (#36889)
• Return the specified field when get dag/dagRun in the REST API (#36641)
• Only iterate over the items if debug is enabled for DagFileProcessorManager (#36761)
• Add a fuzzy/regex pattern-matching for metric allow and block list (#36250)
• Allow custom columns in cli dags list (#35250)
• Make it possible to change the default cron timetable (#34851)
• Some improvements to Airflow IO code (#36259)
• Improve TaskInstance typing hints (#36487)
• Remove dependency of Connexion from auth manager interface (#36209)
• Refactor ExternalDagLink to not create ad hoc TaskInstances (#36135)

Bug Fixes

• Load providers configuration when gunicorn workers start (#38795)
• Fix grid header rendering (#38720)
• Add a task instance dependency for mapped dependencies (#37498)
• Improve stability of remove_task_decorator function (#38649)
• Mark more fields on API as dump-only (#38616)
• Fix total_entries count on the event logs endpoint (#38625)
• Add padding to bottom of log block. (#38610)
• Properly serialize nested attrs classes (#38591)
• Fixing the tz in next run ID info (#38482)
• Show abandoned tasks in Grid View (#38511)
• Apply task instance mutation hook consistently (#38440)
• Override chakra styles to keep dropdowns in filter bar (#38456)
• Store duration in seconds and scale to handle case when a value in the series has a larger unit than the preceding durations. (#38374)
• Don't allow defaults other than None in context parameters, and improve error message (#38015)
• Make postgresql default engine args comply with SA 2.0 (#38362)
• Add return statement to yield within a while loop in triggers (#38389)
• Ensure __exit__ is called in decorator context managers (#38383)
• Make the method BaseAuthManager.is_authorized_custom_view abstract (#37915)
• Add upper limit to planned calendar events calculation (#38310)
• Fix Scheduler in daemon mode doesn't create PID at the specified location (#38117)
• Properly serialize TaskInstancePydantic and DagRunPydantic (#37855)
• Fix graph task state border color (#38084)
• Add back methods removed in security manager (#37997)
• Don't log "403" from worker serve-logs as "Unknown error". (#37933)
• Fix execution data validation error in /get_logs_with_metadata endpoint (#37756)
• Fix task duration selection (#37630)
• Refrain from passing encoding to the SQL engine in SQLAlchemy v2 (#37545)
• Fix 'implicitly coercing SELECT object to scalar subquery' in latest dag run statement (#37505)
• Clean up typing with max_execution_date query builder (#36958)
• Optimize max_execution_date query in single dag case (#33242)
• Fix list dags command for get_dagmodel is None (#36739)
• Load consuming_dags attr eagerly before dataset listener (#36247)

Miscellaneous

• Remove display of param from the UI (#38660)
• Update log level to debug from warning about scheduled_duration metric (#38180)
• Use importlib_metadata with compat to Python 3.10/3.12 stdlib (#38366)
• Refactored __new__ magic method of BaseOperatorMeta to avoid bad mixing classic and decorated operators (#37937)
• Use sys.version_info for determine Python Major.Minor (#38372)
• Add missing deprecated Fab auth manager (#38376)
• Remove unused loop variable from airflow package (#38308)
• Adding max consecutive failed dag runs info in UI (#38229)
• Bump minimum version of blinker add where it requires (#38140)
• Bump follow-redirects from 1.15.4 to 1.15.6 in /airflow/www (#38156)
• Bump Cryptography to > 39.0.0 (#38112)
• Add Python 3.12 support (#36755, #38025, #36595)
• Avoid use of assert outside of the tests (#37718)
• Update ObjectStoragePath for universal_pathlib>=v0.2.2 (#37930)
• Resolve G004: Logging statement uses f-string (#37873)
• Update build and install dependencies. (#37910)
• Bump sanitize-html from 2.11.0 to 2.12.1 in /airflow/www (#37833)
• Update to latest installer versions. (#37754)
• Deprecate smtp configs in airflow settings / local_settings (#37711)
• Deprecate PY* constants into the airflow module (#37575)
• Remove usage of deprecated flask._request_ctx_stack (#37522)
• Remove redundant login attribute in airflow.__init__.py (#37565)
• Upgrade to FAB 4.3.11 (#37233)
• Remove SCHEDULED_DEPS which is no longer used anywhere since 2.0.0 (#37140)
• Replace datetime.datetime.utcnow by airflow.utils.timezone.utcnow in core (#35448)
• Bump aiohttp min version to avoid CVE-2024-23829 and CVE-2024-23334 (#37110)
• Move config related to FAB auth manager to FAB provider (#36232)
• Remove MSSQL support form Airflow core (#36514)
• Remove is_authorized_cluster_activity from auth manager (#36175)
• Create FAB provider and move FAB auth manager in it (#35926)

Doc Only Changes

• Improve timetable documentation (#38505)
• Reorder OpenAPI Spec tags alphabetically (#38717)
• Update UI screenshots in the documentation (#38680, #38403, #38438, #38435)
• Remove section as it's no longer true with dataset expressions PR (#38370)
• Refactor DatasetOrTimeSchedule timetable docs (#37771)
• Migrate executor docs to respective providers (#37728)
• Add directive to render a list of URI schemes (#37700)
• Add doc page with providers deprecations (#37075)
• Add a cross reference to security policy (#37004)
• Improve AIRFLOW__WEBSERVER__BASE_URL docs (#37003)
• Update faq.rst with (hopefully) clearer description of start_date (#36846)
• Update public interface doc re operators (#36767)
• Add exception to templates ref list (#36656)
• Add auth manager interface as public interface (#36312)
• Reference fab provider documentation in Airflow documentation (#36310)
• Create auth manager documentation (#36211)
• Update permission docs (#36120)
• Docstring improvement to _covers_every_hour (#36081)
• Add note that task instance, dag and lifecycle listeners are non-experimental (#36376)

Significant Changes

• Raw HTML code in DAG docs and DAG params descriptions is disabled by default

  To ensure that no malicious javascript can be injected with DAG descriptions or trigger UI forms by DAG authors a new parameter webserver.allow_raw_html_descriptions was added with default value of False. If you trust your DAG authors code and want to allow using raw HTML in DAG descriptions and params, you can restore the previous behavior by setting the configuration value to True.

  To ensure Airflow is secure by default, the raw HTML support in trigger UI has been super-seeded by markdown support via the description_md attribute. If you have been using description_html please migrate to description_md. The custom_html_form is now deprecated. (#35460)

New Features

• AIP-58: Add Airflow ObjectStore (AFS) (AIP-58)
• Add XCom tab to Grid (#35719)
• Add "literal" wrapper to disable field templating (#35017)
• Add task context logging feature to allow forwarding messages to task logs (#32646, #32693, #35857)
• Add Listener hooks for Datasets (#34418, #36247)
• Allow override of navbar text color (#35505)
• Add lightweight serialization for deltalake tables (#35462)
• Add support for serialization of iceberg tables (#35456)
• prev_end_date_success method access (#34528)
• Add task parameter to set custom logger name (#34964)
• Add pyspark decorator (#35247)
• Add trigger as a valid option for the db clean command (#34908)
• Add decorators for external and venv python branching operators (#35043)
• Allow PythonVenvOperator using other index url (#33017)
• Add Python Virtualenv Operator Caching (#33355)
• Introduce a generic export for containerized executor logging (#34903)
• Add ability to clear downstream tis in List Task Instances view (#34529)
• Attribute clear_number to track DAG run being cleared (#34126)
• Add BranchPythonVirtualenvOperator (#33356)
• Allow PythonVenvOperator using other index url (#33017)
• Add CLI notification commands to providers (#33116)
• Use dropdown instead of buttons when there are more than 10 retries in log tab (#36025)

Improvements

• Add multiselect to run state in grid view (#35403)
• Fix warning message in Connection.get_hook in case of ImportError (#36005)
• Add processor_subdir to import_error table to handle multiple dag processors (#35956)
• Consolidate the call of change_state to fail or success in the core executors (#35901)
• Relax mandatory requirement for start_date when schedule=None (#35356)
• Use ExitStack to manage mutation of secrets_backend_list in dag.test (#34620)
• improved visibility of tasks in ActionModal for taskinstance (#35810)
• Create directories based on AIRFLOW_CONFIG path (#35818)
• Implements JSON-string connection representation generator (#35723)
• Move BaseOperatorLink into the separate module (#35032)
• Set mark_end_on_close after set_context (#35761)
• Move external logs links to top of react logs page (#35668)
• Change terminal mode to cbreak in execute_interactive and handle SIGINT (#35602)
• Make raw HTML descriptions configurable (#35460)
• Allow email field to be templated (#35546)
• Hide logical date and run id in trigger UI form (#35284)
• Improved instructions for adding dependencies in TaskFlow (#35406)
• Add optional exit code to list import errors (#35378)
• Limit query result on DB rather than client in synchronize_log_template function (#35366)
• Allow description to be passed in when using variables CLI (#34791)
• Allow optional defaults in required fields with manual triggered dags (#31301)
• Permitting airflow kerberos to run in different modes (#35146)
• Refactor commands to unify daemon context handling (#34945)
• Add extra fields to plugins endpoint (#34913)
• Add description to pools view (#34862)
• Move cli's Connection export and Variable export command print logic to a separate function (#34647)
• Extract and reuse get_kerberos_principle func from get_kerberos_principle (#34936)
• Change type annotation for BaseOperatorLink.operators (#35003)
• Optimise and migrate to SA2-compatible syntax for TaskReschedule (#33720)
• Consolidate the permissions name in SlaMissModelView (#34949)
• Add debug log saying what's being run to EventScheduler (#34808)
• Increase log reader stream loop sleep duration to 1 second (#34789)
• Resolve pydantic deprecation warnings re update_forward_refs (#34657)
• Unify mapped task group lookup logic (#34637)
• Allow filtering event logs by attributes (#34417)
• Make connection login and password TEXT (#32815)
• Ban import Dataset from airflow package in codebase (#34610)
• Use airflow.datasets.Dataset in examples and tests (#34605)
• Enhance task status visibility (#34486)
• Simplify DAG trigger UI (#34567)
• Ban import AirflowException from airflow (#34512)
• Add descriptions for airflow resource config parameters (#34438)
• Simplify trigger name expression (#34356)
• Move definition of Pod*Exceptions to pod_generator (#34346)
• Add deferred tasks to the cluster_activity view Pools Slots (#34275)
• heartbeat failure log message fix (#34160)
• Rename variables for dag runs (#34049)
• Clarify new_state in OpenAPI spec (#34056)
• Remove version top-level element from docker compose files (#33831)
• Remove generic trigger cancelled error log (#33874)
• Use NOT EXISTS subquery instead of tuple_not_in_condition (#33527)
• Allow context key args to not provide a default (#33430)
• Order triggers by - TI priority_weight when assign unassigned triggers (#32318)
• Add metric triggerer_heartbeat (#33320)
• Allow airflow variables export to print to stdout (#33279)
• Workaround failing deadlock when running backfill (#32991)
• add dag_run_ids and task_ids filter for the batch task instance API endpoint (#32705)
• Configurable health check threshold for triggerer (#33089)
• Rework provider manager to treat Airflow core hooks like other provider hooks (#33051)
• Ensure DAG-level references are filled on unmap (#33083)
• Affix webserver access_denied warning to be configurable (#33022)
• Add support for arrays of different data types in the Trigger Form UI (#32734)
• Add a mechanism to warn if executors override existing CLI commands (#33423)

Bug Fixes

• Account for change in UTC offset when calculating next schedule (#35887)
• Add read access to pools for viewer role (#35352)
• Fix gantt chart queued duration when queued_dttm is greater than start_date for deferred tasks (#35984)
• Avoid crushing container when directory is not found on rm (#36050)
• Update reset_user_sessions to work from either CLI or web (#36056)
• Fix UI Grid error when DAG has been removed. (#36028)
• Change Trigger UI to use HTTP POST in web ui (#36026)
• Fix airflow db shell needing an extra key press to exit (#35982)
• Change dag grid overscroll behaviour to auto (#35717)
• Run triggers inline with dag test (#34642)
• Add borderWidthRight to grid for Firefox scrollbar (#35346)
• Fix for infinite recursion due to secrets_masker (#35048)
• Fix write processor_subdir in serialized_dag table (#35661)
• Reload configuration for standalone dag file processor (#35725)
• Long custom operator name overflows in graph view (#35382)
• Add try_number to extra links query (#35317)
• Prevent assignment of non JSON serializable values to DagRun.conf dict (#35096)
• Numeric values in DAG details are incorrectly rendered as timestamps (#35538)
• Fix Scheduler and triggerer crashes in daemon mode when statsd metrics are enabled (#35181)
• Infinite UI redirection loop after deactivating an active user (#35486)
• Bug fix fetch_callback of Partial Subset DAG (#35256)
• Fix DagRun data interval for DeltaDataIntervalTimetable (#35391)
• Fix query in get_dag_by_pickle util function (#35339)
• Fix TriggerDagRunOperator failing to trigger subsequent runs when reset_dag_run=True (#35429)
• Fix weight_rule property type in mappedoperator (#35257)
• Bugfix/prevent concurrency with cached venv (#35258)
• Fix dag serialization (#34042)
• Fix py/url-redirection by replacing request.referrer by get_redirect() (#34237)
• Fix updating variables during variable imports (#33932)
• Use Literal from airflow.typing_compat in Airflow core (#33821)
• Always use Literal from typing_extensions (#33794)

Miscellaneous

• Change default MySQL client to MariaDB (#36243)
• Mark daskexecutor provider as removed (#35965)
• Bump FAB to 4.3.10 (#35991)
• Mark daskexecutor provider as removed (#35965)
• Rename Connection.to_json_dict to Connection.to_dict (#35894)
• Upgrade to Pydantic v2 (#35551)
• Bump moto version to >= 4.2.9 (#35687)
• Use pyarrow-hotfix to mitigate CVE-2023-47248 (#35650)
• Bump axios from 0.26.0 to 1.6.0 in /airflow/www/ (#35624)
• Make docker decorator's type annotation consistent with operator (#35568)
• Add default to navbar_text_color and rm condition in style (#35553)
• Avoid initiating session twice in dag_next_execution (#35539)
• Work around typing issue in examples and providers (#35494)
• Enable TCH004 and TCH005 rules (#35475)
• Humanize log output about retrieved DAG(s) (#35338)
• Switch from Black to Ruff formatter (#35287)
• Upgrade to Flask Application Builder 4.3.9 (#35085)
• D401 Support (#34932, #34933)
• Use requires_access to check read permission on dag instead of checking it explicitly (#34940)
• Deprecate lazy import AirflowException from airflow (#34541)
• View util refactoring on mapped stuff use cases (#34638)
• Bump postcss from 8.4.25 to 8.4.31 in /airflow/www (#34770)
• Refactor Sqlalchemy queries to 2.0 style (#34763, #34665, #32883, #35120)
• Change to lazy loading of io in pandas serializer (#34684)
• Use airflow.models.dag.DAG in examples (#34617)
• Use airflow.exceptions.AirflowException in core (#34510)
• Check that dag_ids passed in request are consistent (#34366)
• Refactors to make code better (#34278, #34113, #34110, #33838, #34260, #34409, #34377, #34350)
• Suspend qubole provider (#33889)
• Generate Python API docs for Google ADS (#33814)
• Improve importing in modules (#33812, #33811, #33810, #33806, #33807, #33805, #33804, #33803, #33801, #33799, #33800, #33797, #33798, #34406, #33808)
• Upgrade Elasticsearch to 8 (#33135)

Doc Only Changes

• Add support for tabs (and other UX components) to docs (#36041)
• Replace architecture diagram of Airflow with diagrams-generated one (#36035)
• Add the section describing the security model of DAG Author capabilities (#36022)
• Enhance docs for zombie tasks (#35825)
• Reflect drop/add support of DB Backends versions in documentation (#35785)
• More detail on mandatory task arguments (#35740)
• Indicate usage of the re2 regex engine in the .airflowignore documentation. (#35663)
• Update best-practices.rst (#35692)
• Update dag-run.rst to mention Airflow's support for extended cron syntax through croniter (#35342)
• Update webserver.rst to include information of supported OAuth2 providers (#35237)
• Add back dag_run to docs (#35142)
• Fix rst code block format (#34708)
• Add typing to concrete taskflow examples (#33417)
• Add concrete examples for accessing context variables from TaskFlow tasks (#33296)
• Fix links in security docs (#33329)

Significant Changes

No significant changes.

Bug Fixes

• Fix pre-mature evaluation of tasks in mapped task group (#34337)
• Add TriggerRule missing value in rest API (#35194)
• Fix Scheduler crash looping when dagrun creation fails (#35135)
• Fix test connection with codemirror and extra (#35122)
• Fix usage of cron-descriptor since BC in v1.3.0 (#34836)
• Fix get_plugin_info for class based listeners. (#35022)
• Some improvements/fixes for dag_run and task_instance endpoints (#34942)
• Fix the dags count filter in webserver home page (#34944)
• Return only the TIs of the readable dags when ~ is provided as a dag_id (#34939)
• Fix triggerer thread crash in daemon mode (#34931)
• Fix wrong plugin schema (#34858)
• Use DAG timezone in TimeSensorAsync (#33406)
• Mark tasks with all_skipped trigger rule as skipped if any task is in upstream_failed state (#34392)
• Add read only validation to read only fields (#33413)

Misc/Internal

• Improve testing harness to separate DB and non-DB tests (#35160, #35333)
• Add pytest db_test markers to our tests (#35264)
• Add pip caching for faster build (#35026)
• Upper bound pendulum requirement to <3.0 (#35336)
• Limit sentry_sdk to 1.33.0 (#35298)
• Fix subtle bug in mocking processor_agent in our tests (#35221)
• Bump @babel/traverse from 7.16.0 to 7.23.2 in /airflow/www (#34988)
• Bump undici from 5.19.1 to 5.26.3 in /airflow/www (#34971)
• Remove unused set from SchedulerJobRunner (#34810)
• Remove warning about max_tis per query > parallelism (#34742)
• Improve modules import in Airflow core by moving some of them into a type-checking block (#33755)
• Fix tests to respond to Python 3.12 handling of utcnow in sentry-sdk (#34946)
• Add connexion<3.0 upper bound (#35218)
• Limit Airflow to < 3.12 (#35123)
• update moto version (#34938)
• Limit WTForms to below 3.1.0 (#34943)

Doc Only Changes

• Fix variables substitution in Airflow Documentation (#34462)
• Added example for defaults in conn.extras (#35165)
• Update datasets.rst issue with running example code (#35035)
• Remove mysql-connector-python from recommended MySQL driver (#34287)
• Fix syntax error in task dependency set_downstream example (#35075)
• Update documentation to enable test connection (#34905)
• Update docs errors.rst - Mention sentry "transport" configuration option (#34912)
• Update dags.rst to put SubDag deprecation note right after the SubDag section heading (#34925)
• Add info on getting variables and config in custom secrets backend (#34834)
• Document BaseExecutor interface in more detail to help users in writing custom executors (#34324)
• Fix broken link to airflow_local_settings.py template (#34826)
• Fixes python_callable function assignment context kwargs example in params.rst (#34759)
• Add missing multiple_outputs=True param in the TaskFlow example (#34812)
• Remove extraneous '>' in provider section name (#34813)
• Fix imports in extra link documentation (#34547)

Significant Changes

No significant changes

Bug Fixes

• Check if the lower of provided values are sensitives in config endpoint (#34712)
• Add support for ZoneInfo and generic UTC to fix datetime serialization (#34683, #34804)
• Fix AttributeError: 'Select' object has no attribute 'count' during the airflow db migrate command (#34348)
• Make dry run optional for patch task instance (#34568)
• Fix non deterministic datetime deserialization (#34492)
• Use iterative loop to look for mapped parent (#34622)
• Fix is_parent_mapped value by checking if any of the parent taskgroup is mapped (#34587)
• Avoid top-level airflow import to avoid circular dependency (#34586)
• Add more exemptions to lengthy metric list (#34531)
• Fix dag warning endpoint permissions (#34355)
• Fix task instance access issue in the batch endpoint (#34315)
• Correcting wrong time showing in grid view (#34179)
• Fix www cluster_activity view not loading due to standaloneDagProcessor templating (#34274)
• Set loglevel=DEBUG in 'Not syncing DAG-level permissions' (#34268)
• Make param validation consistent for DAG validation and triggering (#34248)
• Ensure details panel is shown when any tab is selected (#34136)
• Fix issues related to access_control={} (#34114)
• Fix not found ab_user table in the CLI session (#34120)
• Fix FAB-related logging format interpolation (#34139)
• Fix query bug in next_run_datasets_summary endpoint (#34143)
• Fix for TaskGroup toggles for duplicated labels (#34072)
• Fix the required permissions to clear a TI from the UI (#34123)
• Reuse _run_task_session in mapped render_template_fields (#33309)
• Fix scheduler logic to plan new dag runs by ignoring manual runs (#34027)
• Add missing audit logs for Flask actions add, edit and delete (#34090)
• Hide Irrelevant Dag Processor from Cluster Activity Page (#33611)
• Remove infinite animation for pinwheel, spin for 1.5s (#34020)
• Restore rendering of provider configuration with version_added (#34011)

Doc Only Changes

• Clarify audit log permissions (#34815)
• Add explanation for Audit log users (#34814)
• Import AUTH_REMOTE_USER from FAB in WSGI middleware example (#34721)
• Add information about drop support MsSQL as DB Backend in the future (#34375)
• Document how to use the system's timezone database (#34667)
• Clarify what landing time means in doc (#34608)
• Fix screenshot in dynamic task mapping docs (#34566)
• Fix class reference in Public Interface documentation (#34454)
• Clarify var.value.get and var.json.get usage (#34411)
• Schedule default value description (#34291)
• Docs for triggered_dataset_event (#34410)
• Add DagRun events (#34328)
• Provide tabular overview about trigger form param types (#34285)
• Add link to Amazon Provider Configuration in Core documentation (#34305)
• Add "security infrastructure" paragraph to security model (#34301)
• Change links to SQLAlchemy 1.4 (#34288)
• Add SBOM entry in security documentation (#34261)
• Added more example code for XCom push and pull (#34016)
• Add state utils to Public Airflow Interface (#34059)
• Replace markdown style link with rst style link (#33990)
• Fix broken link to the "UPDATING.md" file (#33583)

Misc/Internal

• Update min-sqlalchemy version to account for latest features used (#34293)
• Fix SesssionExemptMixin spelling (#34696)
• Restrict astroid version < 3 (#34658)
• Fail dag test if defer without triggerer (#34619)
• Fix connections exported output (#34640)
• Don't run isort when creating new alembic migrations (#34636)
• Deprecate numeric type python version in PythonVirtualEnvOperator (#34359)
• Refactor os.path.splitext to Path.* (#34352, #33669)
• Replace = by is for type comparison (#33983)
• Refactor integer division (#34180)
• Refactor: Simplify comparisons (#34181)
• Refactor: Simplify string generation (#34118)
• Replace unnecessary dict comprehension with dict() in core (#33858)
• Change "not all" to "any" for ease of readability (#34259)
• Replace assert by if...raise in code (#34250, #34249)
• Move default timezone to except block (#34245)
• Combine similar if logic in core (#33988)
• Refactor: Consolidate import and usage of random (#34108)
• Consolidate importing of os.path.* (#34060)
• Replace sequence concatenation by unpacking in Airflow core (#33934)
• Refactor unneeded 'continue' jumps around the repo (#33849, #33845, #33846, #33848, #33839, #33844, #33836, #33842)
• Remove [project] section from pyproject.toml (#34014)
• Move the try outside the loop when this is possible in Airflow core (#33975)
• Replace loop by any when looking for a positive value in core (#33985)
• Do not create lists we don't need (#33519)
• Remove useless string join from core (#33969)
• Add TCH001 and TCH002 rules to pre-commit to detect and move type checking modules (#33865)
• Add cancel_trigger_ids to to_cancel dequeue in batch (#33944)
• Avoid creating unnecessary list when parsing stats datadog tags (#33943)
• Replace dict.items by dict.values when key is not used in core (#33940)
• Replace lambdas with comprehensions (#33745)
• Improve modules import in Airflow core by some of them into a type-checking block (#33755)
• Refactor: remove unused state - SHUTDOWN (#33746, #34063, #33893)
• Refactor: Use in-place .sort() (#33743)
• Use literal dict instead of calling dict() in Airflow core (#33762)
• remove unnecessary map and rewrite it using list in Airflow core (#33764)
• Replace lambda by a def method in Airflow core (#33758)
• Replace type func by isinstance in fab_security manager (#33760)
• Replace single quotes by double quotes in all Airflow modules (#33766)
• Merge multiple isinstance calls for the same object in a single call (#33767)
• Use a single statement with multiple contexts instead of nested statements in core (#33769)
• Refactor: Use f-strings (#33734, #33455)
• Refactor: Use random.choices (#33631)
• Use str.splitlines() to split lines (#33592)
• Refactor: Remove useless str() calls (#33629)
• Refactor: Improve detection of duplicates and list sorting (#33675)
• Simplify conditions on len() (#33454)

Significant Changes

CronTriggerTimetable is now less aggressive when trying to skip a run (#33404)

When setting catchup=False, CronTriggerTimetable no longer skips a run if the scheduler does not query the timetable immediately after the previous run has been triggered.

This should not affect scheduling in most cases, but can change the behaviour if a DAG is paused-unpaused to manually skip a run. Previously, the timetable (with catchup=False) would only start a run after a DAG is unpaused, but with this change, the scheduler would try to look at little bit back to schedule the previous run that covers a part of the period when the DAG was paused. This means you will need to keep a DAG paused longer (namely, for the entire cron period to pass) to really skip a run.

Note that this is also the behaviour exhibited by various other cron-based scheduling tools, such as anacron.

conf.set() becomes case insensitive to match conf.get() behavior (#33452)

Also, conf.get() will now break if used with non-string parameters.

conf.set(section, key, value) used to be case sensitive, i.e. conf.set("SECTION", "KEY", value) and conf.set("section", "key", value) were stored as two distinct configurations. This was inconsistent with the behavior of conf.get(section, key), which was always converting the section and key to lower case.

As a result, configuration options set with upper case characters in the section or key were unreachable. That's why we are now converting section and key to lower case in conf.set too.

We also changed a bit the behavior of conf.get(). It used to allow objects that are not strings in the section or key. Doing this will now result in an exception. For instance, conf.get("section", 123) needs to be replaced with conf.get("section", "123").

Bug Fixes

• Ensure that tasks wait for running indirect setup (#33903)
• Respect "soft_fail" for core async sensors (#33403)
• Differentiate 0 and unset as a default param values (#33965)
• Raise 404 from Variable PATCH API if variable is not found (#33885)
• Fix MappedTaskGroup tasks not respecting upstream dependency (#33732)
• Add limit 1 if required first value from query result (#33672)
• Fix UI DAG counts including deleted DAGs (#33778)
• Fix cleaning zombie RESTARTING tasks (#33706)
• SECURITY_MANAGER_CLASS should be a reference to class, not a string (#33690)
• Add back get_url_for_login in security manager (#33660)
• Fix 2.7.0 db migration job errors (#33652)
• Set context inside templates (#33645)
• Treat dag-defined access_control as authoritative if defined (#33632)
• Bind engine before attempting to drop archive tables (#33622)
• Add a fallback in case no first name and last name are set (#33617)
• Sort data before groupby in TIS duration calculation (#33535)
• Stop adding values to rendered templates UI when there is no dagrun (#33516)
• Set strict to True when parsing dates in webserver views (#33512)
• Use dialect.name in custom SA types (#33503)
• Do not return ongoing dagrun when a end_date is less than utcnow (#33488)
• Fix a bug in formatDuration method (#33486)
• Make conf.set case insensitive (#33452)
• Allow timetable to slightly miss catchup cutoff (#33404)
• Respect soft_fail argument when poke is called (#33401)
• Create a new method used to resume the task in order to implement specific logic for operators (#33424)
• Fix DagFileProcessor interfering with dags outside its processor_subdir (#33357)
• Remove the unnecessary <br> text in Provider's view (#33326)
• Respect soft_fail argument when ExternalTaskSensor runs in deferrable mode (#33196)
• Fix handling of default value and serialization of Param class (#33141)
• Check if the dynamically-added index is in the table schema before adding (#32731)
• Fix rendering the mapped parameters when using expand_kwargs method (#32272)
• Fix dependencies for celery and opentelemetry for Python 3.8 (#33579)

Misc/Internal

• Bring back Pydantic 1 compatibility (#34081, #33998)
• Use a trimmed version of README.md for PyPI (#33637)
• Upgrade to Pydantic 2 (#33956)
• Reorganize devel_only extra in Airflow's setup.py (#33907)
• Bumping FAB to 4.3.4 in order to fix issues with filters (#33931)
• Add minimum requirement for sqlalchemy to 1.4.24 (#33892)
• Update version_added field for configs in config file (#33509)
• Replace OrderedDict with plain dict (#33508)
• Consolidate import and usage of itertools (#33479)
• Static check fixes (#33462)
• Import utc from datetime and normalize its import (#33450)
• D401 Support (#33352, #33339, #33337, #33336, #33335, #33333, #33338)
• Fix some missing type hints (#33334)
• D205 Support - Stragglers (#33301, #33298, #33297)
• Refactor: Simplify code (#33160, #33270, #33268, #33267, #33266, #33264, #33292, #33453, #33476, #33567, #33568, #33480, #33753, #33520, #33623)
• Fix Pydantic warning about orm_mode rename (#33220)
• Add MySQL 8.1 to supported versions. (#33576)
• Remove Pydantic limitation for version < 2 (#33507)

Doc only changes

• Add documentation explaining template_ext (and how to override it) (#33735)
• Explain how users can check if python code is top-level (#34006)
• Clarify that DAG authors can also run code in DAG File Processor (#33920)
• Fix broken link in Modules Management page (#33499)
• Fix secrets backend docs (#33471)
• Fix config description for base_log_folder (#33388)

Significant Changes

Remove Python 3.7 support (#30963)

As of now, Python 3.7 is no longer supported by the Python community. Therefore, to use Airflow 2.7.0, you must ensure your Python version is either 3.8, 3.9, 3.10, or 3.11.

Old Graph View is removed (#32958)

The old Graph View is removed. The new Graph View is the default view now.

The trigger UI form is skipped in web UI if no parameters are defined in a DAG (#33351)

If you are using dag_run.conf dictionary and web UI JSON entry to run your DAG you should either:

• Add params to your DAG <https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/params.html#use-params-to-provide-a-trigger-ui-form>_
• Enable the new configuration show_trigger_form_if_no_params to bring back old behaviour

The "db init", "db upgrade" commands and "[database] load_default_connections" configuration options are deprecated (#33136).

Instead, you should use "airflow db migrate" command to create or upgrade database. This command will not create default connections. In order to create default connections you need to run "airflow connections create-default-connections" explicitly, after running "airflow db migrate".

In case of SMTP SSL connection, the context now uses the "default" context (#33070)

The "default" context is Python's default_ssl_contest instead of previously used "none". The default_ssl_context provides a balance between security and compatibility but in some cases, when certificates are old, self-signed or misconfigured, it might not work. This can be configured by setting "ssl_context" in "email" configuration of Airflow.

Setting it to "none" brings back the "none" setting that was used in Airflow 2.6 and before, but it is not recommended due to security reasons ad this setting disables validation of certificates and allows MITM attacks.

Disable default allowing the testing of connections in UI, API and CLI(#32052)

For security reasons, the test connection functionality is disabled by default across Airflow UI, API and CLI. The availability of the functionality can be controlled by the test_connection flag in the core section of the Airflow configuration (airflow.cfg). It can also be controlled by the environment variable AIRFLOW__CORE__TEST_CONNECTION.

The following values are accepted for this config param:

1. Disabled: Disables the test connection functionality and disables the Test Connection button in the UI.

This is also the default value set in the Airflow configuration. 2. Enabled: Enables the test connection functionality and activates the Test Connection button in the UI.

1. Hidden: Disables the test connection functionality and hides the Test Connection button in UI.

For more information on capabilities of users, see the documentation: https://airflow.apache.org/docs/apache-airflow/stable/security/security_model.html#capabilities-of-authenticated-ui-users It is strongly advised to not enable the feature until you make sure that only highly trusted UI/API users have "edit connection" permissions.

The xcomEntries API disables support for the deserialize flag by default (#32176)

For security reasons, the /dags/*/dagRuns/*/taskInstances/*/xcomEntries/* API endpoint now disables the deserialize option to deserialize arbitrary XCom values in the webserver. For backward compatibility, server admins may set the [api] enable_xcom_deserialize_support config to True to enable the flag and restore backward compatibility.

However, it is strongly advised to not enable the feature, and perform deserialization at the client side instead.

Change of the default Celery application name (#32526)

Default name of the Celery application changed from airflow.executors.celery_executor to airflow.providers.celery.executors.celery_executor.

You should change both your configuration and Health check command to use the new name:

• in configuration (celery_app_name configuration in celery section) use airflow.providers.celery.executors.celery_executor
• in your Health check command use airflow.providers.celery.executors.celery_executor.app

The default value for scheduler.max_tis_per_query is changed from 512 to 16 (#32572)

This change is expected to make the Scheduler more responsive.

scheduler.max_tis_per_query needs to be lower than core.parallelism. If both were left to their default value previously, the effective default value of scheduler.max_tis_per_query was 32 (because it was capped at core.parallelism).

To keep the behavior as close as possible to the old config, one can set scheduler.max_tis_per_query = 0, in which case it'll always use the value of core.parallelism.

Some executors have been moved to corresponding providers (#32767)

In order to use the executors, you need to install the providers:

• for Celery executors you need to install apache-airflow-providers-celery package >= 3.3.0
• for Kubernetes executors you need to install apache-airflow-providers-cncf-kubernetes package >= 7.4.0
• For Dask executors you need to install apache-airflow-providers-daskexecutor package in any version

You can achieve it also by installing airflow with [celery], [cncf.kubernetes], [daskexecutor] extras respectively.

Users who base their images on the apache/airflow reference image (not slim) should be unaffected - the base reference image comes with all the three providers installed.

Improvement Changes

PostgreSQL only improvement: Added index on taskinstance table (#30762)

This index seems to have great positive effect in a setup with tens of millions such rows.

New Features

• Add OpenTelemetry to Airflow AIP-49
• Trigger Button - Implement Part 2 of AIP-50 (#31583)
• Removing Executor Coupling from Core Airflow AIP-51
• Automatic setup and teardown tasks AIP-52
• OpenLineage in Airflow AIP-53
• Experimental: Add a cache to Variable and Connection when called at dag parsing time (#30259)
• Enable pools to consider deferred tasks (#32709)
• Allows to choose SSL context for SMTP connection (#33070)
• New gantt tab (#31806)
• Load plugins from providers (#32692)
• Add BranchExternalPythonOperator (#32787, #33360)
• Add option for storing configuration description in providers (#32629)
• Introduce Heartbeat Parameter to Allow Per-LocalTaskJob Configuration (#32313)
• Add Executors discovery and documentation (#32532)
• Add JobState for job state constants (#32549)
• Add config to disable the 'deserialize' XCom API flag (#32176)
• Show task instance in web UI by custom operator name (#31852)
• Add default_deferrable config (#31712)
• Introducing AirflowClusterPolicySkipDag exception (#32013)
• Use reactflow for datasets graph (#31775)
• Add an option to load the dags from db for command tasks run (#32038)
• Add version of chain which doesn't require matched lists (#31927)
• Use operator_name instead of task_type in UI (#31662)
• Add --retry and --retry-delay to airflow db check (#31836)
• Allow skipped task state task_instance_schema.py (#31421)
• Add a new config for celery result_backend engine options (#30426)
• UI Add Cluster Activity Page (#31123, #32446)
• Adding keyboard shortcuts to common actions (#30950)
• Adding more information to kubernetes executor logs (#29929)
• Add support for configuring custom alembic file (#31415)
• Add running and failed status tab for DAGs on the UI (#30429)
• Add multi-select, proposals and labels for trigger form (#31441)
• Making webserver config customizable (#29926)
• Render DAGCode in the Grid View as a tab (#31113)
• Add rest endpoint to get option of configuration (#31056)
• Add section query param in get config rest API (#30936)
• Create metrics to track Scheduled->Queued->Running task state transition times (#30612)
• Mark Task Groups as Success/Failure (#30478)
• Add CLI command to list the provider trigger info (#30822)
• Add Fail Fast feature for DAGs (#29406)

Improvements

• Improve graph nesting logic (#33421)
• Configurable health check threshold for triggerer (#33089, #33084)
• add dag_run_ids and task_ids filter for the batch task instance API endpoint (#32705)
• Ensure DAG-level references are filled on unmap (#33083)
• Add support for arrays of different data types in the Trigger Form UI (#32734)
• Always show gantt and code tabs (#33029)
• Move listener success hook to after SQLAlchemy commit (#32988)
• Rename db upgrade to db migrate and add connections create-default-connections (#32810, #33136)
• Remove old gantt chart and redirect to grid views gantt tab (#32908)
• Adjust graph zoom based on selected task (#32792)
• Call listener on_task_instance_running after rendering templates (#32716)
• Display execution_date in graph view task instance tooltip. (#32527)
• Allow configuration to be contributed by providers (#32604, #32755, #32812)
• Reduce default for max TIs per query, enforce <= parallelism (#32572)
• Store config description in Airflow configuration object (#32669)
• Use isdisjoint instead of not intersection (#32616)
• Speed up calculation of leaves and roots for task groups (#32592)
• Kubernetes Executor Load Time Optimizations (#30727)
• Save DAG parsing time if dag is not schedulable (#30911)
• Updates health check endpoint to include dag_processor status. (#32382)
• Disable default allowing the testing of connections in UI, API and CLI (#32052, #33342)
• Fix config var types under the scheduler section (#32132)
• Allow to sort Grid View alphabetically (#32179)
• Add hostname to triggerer metric [triggers.running] (#32050)
• Improve DAG ORM cleanup code (#30614)
• TriggerDagRunOperator: Add wait_for_completion to template_fields (#31122)
• Open links in new tab that take us away from Airflow UI (#32088)
• Only show code tab when a task is not selected (#31744)
• Add descriptions for celery and dask cert configs (#31822)
• PythonVirtualenvOperator termination log in alert (#31747)
• Migration of all DAG details to existing grid view dag details panel (#31690)
• Add a diagram to help visualize timer metrics (#30650)
• Celery Executor load time optimizations (#31001)
• Update code style for airflow db commands to SQLAlchemy 2.0 style (#31486)
• Mark uses of md5 as "not-used-for-security" in FIPS environments (#31171)
• Add pydantic support to serde (#31565)
• Enable search in note column in DagRun and TaskInstance (#31455)
• Save scheduler execution time by adding new Index idea for dag_run (#30827)
• Save scheduler execution time by caching dags (#30704)
• Support for sorting DAGs by Last Run Date in the web UI (#31234)
• Better typing for Job and JobRunners (#31240)
• Add sorting logic by created_date for fetching triggers (#31151)
• Remove DAGs.can_create on access control doc, adjust test fixture (#30862)
• Split Celery logs into stdout/stderr (#30485)
• Decouple metrics clients and validators into their own modules (#30802)
• Description added for pagination in get_log api (#30729)
• Optimize performance of scheduling mapped tasks (#30372)
• Add sentry transport configuration option (#30419)
• Better message on deserialization error (#30588)

Bug Fixes

• Remove user sessions when resetting password (#33347)
• Gantt chart: Use earliest/oldest ti dates if different than dag run start/end (#33215)
• Fix virtualenv detection for Python virtualenv operator (#33223)
• Correctly log when there are problems trying to chmod airflow.cfg (#33118)
• Pass app context to webserver_config.py (#32759)
• Skip served logs for non-running task try (#32561)
• Fix reload gunicorn workers (#32102)
• Fix future DagRun rarely triggered by race conditions when max_active_runs reached its upper limit. (#31414)
• Fix BaseOperator get_task_instances query (#33054)
• Fix issue with using the various state enum value in logs (#33065)
• Use string concatenation to prepend base URL for log_url (#33063)
• Update graph nodes with operator style attributes (#32822)
• Affix webserver access_denied warning to be configurable (#33022)
• Only load task action modal if user can edit (#32992)
• OpenAPI Spec fix nullable alongside $ref (#32887)
• Make the decorators of PythonOperator sub-classes extend its decorator (#32845)
• Fix check if virtualenv is installed in PythonVirtualenvOperator (#32939)
• Unwrap Proxy before checking __iter__ in is_container() (#32850)
• Override base log folder by using task handler's base_log_folder (#32781)
• Catch arbitrary exception from run_job to prevent zombie scheduler (#32707)
• Fix depends_on_past work for dynamic tasks (#32397)
• Sort extra_links for predictable order in UI. (#32762)
• Fix prefix group false graph (#32764)
• Fix bad delete logic for dagruns (#32684)
• Fix bug in prune_dict where empty dict and list would be removed even in strict mode (#32573)
• Add explicit browsers list and correct rel for blank target links (#32633)
• Handle returned None when multiple_outputs is True (#32625)
• Fix returned value when ShortCircuitOperator condition is falsy and there is not downstream tasks (#32623)
• Fix returned value when ShortCircuitOperator condition is falsy (#32569)
• Fix rendering of dagRunTimeout (#32565)
• Fix permissions on /blocked endpoint (#32571)
• Bugfix, prevent force of unpause on trigger DAG (#32456)
• Fix data interval in cli.dags.trigger command output (#32548)
• Strip whitespaces from airflow connections form (#32292)
• Add timedelta support for applicable arguments of sensors (#32515)
• Fix incorrect default on readonly property in our API (#32510)
• Add xcom map_index as a filter to xcom endpoint (#32453)
• Fix CLI commands when custom timetable is used (#32118)
• Use WebEncoder to encode DagRun.conf in DagRun's list view (#32385)
• Fix logic of the skip_all_except method (#31153)
• Ensure dynamic tasks inside dynamic task group only marks the (#32354)
• Handle the cases that webserver.expose_config is set to non-sensitive-only instead of boolean value (#32261)
• Add retry functionality for handling process termination caused by database network issues (#31998)
• Adapt Notifier for sla_miss_callback (#31887)
• Fix XCOM view (#31807)
• Fix for "Filter dags by tag" flickering on initial load of dags.html (#31578)
• Fix where expanding resizer wouldn't expanse grid view (#31581)
• Fix MappedOperator-BaseOperator attr sync check (#31520)
• Always pass named type_ arg to drop_constraint (#31306)
• Fix bad drop_constraint call in migrations (#31302)
• Resolving problems with redesigned grid view (#31232)
• Support requirepass redis sentinel (#30352)
• Fix webserver crash when calling get /config (#31057)

Misc/Internal

• Modify pathspec version restriction (#33349)
• Refactor: Simplify code in dag_processing (#33161)
• For now limit Pydantic to < 2.0.0 (#33235)
• Refactor: Simplify code in models (#33181)
• Add elasticsearch group to pre-2.7 defaults (#33166)
• Refactor: Simplify dict manipulation in airflow/cli (#33159)
• Remove redundant dict.keys() call (#33158)
• Upgrade ruff to latest 0.0.282 version in pre-commits (#33152)
• Move openlineage configuration to provider (#33124)
• Replace State by TaskInstanceState in Airflow executors (#32627)
• Get rid of Python 2 numeric relics (#33050)
• Remove legacy dag code (#33058)
• Remove legacy task instance modal (#33060)
• Remove old graph view (#32958)
• Move CeleryExecutor to the celery provider (#32526, #32628)
• Move all k8S classes to cncf.kubernetes provider (#32767, #32891)
• Refactor existence-checking SQL to helper (#32790)
• Extract Dask executor to new daskexecutor provider (#32772)
• Remove atlas configuration definition (#32776)
• Add Redis task handler (#31855)
• Move writing configuration for webserver to main (webserver limited) (#32766)
• Improve getting the query count in Airflow API endpoints (#32630)
• Remove click upper bound (#32634)
• Add D400 pydocstyle check - core Airflow only (#31297)
• D205 Support (#31742, #32575, #32213, #32212, #32591, #32449, #32450)
• Bump word-wrap from 1.2.3 to 1.2.4 in /airflow/www (#32680)
• Strong-type all single-state enum values (#32537)
• More strong typed state conversion (#32521)
• SQL query improvements in utils/db.py (#32518)
• Bump semver from 6.3.0 to 6.3.1 in /airflow/www (#32506)
• Bump jsonschema version to 4.18.0 (#32445)
• Bump stylelint from 13.13.1 to 15.10.1 in /airflow/www (#32435)
• Bump tough-cookie from 4.0.0 to 4.1.3 in /airflow/www (#32443)
• upgrade flask-appbuilder (#32054)
• Support Pydantic 2 (#32366)
• Limit click until we fix mypy issues (#32413)
• A couple of minor cleanups (#31890)
• Replace State usages with strong-typed enums (#31735)
• Upgrade ruff to 0.272 (#31966)
• Better error message when serializing callable without name (#31778)
• Improve the views module a bit (#31661)
• Remove asynctest (#31664)
• Refactor sqlalchemy queries to 2.0 style (#31569, #31772, #32350, #32339, #32474, #32645)
• Remove Python 3.7 support (#30963)
• Bring back min-airflow-version for preinstalled providers (#31469)
• Docstring improvements (#31375)
• Improve typing in SchedulerJobRunner (#31285)
• Upgrade ruff to 0.0.262 (#30809)
• Upgrade to MyPy 1.2.0 (#30687)

Docs only changes

• Clarify UI user types in security model (#33021)
• Add links to DAGRun / DAG / Task in templates-ref.rst (#33013)
• Add docs of how to test for DAG Import Errors (#32811)
• Clean-up of our new security page (#32951)
• Cleans up Extras reference page (#32954)
• Update Dag trigger API and command docs (#32696)
• Add deprecation info to the Airflow modules and classes docstring (#32635)
• Formatting installation doc to improve readability (#32502)
• Fix triggerer HA doc (#32454)
• Add type annotation to code examples (#32422)
• Document cron and delta timetables (#32392)
• Update index.rst doc to correct grammar (#32315)
• Fixing small typo in python.py (#31474)
• Separate out and clarify policies for providers (#30657)
• Fix docs: add an "apache" prefix to pip install (#30681)

Significant Changes

Clarifications of the external Health Check mechanism and using Job classes (#31277).

In the past SchedulerJob and other *Job classes are known to have been used to perform external health checks for Airflow components. Those are, however, Airflow DB ORM related classes. The DB models and database structure of Airflow are considered as internal implementation detail, following public interface <https://airflow.apache.org/docs/apache-airflow/stable/public-airflow-interface.html>_). Therefore, they should not be used for external health checks. Instead, you should use the airflow jobs check CLI command (introduced in Airflow 2.1) for that purpose.

Bug Fixes

• Fix calculation of health check threshold for SchedulerJob (#31277)
• Fix timestamp parse failure for k8s executor pod tailing (#31175)
• Make sure that DAG processor job row has filled value in job_type column (#31182)
• Fix section name reference for api_client_retry_configuration (#31174)
• Ensure the KPO runs pod mutation hooks correctly (#31173)
• Remove worrying log message about redaction from the OpenLineage plugin (#31149)
• Move interleave_timestamp_parser config to the logging section (#31102)
• Ensure that we check worker for served logs if no local or remote logs found (#31101)
• Fix MappedTaskGroup import in taskinstance file (#31100)
• Format DagBag.dagbag_report() Output (#31095)
• Mask task attribute on task detail view (#31125)
• Fix template error when iterating None value and fix params documentation (#31078)
• Fix apache-hive extra so it installs the correct package (#31068)
• Fix issue with zip files in DAGs folder when pre-importing Airflow modules (#31061)
• Move TaskInstanceKey to a separate file to fix circular import (#31033, #31204)
• Fix deleting DagRuns and TaskInstances that have a note (#30987)
• Fix airflow providers get command output (#30978)
• Fix Pool schema in the OpenAPI spec (#30973)
• Add support for dynamic tasks with template fields that contain pandas.DataFrame (#30943)
• Use the Task Group explicitly passed to 'partial' if any (#30933)
• Fix order_by request in list DAG rest api (#30926)
• Include node height/width in center-on-task logic (#30924)
• Remove print from dag trigger command (#30921)
• Improve task group UI in new graph (#30918)
• Fix mapped states in grid view (#30916)
• Fix problem with displaying graph (#30765)
• Fix backfill KeyError when try_number out of sync (#30653)
• Re-enable clear and setting state in the TaskInstance UI (#30415)
• Prevent DagRun's state and start_date from being reset when clearing a task in a running DagRun (#30125)

Misc/Internal

• Upper bind dask until they solve a side effect in their test suite (#31259)
• Show task instances affected by clearing in a table (#30633)
• Fix missing models in API documentation (#31021)

Doc only changes

• Improve description of the dag_processing.processes metric (#30891)
• Improve Quick Start instructions (#30820)
• Add section about missing task logs to the FAQ (#30717)
• Mount the config directory in docker compose (#30662)
• Update version_added config field for might_contain_dag and metrics_allow_list (#30969)

Significant Changes

Default permissions of file task handler log directories and files has been changed to "owner + group" writeable (#29506).

Default setting handles case where impersonation is needed and both users (airflow and the impersonated user) have the same group set as main group. Previously the default was also other-writeable and the user might choose to use the other-writeable setting if they wish by configuring file_task_handler_new_folder_permissions and file_task_handler_new_file_permissions in logging section.

SLA callbacks no longer add files to the dag processor manager's queue (#30076)

This stops SLA callbacks from keeping the dag processor manager permanently busy. It means reduced CPU, and fixes issues where SLAs stop the system from seeing changes to existing dag files. Additional metrics added to help track queue state.

The cleanup() method in BaseTrigger is now defined as asynchronous (following async/await) pattern (#30152).

This is potentially a breaking change for any custom trigger implementations that override the cleanup() method and uses synchronous code, however using synchronous operations in cleanup was technically wrong, because the method was executed in the main loop of the Triggerer and it was introducing unnecessary delays impacting other triggers. The change is unlikely to affect any existing trigger implementations.

The gauge scheduler.tasks.running no longer exist (#30374)

The gauge has never been working and its value has always been 0. Having an accurate value for this metric is complex so it has been decided that removing this gauge makes more sense than fixing it with no certainty of the correctness of its value.

Consolidate handling of tasks stuck in queued under new task_queued_timeout config (#30375)

Logic for handling tasks stuck in the queued state has been consolidated, and the all configurations responsible for timing out stuck queued tasks have been deprecated and merged into [scheduler] task_queued_timeout. The configurations that have been deprecated are [kubernetes] worker_pods_pending_timeout, [celery] stalled_task_timeout, and [celery] task_adoption_timeout. If any of these configurations are set, the longest timeout will be respected. For example, if [celery] stalled_task_timeout is 1200, and [scheduler] task_queued_timeout is 600, Airflow will set [scheduler] task_queued_timeout to 1200.

Improvement Changes

Display only the running configuration in configurations view (#28892)

The configurations view now only displays the running configuration. Previously, the default configuration was displayed at the top but it was not obvious whether this default configuration was overridden or not. Subsequently, the non-documented endpoint /configuration?raw=true is deprecated and will be removed in Airflow 3.0. The HTTP response now returns an additional Deprecation header. The /config endpoint on the REST API is the standard way to fetch Airflow configuration programmatically.

Explicit skipped states list for ExternalTaskSensor (#29933)

ExternalTaskSensor now has an explicit skipped_states list

Miscellaneous Changes

Handle OverflowError on exponential backoff in next_run_calculation (#28172)

Maximum retry task delay is set to be 24h (86400s) by default. You can change it globally via core.max_task_retry_delay parameter.

Move Hive macros to the provider (#28538)

The Hive Macros (hive.max_partition, hive.closest_ds_partition) are available only when Hive Provider is installed. Please install Hive Provider > 5.1.0 when using those macros.

New Features

• Skip PythonVirtualenvOperator task when it returns a provided exit code (#30690)
• rename skip_exit_code to skip_on_exit_code and allow providing multiple codes (#30692)
• Add skip_on_exit_code also to ExternalPythonOperator (#30738)
• Add max_active_tis_per_dagrun for Dynamic Task Mapping (#29094)
• Add serializer for pandas dataframe (#30390)
• Deferrable TriggerDagRunOperator (#30292)
• Add command to get DAG Details via CLI (#30432)
• Adding ContinuousTimetable and support for @continuous schedule_interval (#29909)
• Allow customized rules to check if a file has dag (#30104)
• Add a new Airflow conf to specify a SSL ca cert for Kubernetes client (#30048)
• Bash sensor has an explicit retry code (#30080)
• Add filter task upstream/downstream to grid view (#29885)
• Add testing a connection via Airflow CLI (#29892)
• Support deleting the local log files when using remote logging (#29772)
• Blocklist to disable specific metric tags or metric names (#29881)
• Add a new graph inside of the grid view (#29413)
• Add database check_migrations config (#29714)
• add output format arg for cli.dags.trigger (#29224)
• Make json and yaml available in templates (#28930)
• Enable tagged metric names for existing Statsd metric publishing events | influxdb-statsd support (#29093)
• Add arg --yes to db export-archived command. (#29485)
• Make the policy functions pluggable (#28558)
• Add airflow db drop-archived command (#29309)
• Enable individual trigger logging (#27758)
• Implement new filtering options in graph view (#29226)
• Add triggers for ExternalTask (#29313)
• Add command to export purged records to CSV files (#29058)
• Add FileTrigger (#29265)
• Emit DataDog statsd metrics with metadata tags (#28961)
• Add some statsd metrics for dataset (#28907)
• Add --overwrite option to connections import CLI command (#28738)
• Add general-purpose "notifier" concept to DAGs (#28569)
• Add a new conf to wait past_deps before skipping a task (#27710)
• Add Flink on K8s Operator (#28512)
• Allow Users to disable SwaggerUI via configuration (#28354)
• Show mapped task groups in graph (#28392)
• Log FileTaskHandler to work with KubernetesExecutor's multi_namespace_mode (#28436)
• Add a new config for adapting masked secrets to make it easier to prevent secret leakage in logs (#28239)
• List specific config section and its values using the cli (#28334)
• KubernetesExecutor multi_namespace_mode can use namespace list to avoid requiring cluster role (#28047)
• Automatically save and allow restore of recent DAG run configs (#27805)
• Added exclude_microseconds to cli (#27640)

Improvements

• Rename most pod_id usage to pod_name in KubernetesExecutor (#29147)
• Update the error message for invalid use of poke-only sensors (#30821)
• Update log level in scheduler critical section edge case (#30694)
• AIP-51 Removing Executor Coupling from Core Airflow (AIP-51 <https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-51+milestone%3A%22Airflow+2.6.0%22>_)
• Add multiple exit code handling in skip logic for BashOperator (#30739)
• Updated app to support configuring the caching hash method for FIPS v2 (#30675)
• Preload airflow imports before dag parsing to save time (#30495)
• Improve task & run actions UX in grid view (#30373)
• Speed up TaskGroups with caching property of group_id (#30284)
• Use the engine provided in the session (#29804)
• Type related import optimization for Executors (#30361)
• Add more type hints to the code base (#30503)
• Always use self.appbuilder.get_session in security managers (#30233)
• Update SQLAlchemy select() to new style (#30515)
• Refactor out xcom constants from models (#30180)
• Add exception class name to DAG-parsing error message (#30105)
• Rename statsd_allow_list and statsd_block_list to metrics_*_list (#30174)
• Improve serialization of tuples and sets (#29019)
• Make cleanup method in trigger an async one (#30152)
• Lazy load serialization modules (#30094)
• SLA callbacks no longer add files to the dag_processing manager queue (#30076)
• Add task.trigger rule to grid_data (#30130)
• Speed up log template sync by avoiding ORM (#30119)
• Separate cli_parser.py into two modules (#29962)
• Explicit skipped states list for ExternalTaskSensor (#29933)
• Add task state hover highlighting to new graph (#30100)
• Store grid tabs in url params (#29904)
• Use custom Connexion resolver to load lazily (#29992)
• Delay Kubernetes import in secret masker (#29993)
• Delay ConnectionModelView init until it's accessed (#29946)
• Scheduler, make stale DAG deactivation threshold configurable instead of using dag processing timeout (#29446)
• Improve grid view height calculations (#29563)
• Avoid importing executor during conf validation (#29569)
• Make permissions for FileTaskHandler group-writeable and configurable (#29506)
• Add colors in help outputs of Airflow CLI commands #28789 (#29116)
• Add a param for get_dags endpoint to list only unpaused dags (#28713)
• Expose updated_at filter for dag run and task instance endpoints (#28636)
• Increase length of user identifier columns (#29061)
• Update gantt chart UI to display queued state of tasks (#28686)
• Add index on log.dttm (#28944)
• Display only the running configuration in configurations view (#28892)
• Cap dropdown menu size dynamically (#28736)
• Added JSON linter to connection edit / add UI for field extra. On connection edit screen, existing extra data will be displayed indented (#28583)
• Use labels instead of pod name for pod log read in k8s exec (#28546)
• Use time not tries for queued & running re-checks. (#28586)
• CustomTTYColoredFormatter should inherit TimezoneAware formatter (#28439)
• Improve past depends handling in Airflow CLI tasks.run command (#28113)
• Support using a list of callbacks in on_*_callback/sla_miss_callbacks (#28469)
• Better table name validation for db clean (#28246)
• Use object instead of array in config.yml for config template (#28417)
• Add markdown rendering for task notes. (#28245)
• Show mapped task groups in grid view (#28208)
• Add renamed and previous_name in config sections (#28324)
• Speed up most Users/Role CLI commands (#28259)
• Speed up Airflow role list command (#28244)
• Refactor serialization (#28067, #30819, #30823)
• Allow longer pod names for k8s executor / KPO (#27736)
• Updates health check endpoint to include triggerer status (#27755)

Bug Fixes

• Fix static_folder for cli app (#30952)
• Initialize plugins for cli appbuilder (#30934)
• Fix dag file processor heartbeat to run only if necessary (#30899)
• Fix KubernetesExecutor sending state to scheduler (#30872)
• Count mapped upstream only if all are finished (#30641)
• ExternalTaskSensor: add external_task_group_id to template_fields (#30401)
• Improve url detection for task instance details (#30779)
• Use material icons for dag import error banner (#30771)
• Fix misc grid/graph view UI bugs (#30752)
• Add a collapse grid button (#30711)
• Fix d3 dependencies (#30702)
• Simplify logic to resolve tasks stuck in queued despite stalled_task_timeout (#30375)
• When clearing task instances try to get associated DAGs from database (#29065)
• Fix mapped tasks partial arguments when DAG default args are provided (#29913)
• Deactivate DAGs deleted from within zip files (#30608)
• Recover from too old resource version exception by retrieving the latest resource_version (#30425)
• Fix possible race condition when refreshing DAGs (#30392)
• Use custom validator for OpenAPI request body (#30596)
• Fix TriggerDagRunOperator with deferrable parameter (#30406)
• Speed up dag runs deletion (#30330)
• Do not use template literals to construct html elements (#30447)
• Fix deprecation warning in example_sensor_decorator DAG (#30513)
• Avoid logging sensitive information in triggerer job log (#30110)
• Add a new parameter for base sensor to catch the exceptions in poke method (#30293)
• Fix dag run conf encoding with non-JSON serializable values (#28777)
• Added fixes for Airflow to be usable on Windows Dask-Workers (#30249)
• Force DAG last modified time to UTC (#30243)
• Fix EmptySkipOperator in example dag (#30269)
• Make the webserver startup respect update_fab_perms (#30246)
• Ignore error when changing log folder permissions (#30123)
• Disable ordering DagRuns by note (#30043)
• Fix reading logs from finished KubernetesExecutor worker pod (#28817)
• Mask out non-access bits when comparing file modes (#29886)
• Remove Run task action from UI (#29706)
• Fix log tailing issues with legacy log view (#29496)
• Fixes to how DebugExecutor handles sensors (#28528)
• Ensure that pod_mutation_hook is called before logging the pod name (#28534)
• Handle OverflowError on exponential backoff in next_run_calculation (#28172)

Misc/Internal

• Make eager upgrade additional dependencies optional (#30811)
• Upgrade to pip 23.1.1 (#30808)
• Remove protobuf limitation from eager upgrade (#30182)
• Remove protobuf limitation from eager upgrade (#30182)
• Deprecate skip_exit_code in BashOperator (#30734)
• Remove gauge scheduler.tasks.running (#30374)
• Bump json5 to 1.0.2 and eslint-plugin-import to 2.27.5 in /airflow/www (#30568)
• Add tests to PythonOperator (#30362)
• Add asgiref as a core dependency (#30527)
• Discovery safe mode toggle comment clarification (#30459)
• Upgrade moment-timezone package to fix Tehran tz (#30455)
• Bump loader-utils from 2.0.0 to 2.0.4 in /airflow/www (#30319)
• Bump babel-loader from 8.1.0 to 9.1.0 in /airflow/www (#30316)
• DagBag: Use dag.fileloc instead of dag.full_filepath in exception message (#30610)
• Change log level of serialization information (#30239)
• Minor DagRun helper method cleanup (#30092)
• Improve type hinting in stats.py (#30024)
• Limit importlib-metadata backport to < 5.0.0 (#29924)
• Align cncf provider file names with AIP-21 (#29905)
• Upgrade FAB to 4.3.0 (#29766)
• Clear ExecutorLoader cache in tests (#29849)
• Lazy load Task Instance logs in UI (#29827)
• added warning log for max page limit exceeding api calls (#29788)
• Aggressively cache entry points in process (#29625)
• Don't use importlib.metadata to get Version for speed (#29723)
• Upgrade Mypy to 1.0 (#29468)
• Rename db export-cleaned to db export-archived (#29450)
• listener: simplify API by replacing SQLAlchemy event-listening by direct calls (#29289)
• No multi-line log entry for bash env vars (#28881)
• Switch to ruff for faster static checks (#28893)
• Remove horizontal lines in TI logs (#28876)
• Make allowed_deserialization_classes more intuitive (#28829)
• Propagate logs to stdout when in k8s executor pod (#28440, #30860)
• Fix code readability, add docstrings to json_client (#28619)
• AIP-51 - Misc. Compatibility Checks (#28375)
• Fix is_local for LocalKubernetesExecutor (#28288)
• Move Hive macros to the provider (#28538)
• Rerun flaky PinotDB integration test (#28562)
• Add pre-commit hook to check session default value (#28007)
• Refactor get_mapped_group_summaries for web UI (#28374)
• Add support for k8s 1.26 (#28320)
• Replace freezegun with time-machine (#28193)
• Completed D400 for airflow/kubernetes/* (#28212)
• Completed D400 for multiple folders (#27969)
• Drop k8s 1.21 and 1.22 support (#28168)
• Remove unused task_queue attr from k8s scheduler class (#28049)
• Completed D400 for multiple folders (#27767, #27768)

Doc only changes

• Add instructions on how to avoid accidental airflow upgrade/downgrade (#30813)
• Add explicit information about how to write task logs (#30732)
• Better explanation on how to log from tasks (#30746)
• Use correct import path for Dataset (#30617)
• Create audit_logs.rst (#30405)
• Adding taskflow API example for sensors (#30344)
• Add clarification about timezone aware dags (#30467)
• Clarity params documentation (#30345)
• Fix unit for task duration metric (#30273)
• Update dag-run.rst for dead links of cli commands (#30254)
• Add Write efficient Python code section to Reducing DAG complexity (#30158)
• Allow to specify which connection, variable or config are being looked up in the backend using *_lookup_pattern parameters (#29580)
• Add Documentation for notification feature extension (#29191)
• Clarify that executor interface is public but instances are not (#29200)
• Add Public Interface description to Airflow documentation (#28300)
• Add documentation for task group mapping (#28001)
• Some fixes to metrics doc (#30290)

New Features

• TaskRunner: notify of component start and finish (#27855)
• Add DagRun state change to the Listener plugin system(#27113)
• Metric for raw task return codes (#27155)
• Add logic for XComArg to pull specific map indexes (#27771)
• Clear TaskGroup (#26658, #28003)
• Add critical section query duration metric (#27700)
• Add: #23880 :: Audit log for AirflowModelViews(Variables/Connection) (#24079, #27994, #27923)
• Add postgres 15 support (#27444)
• Expand tasks in mapped group at run time (#27491)
• reset commits, clean submodules (#27560)
• scheduler_job, add metric for scheduler loop timer (#27605)
• Allow datasets to be used in taskflow (#27540)
• Add expanded_ti_count to ti context (#27680)
• Add user comment to task instance and dag run (#26457, #27849, #27867)
• Enable copying DagRun JSON to clipboard (#27639)
• Implement extra controls for SLAs (#27557)
• add dag parsed time in DAG view (#27573)
• Add max_wait for exponential_backoff in BaseSensor (#27597)
• Expand tasks in mapped group at parse time (#27158)
• Add disable retry flag on backfill (#23829)
• Adding sensor decorator (#22562)
• Api endpoint update ti (#26165)
• Filtering datasets by recent update events (#26942)
• Support Is /not Null filter for value is None on webui (#26584)
• Add search to datasets list (#26893)
• Split out and handle 'params' in mapped operator (#26100)
• Add authoring API for TaskGroup mapping (#26844)
• Add one_done trigger rule (#26146)
• Create a more efficient airflow dag test command that also has better local logging (#26400)
• Support add/remove permissions to roles commands (#26338)
• Auto tail file logs in Web UI (#26169)
• Add triggerer info to task instance in API (#26249)
• Flag to deserialize value on custom XCom backend (#26343)

Improvements

• Allow depth-first execution (#27827)
• UI: Update offset height if data changes (#27865)
• Improve TriggerRuleDep typing and readability (#27810)
• Make views requiring session, keyword only args (#27790)
• Optimize TI.xcom_pull() with explicit task_ids and map_indexes (#27699)
• Allow hyphens in pod id used by k8s executor (#27737)
• optimise task instances filtering (#27102)
• Use context managers to simplify log serve management (#27756)
• Fix formatting leftovers (#27750)
• Improve task deadlock messaging (#27734)
• Improve "sensor timeout" messaging (#27733)
• Replace urlparse with urlsplit (#27389)
• Align TaskGroup semantics to AbstractOperator (#27723)
• Add new files to parsing queue on every loop of dag processing (#27060)
• Make Kubernetes Executor & Scheduler resilient to error during PMH execution (#27611)
• Separate dataset deps into individual graphs (#27356)
• Use log.exception where more economical than log.error (#27517)
• Move validation branch_task_ids into SkipMixin (#27434)
• Coerce LazyXComAccess to list when pushed to XCom (#27251)
• Update cluster-policies.rst docs (#27362)
• Add warning if connection type already registered within the provider (#27520)
• Activate debug logging in commands with --verbose option (#27447)
• Add classic examples for Python Operators (#27403)
• change .first() to .scalar() (#27323)
• Improve reset_dag_run description (#26755)
• Add examples and howtos about sensors (#27333)
• Make grid view widths adjustable (#27273)
• Sorting plugins custom menu links by category before name (#27152)
• Simplify DagRun.verify_integrity (#26894)
• Add mapped task group info to serialization (#27027)
• Correct the JSON style used for Run config in Grid View (#27119)
• No extra__conn_type__ prefix required for UI behaviors (#26995)
• Improve dataset update blurb (#26878)
• Rename kubernetes config section to kubernetes_executor (#26873)
• decode params for dataset searches (#26941)
• Get rid of the DAGRun details page & rely completely on Grid (#26837)
• Fix scheduler crashloopbackoff when using hostname_callable (#24999)
• Reduce log verbosity in KubernetesExecutor. (#26582)
• Don't iterate tis list twice for no reason (#26740)
• Clearer code for PodGenerator.deserialize_model_file (#26641)
• Don't import kubernetes unless you have a V1Pod (#26496)
• Add updated_at column to DagRun and Ti tables (#26252)
• Move the deserialization of custom XCom Backend to 2.4.0 (#26392)
• Avoid calculating all elements when one item is needed (#26377)
• Add __future__.annotations automatically by isort (#26383)
• Handle list when serializing expand_kwargs (#26369)
• Apply PEP-563 (Postponed Evaluation of Annotations) to core airflow (#26290)
• Add more weekday operator and sensor examples #26071 (#26098)
• Align TaskGroup semantics to AbstractOperator (#27723)

Bug Fixes

• Gracefully handle whole config sections being renamed (#28008)
• Add allow list for imports during deserialization (#27887)
• Soft delete datasets that are no longer referenced in DAG schedules or task outlets (#27828)
• Redirect to home view when there are no valid tags in the URL (#25715)
• Refresh next run datasets info in dags view (#27839)
• Make MappedTaskGroup depend on its expand inputs (#27876)
• Make DagRun state updates for paused DAGs faster (#27725)
• Don't explicitly set include_examples to False on task run command (#27813)
• Fix menu border color (#27789)
• Fix backfill queued task getting reset to scheduled state. (#23720)
• Fix clearing child dag mapped tasks from parent dag (#27501)
• Handle json encoding of V1Pod in task callback (#27609)
• Fix ExternalTaskSensor can't check zipped dag (#27056)
• Avoid re-fetching DAG run in TriggerDagRunOperator (#27635)
• Continue on exception when retrieving metadata (#27665)
• External task sensor fail fix (#27190)
• Add the default None when pop actions (#27537)
• Display parameter values from serialized dag in trigger dag view. (#27482, #27944)
• Move TriggerDagRun conf check to execute (#27035)
• Resolve trigger assignment race condition (#27072)
• Update google_analytics.html (#27226)
• Fix some bug in web ui dags list page (auto-refresh & jump search null state) (#27141)
• Fixed broken URL for docker-compose.yaml (#26721)
• Fix xcom arg.py .zip bug (#26636)
• Fix 404 taskInstance errors and split into two tables (#26575)
• Fix browser warning of improper thread usage (#26551)
• template rendering issue fix (#26390)
• Clear autoregistered DAGs if there are any import errors (#26398)
• Fix from airflow import version lazy import (#26239)
• allow scroll in triggered dag runs modal (#27965)

Misc/Internal

• Remove is_mapped attribute (#27881)
• Simplify FAB table resetting (#27869)
• Fix old-style typing in Base Sensor (#27871)
• Switch (back) to late imports (#27730)
• Completed D400 for multiple folders (#27748)
• simplify notes accordion test (#27757)
• completed D400 for airflow/callbacks/* airflow/cli/* (#27721)
• Completed D400 for airflow/api_connexion/* directory (#27718)
• Completed D400 for airflow/listener/* directory (#27731)
• Completed D400 for airflow/lineage/* directory (#27732)
• Update API & Python Client versions (#27642)
• Completed D400 & D401 for airflow/api/* directory (#27716)
• Completed D400 for multiple folders (#27722)
• Bump minimatch from 3.0.4 to 3.0.8 in /airflow/www (#27688)
• Bump loader-utils from 1.4.1 to 1.4.2 in /airflow/www (#27697)
• Disable nested task mapping for now (#27681)
• bump alembic minimum version (#27629)
• remove unused code.html (#27585)
• Enable python string normalization everywhere (#27588)
• Upgrade dependencies in order to avoid backtracking (#27531)
• Strengthen a bit and clarify importance of triaging issues (#27262)
• Deduplicate type hints (#27508)
• Add stub 'yield' to BaseTrigger.run (#27416)
• Remove upper-bound limit to dask (#27415)
• Limit Dask to under 2022.10.1 (#27383)
• Update old style typing (#26872)
• Enable string normalization for docs (#27269)
• Slightly faster up/downgrade tests (#26939)
• Deprecate use of core get_kube_client in PodManager (#26848)
• Add memray files to gitignore / dockerignore (#27001)
• Bump sphinx and sphinx-autoapi (#26743)
• Simplify RTIF.delete_old_records() (#26667)
• migrate last react files to typescript (#26112)
• Work around pyupgrade edge cases (#26384)

Doc only changes

• Document dag_file_processor_timeouts metric as deprecated (#27067)
• Drop support for PostgreSQL 10 (#27594)
• Update index.rst (#27529)
• Add note about pushing the lazy XCom proxy to XCom (#27250)
• Fix BaseOperator link (#27441)
• [docs] best-practices add use variable with template example. (#27316)
• docs for custom view using plugin (#27244)
• Update graph view and grid view on overview page (#26909)
• Documentation fixes (#26819)
• make consistency on markup title string level (#26696)
• Add documentation to dag test function (#26713)
• Fix broken URL for docker-compose.yaml (#26726)
• Add a note against use of top level code in timetable (#26649)
• Fix example_datasets dag names (#26495)
• Update docs: zip-like effect is now possible in task mapping (#26435)
• changing to task decorator in docs from classic operator use (#25711)

New Features

• Add Data-aware Scheduling (https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-48+milestone%3A%22Airflow+2.4.0%22)
• Add @task.short_circuit TaskFlow decorator (#25752)
• Make execution_date_or_run_id optional in tasks test command (#26114)
• Automatically register DAGs that are used in a context manager (#23592, #26398)
• Add option of sending DAG parser logs to stdout. (#25754)
• Support multiple DagProcessors parsing files from different locations. (#25935)
• Implement ExternalPythonOperator (#25780)
• Make execution_date optional for command dags test (#26111)
• Implement expand_kwargs() against a literal list (#25925)
• Add trigger rule tooltip (#26043)
• Add conf parameter to CLI for airflow dags test (#25900)
• Include scheduled slots in pools view (#26006)
• Add output property to MappedOperator (#25604)
• Add roles delete command to cli (#25854)
• Add Airflow specific warning classes (#25799)
• Add support for TaskGroup in ExternalTaskSensor (#24902)
• Add @task.kubernetes taskflow decorator (#25663)
• Add a way to import Airflow without side-effects (#25832)
• Let timetables control generated run_ids. (#25795)
• Allow per-timetable ordering override in grid view (#25633)
• Grid logs for mapped instances (#25610, #25621, #25611)
• Consolidate to one schedule param (#25410)
• DAG regex flag in backfill command (#23870)
• Adding support for owner links in the Dags view UI (#25280)
• Ability to clear a specific DAG Run's task instances via REST API (#23516)
• Possibility to document DAG with a separate markdown file (#25509)
• Add parsing context to DAG Parsing (#25161)
• Implement CronTriggerTimetable (#23662)
• Add option to mask sensitive data in UI configuration page (#25346)
• Create new databases from the ORM (#24156)
• Implement XComArg.zip(*xcom_args) (#25176)
• Introduce sla_miss metric (#23402)
• Implement map() semantic (#25085)
• Add override method to TaskGroupDecorator (#25160)
• Implement expand_kwargs() (#24989)
• Add parameter to turn off SQL query logging (#24570)
• Add DagWarning model, and a check for missing pools (#23317)
• Add Task Logs to Grid details panel (#24249)
• Added small health check server and endpoint in scheduler(#23905)
• Add built-in External Link for ExternalTaskMarker operator (#23964)
• Add default task retry delay config (#23861)
• Add clear DagRun endpoint. (#23451)
• Add support for timezone as string in cron interval timetable (#23279)
• Add auto-refresh to dags home page (#22900, #24770)

Improvements

• Add more weekday operator and sensor examples #26071 (#26098)
• Add subdir parameter to dags reserialize command (#26170)
• Update zombie message to be more descriptive (#26141)
• Only send an SlaCallbackRequest if the DAG is scheduled (#26089)
• Promote Operator.output more (#25617)
• Upgrade API files to typescript (#25098)
• Less hacky double-rendering prevention in mapped task (#25924)
• Improve Audit log (#25856)
• Remove mapped operator validation code (#25870)
• More DAG(schedule=...) improvements (#25648)
• Reduce operator_name dupe in serialized JSON (#25819)
• Make grid view group/mapped summary UI more consistent (#25723)
• Remove useless statement in task_group_to_grid (#25654)
• Add optional data interval to CronTriggerTimetable (#25503)
• Remove unused code in /grid endpoint (#25481)
• Add and document description fields (#25370)
• Improve Airflow logging for operator Jinja template processing (#25452)
• Update core example DAGs to use @task.branch decorator (#25242)
• Update DAG audit_log route (#25415)
• Change stdout and stderr access mode to append in commands (#25253)
• Remove getTasks from Grid view (#25359)
• Improve taskflow type hints with ParamSpec (#25173)
• Use tables in grid details panes (#25258)
• Explicitly list @dag arguments (#25044)
• More typing in SchedulerJob and TaskInstance (#24912)
• Patch getfqdn with more resilient version (#24981)
• Replace all NBSP characters by whitespaces (#24797)
• Re-serialize all DAGs on airflow db upgrade (#24518)
• Rework contract of try_adopt_task_instances method (#23188)
• Make expand() error vague so it's not misleading (#24018)
• Add enum validation for [webserver]analytics_tool (#24032)
• Add dttm searchable field in audit log (#23794)
• Allow more parameters to be piped through via execute_in_subprocess (#23286)
• Use func.count to count rows (#23657)
• Remove stale serialized dags (#22917)
• AIP45 Remove dag parsing in airflow run local (#21877)
• Add support for queued state in DagRun update endpoint. (#23481)
• Add fields to dagrun endpoint (#23440)
• Use sql_alchemy_conn for celery result backend when result_backend is not set (#24496)

Bug Fixes

• Have consistent types between the ORM and the migration files (#24044, #25869)
• Disallow any dag tags longer than 100 char (#25196)
• Add the dag_id to AirflowDagCycleException message (#26204)
• Properly build URL to retrieve logs independently from system (#26337)
• For worker log servers only bind to IPV6 when dual stack is available (#26222)
• Fix TaskInstance.task not defined before handle_failure (#26040)
• Undo secrets backend config caching (#26223)
• Fix faulty executor config serialization logic (#26191)
• Show DAGs and Datasets menu links based on role permission (#26183)
• Allow setting TaskGroup tooltip via function docstring (#26028)
• Fix RecursionError on graph view of a DAG with many tasks (#26175)
• Fix backfill occasional deadlocking (#26161)
• Fix DagRun.start_date not set during backfill with --reset-dagruns True (#26135)
• Use label instead of id for dynamic task labels in graph (#26108)
• Don't fail DagRun when leaf mapped_task is SKIPPED (#25995)
• Add group prefix to decorated mapped task (#26081)
• Fix UI flash when triggering with dup logical date (#26094)
• Fix Make items nullable for TaskInstance related endpoints to avoid API errors (#26076)
• Fix BranchDateTimeOperator to be timezone-awreness-insensitive (#25944)
• Fix legacy timetable schedule interval params (#25999)
• Fix response schema for list-mapped-task-instance (#25965)
• Properly check the existence of missing mapped TIs (#25788)
• Fix broken auto-refresh on grid view (#25950)
• Use per-timetable ordering in grid UI (#25880)
• Rewrite recursion when parsing DAG into iteration (#25898)
• Find cross-group tasks in iter_mapped_dependants (#25793)
• Fail task if mapping upstream fails (#25757)
• Support / in variable get endpoint (#25774)
• Use cfg default_wrap value for grid logs (#25731)
• Add origin request args when triggering a run (#25729)
• Operator name separate from class (#22834)
• Fix incorrect data interval alignment due to assumption on input time alignment (#22658)
• Return None if an XComArg fails to resolve (#25661)
• Correct json arg help in airflow variables set command (#25726)
• Added MySQL index hint to use ti_state on find_zombies query (#25725)
• Only excluded actually expanded fields from render (#25599)
• Grid, fix toast for axios errors (#25703)
• Fix UI redirect (#26409)
• Require dag_id arg for dags list-runs (#26357)
• Check for queued states for dags auto-refresh (#25695)
• Fix upgrade code for the dag_owner_attributes table (#25579)
• Add map index to task logs api (#25568)
• Ensure that zombie tasks for dags with errors get cleaned up (#25550)
• Make extra link work in UI (#25500)
• Sync up plugin API schema and definition (#25524)
• First/last names can be empty (#25476)
• Refactor DAG pages to be consistent (#25402)
• Check expand_kwargs() input type before unmapping (#25355)
• Filter XCOM by key when calculating map lengths (#24530)
• Fix ExternalTaskSensor not working with dynamic task (#25215)
• Added exception catching to send default email if template file raises any exception (#24943)
• Bring MappedOperator members in sync with BaseOperator (#24034)

Misc/Internal

• Add automatically generated ERD schema for the MetaData DB (#26217)
• Mark serialization functions as internal (#26193)
• Remove remaining deprecated classes and replace them with PEP562 (#26167)
• Move dag_edges and task_group_to_dict to corresponding util modules (#26212)
• Lazily import many modules to improve import speed (#24486, #26239)
• FIX Incorrect typing information (#26077)
• Add missing contrib classes to deprecated dictionaries (#26179)
• Re-configure/connect the ORM after forking to run a DAG processor (#26216)
• Remove cattrs from lineage processing. (#26134)
• Removed deprecated contrib files and replace them with PEP-562 getattr (#26153)
• Make BaseSerialization.serialize "public" to other classes. (#26142)
• Change the template to use human readable task_instance description (#25960)
• Bump moment-timezone from 0.5.34 to 0.5.35 in /airflow/www (#26080)
• Fix Flask deprecation warning (#25753)
• Add CamelCase to generated operations types (#25887)
• Fix migration issues and tighten the CI upgrade/downgrade test (#25869)
• Fix type annotations in SkipMixin (#25864)
• Workaround setuptools editable packages path issue (#25848)
• Bump undici from 5.8.0 to 5.9.1 in /airflow/www (#25801)
• Add custom_operator_name attr to _BranchPythonDecoratedOperator (#25783)
• Clarify filename_template deprecation message (#25749)
• Use ParamSpec to replace ... in Callable (#25658)
• Remove deprecated modules (#25543)
• Documentation on task mapping additions (#24489)
• Remove Smart Sensors (#25507)
• Fix elasticsearch test config to avoid warning on deprecated template (#25520)
• Bump terser from 4.8.0 to 4.8.1 in /airflow/ui (#25178)
• Generate typescript types from rest API docs (#25123)
• Upgrade utils files to typescript (#25089)
• Upgrade remaining context file to typescript. (#25096)
• Migrate files to ts (#25267)
• Upgrade grid Table component to ts. (#25074)
• Skip mapping against mapped ti if it returns None (#25047)
• Refactor js file structure (#25003)
• Move mapped kwargs introspection to separate type (#24971)
• Only assert stuff for mypy when type checking (#24937)
• Bump moment from 2.29.3 to 2.29.4 in /airflow/www (#24885)
• Remove "bad characters" from our codebase (#24841)
• Remove xcom_push flag from BashOperator (#24824)
• Move Flask hook registration to end of file (#24776)
• Upgrade more javascript files to typescript (#24715)
• Clean up task decorator type hints and docstrings (#24667)
• Preserve original order of providers' connection extra fields in UI (#24425)
• Rename charts.css to chart.css (#24531)
• Rename grid.css to chart.css (#24529)
• Misc: create new process group by set_new_process_group utility (#24371)
• Airflow UI fix Prototype Pollution (#24201)
• Bump moto version (#24222)
• Remove unused [github_enterprise] from ref docs (#24033)
• Clean up f-strings in logging calls (#23597)
• Add limit for JPype1 (#23847)
• Simply json responses (#25518)
• Add min attrs version (#26408)

Doc only changes

• Add url prefix setting for Celery Flower (#25986)
• Updating deprecated configuration in examples (#26037)
• Fix wrong link for taskflow tutorial (#26007)
• Reorganize tutorials into a section (#25890)
• Fix concept doc for dynamic task map (#26002)
• Update code examples from "classic" operators to taskflow (#25845, #25657)
• Add instructions on manually fixing MySQL Charset problems (#25938)
• Prefer the local Quick Start in docs (#25888)
• Fix broken link to Trigger Rules (#25840)
• Improve docker documentation (#25735)
• Correctly link to Dag parsing context in docs (#25722)
• Add note on task_instance_mutation_hook usage (#25607)
• Note that TaskFlow API automatically passes data between tasks (#25577)
• Update DAG run to clarify when a DAG actually runs (#25290)
• Update tutorial docs to include a definition of operators (#25012)
• Rewrite the Airflow documentation home page (#24795)
• Fix task-generated mapping example (#23424)
• Add note on subtle logical date change in 2.2.0 (#24413)
• Add missing import in best-practices code example (#25391)

New Features

• Add dynamic task mapping (https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-42+milestone%3A%22Airflow+2.3.0%22)
• New Grid View replaces Tree View (#18675)
• Templated requirements.txt in Python Operators (#17349)
• Allow reuse of decorated tasks (#22941)
• Move the database configuration to a new section (#22284)
• Add SmoothOperator (#22813)
• Make operator's execution_timeout configurable (#22389)
• Events Timetable (#22332)
• Support dag serialization with custom ti_deps rules (#22698)
• Support log download in task log view (#22804)
• support for continue backfill on failures (#22697)
• Add dag-processor cli command (#22305)
• Add possibility to create users in LDAP mode (#22619)
• Add ignore_first_depends_on_past for scheduled jobs (#22491)
• Update base sensor operator to support XCOM return value (#20656)
• Add an option for run id in the ui trigger screen (#21851)
• Enable JSON serialization for connections (#19857)
• Add REST API endpoint for bulk update of DAGs (#19758)
• Add queue button to click-on-DagRun interface. (#21555)
• Add list-import-errors to airflow dags command (#22084)
• Store callbacks in database if standalone_dag_processor config is True. (#21731)
• Add LocalKubernetesExecutor (#19729)
• Add celery.task_timeout_error metric (#21602)
• Airflow db downgrade cli command (#21596)
• Add ALL_SKIPPED trigger rule (#21662)
• Add db clean CLI command for purging old data (#20838)
• Add celery_logging_level (#21506)
• Support different timeout value for dag file parsing (#21501)
• Support generating SQL script for upgrades (#20962)
• Add option to compress Serialized dag data (#21332)
• Branch python operator decorator (#20860)
• Add Audit Log View to Dag View (#20733)
• Add missing StatsD metric for failing SLA Callback notification (#20924)
• Add ShortCircuitOperator configurability for respecting downstream trigger rules (#20044)
• Allow using Markup in page title in Webserver (#20888)
• Add Listener Plugin API that tracks TaskInstance state changes (#20443)
• Add context var hook to inject more env vars (#20361)
• Add a button to set all tasks to skipped (#20455)
• Cleanup pending pods (#20438)
• Add config to warn public deployment exposure in UI (#18557)
• Log filename template records (#20165)
• Added windows extensions (#16110)
• Showing approximate time until next dag_run in Airflow (#20273)
• Extend config window on UI (#20052)
• Add show dag dependencies feature to CLI (#19985)
• Add cli command for 'airflow dags reserialize` (#19471)
• Add missing description field to Pool schema(REST API) (#19841)
• Introduce DagRun action to change state to queued. (#19353)
• Add DAG run details page (#19705)
• Add role export/import to cli tools (#18916)
• Adding dag_id_pattern parameter to the /dags endpoint (#18924)

Improvements

• Show schedule_interval/timetable description in UI (#16931)
• Added column duration to DAG runs view (#19482)
• Enable use of custom conn extra fields without prefix (#22607)
• Initialize finished counter at zero (#23080)
• Improve logging of optional provider features messages (#23037)
• Meaningful error message in resolve_template_files (#23027)
• Update ImportError items instead of deleting and recreating them (#22928)
• Add option --skip-init to db reset command (#22989)
• Support importing connections from files with ".yml" extension (#22872)
• Support glob syntax in .airflowignore files (#21392) (#22051)
• Hide pagination when data is a single page (#22963)
• Support for sorting DAGs in the web UI (#22671)
• Speed up has_access decorator by ~200ms (#22858)
• Add XComArg to lazy-imported list of Airflow module (#22862)
• Add more fields to REST API dags/dag_id/details endpoint (#22756)
• Don't show irrelevant/duplicated/"internal" Task attrs in UI (#22812)
• No need to load whole ti in current_state (#22764)
• Pickle dag exception string fix (#22760)
• Better verification of Localexecutor's parallelism option (#22711)
• log backfill exceptions to sentry (#22704)
• retry commit on MySQL deadlocks during backfill (#22696)
• Add more fields to REST API get DAG(dags/dag_id) endpoint (#22637)
• Use timetable to generate planned days for current year (#22055)
• Disable connection pool for celery worker (#22493)
• Make date picker label visible in trigger dag view (#22379)
• Expose try_number in airflow vars (#22297)
• Add generic connection type (#22310)
• Add a few more fields to the taskinstance finished log message (#22262)
• Pause auto-refresh if scheduler isn't running (#22151)
• Show DagModel details. (#21868)
• Add pip_install_options to PythonVirtualenvOperator (#22158)
• Show import error for airflow dags list CLI command (#21991)
• Pause auto-refresh when page is hidden (#21904)
• Default args type check (#21809)
• Enhance magic methods on XComArg for UX (#21882)
• py files don't have to be checked is_zipfiles in refresh_dag (#21926)
• Fix TaskDecorator type hints (#21881)
• Add 'Show record' option for variables (#21342)
• Use DB where possible for quicker airflow dag subcommands (#21793)
• REST API: add rendered fields in task instance. (#21741)
• Change the default auth backend to session (#21640)
• Don't check if py DAG files are zipped during parsing (#21538)
• Switch XCom implementation to use run_id (#20975)
• Action log on Browse Views (#21569)
• Implement multiple API auth backends (#21472)
• Change logging level details of connection info in get_connection() (#21162)
• Support mssql in airflow db shell (#21511)
• Support config worker_enable_remote_control for celery (#21507)
• Log memory usage in CgroupTaskRunner (#21481)
• Modernize DAG-related URL routes and rename "tree" to "grid" (#20730)
• Move Zombie detection to SchedulerJob (#21181)
• Improve speed to run airflow by 6x (#21438)
• Add more SQL template fields renderers (#21237)
• Simplify fab has access lookup (#19294)
• Log context only for default method (#21244)
• Log trigger status only if at least one is running (#21191)
• Add optional features in providers. (#21074)
• Better multiple_outputs inferral for @task.python (#20800)
• Improve handling of string type and non-attribute template_fields (#21054)
• Remove un-needed deps/version requirements (#20979)
• Correctly specify overloads for TaskFlow API for type-hinting (#20933)
• Introduce notification_sent to SlaMiss view (#20923)
• Rewrite the task decorator as a composition (#20868)
• Add "Greater/Smaller than or Equal" to filters in the browse views (#20602) (#20798)
• Rewrite DAG run retrieval in task command (#20737)
• Speed up creation of DagRun for large DAGs (5k+ tasks) by 25-130% (#20722)
• Make native environment Airflow-flavored like sandbox (#20704)
• Better error when param value has unexpected type (#20648)
• Add filter by state in DagRun REST API (List Dag Runs) (#20485)
• Prevent exponential memory growth in Tasks with custom logging handler (#20541)
• Set default logger in logging Mixin (#20355)
• Reduce deprecation warnings from www (#20378)
• Add hour and minute to time format on x-axis of all charts using nvd3.lineChart (#20002)
• Add specific warning when Task asks for more slots than pool defined with (#20178)
• UI: Update duration column for better human readability (#20112)
• Use Viewer role as example public role (#19215)
• Properly implement DAG param dict copying (#20216)
• ShortCircuitOperator push XCom by returning python_callable result (#20071)
• Add clear logging to tasks killed due to a Dagrun timeout (#19950)
• Change log level for Zombie detection messages (#20204)
• Better confirmation prompts (#20183)
• Only execute TIs of running DagRuns (#20182)
• Check and run migration in commands if necessary (#18439)
• Log only when Zombies exists (#20118)
• Increase length of the email and username (#19932)
• Add more filtering options for TI's in the UI (#19910)
• Dynamically enable "Test Connection" button by connection type (#19792)
• Avoid littering postgres server logs with "could not obtain lock" with HA schedulers (#19842)
• Renamed Connection.get_hook parameter to make it the same as in SqlSensor and SqlOperator. (#19849)
• Add hook_params in SqlSensor using the latest changes from PR #18718. (#18431)
• Speed up webserver boot time by delaying provider initialization (#19709)
• Configurable logging of XCOM value in PythonOperator (#19378)
• Minimize production js files (#19658)
• Add hook_params in BaseSqlOperator (#18718)
• Add missing "end_date" to hash components (#19281)
• More friendly output of the airflow plugins command + add timetables (#19298)
• Add sensor default timeout config (#19119)
• Update taskinstance REST API schema to include dag_run_id field (#19105)
• Adding feature in bash operator to append the user defined env variable to system env variable (#18944)
• Duplicate Connection: Added logic to query if a connection id exists before creating one (#18161)

Bug Fixes

• Use inherited 'trigger_tasks' method (#23016)
• In DAG dependency detector, use class type instead of class name (#21706)
• Fix tasks being wrongly skipped by schedule_after_task_execution (#23181)
• Fix X-Frame enabled behaviour (#23222)
• Allow extra to be nullable in connection payload as per schema(REST API). (#23183)
• Fix dag_id extraction for dag level access checks in web ui (#23015)
• Fix timezone display for logs on UI (#23075)
• Include message in graph errors (#23021)
• Change trigger dropdown left position (#23013)
• Don't add planned tasks for legacy DAG runs (#23007)
• Add dangling rows check for TaskInstance references (#22924)
• Validate the input params in connection CLI command (#22688)
• Fix trigger event payload is not persisted in db (#22944)
• Drop "airflow moved" tables in command db reset (#22990)
• Add max width to task group tooltips (#22978)
• Add template support for external_task_ids. (#22809)
• Allow DagParam to hold falsy values (#22964)
• Fix regression in pool metrics (#22939)
• Priority order tasks even when using pools (#22483)
• Do not clear XCom when resuming from deferral (#22932)
• Handle invalid JSON metadata in get_logs_with_metadata endpoint. (#22898)
• Fix pre-upgrade check for rows dangling w.r.t. dag_run (#22850)
• Fixed backfill interference with scheduler (#22701)
• Support conf param override for backfill runs (#22837)
• Correctly interpolate pool name in PoolSlotsAvailableDep statues (#22807)
• Fix email_on_failure with render_template_as_native_obj (#22770)
• Fix processor cleanup on DagFileProcessorManager (#22685)
• Prevent meta name clash for task instances (#22783)
• remove json parse for gantt chart (#22780)
• Check for missing dagrun should know version (#22752)
• Fixes ScheduleInterval spec (#22635)
• Fixing task status for non-running and non-committed tasks (#22410)
• Do not log the hook connection details even at DEBUG level (#22627)
• Stop crashing when empty logs are received from kubernetes client (#22566)
• Fix bugs about timezone change (#22525)
• Fix entire DAG stops when one task has end_date (#20920)
• Use logger to print message during task execution. (#22488)
• Make sure finalizers are not skipped during exception handling (#22475)
• update smart sensor docs and minor fix on is_smart_sensor_compatible() (#22386)
• Fix run_id k8s and elasticsearch compatibility with Airflow 2.1 (#22385)
• Allow to except_skip None on BranchPythonOperator (#20411)
• Fix incorrect datetime details (DagRun views) (#21357)
• Remove incorrect deprecation warning in secrets backend (#22326)
• Remove RefreshConfiguration workaround for K8s token refreshing (#20759)
• Masking extras in GET /connections/<connection> endpoint (#22227)
• Set queued_dttm when submitting task to directly to executor (#22259)
• Addressed some issues in the tutorial mentioned in discussion #22233 (#22236)
• Change default python executable to python3 for docker decorator (#21973)
• Don't validate that Params are JSON when NOTSET (#22000)
• Add per-DAG delete permissions (#21938)
• Fix handling some None parameters in kubernetes 23 libs. (#21905)
• Fix handling of empty (None) tags in bulk_write_to_db (#21757)
• Fix DAG date range bug (#20507)
• Removed request.referrer from views.py (#21751)
• Make DbApiHook use get_uri from Connection (#21764)
• Fix some migrations (#21670)
• [de]serialize resources on task correctly (#21445)
• Add params dag_id, task_id etc to XCom.serialize_value (#19505)
• Update test connection functionality to use custom form fields (#21330)
• fix all "high" npm vulnerabilities (#21526)
• Fix bug incorrectly removing action from role, rather than permission. (#21483)
• Fix relationship join bug in FAB/SecurityManager with SQLA 1.4 (#21296)
• Use Identity instead of Sequence in SQLAlchemy 1.4 for MSSQL (#21238)
• Ensure on_task_instance_running listener can get at task (#21157)
• Return to the same place when triggering a DAG (#20955)
• Fix task ID deduplication in @task_group (#20870)
• Add downgrade to some FAB migrations (#20874)
• Only validate Params when DAG is triggered (#20802)
• Fix airflow trigger cli (#20781)
• Fix task instances iteration in a pool to prevent blocking (#20816)
• Allow depending to a @task_group as a whole (#20671)
• Use original task's start_date if a task continues after deferral (#20062)
• Disabled edit button in task instances list view page (#20659)
• Fix a package name import error (#20519) (#20519)
• Remove execution_date label when get cleanup pods list (#20417)
• Remove unneeded FAB REST API endpoints (#20487)
• Fix parsing of Cloudwatch log group arn containing slashes (#14667) (#19700)
• Sanity check for MySQL's TIMESTAMP column (#19821)
• Allow using default celery command group with executors subclassed from Celery-based executors. (#18189)
• Move class_permission_name to mixin so it applies to all classes (#18749)
• Adjust trimmed_pod_id and replace '.' with '-' (#19036)
• Pass custom_headers to send_email and send_email_smtp (#19009)
• Ensure catchup=False is used in example dags (#19396)
• Edit permalinks in OpenApi description file (#19244)
• Navigate directly to DAG when selecting from search typeahead list (#18991)
• [Minor] Fix padding on home page (#19025)

Doc only changes

• Update doc for DAG file processing (#23209)
• Replace changelog/updating with release notes and towncrier now (#22003)
• Fix wrong reference in tracking-user-activity.rst (#22745)
• Remove references to rbac = True from docs (#22725)
• Doc: Update description for executor-bound dependencies (#22601)
• Update check-health.rst (#22372)
• Stronger language about Docker Compose customizability (#22304)
• Update logging-tasks.rst (#22116)
• Add example config of sql_alchemy_connect_args (#22045)
• Update best-practices.rst (#22053)
• Add information on DAG pausing/deactivation/deletion (#22025)
• Add brief examples of integration test dags you might want (#22009)
• Run inclusive language check on CHANGELOG (#21980)
• Add detailed email docs for Sendgrid (#21958)
• Add docs for db upgrade / db downgrade (#21879)
• Update modules_management.rst (#21889)
• Fix UPDATING section on SqlAlchemy 1.4 scheme changes (#21887)
• Update TaskFlow tutorial doc to show how to pass "operator-level" args. (#21446)
• Fix doc - replace decreasing by increasing (#21805)
• Add another way to dynamically generate DAGs to docs (#21297)
• Add extra information about time synchronization needed (#21685)
• Update debug.rst docs (#21246)
• Replaces the usage of postgres:// with postgresql:// (#21205)
• Fix task execution process in CeleryExecutor docs (#20783)

Misc/Internal

• Bring back deprecated security manager functions (#23243)
• Replace usage of DummyOperator with EmptyOperator (#22974)
• Deprecate DummyOperator in favor of EmptyOperator (#22832)
• Remove unnecessary python 3.6 conditionals (#20549)
• Bump moment from 2.29.1 to 2.29.2 in /airflow/www (#22873)
• Bump prismjs from 1.26.0 to 1.27.0 in /airflow/www (#22823)
• Bump nanoid from 3.1.23 to 3.3.2 in /airflow/www (#22803)
• Bump minimist from 1.2.5 to 1.2.6 in /airflow/www (#22798)
• Remove dag parsing from db init command (#22531)
• Update our approach for executor-bound dependencies (#22573)
• Use Airflow.Base.metadata in FAB models (#22353)
• Limit docutils to make our documentation pretty again (#22420)
• Add Python 3.10 support (#22050)
• [FEATURE] add 1.22 1.23 K8S support (#21902)
• Remove pandas upper limit now that SQLA is 1.4+ (#22162)
• Patch sql_alchemy_conn if old postgres scheme used (#22333)
• Protect against accidental misuse of XCom.get_value() (#22244)
• Order filenames for migrations (#22168)
• Don't try to auto generate migrations for Celery tables (#22120)
• Require SQLAlchemy 1.4 (#22114)
• bump sphinx-jinja (#22101)
• Add compat shim for SQLAlchemy to avoid warnings (#21959)
• Rename xcom.dagrun_id to xcom.dag_run_id (#21806)
• Deprecate non-JSON conn.extra (#21816)
• Bump upper bound version of jsonschema to 5.0 (#21712)
• Deprecate helper utility days_ago (#21653)
• Remove :type lines now sphinx-autoapi supports type hints (#20951)
• Silence deprecation warning in tests (#20900)
• Use DagRun.run_id instead of execution_date when updating state of TIs (UI & REST API) (#18724)
• Add Context stub to Airflow packages (#20817)
• Update Kubernetes library version (#18797)
• Rename PodLauncher to PodManager (#20576)
• Removes Python 3.6 support (#20467)
• Add deprecation warning for non-json-serializable params (#20174)
• Rename TaskMixin to DependencyMixin (#20297)
• Deprecate passing execution_date to XCom methods (#19825)
• Remove get_readable_dags and get_editable_dags, and get_accessible_dags. (#19961)
• Remove postgres 9.6 support (#19987)
• Removed hardcoded connection types. Check if hook is instance of DbApiHook. (#19639)
• add kubernetes 1.21 support (#19557)
• Add FAB base class and set import_name explicitly. (#19667)
• Removes unused state transitions to handle auto-changing view permissions. (#19153)
• Chore: Use enum for __var and __type members (#19303)
• Use fab models (#19121)
• Consolidate method names between Airflow Security Manager and FAB default (#18726)
• Remove distutils usages for Python 3.10 (#19064)
• Removing redundant max_tis_per_query initialisation on SchedulerJob (#19020)
• Remove deprecated usage of init_role() from API (#18820)
• Remove duplicate code on dbapi hook (#18821)