NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
PyPI · #1072 most downloaded on PyPI
Programmatically author, schedule and monitor data pipelines
Last release 13 days ago
17 Sep 2026
Ships fairly regularly
a new release about every 2 weeks
Nearly every release is documented
notes for 60 of the last 60 stable releases
4 versions withdrawn
withdrawn after publishing
9 years old
300 releases · first in 2017
Nothing published for this version
Do not provide deprecated execution_date in @apply_lineage
No significant changes.
AirflowSecurityManagerV2 leave transactions in the idle in transaction state (#39935)SafeDogStatsdLogger to use get_validator to enable pattern matching (#39370)has_access (#39421)execution_date in @apply_lineage (#39327)sql_alchemy_engine_args config example (#38971)yandex provider to avoid mypy errors (#39990)provider_info_cache decorator (#39750)defer (#39742)idx_last_scheduling_decision on dag_run table (#39275)CronDataIntervalTimetable (#39780)One column per quarter.
Nothing published for this version
Undeprecate BaseXCom.get_one method for now
10.17.0 or later (#38071)If you use Stackdriver logging, you must use Google provider version 10.17.0 or later. Airflow 2.9.1 now passes gcp_log_name to the StackdriverTaskHandler instead of name, and this will fail on earlier provider versions.
This fixes a bug where the log name configured in [logging] remove_base_log_folder was overridden when Airflow configured logging, resulting in task logs going to the wrong destination.
href for nav bar (#39282)firefox (#39261)log_url (#39183)UX (#39119)ux in react dag page (#39122)AUTH_ROLE_PUBLIC is set in check_authentication (#39012)map_index_template so it renders for failed tasks as long as it was defined before the point of failure (#38902)Undeprecate BaseXCom.get_one method for now (#38991)inherit_cache attribute for CreateTableAs custom SA Clause (#38985)SAWarning 'Coercing Subquery object into a select() for use in IN()' (#38926)cartesian product in AirflowSecurityManagerV2 (#38913)methodtools.lru_cache instead of functools.lru_cache in class methods (#37757)airflow dags backfill only if -I / --ignore-first-depends-on-past provided (#38676)TriggerDagRunOperator deprecate execution_date in favor of logical_date (#39285)@deprecated decorator (#39205)is_authorized_custom_view from auth manager to handle custom actions (#39167)minischeduler skip (#38976)undici from 5.28.3 to 5.28.4 in /airflow/www (#38751)PythonOperator op_kwargs (#39242)user and role commands (#39224)k8s 1.29 to supported version in docs (#39168)DagBag class docstring to include all params (#38814)Nothing published for this version
Nothing published for this version
Bump aiohttp min version to avoid CVE-2024-23829 and CVE-2024-23334
Lifecycle events:
on_startingbefore_stoppingDagRun State Change Events:
on_dag_run_runningon_dag_run_successon_dag_run_failedTaskInstance State Change Events:
on_task_instance_runningon_task_instance_successon_task_instance_failedAfter discussion <https://lists.apache.org/thread/r06j306hldg03g2my1pd4nyjxg78b3h4>__
and a voting process <https://lists.apache.org/thread/pgcgmhf6560k8jbsmz8nlyoxosvltph2>__,
the Airflow's PMC and Committers have reached a resolution to no longer maintain MsSQL as a supported Database Backend.
As of Airflow 2.9.0 support of MsSQL has been removed for Airflow Database Backend.
A migration script which can help migrating the database before upgrading to Airflow 2.9.0 is available in
airflow-mssql-migration repo on Github <https://github.com/apache/airflow-mssql-migration>_.
Note that the migration script is provided without support and warranty.
This does not affect the existing provider packages (operators and hooks), DAGs can still access and process data from MsSQL.
Datasets must use a URI that conform to rules laid down in AIP-60, and the value
will be automatically normalized when the DAG file is parsed. See
documentation on Datasets <https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/datasets.html>_ for
a more detailed description on the rules.
You may need to change your Dataset identifiers if they look like a URI, but are used in a less mainstream way, such as relying on the URI's auth section, or have a case-sensitive protocol name.
get_permitted_menu_items in BaseAuthManager has been renamed filter_permitted_menu_items (#37627)The Audit Log event name for REST API events will be prepended with api. or ui., depending on if it came from the Airflow UI or externally.
There are a few caveats though:
Pendulum2 does not support Python 3.12. For Python 3.12 you need to use
Pendulum 3 <https://pendulum.eustace.io/blog/announcing-pendulum-3-0-0.html>_
Minimum SQLAlchemy version supported when Pandas is installed for Python 3.12 is 1.4.36 released in
April 2022. Airflow 2.9.0 increases the minimum supported version of SQLAlchemy to 1.4.36 for all
Python versions.
Not all Providers support Python 3.12. At the initial release of Airflow 2.9.0 the following providers are released without support for Python 3.12:
apache.beam - pending on Apache Beam support for 3.12 <https://github.com/apache/beam/issues/29149>_papermill - pending on Releasing Python 3.12 compatible papermill client version
including this merged issue <https://github.com/nteract/papermill/pull/771>_There's now a limit to the length of data that can be stored in the Rendered Template Fields.
The limit is set to 4096 characters. If the data exceeds this limit, it will be truncated. You can change this limit
by setting the [core]max_template_field_length configuration option in your airflow config.
Xcom table column value type has changed from blob to longblob. This will allow you to store relatively big data in Xcom but process can take a significant amount of time if you have a lot of large data stored in Xcom.
To downgrade from revision: b4078ac230a1, ensure that you don't have Xcom values larger than 65,535 bytes. Otherwise, you'll need to clean those rows or run airflow db clean xcom to clean the Xcom table.
Matomo as an option for analytics_tool. (#38221)hashable (#37465)queuedEvent endpoint to get/delete DatasetDagRunQueue (#37176)DatasetOrTimeSchedule (#36710)on_skipped_callback to BaseOperator (#36374)@task.bash TaskFlow decorator (#30176, #37875)ExternalPythonOperator use version from sys.version_info (#38377)run_id column to log table (#37731)tryNumber to grid task instance tooltip (#37911)ExternalPythonOperator (#37409)Pathlike (#36947)nowait and skip_locked into with_row_locks (#36889)dag/dagRun in the REST API (#36641)Connexion from auth manager interface (#36209)total_entries count on the event logs endpoint (#38625)tz in next run ID info (#38482)chakra styles to keep dropdowns in filter bar (#38456)__exit__ is called in decorator context managers (#38383)BaseAuthManager.is_authorized_custom_view abstract (#37915)/get_logs_with_metadata endpoint (#37756)encoding to the SQL engine in SQLAlchemy v2 (#37545)consuming_dags attr eagerly before dataset listener (#36247)importlib_metadata with compat to Python 3.10/3.12 stdlib (#38366)__new__ magic method of BaseOperatorMeta to avoid bad mixing classic and decorated operators (#37937)sys.version_info for determine Python Major.Minor (#38372)blinker add where it requires (#38140)> 39.0.0 (#38112)assert outside of the tests (#37718)flask._request_ctx_stack (#37522)login attribute in airflow.__init__.py (#37565)datetime.datetime.utcnow by airflow.utils.timezone.utcnow in core (#35448)is_authorized_cluster_activity from auth manager (#36175)exception to templates ref list (#36656)Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Fix incorrect serialization of FixedTimezone
No significant changes.
FixedTimezone (#38139)ObjectStoragePath (#37769)pytest_rewrites (#38095, #38139)pandas to <2.2 (#37748)croniter to fix an issue with 29 Feb cron expressions (#38198)Nothing published for this version
Add "MENU" permission in auth manager
airflow_pre_installed_providers.txt artifact (#37679)BranchDayOfWeekOperator (#37813)ERD generating doc improvement (#37808)Nothing published for this version
Replace other Python 3.11 and 3.12 deprecations
allowed_deserialization_classes flag now follows a glob pattern (#36147).For example if one wants to add the class airflow.tests.custom_class to the
allowed_deserialization_classes list, it can be done by writing the full class
name (airflow.tests.custom_class) or a pattern such as the ones used in glob
search (e.g., airflow.*, airflow.tests.*).
If you currently use a custom regexp path make sure to rewrite it as a glob pattern.
Alternatively, if you still wish to match it as a regexp pattern, add it under the new
list allowed_deserialization_classes_regexp instead.
This was done under the policy that we do not want users like Viewer, Ops, and other users apart from Admin to have access to audit_logs. The intention behind this change is to restrict users with less permissions from viewing user details like First Name, Email etc. from the audit_logs when they are not permitted to.
The impact of this change is that the existing users with non admin rights won't be able to view or access the audit_logs, both from the Browse tab or from the DAG run.
AirflowTimeoutError is no longer except by default through Exception (#35653).The AirflowTimeoutError is now inheriting BaseException instead of
AirflowException->Exception.
See https://docs.python.org/3/library/exceptions.html#exception-hierarchy
This prevents code catching Exception from accidentally
catching AirflowTimeoutError and continuing to run.
AirflowTimeoutError is an explicit intent to cancel the task, and should not
be caught in attempts to handle the error and return some default value.
Catching AirflowTimeoutError is still possible by explicitly excepting
AirflowTimeoutError or BaseException.
This is discouraged, as it may allow the code to continue running even after
such cancellation requests.
Code that previously depended on performing strict cleanup in every situation
after catching Exception is advised to use finally blocks or
context managers. To perform only the cleanup and then automatically
re-raise the exception.
See similar considerations about catching KeyboardInterrupt in
https://docs.python.org/3/library/exceptions.html#KeyboardInterrupt
IMPORT_ERROR from DAG related permissions to view related permissions (#37292)AirflowTaskTimeout to inherit BaseException (#35653)namedtuple (#37168)Treeview function (#37162)access_entity is specified (#37290)dateTimeAttrFormat constant (#37285)@Sentry.enrich_errors (#37002)dryrun auto-fetch (#36941)/variables endpoint (#36820)pendulum.from_timestamp usage (#37160)CLI instead of specific one (#37651)undici from 5.26.3 to 5.28.3 in /airflow/www (#37493)3.12 exclusions in providers/pyproject.toml (#37404)markdown from core dependencies (#37396)pageSize method. (#37319)Python 3.11 and 3.12 deprecations (#37478)airflow_pre_installed_providers.txt into sdist distribution (#37388)universal-pathlib to < 0.2.0 (#37311)queue_when (#36997)config.yml for environment variable sql_alchemy_connect_args (#36526)Alembic to 1.13.1 (#36928)flask-session to <0.6 (#36895)CLI flags available (#37231)otel config descriptions (#37229)Objectstore tutorial with prereqs section (#36983)package/module names (#36927)__init__ of operators automatically (#33786)Nothing published for this version
Nothing published for this version
Nothing published for this version
Get rid of pyarrow-hotfix for CVE-2023-47248
pendulum package set to 3 (#36281).Support for pendulum 2.1.2 will be saved for a while, presumably until the next feature version of Airflow. It is advised to upgrade user code to use pendulum 3 as soon as possible.
We standardized Airflow dependency configuration to follow latest development in Python packaging by
using pyproject.toml. Airflow is now compliant with those accepted PEPs:
PEP-440 Version Identification and Dependency Specification <https://www.python.org/dev/peps/pep-0440/>__PEP-517 A build-system independent format for source trees <https://www.python.org/dev/peps/pep-0517/>__PEP-518 Specifying Minimum Build System Requirements for Python Projects <https://www.python.org/dev/peps/pep-0518/>__PEP-561 Distributing and Packaging Type Information <https://www.python.org/dev/peps/pep-0561/>__PEP-621 Storing project metadata in pyproject.toml <https://www.python.org/dev/peps/pep-0621/>__PEP-660 Editable installs for pyproject.toml based builds (wheel based) <https://www.python.org/dev/peps/pep-0660/>__PEP-685 Comparison of extra names for optional distribution dependencies <https://www.python.org/dev/peps/pep-0685/>__Also we implement multiple license files support coming from Draft, not yet accepted (but supported by hatchling) PEP:
PEP 639 Improving License Clarity with Better Package Metadata <https://peps.python.org/pep-0639/>__This has almost no noticeable impact on users if they are using modern Python packaging and development tools, generally
speaking Airflow should behave as it did before when installing it from PyPI and it should be much easier to install
it for development purposes using pip install -e ".[devel]".
The differences from the user side are:
- (following PEP-685) instead of _ and .
(as it was before in some extras). When you install airflow with such extras (for example dbt.core or
all_dbs) you should use - instead of _ and ..In most modern tools this will work in backwards-compatible way, but in some old version of those tools you might need to
replace _ and . with -. You can also get warnings that the extra you are installing does not exist - but usually
this warning is harmless and the extra is installed anyway. It is, however, recommended to change to use - in extras in your dependency
specifications for all Airflow extras.
Released airflow package does not contain devel, devel-*, doc and doc-gen extras.
Those extras are only available when you install Airflow from sources in --editable mode. This is
because those extras are only used for development and documentation building purposes and are not needed
when you install Airflow for production use. Those dependencies had unspecified and varying behaviour for
released packages anyway and you were not supposed to use them in released packages.
The all and all-* extras were not always working correctly when installing Airflow using constraints
because they were also considered as development-only dependencies. With this change, those dependencies are
now properly handling constraints and they will install properly with constraints, pulling the right set
of providers and dependencies when constraints are used.
The graphviz dependency has been problematic as Airflow required dependency - especially for
ARM-based installations. Graphviz packages require binary graphviz libraries - which is already a
limitation, but they also require to install graphviz Python bindings to be build and installed.
This does not work for older Linux installation but - more importantly - when you try to install
Graphviz libraries for Python 3.8, 3.9 for ARM M1 MacBooks, the packages fail to install because
Python bindings compilation for M1 can only work for Python 3.10+.
This is not a breaking change technically - the CLIs to render the DAGs is still there and IF you already have graphviz installed, it will continue working as it did before. The only problem when it does not work is where you do not have graphviz installed it will raise an error and inform that you need it.
Graphviz will remain to be installed for most users:
The only change will be a new installation of new version of Airflow from the scratch, where graphviz will need to be specified as extra or installed separately in order to enable DAG rendering option.
taskinstance list (#36693)AUTH_ROLE_PUBLIC=admin (#36750)op subtypes (#35536)typing.Union in _infer_multiple_outputs for Python 3.10+ (#36728)multiple_outputs is inferred correctly even when using TypedDict (#36652)Dagrun.update_state (#36712)EventsTimetable schedule past events if catchup=False (#36134)tis_query in _process_executor_events (#36655)call_regular_interval (#36608)DagRun fails while running dag test (#36517)_manage_executor_state by refreshing TIs in batch (#36502)MAX_CONTENT_LENGTH (#36401)kubernetes decorator type annotation consistent with operator (#36405)api/dag/*/dagrun from anonymous user (#36275)DAG.is_fixed_time_schedule (#36370)httpx import in file_task_handler for performance (#36753)pyarrow-hotfix for CVE-2023-47248 (#36697)graphviz dependency optional (#36647)pandas dependency to 1.2.5 for all providers and airflow (#36698)/airflow/www (#36700)docker decorator type annotations (#36406)batch_is_authorized_dag to check if user has permission to read DAGs (#36279)numpy example with practical exercise demonstrating top-level code (#35097)dags.rst with information on DAG pausing (#36540)metrics.rst for param dagrun.schedule_delay (#36404)Nothing published for this version
Use pyarrow-hotfix to mitigate CVE-2023-47248
Raw HTML code in DAG docs and DAG params descriptions is disabled by default
To ensure that no malicious javascript can be injected with DAG descriptions or trigger UI forms by DAG authors
a new parameter webserver.allow_raw_html_descriptions was added with default value of False.
If you trust your DAG authors code and want to allow using raw HTML in DAG descriptions and params, you can restore the previous
behavior by setting the configuration value to True.
To ensure Airflow is secure by default, the raw HTML support in trigger UI has been super-seeded by markdown support via
the description_md attribute. If you have been using description_html please migrate to description_md.
The custom_html_form is now deprecated. (#35460)
prev_end_date_success method access (#34528)List Task Instances view (#34529)clear_number to track DAG run being cleared (#34126)multiselect to run state in grid view (#35403)Connection.get_hook in case of ImportError (#36005)taskinstance (#35810)AIRFLOW_CONFIG path (#35818)JSON-string connection representation generator (#35723)BaseOperatorLink into the separate module (#35032)cbreak in execute_interactive and handle SIGINT (#35602)synchronize_log_template function (#35366)BaseOperatorLink.operators (#35003)SA2-compatible syntax for TaskReschedule (#33720)EventScheduler (#34808)update_forward_refs (#34657)Dataset from airflow package in codebase (#34610)airflow.datasets.Dataset in examples and tests (#34605)version top-level element from docker compose files (#33831)NOT EXISTS subquery instead of tuple_not_in_condition (#33527)triggerer_heartbeat (#33320)airflow variables export to print to stdout (#33279)reset_user_sessions to work from either CLI or web (#36056)overscroll behaviour to auto (#35717)borderWidthRight to grid for Firefox scrollbar (#35346)processor_subdir in serialized_dag table (#35661)get_dag_by_pickle util function (#35339)mappedoperator (#35257)Literal from typing_extensions (#33794)4.3.10 (#35991)Connection.to_json_dict to Connection.to_dict (#35894)moto version to >= 4.2.9 (#35687)pyarrow-hotfix to mitigate CVE-2023-47248 (#35650)axios from 0.26.0 to 1.6.0 in /airflow/www/ (#35624)navbar_text_color and rm condition in style (#35553)dag_next_execution (#35539)TCH004 and TCH005 rules (#35475)AirflowException from airflow (#34541)postcss from 8.4.25 to 8.4.31 in /airflow/www (#34770)airflow.models.dag.DAG in examples (#34617)re2 regex engine in the .airflowignore documentation. (#35663)best-practices.rst (#35692)dag-run.rst to mention Airflow's support for extended cron syntax through croniter (#35342)webserver.rst to include information of supported OAuth2 providers (#35237)rst code block format (#34708)Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Update dags.rst to put SubDag deprecation note right after the SubDag section heading
No significant changes.
codemirror and extra (#35122)get_plugin_info for class based listeners. (#35022)all_skipped trigger rule as skipped if any task is in upstream_failed state (#34392)pendulum requirement to <3.0 (#35336)sentry_sdk to 1.33.0 (#35298)@babel/traverse from 7.16.0 to 7.23.2 in /airflow/www (#34988)undici from 5.19.1 to 5.26.3 in /airflow/www (#34971)SchedulerJobRunner (#34810)max_tis per query > parallelism (#34742)connexion<3.0 upper bound (#35218)< 3.12 (#35123)3.1.0 (#34943)conn.extras (#35165)mysql-connector-python from recommended MySQL driver (#34287)set_downstream example (#35075)airflow_local_settings.py template (#34826)'>' in provider section name (#34813)Nothing published for this version
Deprecate numeric type python version in PythonVirtualEnvOperator
No significant changes
taskgroup is mapped (#34587)cluster_activity view not loading due to standaloneDagProcessor templating (#34274)loglevel=DEBUG in 'Not syncing DAG-level permissions' (#34268)access_control={} (#34114)ab_user table in the CLI session (#34120)next_run_datasets_summary endpoint (#34143)_run_task_session in mapped render_template_fields (#33309)version_added (#34011)AUTH_REMOTE_USER from FAB in WSGI middleware example (#34721)astroid version < 3 (#34658)os.path.splitext to Path.* (#34352, #33669)pyproject.toml (#34014)isinstance in fab_security manager (#33760)isinstance calls for the same object in a single call (#33767)str.splitlines() to split lines (#33592)len() (#33454)Nothing published for this version
When setting catchup=False, CronTriggerTimetable no longer skips a run if the scheduler does not query the timetable immediately after the previous ru
When setting catchup=False, CronTriggerTimetable no longer skips a run if
the scheduler does not query the timetable immediately after the previous run
has been triggered.
This should not affect scheduling in most cases, but can change the behaviour if
a DAG is paused-unpaused to manually skip a run. Previously, the timetable (with
catchup=False) would only start a run after a DAG is unpaused, but with this
change, the scheduler would try to look at little bit back to schedule the
previous run that covers a part of the period when the DAG was paused. This
means you will need to keep a DAG paused longer (namely, for the entire cron
period to pass) to really skip a run.
Note that this is also the behaviour exhibited by various other cron-based
scheduling tools, such as anacron.
conf.set() becomes case insensitive to match conf.get() behavior (#33452)Also, conf.get() will now break if used with non-string parameters.
conf.set(section, key, value) used to be case sensitive, i.e. conf.set("SECTION", "KEY", value)
and conf.set("section", "key", value) were stored as two distinct configurations.
This was inconsistent with the behavior of conf.get(section, key), which was always converting the section and key to lower case.
As a result, configuration options set with upper case characters in the section or key were unreachable.
That's why we are now converting section and key to lower case in conf.set too.
We also changed a bit the behavior of conf.get(). It used to allow objects that are not strings in the section or key.
Doing this will now result in an exception. For instance, conf.get("section", 123) needs to be replaced with conf.get("section", "123").
MappedTaskGroup tasks not respecting upstream dependency (#33732)SECURITY_MANAGER_CLASS should be a reference to class, not a string (#33690)get_url_for_login in security manager (#33660)2.7.0 db migration job errors (#33652)groupby in TIS duration calculation (#33535)dialect.name in custom SA types (#33503)end_date is less than utcnow (#33488)formatDuration method (#33486)conf.set case insensitive (#33452)soft_fail argument when poke is called (#33401)processor_subdir (#33357)<br> text in Provider's view (#33326)soft_fail argument when ExternalTaskSensor runs in deferrable mode (#33196)expand_kwargs method (#32272)Pydantic 1 compatibility (#34081, #33998)Pydantic 2 (#33956)devel_only extra in Airflow's setup.py (#33907)FAB to 4.3.4 in order to fix issues with filters (#33931)sqlalchemy to 1.4.24 (#33892)OrderedDict with plain dict (#33508)Pydantic warning about orm_mode rename (#33220)Pydantic limitation for version < 2 (#33507)Nothing published for this version
Nothing published for this version
Add deprecation info to the Airflow modules and classes docstring
As of now, Python 3.7 is no longer supported by the Python community. Therefore, to use Airflow 2.7.0, you must ensure your Python version is either 3.8, 3.9, 3.10, or 3.11.
The old Graph View is removed. The new Graph View is the default view now.
If you are using dag_run.conf dictionary and web UI JSON entry to run your DAG you should either:
Add params to your DAG <https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/params.html#use-params-to-provide-a-trigger-ui-form>_show_trigger_form_if_no_params to bring back old behaviourInstead, you should use "airflow db migrate" command to create or upgrade database. This command will not create default connections. In order to create default connections you need to run "airflow connections create-default-connections" explicitly, after running "airflow db migrate".
The "default" context is Python's default_ssl_contest instead of previously used "none". The
default_ssl_context provides a balance between security and compatibility but in some cases,
when certificates are old, self-signed or misconfigured, it might not work. This can be configured
by setting "ssl_context" in "email" configuration of Airflow.
Setting it to "none" brings back the "none" setting that was used in Airflow 2.6 and before, but it is not recommended due to security reasons ad this setting disables validation of certificates and allows MITM attacks.
For security reasons, the test connection functionality is disabled by default across Airflow UI,
API and CLI. The availability of the functionality can be controlled by the
test_connection flag in the core section of the Airflow
configuration (airflow.cfg). It can also be controlled by the
environment variable AIRFLOW__CORE__TEST_CONNECTION.
The following values are accepted for this config param:
Disabled: Disables the test connection functionality and
disables the Test Connection button in the UI.This is also the default value set in the Airflow configuration.
2. Enabled: Enables the test connection functionality and
activates the Test Connection button in the UI.
Hidden: Disables the test connection functionality and
hides the Test Connection button in UI.For more information on capabilities of users, see the documentation: https://airflow.apache.org/docs/apache-airflow/stable/security/security_model.html#capabilities-of-authenticated-ui-users It is strongly advised to not enable the feature until you make sure that only highly trusted UI/API users have "edit connection" permissions.
xcomEntries API disables support for the deserialize flag by default (#32176)For security reasons, the /dags/*/dagRuns/*/taskInstances/*/xcomEntries/*
API endpoint now disables the deserialize option to deserialize arbitrary
XCom values in the webserver. For backward compatibility, server admins may set
the [api] enable_xcom_deserialize_support config to True to enable the
flag and restore backward compatibility.
However, it is strongly advised to not enable the feature, and perform deserialization at the client side instead.
Default name of the Celery application changed from airflow.executors.celery_executor to airflow.providers.celery.executors.celery_executor.
You should change both your configuration and Health check command to use the new name:
celery_app_name configuration in celery section) use airflow.providers.celery.executors.celery_executorairflow.providers.celery.executors.celery_executor.appscheduler.max_tis_per_query is changed from 512 to 16 (#32572)This change is expected to make the Scheduler more responsive.
scheduler.max_tis_per_query needs to be lower than core.parallelism.
If both were left to their default value previously, the effective default value of scheduler.max_tis_per_query was 32
(because it was capped at core.parallelism).
To keep the behavior as close as possible to the old config, one can set scheduler.max_tis_per_query = 0,
in which case it'll always use the value of core.parallelism.
In order to use the executors, you need to install the providers:
apache-airflow-providers-celery package >= 3.3.0apache-airflow-providers-cncf-kubernetes package >= 7.4.0apache-airflow-providers-daskexecutor package in any versionYou can achieve it also by installing airflow with [celery], [cncf.kubernetes], [daskexecutor] extras respectively.
Users who base their images on the apache/airflow reference image (not slim) should be unaffected - the base
reference image comes with all the three providers installed.
This index seems to have great positive effect in a setup with tens of millions such rows.
BranchExternalPythonOperator (#32787, #33360)Per-LocalTaskJob Configuration (#32313)AirflowClusterPolicySkipDag exception (#32013)reactflow for datasets graph (#31775)chain which doesn't require matched lists (#31927)--retry and --retry-delay to airflow db check (#31836)section query param in get config rest API (#30936)Scheduled->Queued->Running task state transition times (#30612)db upgrade to db migrate and add connections create-default-connections (#32810, #33136)<= parallelism (#32572)isdisjoint instead of not intersection (#32616)dag_processor status. (#32382)[triggers.running] (#32050)TriggerDagRunOperator: Add wait_for_completion to template_fields (#31122)PythonVirtualenvOperator termination log in alert (#31747)airflow db commands to SQLAlchemy 2.0 style (#31486)validators into their own modules (#30802)get_log api (#30729)Gantt chart: Use earliest/oldest ti dates if different than dag run start/end (#33215)virtualenv detection for Python virtualenv operator (#33223)chmod airflow.cfg (#33118)max_active_runs reached its upper limit. (#31414)get_task_instances query (#33054)$ref (#32887)PythonOperator sub-classes extend its decorator (#32845)virtualenv is installed in PythonVirtualenvOperator (#32939)__iter__ in is_container() (#32850)dagRunTimeout (#32565)/blocked endpoint (#32571)cli.dags.trigger command output (#32548)whitespaces from airflow connections form (#32292)readonly property in our API (#32510)resizer wouldn't expanse grid view (#31581)type_ arg to drop_constraint (#31306)drop_constraint call in migrations (#31302)requirepass redis sentinel (#30352)/config (#31057)dag_processing (#33161)Pydantic to < 2.0.0 (#33235)cncf.kubernetes provider (#32767, #32891)pydocstyle check - core Airflow only (#31297)1.2.3 to 1.2.4 in /airflow/www (#32680)6.3.0 to 6.3.1 in /airflow/www (#32506)4.18.0 (#32445)stylelint from 13.13.1 to 15.10.1 in /airflow/www (#32435)4.0.0 to 4.1.3 in /airflow/www (#32443)Pydantic 2 (#32366)enums (#31735)0.272 (#31966)asynctest (#31664)2.0 style (#31569, #31772, #32350, #32339, #32474, #32645)3.7 support (#30963)0.0.262 (#30809)1.2.0 (#30687)DAGRun / DAG / Task in templates-ref.rst (#33013)Nothing published for this version
Nothing published for this version
Nothing published for this version
Use linear time regular expressions
DagRun.run_id and allow flexibility (#32293)executor_class from Job - fixing backfill for custom executors (#32219)mapIndex to display extra links per mapped task. (#32154)re2 for matching untrusted regex (#32060)dag_dependencies in serialized dag (#32037)None if an XComArg fails to resolve in a multiple_outputs Task (#32027)rendered-templates when map index is not found (#32011)ExternalTaskSensor when there is no task group TIs for the current execution date (#32009)operator_extra_links property serialization in mapped tasks (#31904)xcom_pull and inlets (#31128)on_failure_callback is not invoked when task failed during testing dag. (#30965)ExternalPythonOperator and debug logging level (#30367)task.sensor annotation in type stub (#31954)Pydantic to < 2.0.0 until we solve 2.0.0 incompatibilities (#32312)Pydantic 2 pickiness about model definition (#32307)™ to Airflow in prominent places (#31977)Nothing published for this version
Cascade update of TaskInstance to TaskMap table
map_index to the xcom key when skipping downstream tasks (#31541)/health endpoint (#31529)max_active_tis_per_dagrun for Dynamic Task Mapping (#31406)url_for_asset fallback and 404 on DAG Audit Log (#31233)default_args in nested task groups (#31608)[secrets] backend_kwargs as a sensitive config (#31788)Nothing published for this version
Nothing published for this version
In the past SchedulerJob and other *Job classes are known to have been used to perform external health checks for Airflow components. Those are, howev
Job classes (#31277).In the past SchedulerJob and other *Job classes are known to have been used to perform
external health checks for Airflow components. Those are, however, Airflow DB ORM related classes.
The DB models and database structure of Airflow are considered as internal implementation detail, following
public interface <https://airflow.apache.org/docs/apache-airflow/stable/public-airflow-interface.html>_).
Therefore, they should not be used for external health checks. Instead, you should use the
airflow jobs check CLI command (introduced in Airflow 2.1) for that purpose.
job_type column (#31182)api_client_retry_configuration (#31174)interleave_timestamp_parser config to the logging section (#31102)MappedTaskGroup import in taskinstance file (#31100)apache-hive extra so it installs the correct package (#31068)airflow providers get command output (#30978)pandas.DataFrame (#30943)order_by request in list DAG rest api (#30926)state and start_date from being reset when clearing a task in a running DagRun (#30125)dag_processing.processes metric (#30891)config directory in docker compose (#30662)version_added config field for might_contain_dag and metrics_allow_list (#30969)Nothing published for this version
Nothing published for this version
Nothing published for this version
This is potentially a breaking change for any custom trigger implementations that override the cleanup() method and uses synchronous code, however usi…
Default setting handles case where impersonation is needed and both users (airflow and the impersonated user)
have the same group set as main group. Previously the default was also other-writeable and the user might choose
to use the other-writeable setting if they wish by configuring file_task_handler_new_folder_permissions
and file_task_handler_new_file_permissions in logging section.
This stops SLA callbacks from keeping the dag processor manager permanently busy. It means reduced CPU, and fixes issues where SLAs stop the system from seeing changes to existing dag files. Additional metrics added to help track queue state.
cleanup() method in BaseTrigger is now defined as asynchronous (following async/await) pattern (#30152).This is potentially a breaking change for any custom trigger implementations that override the cleanup()
method and uses synchronous code, however using synchronous operations in cleanup was technically wrong,
because the method was executed in the main loop of the Triggerer and it was introducing unnecessary delays
impacting other triggers. The change is unlikely to affect any existing trigger implementations.
scheduler.tasks.running no longer exist (#30374)The gauge has never been working and its value has always been 0. Having an accurate value for this metric is complex so it has been decided that removing this gauge makes more sense than fixing it with no certainty of the correctness of its value.
task_queued_timeout config (#30375)Logic for handling tasks stuck in the queued state has been consolidated, and the all configurations
responsible for timing out stuck queued tasks have been deprecated and merged into
[scheduler] task_queued_timeout. The configurations that have been deprecated are
[kubernetes] worker_pods_pending_timeout, [celery] stalled_task_timeout, and
[celery] task_adoption_timeout. If any of these configurations are set, the longest timeout will be
respected. For example, if [celery] stalled_task_timeout is 1200, and [scheduler] task_queued_timeout
is 600, Airflow will set [scheduler] task_queued_timeout to 1200.
The configurations view now only displays the running configuration. Previously, the default configuration
was displayed at the top but it was not obvious whether this default configuration was overridden or not.
Subsequently, the non-documented endpoint /configuration?raw=true is deprecated and will be removed in
Airflow 3.0. The HTTP response now returns an additional Deprecation header. The /config endpoint on
the REST API is the standard way to fetch Airflow configuration programmatically.
ExternalTaskSensor now has an explicit skipped_states list
Maximum retry task delay is set to be 24h (86400s) by default. You can change it globally via core.max_task_retry_delay
parameter.
The Hive Macros (hive.max_partition, hive.closest_ds_partition) are available only when Hive Provider is
installed. Please install Hive Provider > 5.1.0 when using those macros.
max_active_tis_per_dagrun for Dynamic Task Mapping (#29094)TriggerDagRunOperator (#30292)Blocklist to disable specific metric tags or metric names (#29881)check_migrations config (#29714)cli.dags.trigger (#29224)db export-archived command. (#29485)airflow db drop-archived command (#29309)FileTrigger (#29265)connections import CLI command (#28738)AIP-51 <https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-51+milestone%3A%22Airflow+2.6.0%22>_)UX in grid view (#30373)select() to new style (#30515)metrics_*_list (#30174)on_*_callback/sla_miss_callbacks (#28469)renamed and previous_name in config sections (#28324)triggerer status (#27755)too old resource version exception by retrieving the latest resource_version (#30425)TriggerDagRunOperator with deferrable parameter (#30406)example_sensor_decorator DAG (#30513)skip_exit_code in BashOperator (#30734)scheduler.tasks.running (#30374)/airflow/www (#30568)/airflow/www (#30319)/airflow/www (#30316)dag.fileloc instead of dag.full_filepath in exception message (#30610)importlib-metadata backport to < 5.0.0 (#29924)importlib.metadata to get Version for speed (#29723)db export-cleaned to db export-archived (#29450)freezegun with time-machine (#28193)airflow/kubernetes/* (#28212)audit_logs.rst (#30405)*_lookup_pattern parameters (#29580)Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Fix DagProcessorJob integration for standalone dag-processor
No significant changes.
dag.partial_subset doesn't mutate task group properties (#30129)airflow dags next-execution cli command (#30117)TriggerRuleDep when the mapped tasks count is 0 (#30084)Nothing published for this version
Nothing published for this version
…time and no timezone was specified) but we raise deprecation warning.
In case of API calls, it was possible that "+" passed as part of the date-time fields were not URL-encoded, and
such date-time fields could pass validation. Such date-time parameters should now be URL-encoded (as %2B).
In case of parameters, we still allow IS8601-compliant date-time (so for example it is possible that
' ' was used instead of T separating date from time and no timezone was specified) but we raise
deprecation warning.
[webserver] expose_hostname changed to False (#29547)The default for [webserver] expose_hostname has been set to False, instead of True. This means administrators must opt-in to expose webserver hostnames to end users.
/dagRuns API should 404 if dag not active (#29860)openapi spec responses by adding additional return type (#29600)prev_logical_date variable offset-aware (#29454)Edgemodifier refactoring w/ labels in TaskGroup edge case (#29410)airflow connections add (#28922)undici from 5.9.1 to 5.19.1 (#29583)v67.2.0 (#29465)ua-parser-js from 0.7.31 to 0.7.33 in /airflow/www (#29172)pytest (#29086)run_id url param when linking to graph/gantt views (#29066)python_callable (#28932)swagger-ui-dist from 3.52.0 to 4.1.3 in /airflow/www (#28824)importlib-metadata backport to < 5.0.0 (#29924, #30069)merge_data() task (#29158)notes param from TriggerDagRunOperator docstring (#29298)schedule param rather than timetable in Timetables docs (#29255)Nothing published for this version
Your coding agent can read these notes before it upgrades. Set up the MCP server →