NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
PyPI, #4038 most downloaded on PyPI
Provider package apache-airflow-providers-openlineage for Apache Airflow
Latest stable
2.21.0
Last release 14 days ago
01 Nov 2026
Ships fairly regularly
a new release about every 2 weeks
Nearly every release is documented
notes for 60 of the last 60 stable releases
Nothing withdrawn
no release was ever pulled
3 years old
132 releases · first in 2023
Drag across the ruler to compare, or tap a release to open it.
Deprecate use of core get_kube_client in PodManager
TaskRunner: notify of component start and finish (#27855)AirflowModelViews(Variables/Connection) (#24079, #27994, #27923)Is /not Null filter for value is None on webui (#26584)one_done trigger rule (#26146)TI.xcom_pull() with explicit task_ids and map_indexes (#27699)urlsplit (#27389)branch_task_ids into SkipMixin (#27434).first() to .scalar() (#27323)howtos about sensors (#27333)extra__conn_type__ prefix required for UI behaviors (#26995)crashloopbackoff when using hostname_callable (#24999)__future__.annotations automatically by isort (#26383)V1Pod in task callback (#27609)taskInstance errors and split into two tables (#26575)autoregistered DAGs if there are any import errors (#26398)from airflow import version lazy import (#26239)is_mapped attribute (#27881)airflow/callbacks/* airflow/cli/* (#27721)airflow/api_connexion/* directory (#27718)airflow/listener/* directory (#27731)airflow/lineage/* directory (#27732)airflow/api/* directory (#27716)minimatch from 3.0.4 to 3.0.8 in /airflow/www (#27688)1.4.1 to 1.4.2 in /airflow/www (#27697)BaseTrigger.run (#27416)2022.10.1 (#27383)memray files to gitignore / dockerignore (#27001)sphinx-autoapi (#26743)RTIF.delete_old_records() (#26667)pyupgrade edge cases (#26384)docker-compose.yaml (#26726)Your coding agent can read these notes before it upgrades. Set up the MCP server →
In order to make airflow dags test more useful as a testing and debugging tool, we no longer run a backfill job and instead run a "local task runner". Users can still backfill their DAGs using the airflow dags backfill command.
KubernetesPodOperator no longer considers any core kubernetes config params, so this section now only applies to kubernetes executor. Renaming it reduces potential for confusion.
ExternalTaskSensor no longer hangs indefinitely when failed_states is set, an execute_date_fn is used, and some but not all of the dependent tasks fail. Instead, an AirflowException is thrown as soon as any of the dependent tasks fail. Any code handling this failure in addition to timeouts should move to caching the AirflowException BaseClass and not only the AirflowSensorTimeout subclass.
The old option will continue to work but will issue deprecation warnings, and will be removed entirely in Airflow 3.
TaskRunner: notify of component start and finish (#27855)
Add DagRun state change to the Listener plugin system(#27113)
Metric for raw task return codes (#27155)
Add logic for XComArg to pull specific map indexes (#27771)
Clear TaskGroup (#26658, #28003)
Add critical section query duration metric (#27700)
Add: #23880 :: Audit log for AirflowModelViews(Variables/Connection) (#24079, #27994, #27923)
Add postgres 15 support (#27444)
Expand tasks in mapped group at run time (#27491)
reset commits, clean submodules (#27560)
scheduler_job, add metric for scheduler loop timer (#27605)
Allow datasets to be used in taskflow (#27540)
Add expanded_ti_count to ti context (#27680)
Add user comment to task instance and dag run (#26457, #27849, #27867)
Enable copying DagRun JSON to clipboard (#27639)
Implement extra controls for SLAs (#27557)
add dag parsed time in DAG view (#27573)
Add max_wait for exponential_backoff in BaseSensor (#27597)
Expand tasks in mapped group at parse time (#27158)
Add disable retry flag on backfill (#23829)
Adding sensor decorator (#22562)
Api endpoint update ti (#26165)
Filtering datasets by recent update events (#26942)
Support Is /not Null filter for value is None on webui (#26584)
Add search to datasets list (#26893)
Split out and handle 'params' in mapped operator (#26100)
Add authoring API for TaskGroup mapping (#26844)
Add one_done trigger rule (#26146)
Create a more efficient airflow dag test command that also has better local logging (#26400)
Support add/remove permissions to roles commands (#26338)
Auto tail file logs in Web UI (#26169)
Add triggerer info to task instance in API (#26249)
Flag to deserialize value on custom XCom backend (#26343)
Allow depth-first execution (#27827)
UI: Update offset height if data changes (#27865)
Improve TriggerRuleDep typing and readability (#27810)
Make views requiring session, keyword only args (#27790)
Optimize TI.xcom_pull() with explicit task_ids and map_indexes (#27699)
Allow hyphens in pod id used by k8s executor (#27737)
optimise task instances filtering (#27102)
Use context managers to simplify log serve management (#27756)
Fix formatting leftovers (#27750)
Improve task deadlock messaging (#27734)
Improve "sensor timeout" messaging (#27733)
Replace urlparse with urlsplit (#27389)
Align TaskGroup semantics to AbstractOperator (#27723)
Add new files to parsing queue on every loop of dag processing (#27060)
Make Kubernetes Executor & Scheduler resilient to error during PMH execution (#27611)
Separate dataset deps into individual graphs (#27356)
Use log.exception where more economical than log.error (#27517)
Move validation branch_task_ids into SkipMixin (#27434)
Coerce LazyXComAccess to list when pushed to XCom (#27251)
Update cluster-policies.rst docs (#27362)
Add warning if connection type already registered within the provider (#27520)
Activate debug logging in commands with --verbose option (#27447)
Add classic examples for Python Operators (#27403)
change .first() to .scalar() (#27323)
Improve reset_dag_run description (#26755)
Add examples and howtos about sensors (#27333)
Make grid view widths adjustable (#27273)
Sorting plugins custom menu links by category before name (#27152)
Simplify DagRun.verify_integrity (#26894)
Add mapped task group info to serialization (#27027)
Correct the JSON style used for Run config in Grid View (#27119)
No extra__conn_type__ prefix required for UI behaviors (#26995)
Improve dataset update blurb (#26878)
Rename kubernetes config section to kubernetes_executor (#26873)
decode params for dataset searches (#26941)
Get rid of the DAGRun details page & rely completely on Grid (#26837)
Fix scheduler crashloopbackoff when using hostname_callable (#24999)
Reduce log verbosity in KubernetesExecutor. (#26582)
Don't iterate tis list twice for no reason (#26740)
Clearer code for PodGenerator.deserialize_model_file (#26641)
Don't import kubernetes unless you have a V1Pod (#26496)
Add updated_at column to DagRun and Ti tables (#26252)
Move the deserialization of custom XCom Backend to 2.4.0 (#26392)
Avoid calculating all elements when one item is needed (#26377)
Add __future__.annotations automatically by isort (#26383)
Handle list when serializing expand_kwargs (#26369)
Apply PEP-563 (Postponed Evaluation of Annotations) to core airflow (#26290)
Add more weekday operator and sensor examples #26071 (#26098)
Align TaskGroup semantics to AbstractOperator (#27723)
Gracefully handle whole config sections being renamed (#28008)
Add allow list for imports during deserialization (#27887)
Soft delete datasets that are no longer referenced in DAG schedules or task outlets (#27828)
Redirect to home view when there are no valid tags in the URL (#25715)
Refresh next run datasets info in dags view (#27839)
Make MappedTaskGroup depend on its expand inputs (#27876)
Make DagRun state updates for paused DAGs faster (#27725)
Don't explicitly set include_examples to False on task run command (#27813)
Fix menu border color (#27789)
Fix backfill queued task getting reset to scheduled state. (#23720)
Fix clearing child dag mapped tasks from parent dag (#27501)
Handle json encoding of V1Pod in task callback (#27609)
Fix ExternalTaskSensor can't check zipped dag (#27056)
Avoid re-fetching DAG run in TriggerDagRunOperator (#27635)
Continue on exception when retrieving metadata (#27665)
External task sensor fail fix (#27190)
Add the default None when pop actions (#27537)
Display parameter values from serialized dag in trigger dag view. (#27482, #27944)
Move TriggerDagRun conf check to execute (#27035)
Resolve trigger assignment race condition (#27072)
Update google_analytics.html (#27226)
Fix some bug in web ui dags list page (auto-refresh & jump search null state) (#27141)
Fixed broken URL for docker-compose.yaml (#26721)
Fix xcom arg.py .zip bug (#26636)
Fix 404 taskInstance errors and split into two tables (#26575)
Fix browser warning of improper thread usage (#26551)
template rendering issue fix (#26390)
Clear autoregistered DAGs if there are any import errors (#26398)
Fix from airflow import version lazy import (#26239)
allow scroll in triggered dag runs modal (#27965)
Remove is_mapped attribute (#27881)
Simplify FAB table resetting (#27869)
Fix old-style typing in Base Sensor (#27871)
Switch (back) to late imports (#27730)
Completed D400 for multiple folders (#27748)
simplify notes accordion test (#27757)
completed D400 for airflow/callbacks/* airflow/cli/* (#27721)
Completed D400 for airflow/api_connexion/* directory (#27718)
Completed D400 for airflow/listener/* directory (#27731)
Completed D400 for airflow/lineage/* directory (#27732)
Update API & Python Client versions (#27642)
Completed D400 & D401 for airflow/api/* directory (#27716)
Completed D400 for multiple folders (#27722)
Bump minimatch from 3.0.4 to 3.0.8 in /airflow/www (#27688)
Bump loader-utils from 1.4.1 to 1.4.2 ``in ``/airflow/www (#27697)
Disable nested task mapping for now (#27681)
bump alembic minimum version (#27629)
remove unused code.html (#27585)
Enable python string normalization everywhere (#27588)
Upgrade dependencies in order to avoid backtracking (#27531)
Strengthen a bit and clarify importance of triaging issues (#27262)
Deduplicate type hints (#27508)
Add stub 'yield' to BaseTrigger.run (#27416)
Remove upper-bound limit to dask (#27415)
Limit Dask to under 2022.10.1 (#27383)
Update old style typing (#26872)
Enable string normalization for docs (#27269)
Slightly faster up/downgrade tests (#26939)
Deprecate use of core get_kube_client in PodManager (#26848)
Add memray files to gitignore / dockerignore (#27001)
Bump sphinx and sphinx-autoapi (#26743)
Simplify RTIF.delete_old_records() (#26667)
migrate last react files to typescript (#26112)
Work around pyupgrade edge cases (#26384)
Document dag_file_processor_timeouts metric as deprecated (#27067)
Drop support for PostgreSQL 10 (#27594)
Update index.rst (#27529)
Add note about pushing the lazy XCom proxy to XCom (#27250)
Fix BaseOperator link (#27441)
[docs] best-practices add use variable with template example. (#27316)
docs for custom view using plugin (#27244)
Update graph view and grid view on overview page (#26909)
Documentation fixes (#26819)
make consistency on markup title string level (#26696)
Add documentation to dag test function (#26713)
Fix broken URL for docker-compose.yaml (#26726)
Add a note against use of top level code in timetable (#26649)
Fix example_datasets dag names (#26495)
Update docs: zip-like effect is now possible in task mapping (#26435)
changing to task decorator in docs from classic operator use (#25711)
Release Date: 2025-07-06
[OpenLineage] Added operator_provider_version to task event (#52468)
fix non existing 'ti.dag_run' access in openlineage provider (#51932)
Fix type import to AbstractOperator (#51773)
Move 'BaseHook' implementation to task SDK (#51873)
Disable UP038 ruff rule and revert mandatory 'X | Y' in insintance checks (#52644)
Add a bunch of no-redef ignores so Mypy is happy (#52507)
chore: use task_instance as source for all airflow identifiers used in listener (#52339)
Drop support for Python 3.9 (#52072)
Replace 'models.BaseOperator' to Task SDK one for Standard Provider (#52292)
Use BaseSensorOperator from task sdk in providers (#52296)
nit: bump openlineage libraries requirement to 1.34 (#52075)
Fixing ruff static check failures on main (#51937)
[OpenLineage] Added operator_provider_version to task event (#52468)
fix non existing 'ti.dag_run' access in openlineage provider (#51932)
Fix type import to AbstractOperator (#51773)
Move 'BaseHook' implementation to task SDK (#51873)
Disable UP038 ruff rule and revert mandatory 'X | Y' in insintance checks (#52644)
Add a bunch of no-redef ignores so Mypy is happy (#52507)
chore: use task_instance as source for all airflow identifiers used in listener (#52339)
Drop support for Python 3.9 (#52072)
Replace 'models.BaseOperator' to Task SDK one for Standard Provider (#52292)
Use BaseSensorOperator from task sdk in providers (#52296)
nit: bump openlineage libraries requirement to 1.34 (#52075)
Fixing ruff static check failures on main (#51937)
Nothing published for this version
Remove remaining deprecated classes and replace them with PEP562
@task.short_circuit TaskFlow decorator (#25752)execution_date_or_run_id optional in tasks test command (#26114)DagProcessors parsing files from different locations. (#25935)ExternalPythonOperator (#25780)dags test (#26111)expand_kwargs() against a literal list (#25925)output property to MappedOperator (#25604)TaskGroup in ExternalTaskSensor (#24902)@task.kubernetes taskflow decorator (#25663)schedule param (#25410)CronTriggerTimetable (#23662)XComArg.zip(*xcom_args) (#25176)sla_miss metric (#23402)map() semantic (#25085)expand_kwargs() (#24989)DagWarning model, and a check for missing pools (#23317)ExternalTaskMarker operator (#23964)SlaCallbackRequest if the DAG is scheduled (#26089)Operator.output more (#25617)hacky double-rendering prevention in mapped task (#25924)DAG(schedule=...) improvements (#25648)operator_name dupe in serialized JSON (#25819)task_group_to_grid (#25654)CronTriggerTimetable (#25503)/grid endpoint (#25481)@task.branch decorator (#25242)audit_log route (#25415)getTasks from Grid view (#25359)@dag arguments (#25044)SchedulerJob and TaskInstance (#24912)getfqdn with more resilient version (#24981)NBSP characters by whitespaces (#24797)airflow db upgrade (#24518)expand() error vague so it's not misleading (#24018)[webserver]analytics_tool (#24032)dttm searchable field in audit log (#23794)execute_in_subprocess (#23286)func.count to count rows (#23657)sql_alchemy_conn for celery result backend when result_backend is not set (#24496)AirflowDagCycleException message (#26204)TaskInstance.task not defined before handle_failure (#26040)DAGs and Datasets menu links based on role permission (#26183)TaskGroup tooltip via function docstring (#26028)DagRun.start_date not set during backfill with --reset-dagruns True (#26135)mapped_task is SKIPPED (#25995)TaskInstance related endpoints to avoid API errors (#26076)BranchDateTimeOperator to be timezone-awreness-insensitive (#25944)list-mapped-task-instance (#25965)iter_mapped_dependants (#25793)/ in variable get endpoint (#25774)XComArg fails to resolve (#25661)json arg help in airflow variables set command (#25726)ti_state on find_zombies query (#25725)axios errors (#25703)dag_owner_attributes table (#25579)expand_kwargs() input type before unmapping (#25355)ExternalTaskSensor not working with dynamic task (#25215)MappedOperator members in sync with BaseOperator (#24034)ERD schema for the MetaData DB (#26217)PEP562 (#26167)dag_edges and task_group_to_dict to corresponding util modules (#26212)ORM after forking to run a DAG processor (#26216)PEP-562 getattr (#26153)BaseSerialization.serialize "public" to other classes. (#26142)moment-timezone from 0.5.34 to 0.5.35 in /airflow/www (#26080)CamelCase to generated operations types (#25887)SkipMixin (#25864)undici from 5.8.0 to 5.9.1 in /airflow/www (#25801)_BranchPythonDecoratedOperator (#25783)filename_template deprecation message (#25749)ParamSpec to replace ... in Callable (#25658)elasticsearch test config to avoid warning on deprecated template (#25520)terser from 4.8.0 to 4.8.1 in /airflow/ui (#25178)typescript types from rest API docs (#25123)typescript (#25089)typescript. (#25096)ts (#25267)ts. (#25074)ti if it returns None (#25047)js file structure (#25003)moment from 2.29.3 to 2.29.4 in /airflow/www (#24885)xcom_push flag from BashOperator (#24824)typescript (#24715)charts.css to chart.css (#24531)grid.css to chart.css (#24529)set_new_process_group utility (#24371)moto version (#24222)[github_enterprise] from ref docs (#24033)f-strings in logging calls (#23597)JPype1 (#23847)Celery Flower (#25986)MySQL Charset problems (#25938)Trigger Rules (#25840)task_instance_mutation_hook usage (#25607)task-generated mapping example (#23424)2.2.0 (#24413)Release Date: 2025-06-18
feat: Add NominalTimeRunFacet to all OL events (#51404)
feat: Add TagsJobFacet to DAGRun OpenLineage events (#51303)
feat: Add Airflow-specific OL system tests validation and more tests (#51084)
feat: merge TimeDeltaSensorAsync to TimeDeltaSensor (#51133)
expose OpenLineage's lineage_root_* macros in plugin (#50532)
fix: Use task owner for TASK level Ownership facet (#51305)
Fix openlineage doc error (#51356)
Fix OpenLineage macro _get_logical_date (#51210)
Fix failing static checks (#51197)
Fix simple grammar mistakes in doc (#51138)
Fixes issue RuntimeTaskInstance does not contain log_url | added during taskrunner startup (#50376)
nit: task-level facets should not overwrite integration-level facets (#51690)
Make duration in 'List Dag Run' page sortable (#51495)
import MappedOperator from airflow.sdk.definitions.mappedoperator (#51492)
feat: Add NominalTimeRunFacet to all OL events (#51404)
feat: Add TagsJobFacet to DAGRun OpenLineage events (#51303)
feat: Add Airflow-specific OL system tests validation and more tests (#51084)
feat: merge TimeDeltaSensorAsync to TimeDeltaSensor (#51133)
expose OpenLineage's lineage_root_* macros in plugin (#50532)
fix: Use task owner for TASK level Ownership facet (#51305)
Fix openlineage doc error (#51356)
Fix OpenLineage macro _get_logical_date (#51210)
Fix failing static checks (#51197)
Fix simple grammar mistakes in doc (#51138)
Fixes issue RuntimeTaskInstance does not contain log_url | added during taskrunner startup (#50376)
nit: task-level facets should not overwrite integration-level facets (#51690)
Make duration in 'List Dag Run' page sortable (#51495)
import MappedOperator from airflow.sdk.definitions.mappedoperator (#51492)
Nothing published for this version
fix all "high" npm vulnerabilities
requirements.txt in Python Operators (#17349)SmoothOperator (#22813)execution_timeout configurable (#22389)ti_deps rules (#22698)dag-processor cli command (#22305)ignore_first_depends_on_past for scheduled jobs (#22491)list-import-errors to airflow dags command (#22084)standalone_dag_processor config is True. (#21731)celery.task_timeout_error metric (#21602)db downgrade cli command (#21596)ALL_SKIPPED trigger rule (#21662)db clean CLI command for purging old data (#20838)celery_logging_level (#21506)ShortCircuitOperator configurability for respecting downstream trigger rules (#20044)dag_id_pattern parameter to the /dags endpoint (#18924)--skip-init to db reset command (#22989).airflowignore files (#21392) (#22051)has_access decorator by ~200ms (#22858)try_number in airflow vars (#22297)airflow dags list CLI command (#21991)is_zipfiles in refresh_dag (#21926)airflow dag subcommands (#21793)py DAG files are zipped during parsing (#21538)run_id (#20975)get_connection() (#21162)worker_enable_remote_control for celery (#21507)CgroupTaskRunner (#21481)SchedulerJob (#21181)airflow by 6x (#21438)template_fields (#21054)ShortCircuitOperator push XCom by returning python_callable result (#20071)Connection.get_hook parameter to make it the same as in SqlSensor and SqlOperator. (#19849)XCOM value in PythonOperator (#19378)hook_params in BaseSqlOperator (#18718)taskinstance REST API schema to include dag_run_id field (#19105)extra to be nullable in connection payload as per schema(REST API). (#23183)dag_id extraction for dag level access checks in web ui (#23015)CLI command (#22688)db reset (#22990)external_task_ids. (#22809)DagParam to hold falsy values (#22964)get_logs_with_metadata endpoint. (#22898)PoolSlotsAvailableDep statues (#22807)email_on_failure with render_template_as_native_obj (#22770)DagFileProcessorManager (#22685)ScheduleInterval spec (#22635)is_smart_sensor_compatible() (#22386)run_id k8s and elasticsearch compatibility with Airflow 2.1 (#22385)except_skip None on BranchPythonOperator (#20411)RefreshConfiguration workaround for K8s token refreshing (#20759)/connections/<connection> endpoint (#22227)queued_dttm when submitting task to directly to executor (#22259)bulk_write_to_db (#21757)request.referrer from views.py (#21751)DbApiHook use get_uri from Connection (#21764)dag_id, task_id etc to XCom.serialize_value (#19505)on_task_instance_running listener can get at task (#21157)@task_group (#20870)airflow trigger cli (#20781)@task_group as a whole (#20671)start_date if a task continues after deferral (#20062)execution_date label when get cleanup pods list (#20417)class_permission_name to mixin so it applies to all classes (#18749)catchup=False is used in example dags (#19396)towncrier now (#22003)rbac = True from docs (#22725)sql_alchemy_connect_args (#22045)db upgrade / db downgrade (#21879)postgres:// with postgresql:// (#21205)CeleryExecutor docs (#20783)DummyOperator with EmptyOperator (#22974)DummyOperator in favor of EmptyOperator (#22832)moment from 2.29.1 to 2.29.2 in /airflow/www (#22873)prismjs from 1.26.0 to 1.27.0 in /airflow/www (#22823)nanoid from 3.1.23 to 3.3.2 in /airflow/www (#22803)minimist from 1.2.5 to 1.2.6 in /airflow/www (#22798)Airflow.Base.metadata in FAB models (#22353)sql_alchemy_conn if old postgres scheme used (#22333)xcom.dagrun_id to xcom.dag_run_id (#21806)conn.extra (#21816)jsonschema to 5.0 (#21712)days_ago (#21653):type lines now sphinx-autoapi supports type hints (#20951)DagRun.run_id instead of execution_date when updating state of TIs (UI & REST API) (#18724)PodLauncher to PodManager (#20576)get_readable_dags and get_editable_dags, and get_accessible_dags. (#19961)__var and __type members (#19303)max_tis_per_query initialisation on SchedulerJob (#19020)init_role() from API (#18820)Release Date: 2025-05-18
Note
This release of provider is only available for Airflow 2.10+ as explained in the Apache Airflow providers support policy < https://github.com/apache/airflow/blob/main/PROVIDERS.rst#minimum-supported-version-of-airflow-for-community-managed-providers >_.
add root parent information to OpenLineage events (#49237)
feat: Add DAG versioning information to OpenLineage events (#48741)
Improve execution time messages for DAG or Task not found (#49352)
Remove AIRFLOW_2_10_PLUS conditions (#49877)
Bump min Airflow version in providers to 2.10 (#49843)
nit: Remove duplicate warning when no OL metadata returned (#50350)
Note
This release of provider is only available for Airflow 2.10+ as explained in the Apache Airflow providers support policy <https://github.com/apache/airflow/blob/main/PROVIDERS.rst#minimum-supported-version-of-airflow-for-community-managed-providers>_.
add root parent information to OpenLineage events (#49237)
feat: Add DAG versioning information to OpenLineage events (#48741)
Improve execution time messages for DAG or Task not found (#49352)
Remove AIRFLOW_2_10_PLUS conditions (#49877)
Bump min Airflow version in providers to 2.10 (#49843)
nit: Remove duplicate warning when no OL metadata returned (#50350)
Nothing published for this version
Add deprecation notice for SubDagOperator
cwd for BashOperator (#17751)RESTARTING state (#16681)insert_args for support transfer replace (#15825)default_args for TaskGroup (#16557)kinit options [-f|-F] and [-a|-A] (#17816)DaskExecutor using Dask Worker Resources (#16829, #18720)processor_poll_interval to scheduler_idle_sleep_time (#18704)dagrun_conf (#18655)TaskInstanceModelView (#18438)Variable.update method and improving detection of variable key collisions (#18159)TaskInstance and TaskReschedule PK from execution_date to run_id (#17719)TaskGroup support in BaseOperator.chain() (#17456)template_ext attribute to show it in UI (#17985)robots.txt and X-Robots-Tag header (#17946)BranchDayOfWeekOperator, DayOfWeekSensor (#17940)none_failed_or_skipped by none_failed_min_one_success trigger rule (#17683)[core] store_dag_code & use DB to get Dag Code (#16342)task_concurrency to max_active_tis_per_dag (#17708)execution_date with run_id in airflow tasks run command (#16666)worker_log_server_port option to the logging section (#17621)SubDagOperator (#17488)template_fields_renderers (#17321)airflow celery stop to accept the pid file. (#17278)airflow_local_settings (#17195)AirflowException str when BashOperator fails. (#17151)SQLite or SequentialExecutor (#17133)init_containers defined in pod_override (#17537)airflow db init/upgrade migrations and setup in parallel. (#17078)chain() and cross_downstream() to support XComArgs (#16732)serve-logs and LocalExecutor (#16644)test_cycle to check_cycle (#16617)LocalExecutor (#16623)DbApiHook instance attribute (#16521, #17423)dag.sub_dag with dag.partial_subset (#16179)AirflowSensorTimeout as immediate failure without retrying (#12058)action_clear view (#15980)[core] dag_concurrency) settings for easier understanding (#16267, #18730)SKIPPED should not be logged again as SUCCESS (#14822)start_date for cleared tasks (#18708)AirflowDateTimePickerWidget a required field (#18602)retry_exponential_backoff divide by zero error when retry delay is zero (#17003){{ task.x }} attributes from within templates (#18516)sys.path (#18384)dag_tag rows that are now unused (#8231)wait_for_downstream dep (#18338)run_finished_callback for Debug Executor (#17983)XCom.get_one return full, not abbreviated values (#18274)XCom.set (#18240)_check_for_stalled_adopted_tasks method (#18208)StandardTaskRunner (#17967)self._error_file (#15947)DateTimeSensor (#17959)traceback.html (#17942)DagRunState enum query for MySQLdb driver (#17886)utf8mb3_general_ci collation for MySQL (#17729)TaskInstance does not work #17535 (#17548)DAG.cli() (#17105)None comparison in model_list template (#16893)cached_property module (#16710)Decimal (#16383)dag_id and empty subdir (#16513)dag.fileloc when using the @dag decorator (#16384)airflow/www/views.py (#15940)reschedule state (#17305, #18806)dagbag_size documentation (#18824)search_path set up instructions (#17600)AIRFLOW_GID from Docker images (#18747)sla_miss_callback section to the documentation (#18305)closer.lua script for downloading sources (#18179)DAG.is_active read-only in API (#17667)XCom.clear for data lifecycle management (#17589)pod_template_file (#16861)Jed and TP (#16671)flask-ouathlib to flask-oauthlib in Upgrading docs (#16320)Elasticsearch (#16275)dag_concurrency (#16177)default_pool slots (#15997)KubernetesExecutor git-sync pod template file (#15904)render_template_as_native_obj (#16534)POST to PATCH (#16511)BranchPythonOperator (#18623)boto3 to <1.19 (#18389)airflow.security.kerberos module (#18258)range(len()) to enumerate (#18174)main builds (#18035)tenacity (#17593)numpy dependency (#17594)mysql-connector-python to latest version (#17596)pandas an optional core dependency (#17575)airflow/utils/db.py (#17090)click to 8.x (#16779)dag.clear method (#16086)DAG_ACTIONS constant (#16232)_get_all_non_dag_permissions method (#16317)docutils to <0.17 until breaking behaviour is fixed (#16133)TaskInstance.log_filepath attribute (#15217)airflow/www/app.py (#15956)plyvel to google provider extra (#15812)find_permission_view_menu for get_permission wrapper (#16377)fab_logging_level to WARNING (#18783)CeleryKubernetesExecutor (#18441)Release Date: 2025-04-19
feat: Add support for task's manual state change notification in OpenLineage listener (#49040)
feat: Explicitly propagate airflow logging level to OL client (#49108)
feat: Add owner_links in DAG object in airflow facet (#49085)
gate import behind Airflow 2 path (#49209)
remove superfluous else block (#49199)
chore: Update requirement for openlineage client to >=1.31.0 (#49083)
feat: Add support for task's manual state change notification in OpenLineage listener (#49040)
feat: Explicitly propagate airflow logging level to OL client (#49108)
feat: Add owner_links in DAG object in airflow facet (#49085)
gate import behind Airflow 2 path (#49209)
remove superfluous else block (#49199)
chore: Update requirement for openlineage client to >=1.31.0 (#49083)
Nothing published for this version
Fix task retries when they receive sigkill and have retries and properly handle sigterm
sigkill and have retries and properly handle sigterm (#16301)default_impersonation config (#17229)kubernetes cleanup-pods which fails on invalid label key (#17298)TaskInstance does not show queued_by_job_id & external_executor_id (#17179)SecretsMasker is not configured (#17101)__init_subclass__ in subclasses of BaseOperator (#17027)priority_weight during parsing (#16765)deps` and task_group`` during DAG Serialization (#16734)AttributeError: datetime.timezone object has no attribute name (#16599)UserModelView controls. (#17431)secret_key mis-configured (#17410)execution_date in task_instance.refresh_from_db (#16809)LocalTaskJob during task exit (#16289)SQLAlchemy<1.4 constraint (#16630)dnspython (#16698)FlaskAppBuilder 3.3.2+ (#17208)LocalExecutor (#16623)DagFileProcessor and DagFileProcessorProcess out of scheduler_job.py (#16581)Release Date: 2025-04-14
Move ObjectStoragePath and attach to Task SDK (#48906)
Make '@task' import from airflow.sdk (#48896)
Move ObjectStoragePath and attach to Task SDK (#48906)
Make '@task' import from airflow.sdk (#48896)
Nothing published for this version
Only allow the webserver to request from the worker log server
CeleryKubernetesExecutor (#16700)token (#16474)LocalTaskJob (#16852)Release Date: 2025-04-09
fix: Adjust OpenLineage DefaultExtractor for RuntimeTaskInstance in Airflow 3 (#47673)
Stop trying to reconfigure the ORM in the OpenLineage workers (#47580)
fix: Re-add configuring orm for OpenLineage's listener on scheduler (#48049)
fix: remove use of get_inlet_defs and get_outlet_defs from OpenLineage (#48792)
Make datetime objects in Context as Pendulum objects (#48592)
fix: OpenLineage BaseExtractor's on_failure should call on_complete by default (#48456)
Fix OL tests due to decorators move to standard provider (#48808)
add OpenLineage configuration injection to SparkSubmitOperator (#47508)
feat: Add dagrun's end_date and duration to OL facet (#47901)
Use TaskInstance ID as FK in TaskReschedule instead of the multiple fields (#47459)
serialize http transports contained in composite transport (#47444)
Implement task-level "on" callbacks in sdk (#48002)
Calculate retry eligibility before task runs (#47996)
Implement triggering_asset_events in task sdk (#48650)
nit: log more details about OpenLineage exceptions being caught (#48459)
Add backcompat to openlineage provider method (#48406)
fix: Adjust OpenLineage DefaultExtractor for RuntimeTaskInstance in Airflow 3 (#47673)
Stop trying to reconfigure the ORM in the OpenLineage workers (#47580)
fix: Re-add configuring orm for OpenLineage's listener on scheduler (#48049)
fix: remove use of get_inlet_defs and get_outlet_defs from OpenLineage (#48792)
Make datetime objects in Context as Pendulum objects (#48592)
fix: OpenLineage BaseExtractor's on_failure should call on_complete by default (#48456)
Fix OL tests due to decorators move to standard provider (#48808)
add OpenLineage configuration injection to SparkSubmitOperator (#47508)
feat: Add dagrun's end_date and duration to OL facet (#47901)
Use TaskInstance ID as FK in TaskReschedule instead of the multiple fields (#47459)
serialize http transports contained in composite transport (#47444)
Implement task-level "on" callbacks in sdk (#48002)
Calculate retry eligibility before task runs (#47996)
Implement triggering_asset_events in task sdk (#48650)
nit: log more details about OpenLineage exceptions being caught (#48459)
Add backcompat to openlineage provider method (#48406)
Nothing published for this version
Nothing published for this version
Nothing published for this version
Fix normalize-url vulnerability
PyPI (#16594)dag_run.conf is a dict (#15057)only_active parameter to /dags endpoint (#14306)filebeat 7 (#14625)passphrase and private_key to default sensitive field names (#16392)airflow task run without --local/--raw for KubeExecutor (#16108)log.exception when there is no exception (#16047)/ (#16018)Release Date: 2025-03-13
fix: OpenLineage serialization of dataset timetables for Airflow 2.9 (#47150)
chore: Update description of 'execution_timeout' in OpenLineage provider.yaml (#47448)
Remove the old 'task run' commands and LocalTaskJob (#47453)
Move task_sdk to a standalone task-sdk distribution (#47451)
Move uuid6 to be devel dependency of openlineage (#47464)
revert removing 'external_trigger' from OpenLineage provider (#47383)
Implement stale dag bundle cleanup (#46503)
Replace 'external_trigger' check with DagRunType (#45961)
Runtime context shouldn't have start_date as a key (#46961)
Upgrade flit to 3.11.0 (#46938)
fix: OpenLineage serialization of dataset timetables for Airflow 2.9 (#47150)
chore: Update description of 'execution_timeout' in OpenLineage provider.yaml (#47448)
Remove the old 'task run' commands and LocalTaskJob (#47453)
Move task_sdk to a standalone task-sdk distribution (#47451)
Move uuid6 to be devel dependency of openlineage (#47464)
revert removing 'external_trigger' from OpenLineage provider (#47383)
Implement stale dag bundle cleanup (#46503)
Replace 'external_trigger' check with DagRunType (#45961)
Runtime context shouldn't have start_date as a key (#46961)
Upgrade flit to 3.11.0 (#46938)
Nothing published for this version
- Remove AirflowContextDeprecationWarning as all context should be clean for Airflow 3
Release Date: 2025-02-26
change listener API, add basic support for task instance listeners in TaskSDK, make OpenLineage provider support Airflow 3's listener interface (#45294)
feat: Add ProcessingEngineRunFacet to all OL events (#46283)
feat: automatically inject OL transport info into spark jobs (#45326)
feat: Add OpenLineage support for some SQL to GCS operators (#45242)
feat: automatically inject OL info into spark job in DataprocCreateBatchOperator (#44612)
Update OpenLineage emmiter to cope with nullable logical_date (#46722)
fix: OL sql parsing add try-except for sqlalchemy engine (#46366)
OpenLineage: Include 'AirflowDagRunFacet' in complete/failed events (#45615)
Adding uuid6 as a dependency for openlineage (#46653)
Remove AirflowContextDeprecationWarning as all context should be clean for Airflow 3 (#46601)
Remove Airflow 3 Deprecation Warning dependency in OTel Provider (#46600)
AIP-72: Move Secrets Masker to task SDK (#46375)
Add run_after column to DagRun model (#45732)
Remove old lineage stuff (#45260)
Start porting mapped task to SDK (#45627)
chore: Update docstring for DatabaseInfo in OpenLineage provider (#45638)
Remove classes from 'typing_compat' that can be imported directly (#45589)
updated 404 hyperlink to gcstogcsoperator (#45311)
pass error for on_task_instance_failed in task sdk (#46941)
change listener API, add basic support for task instance listeners in TaskSDK, make OpenLineage provider support Airflow 3's listener interface (#45294)
feat: Add ProcessingEngineRunFacet to all OL events (#46283)
feat: automatically inject OL transport info into spark jobs (#45326)
feat: Add OpenLineage support for some SQL to GCS operators (#45242)
feat: automatically inject OL info into spark job in DataprocCreateBatchOperator (#44612)
Update OpenLineage emmiter to cope with nullable logical_date (#46722)
fix: OL sql parsing add try-except for sqlalchemy engine (#46366)
OpenLineage: Include 'AirflowDagRunFacet' in complete/failed events (#45615)
Adding uuid6 as a dependency for openlineage (#46653)
Remove AirflowContextDeprecationWarning as all context should be clean for Airflow 3 (#46601)
Remove Airflow 3 Deprecation Warning dependency in OTel Provider (#46600)
AIP-72: Move Secrets Masker to task SDK (#46375)
Add run_after column to DagRun model (#45732)
Remove old lineage stuff (#45260)
Start porting mapped task to SDK (#45627)
chore: Update docstring for DatabaseInfo in OpenLineage provider (#45638)
Remove classes from 'typing_compat' that can be imported directly (#45589)
updated 404 hyperlink to gcstogcsoperator (#45311)
pass error for on_task_instance_failed in task sdk (#46941)
Bump stylelint to remove vulnerable sub-dependency
New Features """"""""""""
PythonVirtualenvDecorator to Taskflow API (#14761)Taskgroup decorator (#15034)SubprocessHook for running commands from operators (#13423)WeekDayBranchOperator (#13997)worker_pod_pending_timeout support (#15263)template_fields_renderers additions (#15130)AirflowSkipException on exit code 99 (by default, configurable) (#13421) (#14963)airflow jobs check CLI command to check health of jobs (Scheduler etc) (#14519)DateTimeBranchOperator to BranchDateTimeOperator (#14720)Improvements """"""""""""
DbApiHook (#15581)apply_default to subclasses of BaseOperator (#15667)KubernetesExecutor pod templates to allow access to IAM permissions (#15669)airflow db check-migrations (#15662)secret_key when Webserver > 1 (#15546)JSONFormatter (#15414)on_failure_callback when SIGTERM is received (#15172)worker_refresh_interval to 6000 seconds (#14970)[celery] default_queue config to [operators] default_queue to re-use between executors (#14699)Bug Fixes """""""""
updateTaskInstancesState API endpoint when dry_run not passed (#15889)drawDagStatsForDag in dags.html (#13884)NotPreviouslySkippedDep (#13933)KubernetesExecutor (#14795)KubernetesPodOperator (#15388)dag.partial_subset (#13700) (#15308)pod_id for KubernetesPodOperator (#15445)pod_id ends with hyphen in KubernetesPodOperator (#15443)pool_slots > 1 (#15426)sync-perm to work correctly when update_fab_perms = False (#14847)GCSObjectsWtihPrefixExistenceSensor (#14179)CeleryKubernetesExecutor bug (#13247)StackdriverTaskHandler (#13784)func.sum may return Decimal that break rest APIs (#15585)AlreadyExists exception when the execution_date is same (#15174)sync_metadata inside DagFileProcessorManager (#15121)docker-py update to resolve docker op issues (#15731)user_id from API schema (#15117)airflow info work with pipes (#14528)CollectionInfo in all Collections that have total_entries (#14366)task_instance_mutation_hook when importing airflow.models.dagrun (#15851)Doc only changes """"""""""""""""
markdownlint and yamllint config files (#15682)git_sync_template.yaml (#13197)Misc/Internal """""""""""""
logging.exception redundancy (#14823)stylelint to remove vulnerable sub-dependency (#15784)ssri from 6.0.1 to 6.0.2 in /airflow/www (#15437)datepicker for task instance detail view (#15284)tableau extra (#13595)cached_property on Python 3.8 where possible (#14606)flynt. (#13732)jquery ready instead of vanilla js (#15258)Webpack entries (#14551)We’ve tried to make as few breaking changes as possible and to provide deprecation path in the code, especially in the case of anything called in the…
The full changelog is about 3,000 lines long (already excluding everything backported to 1.10), so for now I’ll simply share some of the major features in 2.0.0 compared to 1.10.14:
(Known in 2.0.0alphas as Functional DAGs.)
DAGs are now much much nicer to author especially when using PythonOperator. Dependencies are handled more clearly and XCom is nicer to use
A quick teaser of what DAGs can now look like:
from airflow.decorators import dag, task
from airflow.utils.dates import days_ago
@dag(default_args={'owner': 'airflow'}, schedule_interval=None, start_date=days_ago(2))
def tutorial_taskflow_api_etl():
@task
def extract():
return {"1001": 301.27, "1002": 433.21, "1003": 502.22}
@task
def transform(order_data_dict: dict) -> dict:
total_order_value = 0
for value in order_data_dict.values():
total_order_value += value
return {"total_order_value": total_order_value}
@task()
def load(total_order_value: float):
print("Total order value is: %.2f" % total_order_value)
order_data = extract()
order_summary = transform(order_data)
load(order_summary["total_order_value"])
tutorial_etl_dag = tutorial_taskflow_api_etl()
We now have a fully supported, no-longer-experimental API with a comprehensive OpenAPI specification
Read more here:
REST API Documentation.
As part of AIP-15 (Scheduler HA+performance) and other work Kamil did, we significantly improved the performance of the Airflow Scheduler. It now starts tasks much, MUCH quicker.
Over at Astronomer.io we’ve benchmarked the scheduler—it’s fast (we had to triple check the numbers as we don’t quite believe them at first!)
It’s now possible and supported to run more than a single scheduler instance. This is super useful for both resiliency (in case a scheduler goes down) and scheduling performance.
To fully use this feature you need Postgres 9.6+ or MySQL 8+ (MySQL 5, and MariaDB won’t work with more than one scheduler I’m afraid).
There’s no config or other set up required to run more than one scheduler—just start up a scheduler somewhere else (ensuring it has access to the DAG files) and it will cooperate with your existing schedulers through the database.
For more information, read the Scheduler HA documentation.
SubDAGs were commonly used for grouping tasks in the UI, but they had many drawbacks in their execution behaviour (primarirly that they only executed a single task in parallel!) To improve this experience, we’ve introduced “Task Groups”: a method for organizing tasks which provides the same grouping behaviour as a subdag without any of the execution-time drawbacks.
SubDAGs will still work for now, but we think that any previous use of SubDAGs can now be replaced with task groups. If you find an example where this isn’t the case, please let us know by opening an issue on GitHub
For more information, check out the Task Group documentation.
We’ve given the Airflow UI a visual refresh and updated some of the styling. Check out the UI section of the docs for screenshots.
We have also added an option to auto-refresh task states in Graph View so you no longer need to continuously press the refresh button :).
If you make heavy use of sensors in your Airflow cluster, you might find that sensor execution takes up a significant proportion of your cluster even with “reschedule” mode. To improve this, we’ve added a new mode called “Smart Sensors”.
This feature is in “early-access”: it’s been well-tested by AirBnB and is “stable”/usable, but we reserve the right to make backwards-incompatible changes to it in a future release (if we have to. We’ll try very hard not to!)
For Airflow 2.0, we have re-architected the KubernetesExecutor in a fashion that is simultaneously faster, easier to understand, and more flexible for Airflow users. Users will now be able to access the full Kubernetes API to create a .yaml pod_template_file instead of specifying parameters in their airflow.cfg.
We have also replaced the executor_config dictionary with the pod_override parameter, which takes a Kubernetes V1Pod object for a 1:1 setting override. These changes have removed over three thousand lines of code from the KubernetesExecutor, which makes it run faster and creates fewer potential errors.
Airflow 2.0 is not a monolithic “one to rule them all” package. We’ve split Airflow into core and 61 (for now) provider packages. Each provider package is for either a particular external service (Google, Amazon, Microsoft, Snowflake), a database (Postgres, MySQL), or a protocol (HTTP/FTP). Now you can create a custom Airflow installation from “building” blocks and choose only what you need, plus add whatever other requirements you might have. Some of the common providers are installed automatically (ftp, http, imap, sqlite) as they are commonly used. Other providers are automatically installed when you choose appropriate extras when installing Airflow.
The provider architecture should make it much easier to get a fully customized, yet consistent runtime with the right set of Python dependencies.
But that’s not all: you can write your own custom providers and add things like custom connection types, customizations of the Connection Forms, and extra links to your operators in a manageable way. You can build your own provider and install it as a Python package and have your customizations visible right in the Airflow UI.
Security
As part of Airflow 2.0 effort, there has been a conscious focus on Security and reducing areas of exposure. This is represented across different functional areas in different forms. For example, in the new REST API, all operations now require authorization. Similarly, in the configuration settings, the Fernet key is now required to be specified.
Configuration in the form of the airflow.cfg file has been rationalized further in distinct sections, specifically around “core”. Additionally, a significant amount of configuration options have been deprecated or moved to individual component-specific configuration files, such as the pod-template-file for Kubernetes execution-related configuration.
We’ve tried to make as few breaking changes as possible and to provide deprecation path in the code, especially in the case of anything called in the DAG. That said, please read through UPDATING.md to check what might affect you. For example: We re-organized the layout of operators (they now all live under airflow.providers.*) but the old names should continue to work - you’ll just notice a lot of DeprecationWarnings that need to be fixed up.
Release Date: 2024-12-26
Note
This release of provider is only available for Airflow 2.9+ as explained in the Apache Airflow providers support policy .
Warning
All deprecated classes, parameters and features have been removed from the OpenLineage provider package. The following breaking changes were introduced:
Utils
Removed normalize_sql function from openlineage.utils module.
Remove Provider Deprecations in OpenLineage (#44636)
add clear_number to OpenLineage's dagrun-level event run id generation (#44617)
utilize more information to deterministically generate OpenLineage run_id (#43936)
feat: automatically inject OL info into spark job in DataprocSubmitJobOperator (#44477)
Remove references to AIRFLOW_V_2_9_PLUS (#44987)
Bump minimum Airflow version in providers to Airflow 2.9.0 (#44956)
Consistent way of checking Airflow version in providers (#44686)
add basic system tests for OpenLineage (#43643)
Move Asset user facing components to task_sdk (#43773)
Rename execution_date to logical_date across codebase (#43902)
Note
This release of provider is only available for Airflow 2.9+ as explained in the Apache Airflow providers support policy.
Warning
All deprecated classes, parameters and features have been removed from the OpenLineage provider package. The following breaking changes were introduced:
Utils
Removed normalize_sql function from openlineage.utils module.
Remove Provider Deprecations in OpenLineage (#44636)
add clear_number to OpenLineage's dagrun-level event run id generation (#44617)
utilize more information to deterministically generate OpenLineage run_id (#43936)
feat: automatically inject OL info into spark job in DataprocSubmitJobOperator (#44477)
Remove references to AIRFLOW_V_2_9_PLUS (#44987)
Bump minimum Airflow version in providers to Airflow 2.9.0 (#44956)
Consistent way of checking Airflow version in providers (#44686)
add basic system tests for OpenLineage (#43643)
Move Asset user facing components to task_sdk (#43773)
Rename execution_date to logical_date across codebase (#43902)
Nothing published for this version
Nothing published for this version
- Add support for semicolon stripping to DbApiHook, PrestoHook, and TrinoHook
Release Date: 2024-11-18
Add support for semicolon stripping to DbApiHook, PrestoHook, and TrinoHook (#41916)
Add ProcessingEngineRunFacet to OL DAG Start event (#43213)
serialize asset/dataset timetable conditions in OpenLineage info also for Airflow 2 (#43434)
OpenLineage: accept whole config when instantiating OpenLineageClient. (#43740)
Temporarily limit openlineage to <1.24.0 (#43732)
Move python operator to Standard provider (#42081)
Nothing published for this version
- feat(providers/openlineage): Use asset in common provider
Release Date: 2024-10-31
feat(providers/openlineage): Use asset in common provider (#43111)
Ignore attr-defined for compat import (#43301)
nit: remove taskgroup's tooltip from OL's AirflowJobFacet (#43152)
require 1.2.1 common.compat for openlineage provider (#43039)
feat(providers/openlineage): Use asset in common provider (#43111)
Ignore attr-defined for compat import (#43301)
nit: remove taskgroup's tooltip from OL's AirflowJobFacet (#43152)
require 1.2.1 common.compat for openlineage provider (#43039)
Nothing published for this version
- Change imports to use Standard provider for BashOperator
Release Date: 2024-10-14
Change imports to use Standard provider for BashOperator (#42252)
Drop python3.8 support core and providers (#42766)
Rename dataset related python variable names to asset (#41348)
Change imports to use Standard provider for BashOperator (#42252)
Drop python3.8 support core and providers (#42766)
Rename dataset related python variable names to asset (#41348)
Nothing published for this version
- fix: OpenLineage dag start event not being emitted
Release Date: 2024-10-01
fix: OpenLineage dag start event not being emitted (#42448)
fix: typo in error stack trace formatting for clearer output (#42017)
Nothing published for this version
- feat: notify about potential serialization failures when sending DagRun, don't serialize unnecessary params, guard listener for exceptions
Release Date: 2024-09-24
feat: notify about potential serialization failures when sending DagRun, don't serialize unnecessary params, guard listener for exceptions (#41690)
fix: cast list to flattened string in openlineage InfoJsonEncodable (#41786)
chore: bump OL provider dependencies versions (#42059)
move to dag_run.logical_date from execution date in OpenLineage provider (#41889)
Unify DAG schedule args and change default to None (#41453)
feat: notify about potential serialization failures when sending DagRun, don't serialize unnecessary params, guard listener for exceptions (#41690)
fix: cast list to flattened string in openlineage InfoJsonEncodable (#41786)
chore: bump OL provider dependencies versions (#42059)
move to dag_run.logical_date from execution date in OpenLineage provider (#41889)
Unify DAG schedule args and change default to None (#41453)
Nothing published for this version
- feat: remove openlineage client deprecated from_environment() method
Release Date: 2024-08-28
Note
This release of provider is only available for Airflow 2.8+ as explained in the Apache Airflow providers support policy .
feat: add debug facet to all OpenLineage events (#41217)
feat: add fileloc to DAG info in AirflowRunFacet (#41311)
feat: remove openlineage client deprecated from_environment() method (#41310)
feat: openlineage listener captures hook-level lineage (#41482)
fix: get task dependencies without serializing task tree to string (#41494)
fix: return empty data instead of None when OpenLineage on_start method is missing (#41268)
fix: replace dagTree with downstream_task_ids (#41587)
Bump minimum Airflow version in providers to Airflow 2.8.0 (#41396)
chore: remove openlineage deprecation warnings (#41284)
Note
This release of provider is only available for Airflow 2.8+ as explained in the Apache Airflow providers support policy.
feat: add debug facet to all OpenLineage events (#41217)
feat: add fileloc to DAG info in AirflowRunFacet (#41311)
feat: remove openlineage client deprecated from_environment() method (#41310)
feat: openlineage listener captures hook-level lineage (#41482)
fix: get task dependencies without serializing task tree to string (#41494)
fix: return empty data instead of None when OpenLineage on_start method is missing (#41268)
fix: replace dagTree with downstream_task_ids (#41587)
Bump minimum Airflow version in providers to Airflow 2.8.0 (#41396)
chore: remove openlineage deprecation warnings (#41284)
Nothing published for this version
Nothing published for this version
- openlineage: remove deprecated parentRun facet key
Release Date: 2024-08-06
Add AirflowRun on COMPLETE/FAIL events (#40996)
openlineage: extend custom_run_facets to also be executed on complete and fail (#40953)
openlineage: migrate OpenLineage provider to V2 facets. (#39530)
openlineage: Add AirflowRunFacet for dag runEvents (#40854)
[AIP-62] Translate AIP-60 URI to OpenLineage (#40173)
Ability to add custom facet in OpenLineage events (#38982)
openlineage: add method to common.compat to not force hooks to try/except every 2.10 hook lineage call (#40812)
openlineage: use airflow provided getters from conf (#40790)
openlineage: add config to include 'full' task info based on conf setting (#40589)
Add TaskInstance log_url to OpenLineage facet (#40797)
openlineage: add deferrable information to task info in airflow run facet (#40682)
Adjust default extractor's on_failure detection for airflow 2.10 fix (#41094)
openlineage: make value of slots in attrs.define consistent across all OL usages (#40992)
Set 'slots' to True for facets used in DagRun (#40972)
openlineage: fix / add some task attributes in AirflowRunFacet (#40725)
openlineage: replace dt.now with airflow.utils.timezone.utcnow (#40887)
openlineage: remove deprecated parentRun facet key (#40681)
Nothing published for this version
- fix openlineage parsing dag tree with MappedOperator
Release Date: 2024-07-12
fix openlineage parsing dag tree with MappedOperator (#40621)
Nothing published for this version
- fix: sqa deprecations for airflow providers
Release Date: 2024-07-04
local task job: add timeout, to not kill on_task_instance_success listener prematurely (#39890)
openlineage: add some debug logging around sql parser call sites (#40200)
Add task SLA and queued datetime information to AirflowRunFacet (#40091)
Add error stacktrace to OpenLineage task event (#39813)
Introduce AirflowJobFacet and AirflowStateRunFacet (#39520)
Use UUIDv7 for OpenLineage runIds (#39889)
openlineage: execute extraction and message sending in separate process (#40078)
Add few removed Task properties in AirflowRunFacet (#40371)
openlineage, redshift: do not call DB for schemas below Airflow 2.10 (#40197)
fix: scheduler crashing with OL provider on airflow standalone (#40459)
nit: fix logging level (#40461)
fix: provide stack trace under proper key in OL facet (#40372)
fix: sqa deprecations for airflow providers (#39293)
Enable enforcing pydocstyle rule D213 in ruff. (#40448)
local task job: add timeout, to not kill on_task_instance_success listener prematurely (#39890)
openlineage: add some debug logging around sql parser call sites (#40200)
Add task SLA and queued datetime information to AirflowRunFacet (#40091)
Add error stacktrace to OpenLineage task event (#39813)
Introduce AirflowJobFacet and AirflowStateRunFacet (#39520)
Use UUIDv7 for OpenLineage runIds (#39889)
openlineage: execute extraction and message sending in separate process (#40078)
Add few removed Task properties in AirflowRunFacet (#40371)
openlineage, redshift: do not call DB for schemas below Airflow 2.10 (#40197)
fix: scheduler crashing with OL provider on airflow standalone (#40459)
nit: fix logging level (#40461)
fix: provide stack trace under proper key in OL facet (#40372)
fix: sqa deprecations for airflow providers (#39293)
Enable enforcing pydocstyle rule D213 in ruff. (#40448)
Nothing published for this version
Nothing published for this version
In Airflow 2.10.0, we fix the way try_number works. For Airflow >= 2.10.0, use apache-airflow-providers-openlineage >= 1.8.0 . Older versions of Airfl
Release Date: 2024-05-30
Warning
In Airflow 2.10.0, we fix the way try_number works. For Airflow >= 2.10.0, use apache-airflow-providers-openlineage >= 1.8.0 . Older versions of Airflow are not affected, In case you run an incompatible version an exception will be raised asking you to upgrade provider version.
Scheduler to handle incrementing of try_number (#39336)
fix: Prevent error when extractor can't be imported (#39736)
Re-configure ORM in spawned OpenLineage process in scheduler. (#39735)
chore: Update conf retrieval docstring and adjust pool_size (#39721)
Remove 'openlineage.common' dependencies in Google and Snowflake providers. (#39614)
Use 'ProcessPoolExecutor' over 'ThreadPoolExecutor'. (#39235)
misc: Add custom provider runtime checks (#39609)
Faster 'airflow_version' imports (#39552)
Simplify 'airflow_version' imports (#39497)
openlineage: notify that logged exception was caught (#39493)
chore: Add more OpenLineage logs to facilitate debugging (#39136)
Warning
In Airflow 2.10.0, we fix the way try_number works. For Airflow >= 2.10.0, use apache-airflow-providers-openlineage >= 1.8.0. Older versions of Airflow are not affected, In case you run an incompatible version an exception will be raised asking you to upgrade provider version.
Scheduler to handle incrementing of try_number (#39336)
fix: Prevent error when extractor can't be imported (#39736)
Re-configure ORM in spawned OpenLineage process in scheduler. (#39735)
chore: Update conf retrieval docstring and adjust pool_size (#39721)
Remove 'openlineage.common' dependencies in Google and Snowflake providers. (#39614)
Use 'ProcessPoolExecutor' over 'ThreadPoolExecutor'. (#39235)
misc: Add custom provider runtime checks (#39609)
Faster 'airflow_version' imports (#39552)
Simplify 'airflow_version' imports (#39497)
openlineage: notify that logged exception was caught (#39493)
chore: Add more OpenLineage logs to facilitate debugging (#39136)
Nothing published for this version
- openlineage, snowflake: do not run external queries for Snowflake
Release Date: 2024-05-06
openlineage, snowflake: do not run external queries for Snowflake (#39113)
Nothing published for this version
for the record, generated with: ~/node_modules/github-changes/bin/index.js -o airbnb -r airflow --only-pulls --use-commit-body --between-tags 1.6.2...
Raw changelog
for the record, generated with:
~/node_modules/github-changes/bin/index.js -o airbnb -r airflow --only-pulls --use-commit-body --between-tags 1.6.2...1.7.0rc1 --token {your_gh_token_here}
Session.remove after each run, to survive DB restart (@KMK-ONLINE)Release Date: 2024-04-13
Add lineage_job_namespace and lineage_job_name OpenLineage macros (#38829)
openlineage: add 'opt-in' option (#37725)
fix: Remove redundant operator information from facets (#38264)
fix: disabled_for_operators now stops whole event emission (#38033)
fix: Add fallbacks when retrieving Airflow configuration to avoid errors being raised (#37994)
fix: Fix parent id macro and remove unused utils (#37877)
Avoid use of 'assert' outside of the tests (#37718)
Add default for 'task' on TaskInstance / fix attrs on TaskInstancePydantic (#37854)
Add lineage_job_namespace and lineage_job_name OpenLineage macros (#38829)
openlineage: add 'opt-in' option (#37725)
fix: Remove redundant operator information from facets (#38264)
fix: disabled_for_operators now stops whole event emission (#38033)
fix: Add fallbacks when retrieving Airflow configuration to avoid errors being raised (#37994)
fix: Fix parent id macro and remove unused utils (#37877)
Avoid use of 'assert' outside of the tests (#37718)
Add default for 'task' on TaskInstance / fix attrs on TaskInstancePydantic (#37854)
Nothing published for this version
[scheduler] the notion of DAG runs allows for more parallelization, and controls around scheduling (max number of running task instance per DAG, max n
v1.6.0 brings:
Thanks to everyone in the community for all the PRs (stellar contributions!), comments and issue reporting.
Release Date: 2024-03-08
feat: Add OpenLineage metrics for event_size and extraction time (#37797)
feat: Add OpenLineage support for File and User Airflow's lineage entities (#37744)
[OpenLineage] Add support for JobTypeJobFacet properties. (#37255)
chore: Update comments and logging in OpenLineage ExtractorManager (#37622)
fix: Check if operator is disabled in DefaultExtractor.extract_on_complete (#37392)
docs: Update whole OpenLineage Provider docs. (#37620)
feat: Add OpenLineage metrics for event_size and extraction time (#37797)
feat: Add OpenLineage support for File and User Airflow's lineage entities (#37744)
[OpenLineage] Add support for JobTypeJobFacet properties. (#37255)
chore: Update comments and logging in OpenLineage ExtractorManager (#37622)
fix: Check if operator is disabled in DefaultExtractor.extract_on_complete (#37392)
docs: Update whole OpenLineage Provider docs. (#37620)
Nothing published for this version
v1.5.0 is a huge release. Tons of important features.
v1.5.0 is a huge release. Tons of important features.
Make sure to run airflow upgradedb after you upgrade
Improvements:
airflow upgradedb to get your database up to date as you upgrade AirflowRelease Date: 2024-01-27
feat: Add dag_id when generating OpenLineage run_id for task instance. (#36659)
feat: Add dag_id when generating OpenLineage run_id for task instance. (#36659)
Nothing published for this version
Nothing published for this version
Python3 compatibility improvements
Release Date: 2024-01-10
Add OpenLineage support for Redshift SQL. (#35794)
Nothing published for this version
Release Date: 2023-12-27 ### Bug Fixes - Fix typo.
Release Date: 2023-12-27
Fix typo. (#36362)
Nothing published for this version
- feat: Add parent_run_id for COMPLETE and FAIL events
Release Date: 2023-12-12
feat: Add parent_run_id for COMPLETE and FAIL events (#36067)
Add basic metrics to stats collector. (#35368)
fix: Repair run_id for OpenLineage FAIL events (#36051)
Fix and reapply templates for provider documentation (#35686)
Remove ClassVar annotations. (#36084)
feat: Add parent_run_id for COMPLETE and FAIL events (#36067)
Add basic metrics to stats collector. (#35368)
fix: Repair run_id for OpenLineage FAIL events (#36051)
Fix and reapply templates for provider documentation (#35686)
Remove ClassVar annotations. (#36084)
Nothing published for this version
- Make schema filter uppercase in 'create_filter_clauses'
Release Date: 2023-11-12
Make schema filter uppercase in 'create_filter_clauses' (#35428)
Make schema filter uppercase in 'create_filter_clauses' (#35428)
Nothing published for this version
One column per quarter.