New Features

• Add Data-aware Scheduling (https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-48+milestone%3A%22Airflow+2.4.0%22)
• Add @task.short_circuit TaskFlow decorator (#25752)
• Make execution_date_or_run_id optional in tasks test command (#26114)
• Automatically register DAGs that are used in a context manager (#23592, #26398)
• Add option of sending DAG parser logs to stdout. (#25754)
• Support multiple DagProcessors parsing files from different locations. (#25935)
• Implement ExternalPythonOperator (#25780)
• Make execution_date optional for command dags test (#26111)
• Implement expand_kwargs() against a literal list (#25925)
• Add trigger rule tooltip (#26043)
• Add conf parameter to CLI for airflow dags test (#25900)
• Include scheduled slots in pools view (#26006)
• Add output property to MappedOperator (#25604)
• Add roles delete command to cli (#25854)
• Add Airflow specific warning classes (#25799)
• Add support for TaskGroup in ExternalTaskSensor (#24902)
• Add @task.kubernetes taskflow decorator (#25663)
• Add a way to import Airflow without side-effects (#25832)
• Let timetables control generated run_ids. (#25795)
• Allow per-timetable ordering override in grid view (#25633)
• Grid logs for mapped instances (#25610, #25621, #25611)
• Consolidate to one schedule param (#25410)
• DAG regex flag in backfill command (#23870)
• Adding support for owner links in the Dags view UI (#25280)
• Ability to clear a specific DAG Run's task instances via REST API (#23516)
• Possibility to document DAG with a separate markdown file (#25509)
• Add parsing context to DAG Parsing (#25161)
• Implement CronTriggerTimetable (#23662)
• Add option to mask sensitive data in UI configuration page (#25346)
• Create new databases from the ORM (#24156)
• Implement XComArg.zip(*xcom_args) (#25176)
• Introduce sla_miss metric (#23402)
• Implement map() semantic (#25085)
• Add override method to TaskGroupDecorator (#25160)
• Implement expand_kwargs() (#24989)
• Add parameter to turn off SQL query logging (#24570)
• Add DagWarning model, and a check for missing pools (#23317)
• Add Task Logs to Grid details panel (#24249)
• Added small health check server and endpoint in scheduler(#23905)
• Add built-in External Link for ExternalTaskMarker operator (#23964)
• Add default task retry delay config (#23861)
• Add clear DagRun endpoint. (#23451)
• Add support for timezone as string in cron interval timetable (#23279)
• Add auto-refresh to dags home page (#22900, #24770)

Improvements

• Add more weekday operator and sensor examples #26071 (#26098)
• Add subdir parameter to dags reserialize command (#26170)
• Update zombie message to be more descriptive (#26141)
• Only send an SlaCallbackRequest if the DAG is scheduled (#26089)
• Promote Operator.output more (#25617)
• Upgrade API files to typescript (#25098)
• Less hacky double-rendering prevention in mapped task (#25924)
• Improve Audit log (#25856)
• Remove mapped operator validation code (#25870)
• More DAG(schedule=...) improvements (#25648)
• Reduce operator_name dupe in serialized JSON (#25819)
• Make grid view group/mapped summary UI more consistent (#25723)
• Remove useless statement in task_group_to_grid (#25654)
• Add optional data interval to CronTriggerTimetable (#25503)
• Remove unused code in /grid endpoint (#25481)
• Add and document description fields (#25370)
• Improve Airflow logging for operator Jinja template processing (#25452)
• Update core example DAGs to use @task.branch decorator (#25242)
• Update DAG audit_log route (#25415)
• Change stdout and stderr access mode to append in commands (#25253)
• Remove getTasks from Grid view (#25359)
• Improve taskflow type hints with ParamSpec (#25173)
• Use tables in grid details panes (#25258)
• Explicitly list @dag arguments (#25044)
• More typing in SchedulerJob and TaskInstance (#24912)
• Patch getfqdn with more resilient version (#24981)
• Replace all NBSP characters by whitespaces (#24797)
• Re-serialize all DAGs on airflow db upgrade (#24518)
• Rework contract of try_adopt_task_instances method (#23188)
• Make expand() error vague so it's not misleading (#24018)
• Add enum validation for [webserver]analytics_tool (#24032)
• Add dttm searchable field in audit log (#23794)
• Allow more parameters to be piped through via execute_in_subprocess (#23286)
• Use func.count to count rows (#23657)
• Remove stale serialized dags (#22917)
• AIP45 Remove dag parsing in airflow run local (#21877)
• Add support for queued state in DagRun update endpoint. (#23481)
• Add fields to dagrun endpoint (#23440)
• Use sql_alchemy_conn for celery result backend when result_backend is not set (#24496)

Bug Fixes

• Have consistent types between the ORM and the migration files (#24044, #25869)
• Disallow any dag tags longer than 100 char (#25196)
• Add the dag_id to AirflowDagCycleException message (#26204)
• Properly build URL to retrieve logs independently from system (#26337)
• For worker log servers only bind to IPV6 when dual stack is available (#26222)
• Fix TaskInstance.task not defined before handle_failure (#26040)
• Undo secrets backend config caching (#26223)
• Fix faulty executor config serialization logic (#26191)
• Show DAGs and Datasets menu links based on role permission (#26183)
• Allow setting TaskGroup tooltip via function docstring (#26028)
• Fix RecursionError on graph view of a DAG with many tasks (#26175)
• Fix backfill occasional deadlocking (#26161)
• Fix DagRun.start_date not set during backfill with --reset-dagruns True (#26135)
• Use label instead of id for dynamic task labels in graph (#26108)
• Don't fail DagRun when leaf mapped_task is SKIPPED (#25995)
• Add group prefix to decorated mapped task (#26081)
• Fix UI flash when triggering with dup logical date (#26094)
• Fix Make items nullable for TaskInstance related endpoints to avoid API errors (#26076)
• Fix BranchDateTimeOperator to be timezone-awreness-insensitive (#25944)
• Fix legacy timetable schedule interval params (#25999)
• Fix response schema for list-mapped-task-instance (#25965)
• Properly check the existence of missing mapped TIs (#25788)
• Fix broken auto-refresh on grid view (#25950)
• Use per-timetable ordering in grid UI (#25880)
• Rewrite recursion when parsing DAG into iteration (#25898)
• Find cross-group tasks in iter_mapped_dependants (#25793)
• Fail task if mapping upstream fails (#25757)
• Support / in variable get endpoint (#25774)
• Use cfg default_wrap value for grid logs (#25731)
• Add origin request args when triggering a run (#25729)
• Operator name separate from class (#22834)
• Fix incorrect data interval alignment due to assumption on input time alignment (#22658)
• Return None if an XComArg fails to resolve (#25661)
• Correct json arg help in airflow variables set command (#25726)
• Added MySQL index hint to use ti_state on find_zombies query (#25725)
• Only excluded actually expanded fields from render (#25599)
• Grid, fix toast for axios errors (#25703)
• Fix UI redirect (#26409)
• Require dag_id arg for dags list-runs (#26357)
• Check for queued states for dags auto-refresh (#25695)
• Fix upgrade code for the dag_owner_attributes table (#25579)
• Add map index to task logs api (#25568)
• Ensure that zombie tasks for dags with errors get cleaned up (#25550)
• Make extra link work in UI (#25500)
• Sync up plugin API schema and definition (#25524)
• First/last names can be empty (#25476)
• Refactor DAG pages to be consistent (#25402)
• Check expand_kwargs() input type before unmapping (#25355)
• Filter XCOM by key when calculating map lengths (#24530)
• Fix ExternalTaskSensor not working with dynamic task (#25215)
• Added exception catching to send default email if template file raises any exception (#24943)
• Bring MappedOperator members in sync with BaseOperator (#24034)

Misc/Internal

• Add automatically generated ERD schema for the MetaData DB (#26217)
• Mark serialization functions as internal (#26193)
• Remove remaining deprecated classes and replace them with PEP562 (#26167)
• Move dag_edges and task_group_to_dict to corresponding util modules (#26212)
• Lazily import many modules to improve import speed (#24486, #26239)
• FIX Incorrect typing information (#26077)
• Add missing contrib classes to deprecated dictionaries (#26179)
• Re-configure/connect the ORM after forking to run a DAG processor (#26216)
• Remove cattrs from lineage processing. (#26134)
• Removed deprecated contrib files and replace them with PEP-562 getattr (#26153)
• Make BaseSerialization.serialize "public" to other classes. (#26142)
• Change the template to use human readable task_instance description (#25960)
• Bump moment-timezone from 0.5.34 to 0.5.35 in /airflow/www (#26080)
• Fix Flask deprecation warning (#25753)
• Add CamelCase to generated operations types (#25887)
• Fix migration issues and tighten the CI upgrade/downgrade test (#25869)
• Fix type annotations in SkipMixin (#25864)
• Workaround setuptools editable packages path issue (#25848)
• Bump undici from 5.8.0 to 5.9.1 in /airflow/www (#25801)
• Add custom_operator_name attr to _BranchPythonDecoratedOperator (#25783)
• Clarify filename_template deprecation message (#25749)
• Use ParamSpec to replace ... in Callable (#25658)
• Remove deprecated modules (#25543)
• Documentation on task mapping additions (#24489)
• Remove Smart Sensors (#25507)
• Fix elasticsearch test config to avoid warning on deprecated template (#25520)
• Bump terser from 4.8.0 to 4.8.1 in /airflow/ui (#25178)
• Generate typescript types from rest API docs (#25123)
• Upgrade utils files to typescript (#25089)
• Upgrade remaining context file to typescript. (#25096)
• Migrate files to ts (#25267)
• Upgrade grid Table component to ts. (#25074)
• Skip mapping against mapped ti if it returns None (#25047)
• Refactor js file structure (#25003)
• Move mapped kwargs introspection to separate type (#24971)
• Only assert stuff for mypy when type checking (#24937)
• Bump moment from 2.29.3 to 2.29.4 in /airflow/www (#24885)
• Remove "bad characters" from our codebase (#24841)
• Remove xcom_push flag from BashOperator (#24824)
• Move Flask hook registration to end of file (#24776)
• Upgrade more javascript files to typescript (#24715)
• Clean up task decorator type hints and docstrings (#24667)
• Preserve original order of providers' connection extra fields in UI (#24425)
• Rename charts.css to chart.css (#24531)
• Rename grid.css to chart.css (#24529)
• Misc: create new process group by set_new_process_group utility (#24371)
• Airflow UI fix Prototype Pollution (#24201)
• Bump moto version (#24222)
• Remove unused [github_enterprise] from ref docs (#24033)
• Clean up f-strings in logging calls (#23597)
• Add limit for JPype1 (#23847)
• Simply json responses (#25518)
• Add min attrs version (#26408)

Doc only changes

• Add url prefix setting for Celery Flower (#25986)
• Updating deprecated configuration in examples (#26037)
• Fix wrong link for taskflow tutorial (#26007)
• Reorganize tutorials into a section (#25890)
• Fix concept doc for dynamic task map (#26002)
• Update code examples from "classic" operators to taskflow (#25845, #25657)
• Add instructions on manually fixing MySQL Charset problems (#25938)
• Prefer the local Quick Start in docs (#25888)
• Fix broken link to Trigger Rules (#25840)
• Improve docker documentation (#25735)
• Correctly link to Dag parsing context in docs (#25722)
• Add note on task_instance_mutation_hook usage (#25607)
• Note that TaskFlow API automatically passes data between tasks (#25577)
• Update DAG run to clarify when a DAG actually runs (#25290)
• Update tutorial docs to include a definition of operators (#25012)
• Rewrite the Airflow documentation home page (#24795)
• Fix task-generated mapping example (#23424)
• Add note on subtle logical date change in 2.2.0 (#24413)
• Add missing import in best-practices code example (#25391)

New Features

• Add dynamic task mapping (https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-42+milestone%3A%22Airflow+2.3.0%22)
• New Grid View replaces Tree View (#18675)
• Templated requirements.txt in Python Operators (#17349)
• Allow reuse of decorated tasks (#22941)
• Move the database configuration to a new section (#22284)
• Add SmoothOperator (#22813)
• Make operator's execution_timeout configurable (#22389)
• Events Timetable (#22332)
• Support dag serialization with custom ti_deps rules (#22698)
• Support log download in task log view (#22804)
• support for continue backfill on failures (#22697)
• Add dag-processor cli command (#22305)
• Add possibility to create users in LDAP mode (#22619)
• Add ignore_first_depends_on_past for scheduled jobs (#22491)
• Update base sensor operator to support XCOM return value (#20656)
• Add an option for run id in the ui trigger screen (#21851)
• Enable JSON serialization for connections (#19857)
• Add REST API endpoint for bulk update of DAGs (#19758)
• Add queue button to click-on-DagRun interface. (#21555)
• Add list-import-errors to airflow dags command (#22084)
• Store callbacks in database if standalone_dag_processor config is True. (#21731)
• Add LocalKubernetesExecutor (#19729)
• Add celery.task_timeout_error metric (#21602)
• Airflow db downgrade cli command (#21596)
• Add ALL_SKIPPED trigger rule (#21662)
• Add db clean CLI command for purging old data (#20838)
• Add celery_logging_level (#21506)
• Support different timeout value for dag file parsing (#21501)
• Support generating SQL script for upgrades (#20962)
• Add option to compress Serialized dag data (#21332)
• Branch python operator decorator (#20860)
• Add Audit Log View to Dag View (#20733)
• Add missing StatsD metric for failing SLA Callback notification (#20924)
• Add ShortCircuitOperator configurability for respecting downstream trigger rules (#20044)
• Allow using Markup in page title in Webserver (#20888)
• Add Listener Plugin API that tracks TaskInstance state changes (#20443)
• Add context var hook to inject more env vars (#20361)
• Add a button to set all tasks to skipped (#20455)
• Cleanup pending pods (#20438)
• Add config to warn public deployment exposure in UI (#18557)
• Log filename template records (#20165)
• Added windows extensions (#16110)
• Showing approximate time until next dag_run in Airflow (#20273)
• Extend config window on UI (#20052)
• Add show dag dependencies feature to CLI (#19985)
• Add cli command for 'airflow dags reserialize` (#19471)
• Add missing description field to Pool schema(REST API) (#19841)
• Introduce DagRun action to change state to queued. (#19353)
• Add DAG run details page (#19705)
• Add role export/import to cli tools (#18916)
• Adding dag_id_pattern parameter to the /dags endpoint (#18924)

Improvements

• Show schedule_interval/timetable description in UI (#16931)
• Added column duration to DAG runs view (#19482)
• Enable use of custom conn extra fields without prefix (#22607)
• Initialize finished counter at zero (#23080)
• Improve logging of optional provider features messages (#23037)
• Meaningful error message in resolve_template_files (#23027)
• Update ImportError items instead of deleting and recreating them (#22928)
• Add option --skip-init to db reset command (#22989)
• Support importing connections from files with ".yml" extension (#22872)
• Support glob syntax in .airflowignore files (#21392) (#22051)
• Hide pagination when data is a single page (#22963)
• Support for sorting DAGs in the web UI (#22671)
• Speed up has_access decorator by ~200ms (#22858)
• Add XComArg to lazy-imported list of Airflow module (#22862)
• Add more fields to REST API dags/dag_id/details endpoint (#22756)
• Don't show irrelevant/duplicated/"internal" Task attrs in UI (#22812)
• No need to load whole ti in current_state (#22764)
• Pickle dag exception string fix (#22760)
• Better verification of Localexecutor's parallelism option (#22711)
• log backfill exceptions to sentry (#22704)
• retry commit on MySQL deadlocks during backfill (#22696)
• Add more fields to REST API get DAG(dags/dag_id) endpoint (#22637)
• Use timetable to generate planned days for current year (#22055)
• Disable connection pool for celery worker (#22493)
• Make date picker label visible in trigger dag view (#22379)
• Expose try_number in airflow vars (#22297)
• Add generic connection type (#22310)
• Add a few more fields to the taskinstance finished log message (#22262)
• Pause auto-refresh if scheduler isn't running (#22151)
• Show DagModel details. (#21868)
• Add pip_install_options to PythonVirtualenvOperator (#22158)
• Show import error for airflow dags list CLI command (#21991)
• Pause auto-refresh when page is hidden (#21904)
• Default args type check (#21809)
• Enhance magic methods on XComArg for UX (#21882)
• py files don't have to be checked is_zipfiles in refresh_dag (#21926)
• Fix TaskDecorator type hints (#21881)
• Add 'Show record' option for variables (#21342)
• Use DB where possible for quicker airflow dag subcommands (#21793)
• REST API: add rendered fields in task instance. (#21741)
• Change the default auth backend to session (#21640)
• Don't check if py DAG files are zipped during parsing (#21538)
• Switch XCom implementation to use run_id (#20975)
• Action log on Browse Views (#21569)
• Implement multiple API auth backends (#21472)
• Change logging level details of connection info in get_connection() (#21162)
• Support mssql in airflow db shell (#21511)
• Support config worker_enable_remote_control for celery (#21507)
• Log memory usage in CgroupTaskRunner (#21481)
• Modernize DAG-related URL routes and rename "tree" to "grid" (#20730)
• Move Zombie detection to SchedulerJob (#21181)
• Improve speed to run airflow by 6x (#21438)
• Add more SQL template fields renderers (#21237)
• Simplify fab has access lookup (#19294)
• Log context only for default method (#21244)
• Log trigger status only if at least one is running (#21191)
• Add optional features in providers. (#21074)
• Better multiple_outputs inferral for @task.python (#20800)
• Improve handling of string type and non-attribute template_fields (#21054)
• Remove un-needed deps/version requirements (#20979)
• Correctly specify overloads for TaskFlow API for type-hinting (#20933)
• Introduce notification_sent to SlaMiss view (#20923)
• Rewrite the task decorator as a composition (#20868)
• Add "Greater/Smaller than or Equal" to filters in the browse views (#20602) (#20798)
• Rewrite DAG run retrieval in task command (#20737)
• Speed up creation of DagRun for large DAGs (5k+ tasks) by 25-130% (#20722)
• Make native environment Airflow-flavored like sandbox (#20704)
• Better error when param value has unexpected type (#20648)
• Add filter by state in DagRun REST API (List Dag Runs) (#20485)
• Prevent exponential memory growth in Tasks with custom logging handler (#20541)
• Set default logger in logging Mixin (#20355)
• Reduce deprecation warnings from www (#20378)
• Add hour and minute to time format on x-axis of all charts using nvd3.lineChart (#20002)
• Add specific warning when Task asks for more slots than pool defined with (#20178)
• UI: Update duration column for better human readability (#20112)
• Use Viewer role as example public role (#19215)
• Properly implement DAG param dict copying (#20216)
• ShortCircuitOperator push XCom by returning python_callable result (#20071)
• Add clear logging to tasks killed due to a Dagrun timeout (#19950)
• Change log level for Zombie detection messages (#20204)
• Better confirmation prompts (#20183)
• Only execute TIs of running DagRuns (#20182)
• Check and run migration in commands if necessary (#18439)
• Log only when Zombies exists (#20118)
• Increase length of the email and username (#19932)
• Add more filtering options for TI's in the UI (#19910)
• Dynamically enable "Test Connection" button by connection type (#19792)
• Avoid littering postgres server logs with "could not obtain lock" with HA schedulers (#19842)
• Renamed Connection.get_hook parameter to make it the same as in SqlSensor and SqlOperator. (#19849)
• Add hook_params in SqlSensor using the latest changes from PR #18718. (#18431)
• Speed up webserver boot time by delaying provider initialization (#19709)
• Configurable logging of XCOM value in PythonOperator (#19378)
• Minimize production js files (#19658)
• Add hook_params in BaseSqlOperator (#18718)
• Add missing "end_date" to hash components (#19281)
• More friendly output of the airflow plugins command + add timetables (#19298)
• Add sensor default timeout config (#19119)
• Update taskinstance REST API schema to include dag_run_id field (#19105)
• Adding feature in bash operator to append the user defined env variable to system env variable (#18944)
• Duplicate Connection: Added logic to query if a connection id exists before creating one (#18161)

Bug Fixes

• Use inherited 'trigger_tasks' method (#23016)
• In DAG dependency detector, use class type instead of class name (#21706)
• Fix tasks being wrongly skipped by schedule_after_task_execution (#23181)
• Fix X-Frame enabled behaviour (#23222)
• Allow extra to be nullable in connection payload as per schema(REST API). (#23183)
• Fix dag_id extraction for dag level access checks in web ui (#23015)
• Fix timezone display for logs on UI (#23075)
• Include message in graph errors (#23021)
• Change trigger dropdown left position (#23013)
• Don't add planned tasks for legacy DAG runs (#23007)
• Add dangling rows check for TaskInstance references (#22924)
• Validate the input params in connection CLI command (#22688)
• Fix trigger event payload is not persisted in db (#22944)
• Drop "airflow moved" tables in command db reset (#22990)
• Add max width to task group tooltips (#22978)
• Add template support for external_task_ids. (#22809)
• Allow DagParam to hold falsy values (#22964)
• Fix regression in pool metrics (#22939)
• Priority order tasks even when using pools (#22483)
• Do not clear XCom when resuming from deferral (#22932)
• Handle invalid JSON metadata in get_logs_with_metadata endpoint. (#22898)
• Fix pre-upgrade check for rows dangling w.r.t. dag_run (#22850)
• Fixed backfill interference with scheduler (#22701)
• Support conf param override for backfill runs (#22837)
• Correctly interpolate pool name in PoolSlotsAvailableDep statues (#22807)
• Fix email_on_failure with render_template_as_native_obj (#22770)
• Fix processor cleanup on DagFileProcessorManager (#22685)
• Prevent meta name clash for task instances (#22783)
• remove json parse for gantt chart (#22780)
• Check for missing dagrun should know version (#22752)
• Fixes ScheduleInterval spec (#22635)
• Fixing task status for non-running and non-committed tasks (#22410)
• Do not log the hook connection details even at DEBUG level (#22627)
• Stop crashing when empty logs are received from kubernetes client (#22566)
• Fix bugs about timezone change (#22525)
• Fix entire DAG stops when one task has end_date (#20920)
• Use logger to print message during task execution. (#22488)
• Make sure finalizers are not skipped during exception handling (#22475)
• update smart sensor docs and minor fix on is_smart_sensor_compatible() (#22386)
• Fix run_id k8s and elasticsearch compatibility with Airflow 2.1 (#22385)
• Allow to except_skip None on BranchPythonOperator (#20411)
• Fix incorrect datetime details (DagRun views) (#21357)
• Remove incorrect deprecation warning in secrets backend (#22326)
• Remove RefreshConfiguration workaround for K8s token refreshing (#20759)
• Masking extras in GET /connections/<connection> endpoint (#22227)
• Set queued_dttm when submitting task to directly to executor (#22259)
• Addressed some issues in the tutorial mentioned in discussion #22233 (#22236)
• Change default python executable to python3 for docker decorator (#21973)
• Don't validate that Params are JSON when NOTSET (#22000)
• Add per-DAG delete permissions (#21938)
• Fix handling some None parameters in kubernetes 23 libs. (#21905)
• Fix handling of empty (None) tags in bulk_write_to_db (#21757)
• Fix DAG date range bug (#20507)
• Removed request.referrer from views.py (#21751)
• Make DbApiHook use get_uri from Connection (#21764)
• Fix some migrations (#21670)
• [de]serialize resources on task correctly (#21445)
• Add params dag_id, task_id etc to XCom.serialize_value (#19505)
• Update test connection functionality to use custom form fields (#21330)
• fix all "high" npm vulnerabilities (#21526)
• Fix bug incorrectly removing action from role, rather than permission. (#21483)
• Fix relationship join bug in FAB/SecurityManager with SQLA 1.4 (#21296)
• Use Identity instead of Sequence in SQLAlchemy 1.4 for MSSQL (#21238)
• Ensure on_task_instance_running listener can get at task (#21157)
• Return to the same place when triggering a DAG (#20955)
• Fix task ID deduplication in @task_group (#20870)
• Add downgrade to some FAB migrations (#20874)
• Only validate Params when DAG is triggered (#20802)
• Fix airflow trigger cli (#20781)
• Fix task instances iteration in a pool to prevent blocking (#20816)
• Allow depending to a @task_group as a whole (#20671)
• Use original task's start_date if a task continues after deferral (#20062)
• Disabled edit button in task instances list view page (#20659)
• Fix a package name import error (#20519) (#20519)
• Remove execution_date label when get cleanup pods list (#20417)
• Remove unneeded FAB REST API endpoints (#20487)
• Fix parsing of Cloudwatch log group arn containing slashes (#14667) (#19700)
• Sanity check for MySQL's TIMESTAMP column (#19821)
• Allow using default celery command group with executors subclassed from Celery-based executors. (#18189)
• Move class_permission_name to mixin so it applies to all classes (#18749)
• Adjust trimmed_pod_id and replace '.' with '-' (#19036)
• Pass custom_headers to send_email and send_email_smtp (#19009)
• Ensure catchup=False is used in example dags (#19396)
• Edit permalinks in OpenApi description file (#19244)
• Navigate directly to DAG when selecting from search typeahead list (#18991)
• [Minor] Fix padding on home page (#19025)

Doc only changes

• Update doc for DAG file processing (#23209)
• Replace changelog/updating with release notes and towncrier now (#22003)
• Fix wrong reference in tracking-user-activity.rst (#22745)
• Remove references to rbac = True from docs (#22725)
• Doc: Update description for executor-bound dependencies (#22601)
• Update check-health.rst (#22372)
• Stronger language about Docker Compose customizability (#22304)
• Update logging-tasks.rst (#22116)
• Add example config of sql_alchemy_connect_args (#22045)
• Update best-practices.rst (#22053)
• Add information on DAG pausing/deactivation/deletion (#22025)
• Add brief examples of integration test dags you might want (#22009)
• Run inclusive language check on CHANGELOG (#21980)
• Add detailed email docs for Sendgrid (#21958)
• Add docs for db upgrade / db downgrade (#21879)
• Update modules_management.rst (#21889)
• Fix UPDATING section on SqlAlchemy 1.4 scheme changes (#21887)
• Update TaskFlow tutorial doc to show how to pass "operator-level" args. (#21446)
• Fix doc - replace decreasing by increasing (#21805)
• Add another way to dynamically generate DAGs to docs (#21297)
• Add extra information about time synchronization needed (#21685)
• Update debug.rst docs (#21246)
• Replaces the usage of postgres:// with postgresql:// (#21205)
• Fix task execution process in CeleryExecutor docs (#20783)

Misc/Internal

• Bring back deprecated security manager functions (#23243)
• Replace usage of DummyOperator with EmptyOperator (#22974)
• Deprecate DummyOperator in favor of EmptyOperator (#22832)
• Remove unnecessary python 3.6 conditionals (#20549)
• Bump moment from 2.29.1 to 2.29.2 in /airflow/www (#22873)
• Bump prismjs from 1.26.0 to 1.27.0 in /airflow/www (#22823)
• Bump nanoid from 3.1.23 to 3.3.2 in /airflow/www (#22803)
• Bump minimist from 1.2.5 to 1.2.6 in /airflow/www (#22798)
• Remove dag parsing from db init command (#22531)
• Update our approach for executor-bound dependencies (#22573)
• Use Airflow.Base.metadata in FAB models (#22353)
• Limit docutils to make our documentation pretty again (#22420)
• Add Python 3.10 support (#22050)
• [FEATURE] add 1.22 1.23 K8S support (#21902)
• Remove pandas upper limit now that SQLA is 1.4+ (#22162)
• Patch sql_alchemy_conn if old postgres scheme used (#22333)
• Protect against accidental misuse of XCom.get_value() (#22244)
• Order filenames for migrations (#22168)
• Don't try to auto generate migrations for Celery tables (#22120)
• Require SQLAlchemy 1.4 (#22114)
• bump sphinx-jinja (#22101)
• Add compat shim for SQLAlchemy to avoid warnings (#21959)
• Rename xcom.dagrun_id to xcom.dag_run_id (#21806)
• Deprecate non-JSON conn.extra (#21816)
• Bump upper bound version of jsonschema to 5.0 (#21712)
• Deprecate helper utility days_ago (#21653)
• Remove :type lines now sphinx-autoapi supports type hints (#20951)
• Silence deprecation warning in tests (#20900)
• Use DagRun.run_id instead of execution_date when updating state of TIs (UI & REST API) (#18724)
• Add Context stub to Airflow packages (#20817)
• Update Kubernetes library version (#18797)
• Rename PodLauncher to PodManager (#20576)
• Removes Python 3.6 support (#20467)
• Add deprecation warning for non-json-serializable params (#20174)
• Rename TaskMixin to DependencyMixin (#20297)
• Deprecate passing execution_date to XCom methods (#19825)
• Remove get_readable_dags and get_editable_dags, and get_accessible_dags. (#19961)
• Remove postgres 9.6 support (#19987)
• Removed hardcoded connection types. Check if hook is instance of DbApiHook. (#19639)
• add kubernetes 1.21 support (#19557)
• Add FAB base class and set import_name explicitly. (#19667)
• Removes unused state transitions to handle auto-changing view permissions. (#19153)
• Chore: Use enum for __var and __type members (#19303)
• Use fab models (#19121)
• Consolidate method names between Airflow Security Manager and FAB default (#18726)
• Remove distutils usages for Python 3.10 (#19064)
• Removing redundant max_tis_per_query initialisation on SchedulerJob (#19020)
• Remove deprecated usage of init_role() from API (#18820)
• Remove duplicate code on dbapi hook (#18821)

New Features

• AIP-39: Add (customizable) Timetable class to Airflow for richer scheduling behaviour (#15397, #16030, #16352, #17030, #17122, #17414, #17552, #17755, #17989, #18084, #18088, #18244, #18266, #18420, #18434, #18421, #18475, #18499, #18573, #18522, #18729, #18706, #18742, #18786, #18804)
• AIP-40: Add Deferrable "Async" Tasks (#15389, #17564, #17565, #17601, #17745, #17747, #17748, #17875, #17876, #18129, #18210, #18214, #18552, #18728, #18414)
• Add a Docker Taskflow decorator (#15330, #18739)
• Add Airflow Standalone command (#15826)
• Display alert messages on dashboard from local settings (#18284)
• Advanced Params using json-schema (#17100)
• Ability to test connections from UI or API (#15795, #18750)
• Add Next Run to UI (#17732)
• Add default weight rule configuration option (#18627)
• Add a calendar field to choose the execution date of the DAG when triggering it (#16141)
• Allow setting specific cwd for BashOperator (#17751)
• Show import errors in DAG views (#17818)
• Add pre/post execution hooks [Experimental] (#17576)
• Added table to view providers in Airflow ui under admin tab (#15385)
• Adds secrets backend/logging/auth information to provider yaml (#17625)
• Add date format filters to Jinja environment (#17451)
• Introduce RESTARTING state (#16681)
• Webserver: Unpause DAG on manual trigger (#16569)
• API endpoint to create new user (#16609)
• Add insert_args for support transfer replace (#15825)
• Add recursive flag to glob in filesystem sensor (#16894)
• Add conn to jinja template context (#16686)
• Add default_args for TaskGroup (#16557)
• Allow adding duplicate connections from UI (#15574)
• Allow specifying multiple URLs via the CORS config option (#17941)
• Implement API endpoint for DAG deletion (#17980)
• Add DAG run endpoint for marking a dagrun success or failed(#17839)
• Add support for kinit options [-f|-F] and [-a|-A] (#17816)
• Queue support for DaskExecutor using Dask Worker Resources (#16829, #18720)
• Make auto refresh interval configurable (#18107)

Improvements

• Small improvements for Airflow UI (#18715, #18795)
• Rename processor_poll_interval to scheduler_idle_sleep_time (#18704)
• Check the allowed values for the logging level (#18651)
• Fix error on triggering a dag that doesn't exist using dagrun_conf (#18655)
• Add muldelete action to TaskInstanceModelView (#18438)
• Avoid importing DAGs during clean DB installation (#18450)
• Require can_edit on DAG privileges to modify TaskInstances and DagRuns (#16634)
• Make Kubernetes job description fit on one log line (#18377)
• Always draw borders if task instance state is null or undefined (#18033)
• Inclusive Language (#18349)
• Improved log handling for zombie tasks (#18277)
• Adding Variable.update method and improving detection of variable key collisions (#18159)
• Add note about params on trigger DAG page (#18166)
• Change TaskInstance and TaskReschedule PK from execution_date to run_id (#17719)
• Adding TaskGroup support in BaseOperator.chain() (#17456)
• Advises the kernel to not cache log files generated by Airflow (#18054)
• Allow filtering DAGS by tags in the REST API (#18090)
• Optimize imports of Providers Manager (#18052)
• Adds capability of Warnings for incompatible community providers (#18020)
• Serialize the template_ext attribute to show it in UI (#17985)
• Add robots.txt and X-Robots-Tag header (#17946)
• Refactor BranchDayOfWeekOperator, DayOfWeekSensor (#17940)
• Update error message to guide the user into self-help mostly (#17929)
• Update to Celery 5 (#17397)
• Avoid redirect loop for users with no permissions (#17838)
• Improve graph view load time for dags with open groups (#17821)
• Add links to provider's documentation (#17736)
• Remove Marshmallow schema warnings (#17753)
• Rename none_failed_or_skipped by none_failed_min_one_success trigger rule (#17683)
• Remove [core] store_dag_code & use DB to get Dag Code (#16342)
• Rename task_concurrency to max_active_tis_per_dag (#17708)
• Import Hooks lazily individually in providers manager (#17682)
• Adding support for multiple task-ids in the external task sensor (#17339)
• Replace execution_date with run_id in airflow tasks run command (#16666)
• Avoid endless redirect loop when user has no roles (#17613)
• Make output from users cli command more consistent (#17642)
• Open relative extra links in place (#17477)
• Move worker_log_server_port option to the logging section (#17621)
• Use gunicorn to serve logs generated by worker (#17591)
• Improve validation of Group id (#17578)
• Simplify 404 page (#17501)
• Add XCom.clear so it's hookable in custom XCom backend (#17405)
• Add deprecation notice for SubDagOperator (#17488)
• Support DAGS folder being in different location on scheduler and runners (#16860)
• Remove /dagrun/create and disable edit form generated by F.A.B (#17376)
• Enable specifying dictionary paths in template_fields_renderers (#17321)
• error early if virtualenv is missing (#15788)
• Handle connection parameters added to Extra and custom fields (#17269)
• Fix airflow celery stop to accept the pid file. (#17278)
• Remove DAG refresh buttons (#17263)
• Deprecate dummy trigger rule in favor of always (#17144)
• Be verbose about failure to import airflow_local_settings (#17195)
• Include exit code in AirflowException str when BashOperator fails. (#17151)
• Adding EdgeModifier support for chain() (#17099)
• Only allows supported field types to be used in custom connections (#17194)
• Secrets backend failover (#16404)
• Warn on Webserver when using SQLite or SequentialExecutor (#17133)
• Extend init_containers defined in pod_override (#17537)
• Client-side filter dag dependencies (#16253)
• Improve executor validation in CLI (#17071)
• Prevent running airflow db init/upgrade migrations and setup in parallel. (#17078)
• Update chain() and cross_downstream() to support XComArgs (#16732)
• Improve graph view refresh (#16696)
• When a task instance fails with exception, log it (#16805)
• Set process title for serve-logs and LocalExecutor (#16644)
• Rename test_cycle to check_cycle (#16617)
• Set process title for Worker when using LocalExecutor (#16623)
• Add schema as DbApiHook instance attribute (#16521, #17423)
• Improve compatibility with MSSQL (#9973)
• Add transparency for unsupported connection type (#16220)
• Call resource based fab methods (#16190)
• Format more dates with timezone (#16129)
• Replace deprecated dag.sub_dag with dag.partial_subset (#16179)
• Treat AirflowSensorTimeout as immediate failure without retrying (#12058)
• Marking success/failed automatically clears failed downstream tasks (#13037)
• Add close/open indicator for import dag errors (#16073)
• Add collapsible import errors (#16072)
• Always return a response in TI's action_clear view (#15980)
• Add cli command to delete user by email (#15873)
• Use resource and action names for FAB permissions (#16410)
• Rename DAG concurrency ([core] dag_concurrency) settings for easier understanding (#16267, #18730)
• Calendar UI improvements (#16226)
• Refactor: SKIPPED should not be logged again as SUCCESS (#14822)
• Remove version limits for dnspython (#18046, #18162)
• Accept custom run ID in TriggerDagRunOperator (#18788)

Bug Fixes

• Make REST API patch user endpoint work the same way as the UI (#18757)
• Properly set start_date for cleared tasks (#18708)
• Ensure task_instance exists before running update on its state(REST API) (#18642)
• Make AirflowDateTimePickerWidget a required field (#18602)
• Retry deadlocked transactions on deleting old rendered task fields (#18616)
• Fix retry_exponential_backoff divide by zero error when retry delay is zero (#17003)
• Improve how UI handles datetimes (#18611, #18700)
• Bugfix: dag_bag.get_dag should return None, not raise exception (#18554)
• Only show the task modal if it is a valid instance (#18570)
• Fix accessing rendered {{ task.x }} attributes from within templates (#18516)
• Add missing email type of connection (#18502)
• Don't use flash for "same-page" UI messages. (#18462)
• Fix task group tooltip (#18406)
• Properly fix dagrun update state endpoint (#18370)
• Properly handle ti state difference between executor and scheduler (#17819)
• Fix stuck "queued" tasks in KubernetesExecutor (#18152)
• Don't permanently add zip DAGs to sys.path (#18384)
• Fix random deadlocks in MSSQL database (#18362)
• Deactivating DAGs which have been removed from files (#17121)
• When syncing dags to db remove dag_tag rows that are now unused (#8231)
• Graceful scheduler shutdown on error (#18092)
• Fix mini scheduler not respecting wait_for_downstream dep (#18338)
• Pass exception to run_finished_callback for Debug Executor (#17983)
• Make XCom.get_one return full, not abbreviated values (#18274)
• Use try/except when closing temporary file in task_runner (#18269)
• show next run if not none (#18273)
• Fix DB session handling in XCom.set (#18240)
• Sort adopted tasks in _check_for_stalled_adopted_tasks method (#18208)
• Fix external_executor_id not being set for manually run jobs (#17207)
• Fix deleting of zipped Dags in Serialized Dag Table (#18243)
• Return explicit error on user-add for duplicated email (#18224)
• Remove loading dots even when last run data is empty (#18230)
• Swap dag import error dropdown icons (#18207)
• Automatically create section when migrating config (#16814)
• Set encoding to utf-8 by default while reading task logs (#17965)
• Set task state to failed when pod is DELETED while running (#18095)
• Apply parent dag permissions to subdags (#18160)
• Change id collation for MySQL to case-sensitive (#18072)
• Logs task launch exception in StandardTaskRunner (#17967)
• Applied permissions to self._error_file (#15947)
• Hide variable import form if user lacks permission (#18000)
• Fix blank dag dependencies view (#17990)
• Add missing menu access for dag dependencies and configurations pages (#17450)
• Fix passing Jinja templates in DateTimeSensor (#17959)
• Fixing bug which restricted the visibility of ImportErrors (#17924)
• Fix grammar in traceback.html (#17942)
• Fix DagRunState enum query for MySQLdb driver (#17886)
• Fixed button size in "Actions" group. (#17902)
• Fix log links on graph TI modal (#17862)
• Only show import errors for DAGs a user can access (#17835)
• Show all import_errors from zip files (#17759)
• fix EXTRA_LOGGER_NAMES param and related docs (#17808)
• Use one interpreter for Airflow and gunicorn (#17805)
• Fix: Mysql 5.7 id utf8mb3 (#14535)
• Fix dag_processing.last_duration metric random holes (#17769)
• Automatically use utf8mb3_general_ci collation for MySQL (#17729)
• Fix wrong query on running tis (#17631)
• Add root to tree refresh url (#17633)
• Do not delete running DAG from the UI (#17630)
• Rescue if a DagRun's DAG was removed from db (#17544)
• fix: filter condition of TaskInstance does not work #17535 (#17548)
• Regression on pid reset to allow task start after heartbeat (#17333)
• Dont use TaskInstance in CeleryExecutor.trigger_tasks (#16248)
• Remove locks for upgrades in MSSQL (#17213)
• Create virtualenv via python call (#17156)
• Ensure a DAG is acyclic when running DAG.cli() (#17105)
• Translate non-ascii characters (#17057)
• Change the logic of None comparison in model_list template (#16893)
• Have UI and POST /task_instances_state API endpoint have same behaviour (#16539)
• ensure task is skipped if missing sla (#16719)
• Fix direct use of cached_property module (#16710)
• Fix TI success confirm page (#16650)
• Cattrs 1.7.0 released by the end of May 2021 break lineage usage (#16173)
• Fix Celery executor getting stuck randomly because of reset_signals in multiprocessing (#15989)
• Modify return value check in python virtualenv jinja template (#16049)
• Fix dag dependency search (#15924)
• Make custom JSON encoder support Decimal (#16383)
• Bugfix: Allow clearing tasks with just dag_id and empty subdir (#16513)
• Convert port value to a number before calling test connection (#16497)
• Handle missing/null serialized DAG dependencies (#16393)
• Correctly set dag.fileloc when using the @dag decorator (#16384)
• Fix TI success/failure links (#16233)
• Correctly implement autocomplete early return in airflow/www/views.py (#15940)
• Backport fix to allow pickling of Loggers to Python 3.6 (#18798)
• Fix bug that Backfill job fail to run when there are tasks run into reschedule state (#17305, #18806)

Doc only changes

• Update dagbag_size documentation (#18824)
• Update documentation about bundle extras (#18828)
• Fix wrong Postgres search_path set up instructions (#17600)
• Remove AIRFLOW_GID from Docker images (#18747)
• Improve error message for BranchPythonOperator when no task_id to follow (#18471)
• Improve guidance to users telling them what to do on import timeout (#18478)
• Explain scheduler fine-tuning better (#18356)
• Added example JSON for airflow pools import (#18376)
• Add sla_miss_callback section to the documentation (#18305)
• Explain sentry default environment variable for subprocess hook (#18346)
• Refactor installation pages (#18282)
• Improves installing from sources pages for all components (#18251)
• Doc: Use closer.lua script for downloading sources (#18179)
• Improves quick-start docker-compose warnings and documentation (#18164)
• Production-level support for MSSQL (#18382)
• Update non-working example in documentation (#18067)
• Improve the description of how to handle dynamic task generation (#17963)
• Remove default_args pattern + added get_current_context() use for Core Airflow example DAGs (#16866)
• Update max_tis_per_query to better render on the webpage (#17971)
• Adds Github Oauth example with team based authorization (#17896)
• Update docker.rst (#17882)
• Example xcom update (#17749)
• Add doc warning about connections added via envvars (#17915)
• fix wrong documents around upgrade-check.rst (#17903)
• Add Brent to Committers list (#17873)
• Improves documentation about modules management (#17757)
• Remove deprecated metrics from metrics.rst (#17772)
• Make sure "production-readiness" of docker-compose is well explained (#17731)
• Doc: Update Upgrade to v2 docs with Airflow 1.10.x EOL dates (#17710)
• Doc: Replace deprecated param from docstrings (#17709)
• docs(dagowner): describe dag owner more carefully (#17699)
• docs(impersonation): update note so avoid misintrepretation (#17701)
• Docs: Make DAG.is_active read-only in API (#17667)
• Improve cross-links to operators and hooks references (#17622)
• Update documentation regarding Python 3.9 support (#17611)
• Fix MySQL database character set instruction (#17603)
• Document overriding XCom.clear for data lifecycle management (#17589)
• Path correction in docs for airflow core (#17567)
• docs(celery): reworded, add actual multiple queues example (#17541)
• Doc: Add FAQ to speed up parsing with tons of dag files (#17519)
• Improve image building documentation for new users (#17409)
• Doc: Strip unnecessary arguments from MariaDB JIRA URL (#17296)
• Update warning about MariaDB and multiple schedulers (#17287)
• Doc: Recommend using same configs on all Airflow components (#17146)
• Move docs about masking to a new page (#17007)
• Docs: Suggest use of Env vars instead of Airflow Vars in best practises doc (#16926)
• Docs: Better description for pod_template_file (#16861)
• Add Aneesh Joseph as Airflow Committer (#16835)
• Docs: Added new pipeline example for the tutorial docs (#16548)
• Remove upstart from docs (#16672)
• Add new committers: Jed and TP (#16671)
• Docs: Fix flask-ouathlib to flask-oauthlib in Upgrading docs (#16320)
• Docs: Fix creating a connection docs (#16312)
• Docs: Fix url for Elasticsearch (#16275)
• Small improvements for README.md files (#16244)
• Fix docs for dag_concurrency (#16177)
• Check syntatic correctness for code-snippets (#16005)
• Improvements for Docker Image docs (#14843)
• Add proper link for wheel packages in docs. (#15999)
• Add Docs for default_pool slots (#15997)
• Add memory usage warning in quick-start documentation (#15967)
• Update example KubernetesExecutor git-sync pod template file (#15904)
• Docs: Fix Taskflow API docs (#16574)
• Added new pipeline example for the tutorial docs (#16084)
• Updating the DAG docstring to include render_template_as_native_obj (#16534)
• Update docs on setting up SMTP (#16523)
• Docs: Fix API verb from POST to PATCH (#16511)

Misc/Internal

• Renaming variables to be consistent with code logic (#18685)
• Simplify strings previously split across lines (#18679)
• fix exception string of BranchPythonOperator (#18623)
• Add multiple roles when creating users (#18617)
• Move FABs base Security Manager into Airflow. (#16647)
• Remove unnecessary css state colors (#18461)
• Update boto3 to <1.19 (#18389)
• Improve coverage for airflow.security.kerberos module (#18258)
• Fix kinesis test (#18337)
• Fix provider test acessing importlib-resources (#18228)
• Silence warnings in tests from using SubDagOperator (#18275)
• Fix usage of range(len()) to enumerate (#18174)
• Test coverage on the autocomplete view (#15943)
• Update version added fields in airflow/config_templates/config.yml (#18128)
• Add "packaging" to core requirements (#18122)
• Adds LoggingMixins to BaseTrigger (#18106)
• Limit colorlog version (6.x is incompatible) (#18099)
• Fix building docs in main builds (#18035)
• Remove upper-limit on tenacity (#17593)
• Remove redundant numpy dependency (#17594)
• Bump mysql-connector-python to latest version (#17596)
• Make pandas an optional core dependency (#17575)
• Add more typing to airflow.utils.helpers (#15582)
• Chore: Some code cleanup in airflow/utils/db.py (#17090)
• Refactor: Remove processor_factory from DAG processing (#16659)
• Remove AbstractDagFileProcessorProcess from dag processing (#16816)
• Update TaskGroup typing (#16811)
• Update click to 8.x (#16779)
• Remove remaining Pylint disables (#16760)
• Remove duplicated try, there is already a try in create_session (#16701)
• Removes pylint from our toolchain (#16682)
• Refactor usage of unneeded function call (#16653)
• Add type annotations to setup.py (#16658)
• Remove SQLAlchemy <1.4 constraint (#16630) (Note: our dependencies still have a requirement on <1.4)
• Refactor dag.clear method (#16086)
• Use DAG_ACTIONS constant (#16232)
• Use updated _get_all_non_dag_permissions method (#16317)
• Add updated-name wrappers for built-in FAB methods (#16077)
• Pins docutils to <0.17 until breaking behaviour is fixed (#16133)
• Remove TaskInstance.log_filepath attribute (#15217)
• Removes unnecessary function call in airflow/www/app.py (#15956)
• Move plyvel to google provider extra (#15812)
• Update permission migrations to use new naming scheme (#16400)
• Use resource and action names for FAB (#16380)
• Swap out calls to find_permission_view_menu for get_permission wrapper (#16377)
• Fix deprecated default for fab_logging_level to WARNING (#18783)
• Allow running tasks from UI when using CeleryKubernetesExecutor (#18441)

Bug Fixes

• Don't crash attempting to mask secrets in dict with non-string keys (#16601)
• Always install sphinx_airflow_theme from PyPI (#16594)
• Remove limitation for elasticsearch library (#16553)
• Adding extra requirements for build and runtime of the PROD image. (#16170)
• Cattrs 1.7.0 released by the end of May 2021 break lineage usage (#16173)
• Removes unnecessary packages from setup_requires (#16139)
• Pins docutils to <0.17 until breaking behaviour is fixed (#16133)
• Improvements for Docker Image docs (#14843)
• Ensure that dag_run.conf is a dict (#15057)
• Fix CLI connections import and migrate logic from secrets to Connection model (#15425)
• Fix Dag Details start date bug (#16206)
• Fix DAG run state not updated while DAG is paused (#16343)
• Allow null value for operator field in task_instance schema(REST API) (#16516)
• Avoid recursion going too deep when redacting logs (#16491)
• Backfill: Don't create a DagRun if no tasks match task regex (#16461)
• Tree View UI for larger DAGs & more consistent spacing in Tree View (#16522)
• Correctly handle None returns from Query.scalar() (#16345)
• Adding only_active parameter to /dags endpoint (#14306)
• Don't show stale Serialized DAGs if they are deleted in DB (#16368)
• Make REST API List DAGs endpoint consistent with UI/CLI behaviour (#16318)
• Support remote logging in elasticsearch with filebeat 7 (#14625)
• Queue tasks with higher priority and earlier execution_date first. (#15210)
• Make task ID on legend have enough width and width of line chart to be 100%. (#15915)
• Fix normalize-url vulnerability (#16375)
• Validate retries value on init for better errors (#16415)
• add num_runs query param for tree refresh (#16437)
• Fix templated default/example values in config ref docs (#16442)
• Add passphrase and private_key to default sensitive field names (#16392)
• Fix tasks in an infinite slots pool were never scheduled (#15247)
• Fix Orphaned tasks stuck in CeleryExecutor as running (#16550)
• Don't fail to log if we can't redact something (#16118)
• Set max tree width to 1200 pixels (#16067)
• Fill the "job_id" field for airflow task run without --local/--raw for KubeExecutor (#16108)
• Fixes problem where conf variable was used before initialization (#16088)
• Fix apply defaults for task decorator (#16085)
• Parse recently modified files even if just parsed (#16075)
• Ensure that we don't try to mask empty string in logs (#16057)
• Don't die when masking log.exception when there is no exception (#16047)
• Restores apply_defaults import in base_sensor_operator (#16040)
• Fix auto-refresh in tree view When webserver ui is not in / (#16018)
• Fix dag.clear() to set multiple dags to running when necessary (#15382)
• Fix Celery executor getting stuck randomly because of reset_signals in multiprocessing (#15989)

Release Date: 2021-08-02

Features

Add support for non-RSA type key for SFTP hook (#16314)

New Features """"""""""""

• Add PythonVirtualenvDecorator to Taskflow API (#14761)
• Add Taskgroup decorator (#15034)
• Create a DAG Calendar View (#15423)
• Create cross-DAG dependencies view (#13199)
• Add rest API to query for providers (#13394)
• Mask passwords and sensitive info in task logs and UI (#15599)
• Add SubprocessHook for running commands from operators (#13423)
• Add DAG Timeout in UI page "DAG Details" (#14165)
• Add WeekDayBranchOperator (#13997)
• Add JSON linter to DAG Trigger UI (#13551)
• Add DAG Description Doc to Trigger UI Page (#13365)
• Add airflow webserver URL into SLA miss email. (#13249)
• Add read only REST API endpoints for users (#14735)
• Add files to generate Airflow's Python SDK (#14739)
• Add dynamic fields to snowflake connection (#14724)
• Add read only REST API endpoint for roles and permissions (#14664)
• Add new datetime branch operator (#11964)
• Add Google leveldb hook and operator (#13109) (#14105)
• Add plugins endpoint to the REST API (#14280)
• Add worker_pod_pending_timeout support (#15263)
• Add support for labeling DAG edges (#15142)
• Add CUD REST API endpoints for Roles (#14840)
• Import connections from a file (#15177)
• A bunch of template_fields_renderers additions (#15130)
• Add REST API query sort and order to some endpoints (#14895)
• Add timezone context in new ui (#15096)
• Add query mutations to new UI (#15068)
• Add different modes to sort dag files for parsing (#15046)
• Auto refresh on Tree View (#15474)
• BashOperator to raise AirflowSkipException on exit code 99 (by default, configurable) (#13421) (#14963)
• Clear tasks by task ids in REST API (#14500)
• Support jinja2 native Python types (#14603)
• Allow celery workers without gossip or mingle modes (#13880)
• Add airflow jobs check CLI command to check health of jobs (Scheduler etc) (#14519)
• Rename DateTimeBranchOperator to BranchDateTimeOperator (#14720)

Improvements """"""""""""

• Add optional result handler callback to DbApiHook (#15581)
• Update Flask App Builder limit to recently released 3.3 (#15792)
• Prevent creating flask sessions on REST API requests (#15295)
• Sync DAG specific permissions when parsing (#15311)
• Increase maximum length of pool name on Tasks to 256 characters (#15203)
• Enforce READ COMMITTED isolation when using mysql (#15714)
• Auto-apply apply_default to subclasses of BaseOperator (#15667)
• Emit error on duplicated DAG ID (#15302)
• Update KubernetesExecutor pod templates to allow access to IAM permissions (#15669)
• More verbose logs when running airflow db check-migrations (#15662)
• When one_success mark task as failed if no success (#15467)
• Add an option to trigger a dag w/o changing conf (#15591)
• Add Airflow UI instance_name configuration option (#10162)
• Add a decorator to retry functions with DB transactions (#14109)
• Add return to PythonVirtualenvOperator's execute method (#14061)
• Add verify_ssl config for kubernetes (#13516)
• Add description about secret_key when Webserver > 1 (#15546)
• Add Traceback in LogRecord in JSONFormatter (#15414)
• Add support for arbitrary json in conn uri format (#15100)
• Adds description field in variable (#12413) (#15194)
• Add logs to show last modified in SFTP, FTP and Filesystem sensor (#15134)
• Execute on_failure_callback when SIGTERM is received (#15172)
• Allow hiding of all edges when highlighting states (#15281)
• Display explicit error in case UID has no actual username (#15212)
• Serve logs with Scheduler when using Local or Sequential Executor (#15557)
• Deactivate trigger, refresh, and delete controls on dag detail view. (#14144)
• Turn off autocomplete for connection forms (#15073)
• Increase default worker_refresh_interval to 6000 seconds (#14970)
• Only show User's local timezone if it's not UTC (#13904)
• Suppress LOG/WARNING for a few tasks CLI for better CLI experience (#14567)
• Configurable API response (CORS) headers (#13620)
• Allow viewers to see all docs links (#14197)
• Update Tree View date ticks (#14141)
• Make the tooltip to Pause / Unpause a DAG clearer (#13642)
• Warn about precedence of env var when getting variables (#13501)
• Move [celery] default_queue config to [operators] default_queue to re-use between executors (#14699)

Bug Fixes """""""""

• Fix 500 error from updateTaskInstancesState API endpoint when dry_run not passed (#15889)
• Ensure that task preceding a PythonVirtualenvOperator doesn't fail (#15822)
• Prevent mixed case env vars from crashing processes like worker (#14380)
• Fixed type annotations in DAG decorator (#15778)
• Fix on_failure_callback when task receive SIGKILL (#15537)
• Fix dags table overflow (#15660)
• Fix changing the parent dag state on subdag clear (#15562)
• Fix reading from zip package to default to text (#13962)
• Fix wrong parameter for drawDagStatsForDag in dags.html (#13884)
• Fix QueuedLocalWorker crashing with EOFError (#13215)
• Fix typo in NotPreviouslySkippedDep (#13933)
• Fix parallelism after KubeExecutor pod adoption (#15555)
• Fix kube client on mac with keepalive enabled (#15551)
• Fixes wrong limit for dask for python>3.7 (should be <3.7) (#15545)
• Fix Task Adoption in KubernetesExecutor (#14795)
• Fix timeout when using XCom with KubernetesPodOperator (#15388)
• Fix deprecated provider aliases in "extras" not working (#15465)
• Fixed default XCom deserialization. (#14827)
• Fix used_group_ids in dag.partial_subset (#13700) (#15308)
• Further fix trimmed pod_id for KubernetesPodOperator (#15445)
• Bugfix: Invalid name when trimmed pod_id ends with hyphen in KubernetesPodOperator (#15443)
• Fix incorrect slots stats when TI pool_slots > 1 (#15426)
• Fix DAG last run link (#15327)
• Fix sync-perm to work correctly when update_fab_perms = False (#14847)
• Fixes limits on Arrow for plexus test (#14781)
• Fix UI bugs in tree view (#14566)
• Fix AzureDataFactoryHook failing to instantiate its connection (#14565)
• Fix permission error on non-POSIX filesystem (#13121)
• Fix spelling in "ignorable" (#14348)
• Fix get_context_data doctest import (#14288)
• Correct typo in GCSObjectsWtihPrefixExistenceSensor (#14179)
• Fix order of failed deps (#14036)
• Fix critical CeleryKubernetesExecutor bug (#13247)
• Fix four bugs in StackdriverTaskHandler (#13784)
• func.sum may return Decimal that break rest APIs (#15585)
• Persist tags params in pagination (#15411)
• API: Raise AlreadyExists exception when the execution_date is same (#15174)
• Remove duplicate call to sync_metadata inside DagFileProcessorManager (#15121)
• Extra docker-py update to resolve docker op issues (#15731)
• Ensure executors end method is called (#14085)
• Remove user_id from API schema (#15117)
• Prevent clickable bad links on disabled pagination (#15074)
• Acquire lock on db for the time of migration (#10151)
• Skip SLA check only if SLA is None (#14064)
• Print right version in airflow info command (#14560)
• Make airflow info work with pipes (#14528)
• Rework client-side script for connection form. (#14052)
• API: Add CollectionInfo in all Collections that have total_entries (#14366)
• Fix task_instance_mutation_hook when importing airflow.models.dagrun (#15851)

Doc only changes """"""""""""""""

• Fix docstring of SqlSensor (#15466)
• Small changes on "DAGs and Tasks documentation" (#14853)
• Add note on changes to configuration options (#15696)
• Add docs to the markdownlint and yamllint config files (#15682)
• Rename old "Experimental" API to deprecated in the docs. (#15653)
• Fix documentation error in git_sync_template.yaml (#13197)
• Fix doc link permission name (#14972)
• Fix link to Helm chart docs (#14652)
• Fix docstrings for Kubernetes code (#14605)
• docs: Capitalize & minor fixes (#14283) (#14534)
• Fixed reading from zip package to default to text. (#13984)
• An initial rework of the "Concepts" docs (#15444)
• Improve docstrings for various modules (#15047)
• Add documentation on database connection URI (#14124)
• Add Helm Chart logo to docs index (#14762)
• Create a new documentation package for Helm Chart (#14643)
• Add docs about supported logging levels (#14507)
• Update docs about tableau and salesforce provider (#14495)
• Replace deprecated doc links to the correct one (#14429)
• Refactor redundant doc url logic to use utility (#14080)
• docs: NOTICE: Updated 2016-2019 to 2016-now (#14248)
• Skip DAG perm sync during parsing if possible (#15464)
• Add picture and examples for Edge Labels (#15310)
• Add example DAG & how-to guide for sqlite (#13196)
• Add links to new modules for deprecated modules (#15316)
• Add note in Updating.md about FAB data model change (#14478)

Misc/Internal """""""""""""

• Fix logging.exception redundancy (#14823)
• Bump stylelint to remove vulnerable sub-dependency (#15784)
• Add resolution to force dependencies to use patched version of lodash (#15777)
• Update croniter to 1.0.x series (#15769)
• Get rid of Airflow 1.10 in Breeze (#15712)
• Run helm chart tests in parallel (#15706)
• Bump ssri from 6.0.1 to 6.0.2 in /airflow/www (#15437)
• Remove the limit on Gunicorn dependency (#15611)
• Better "dependency already registered" warning message for tasks #14613 (#14860)
• Pin pandas-gbq to <0.15.0 (#15114)
• Use Pip 21.* to install airflow officially (#15513)
• Bump mysqlclient to support the 1.4.x and 2.x series (#14978)
• Finish refactor of DAG resource name helper (#15511)
• Refactor/Cleanup Presentation of Graph Task and Path Highlighting (#15257)
• Standardize default fab perms (#14946)
• Remove datepicker for task instance detail view (#15284)
• Turn provider's import warnings into debug logs (#14903)
• Remove left-over fields from required in provider_info schema. (#14119)
• Deprecate tableau extra (#13595)
• Use built-in cached_property on Python 3.8 where possible (#14606)
• Clean-up JS code in UI templates (#14019)
• Bump elliptic from 6.5.3 to 6.5.4 in /airflow/www (#14668)
• Switch to f-strings using flynt. (#13732)
• use jquery ready instead of vanilla js (#15258)
• Migrate task instance log (ti_log) js (#15309)
• Migrate graph js (#15307)
• Migrate dags.html javascript (#14692)
• Removes unnecessary AzureContainerInstance connection type (#15514)
• Separate Kubernetes pod_launcher from core airflow (#15165)
• update remaining old import paths of operators (#15127)
• Remove broken and undocumented "demo mode" feature (#14601)
• Simplify configuration/legibility of Webpack entries (#14551)
• remove inline tree js (#14552)
• Js linting and inline migration for simple scripts (#14215)
• Remove use of repeated constant in AirflowConfigParser (#14023)
• Deprecate email credentials from environment variables. (#13601)
• Remove unused 'context' variable in task_instance.py (#14049)
• Disable suppress_logs_and_warning in cli when debugging (#13180)

The full changelog is about 3,000 lines long (already excluding everything backported to 1.10), so for now I’ll simply share some of the major features in 2.0.0 compared to 1.10.14:

A new way of writing dags: the TaskFlow API (AIP-31)

(Known in 2.0.0alphas as Functional DAGs.)

DAGs are now much much nicer to author especially when using PythonOperator. Dependencies are handled more clearly and XCom is nicer to use

A quick teaser of what DAGs can now look like:

from airflow.decorators import dag, task
from airflow.utils.dates import days_ago

@dag(default_args={'owner': 'airflow'}, schedule_interval=None, start_date=days_ago(2))
def tutorial_taskflow_api_etl():
   @task
   def extract():
       return {"1001": 301.27, "1002": 433.21, "1003": 502.22}

   @task
   def transform(order_data_dict: dict) -> dict:
       total_order_value = 0

       for value in order_data_dict.values():
           total_order_value += value

       return {"total_order_value": total_order_value}

   @task()
   def load(total_order_value: float):

       print("Total order value is: %.2f" % total_order_value)

   order_data = extract()
   order_summary = transform(order_data)
   load(order_summary["total_order_value"])

tutorial_etl_dag = tutorial_taskflow_api_etl()

Fully specified REST API (AIP-32)

We now have a fully supported, no-longer-experimental API with a comprehensive OpenAPI specification

Read more here:

REST API Documentation.

Massive Scheduler performance improvements

As part of AIP-15 (Scheduler HA+performance) and other work Kamil did, we significantly improved the performance of the Airflow Scheduler. It now starts tasks much, MUCH quicker.

Over at Astronomer.io we’ve benchmarked the scheduler—it’s fast (we had to triple check the numbers as we don’t quite believe them at first!)

Scheduler is now HA compatible (AIP-15)

It’s now possible and supported to run more than a single scheduler instance. This is super useful for both resiliency (in case a scheduler goes down) and scheduling performance.

To fully use this feature you need Postgres 9.6+ or MySQL 8+ (MySQL 5, and MariaDB won’t work with more than one scheduler I’m afraid).

There’s no config or other set up required to run more than one scheduler—just start up a scheduler somewhere else (ensuring it has access to the DAG files) and it will cooperate with your existing schedulers through the database.

For more information, read the Scheduler HA documentation.

Task Groups (AIP-34)

SubDAGs were commonly used for grouping tasks in the UI, but they had many drawbacks in their execution behaviour (primarirly that they only executed a single task in parallel!) To improve this experience, we’ve introduced “Task Groups”: a method for organizing tasks which provides the same grouping behaviour as a subdag without any of the execution-time drawbacks.

SubDAGs will still work for now, but we think that any previous use of SubDAGs can now be replaced with task groups. If you find an example where this isn’t the case, please let us know by opening an issue on GitHub

For more information, check out the Task Group documentation.

Refreshed UI

We’ve given the Airflow UI a visual refresh and updated some of the styling. Check out the UI section of the docs for screenshots.

We have also added an option to auto-refresh task states in Graph View so you no longer need to continuously press the refresh button :).

Smart Sensors for reduced load from sensors (AIP-17)

If you make heavy use of sensors in your Airflow cluster, you might find that sensor execution takes up a significant proportion of your cluster even with “reschedule” mode. To improve this, we’ve added a new mode called “Smart Sensors”.

This feature is in “early-access”: it’s been well-tested by AirBnB and is “stable”/usable, but we reserve the right to make backwards-incompatible changes to it in a future release (if we have to. We’ll try very hard not to!)

Simplified KubernetesExecutor

For Airflow 2.0, we have re-architected the KubernetesExecutor in a fashion that is simultaneously faster, easier to understand, and more flexible for Airflow users. Users will now be able to access the full Kubernetes API to create a .yaml pod_template_file instead of specifying parameters in their airflow.cfg.

We have also replaced the executor_config dictionary with the pod_override parameter, which takes a Kubernetes V1Pod object for a 1:1 setting override. These changes have removed over three thousand lines of code from the KubernetesExecutor, which makes it run faster and creates fewer potential errors.

Airflow core and providers: Splitting Airflow into 60+ packages

Airflow 2.0 is not a monolithic “one to rule them all” package. We’ve split Airflow into core and 61 (for now) provider packages. Each provider package is for either a particular external service (Google, Amazon, Microsoft, Snowflake), a database (Postgres, MySQL), or a protocol (HTTP/FTP). Now you can create a custom Airflow installation from “building” blocks and choose only what you need, plus add whatever other requirements you might have. Some of the common providers are installed automatically (ftp, http, imap, sqlite) as they are commonly used. Other providers are automatically installed when you choose appropriate extras when installing Airflow.

The provider architecture should make it much easier to get a fully customized, yet consistent runtime with the right set of Python dependencies.

But that’s not all: you can write your own custom providers and add things like custom connection types, customizations of the Connection Forms, and extra links to your operators in a manageable way. You can build your own provider and install it as a Python package and have your customizations visible right in the Airflow UI.

Security

As part of Airflow 2.0 effort, there has been a conscious focus on Security and reducing areas of exposure. This is represented across different functional areas in different forms. For example, in the new REST API, all operations now require authorization. Similarly, in the configuration settings, the Fernet key is now required to be specified.

Configuration

Configuration in the form of the airflow.cfg file has been rationalized further in distinct sections, specifically around “core”. Additionally, a significant amount of configuration options have been deprecated or moved to individual component-specific configuration files, such as the pod-template-file for Kubernetes execution-related configuration.

We’ve tried to make as few breaking changes as possible and to provide deprecation path in the code, especially in the case of anything called in the DAG. That said, please read through UPDATING.md to check what might affect you. For example: We re-organized the layout of operators (they now all live under airflow.providers.*) but the old names should continue to work - you’ll just notice a lot of DeprecationWarnings that need to be fixed up.

Release Date: 2021-05-06

Features

Undeprecate private_key option in SFTPHook (#15348)

Add logs to show last modified in SFTP, FTP and Filesystem sensor (#15134)

Release Date: 2021-03-07

Features

SFTPHook private_key_pass extra param is deprecated and renamed to private_key_passphrase, for consistency with arguments' naming in SSHHook

Bug fixes

Corrections in docs and tools after releasing provider RCs (#14082)

Release Date: 2021-02-08

Updated documentation and readme files.

Features

Add retryer to SFTP hook connection (#13065)

Release Date: 2020-12-14

Initial version of the provider.

Previous Next

Was this entry helpful?

Suggest a change on this page