apache-airflow-providers-openlineage

Bug Fixes

• Only allow the webserver to request from the worker log server (#16754)
• Fix "Invalid JSON configuration, must be a dict" bug (#16648)
• Fix CeleryKubernetesExecutor (#16700)
• Mask value if the key is token (#16474)
• Fix impersonation issue with LocalTaskJob (#16852)

Misc

• Add Python 3.9 support (#15515)

Bug Fixes

• Don't crash attempting to mask secrets in dict with non-string keys (#16601)
• Always install sphinx_airflow_theme from PyPI (#16594)
• Remove limitation for elasticsearch library (#16553)
• Adding extra requirements for build and runtime of the PROD image. (#16170)
• Cattrs 1.7.0 released by the end of May 2021 break lineage usage (#16173)
• Removes unnecessary packages from setup_requires (#16139)
• Pins docutils to <0.17 until breaking behaviour is fixed (#16133)
• Improvements for Docker Image docs (#14843)
• Ensure that dag_run.conf is a dict (#15057)
• Fix CLI connections import and migrate logic from secrets to Connection model (#15425)
• Fix Dag Details start date bug (#16206)
• Fix DAG run state not updated while DAG is paused (#16343)
• Allow null value for operator field in task_instance schema(REST API) (#16516)
• Avoid recursion going too deep when redacting logs (#16491)
• Backfill: Don't create a DagRun if no tasks match task regex (#16461)
• Tree View UI for larger DAGs & more consistent spacing in Tree View (#16522)
• Correctly handle None returns from Query.scalar() (#16345)
• Adding only_active parameter to /dags endpoint (#14306)
• Don't show stale Serialized DAGs if they are deleted in DB (#16368)
• Make REST API List DAGs endpoint consistent with UI/CLI behaviour (#16318)
• Support remote logging in elasticsearch with filebeat 7 (#14625)
• Queue tasks with higher priority and earlier execution_date first. (#15210)
• Make task ID on legend have enough width and width of line chart to be 100%. (#15915)
• Fix normalize-url vulnerability (#16375)
• Validate retries value on init for better errors (#16415)
• add num_runs query param for tree refresh (#16437)
• Fix templated default/example values in config ref docs (#16442)
• Add passphrase and private_key to default sensitive field names (#16392)
• Fix tasks in an infinite slots pool were never scheduled (#15247)
• Fix Orphaned tasks stuck in CeleryExecutor as running (#16550)
• Don't fail to log if we can't redact something (#16118)
• Set max tree width to 1200 pixels (#16067)
• Fill the "job_id" field for airflow task run without --local/--raw for KubeExecutor (#16108)
• Fixes problem where conf variable was used before initialization (#16088)
• Fix apply defaults for task decorator (#16085)
• Parse recently modified files even if just parsed (#16075)
• Ensure that we don't try to mask empty string in logs (#16057)
• Don't die when masking log.exception when there is no exception (#16047)
• Restores apply_defaults import in base_sensor_operator (#16040)
• Fix auto-refresh in tree view When webserver ui is not in / (#16018)
• Fix dag.clear() to set multiple dags to running when necessary (#15382)
• Fix Celery executor getting stuck randomly because of reset_signals in multiprocessing (#15989)

Release Date: 2025-02-26

Features

change listener API, add basic support for task instance listeners in TaskSDK, make OpenLineage provider support Airflow 3's listener interface (#45294)

feat: Add ProcessingEngineRunFacet to all OL events (#46283)

feat: automatically inject OL transport info into spark jobs (#45326)

feat: Add OpenLineage support for some SQL to GCS operators (#45242)

feat: automatically inject OL info into spark job in DataprocCreateBatchOperator (#44612)

Bug Fixes

Update OpenLineage emmiter to cope with nullable logical_date (#46722)

fix: OL sql parsing add try-except for sqlalchemy engine (#46366)

OpenLineage: Include 'AirflowDagRunFacet' in complete/failed events (#45615)

Misc

Adding uuid6 as a dependency for openlineage (#46653)

Remove AirflowContextDeprecationWarning as all context should be clean for Airflow 3 (#46601)

Remove Airflow 3 Deprecation Warning dependency in OTel Provider (#46600)

AIP-72: Move Secrets Masker to task SDK (#46375)

Add run_after column to DagRun model (#45732)

Remove old lineage stuff (#45260)

Start porting mapped task to SDK (#45627)

chore: Update docstring for DatabaseInfo in OpenLineage provider (#45638)

Remove classes from 'typing_compat' that can be imported directly (#45589)

updated 404 hyperlink to gcstogcsoperator (#45311)

pass error for on_task_instance_failed in task sdk (#46941)

New Features """"""""""""

• Add PythonVirtualenvDecorator to Taskflow API (#14761)
• Add Taskgroup decorator (#15034)
• Create a DAG Calendar View (#15423)
• Create cross-DAG dependencies view (#13199)
• Add rest API to query for providers (#13394)
• Mask passwords and sensitive info in task logs and UI (#15599)
• Add SubprocessHook for running commands from operators (#13423)
• Add DAG Timeout in UI page "DAG Details" (#14165)
• Add WeekDayBranchOperator (#13997)
• Add JSON linter to DAG Trigger UI (#13551)
• Add DAG Description Doc to Trigger UI Page (#13365)
• Add airflow webserver URL into SLA miss email. (#13249)
• Add read only REST API endpoints for users (#14735)
• Add files to generate Airflow's Python SDK (#14739)
• Add dynamic fields to snowflake connection (#14724)
• Add read only REST API endpoint for roles and permissions (#14664)
• Add new datetime branch operator (#11964)
• Add Google leveldb hook and operator (#13109) (#14105)
• Add plugins endpoint to the REST API (#14280)
• Add worker_pod_pending_timeout support (#15263)
• Add support for labeling DAG edges (#15142)
• Add CUD REST API endpoints for Roles (#14840)
• Import connections from a file (#15177)
• A bunch of template_fields_renderers additions (#15130)
• Add REST API query sort and order to some endpoints (#14895)
• Add timezone context in new ui (#15096)
• Add query mutations to new UI (#15068)
• Add different modes to sort dag files for parsing (#15046)
• Auto refresh on Tree View (#15474)
• BashOperator to raise AirflowSkipException on exit code 99 (by default, configurable) (#13421) (#14963)
• Clear tasks by task ids in REST API (#14500)
• Support jinja2 native Python types (#14603)
• Allow celery workers without gossip or mingle modes (#13880)
• Add airflow jobs check CLI command to check health of jobs (Scheduler etc) (#14519)
• Rename DateTimeBranchOperator to BranchDateTimeOperator (#14720)

Improvements """"""""""""

• Add optional result handler callback to DbApiHook (#15581)
• Update Flask App Builder limit to recently released 3.3 (#15792)
• Prevent creating flask sessions on REST API requests (#15295)
• Sync DAG specific permissions when parsing (#15311)
• Increase maximum length of pool name on Tasks to 256 characters (#15203)
• Enforce READ COMMITTED isolation when using mysql (#15714)
• Auto-apply apply_default to subclasses of BaseOperator (#15667)
• Emit error on duplicated DAG ID (#15302)
• Update KubernetesExecutor pod templates to allow access to IAM permissions (#15669)
• More verbose logs when running airflow db check-migrations (#15662)
• When one_success mark task as failed if no success (#15467)
• Add an option to trigger a dag w/o changing conf (#15591)
• Add Airflow UI instance_name configuration option (#10162)
• Add a decorator to retry functions with DB transactions (#14109)
• Add return to PythonVirtualenvOperator's execute method (#14061)
• Add verify_ssl config for kubernetes (#13516)
• Add description about secret_key when Webserver > 1 (#15546)
• Add Traceback in LogRecord in JSONFormatter (#15414)
• Add support for arbitrary json in conn uri format (#15100)
• Adds description field in variable (#12413) (#15194)
• Add logs to show last modified in SFTP, FTP and Filesystem sensor (#15134)
• Execute on_failure_callback when SIGTERM is received (#15172)
• Allow hiding of all edges when highlighting states (#15281)
• Display explicit error in case UID has no actual username (#15212)
• Serve logs with Scheduler when using Local or Sequential Executor (#15557)
• Deactivate trigger, refresh, and delete controls on dag detail view. (#14144)
• Turn off autocomplete for connection forms (#15073)
• Increase default worker_refresh_interval to 6000 seconds (#14970)
• Only show User's local timezone if it's not UTC (#13904)
• Suppress LOG/WARNING for a few tasks CLI for better CLI experience (#14567)
• Configurable API response (CORS) headers (#13620)
• Allow viewers to see all docs links (#14197)
• Update Tree View date ticks (#14141)
• Make the tooltip to Pause / Unpause a DAG clearer (#13642)
• Warn about precedence of env var when getting variables (#13501)
• Move [celery] default_queue config to [operators] default_queue to re-use between executors (#14699)

Bug Fixes """""""""

• Fix 500 error from updateTaskInstancesState API endpoint when dry_run not passed (#15889)
• Ensure that task preceding a PythonVirtualenvOperator doesn't fail (#15822)
• Prevent mixed case env vars from crashing processes like worker (#14380)
• Fixed type annotations in DAG decorator (#15778)
• Fix on_failure_callback when task receive SIGKILL (#15537)
• Fix dags table overflow (#15660)
• Fix changing the parent dag state on subdag clear (#15562)
• Fix reading from zip package to default to text (#13962)
• Fix wrong parameter for drawDagStatsForDag in dags.html (#13884)
• Fix QueuedLocalWorker crashing with EOFError (#13215)
• Fix typo in NotPreviouslySkippedDep (#13933)
• Fix parallelism after KubeExecutor pod adoption (#15555)
• Fix kube client on mac with keepalive enabled (#15551)
• Fixes wrong limit for dask for python>3.7 (should be <3.7) (#15545)
• Fix Task Adoption in KubernetesExecutor (#14795)
• Fix timeout when using XCom with KubernetesPodOperator (#15388)
• Fix deprecated provider aliases in "extras" not working (#15465)
• Fixed default XCom deserialization. (#14827)
• Fix used_group_ids in dag.partial_subset (#13700) (#15308)
• Further fix trimmed pod_id for KubernetesPodOperator (#15445)
• Bugfix: Invalid name when trimmed pod_id ends with hyphen in KubernetesPodOperator (#15443)
• Fix incorrect slots stats when TI pool_slots > 1 (#15426)
• Fix DAG last run link (#15327)
• Fix sync-perm to work correctly when update_fab_perms = False (#14847)
• Fixes limits on Arrow for plexus test (#14781)
• Fix UI bugs in tree view (#14566)
• Fix AzureDataFactoryHook failing to instantiate its connection (#14565)
• Fix permission error on non-POSIX filesystem (#13121)
• Fix spelling in "ignorable" (#14348)
• Fix get_context_data doctest import (#14288)
• Correct typo in GCSObjectsWtihPrefixExistenceSensor (#14179)
• Fix order of failed deps (#14036)
• Fix critical CeleryKubernetesExecutor bug (#13247)
• Fix four bugs in StackdriverTaskHandler (#13784)
• func.sum may return Decimal that break rest APIs (#15585)
• Persist tags params in pagination (#15411)
• API: Raise AlreadyExists exception when the execution_date is same (#15174)
• Remove duplicate call to sync_metadata inside DagFileProcessorManager (#15121)
• Extra docker-py update to resolve docker op issues (#15731)
• Ensure executors end method is called (#14085)
• Remove user_id from API schema (#15117)
• Prevent clickable bad links on disabled pagination (#15074)
• Acquire lock on db for the time of migration (#10151)
• Skip SLA check only if SLA is None (#14064)
• Print right version in airflow info command (#14560)
• Make airflow info work with pipes (#14528)
• Rework client-side script for connection form. (#14052)
• API: Add CollectionInfo in all Collections that have total_entries (#14366)
• Fix task_instance_mutation_hook when importing airflow.models.dagrun (#15851)

Doc only changes """"""""""""""""

• Fix docstring of SqlSensor (#15466)
• Small changes on "DAGs and Tasks documentation" (#14853)
• Add note on changes to configuration options (#15696)
• Add docs to the markdownlint and yamllint config files (#15682)
• Rename old "Experimental" API to deprecated in the docs. (#15653)
• Fix documentation error in git_sync_template.yaml (#13197)
• Fix doc link permission name (#14972)
• Fix link to Helm chart docs (#14652)
• Fix docstrings for Kubernetes code (#14605)
• docs: Capitalize & minor fixes (#14283) (#14534)
• Fixed reading from zip package to default to text. (#13984)
• An initial rework of the "Concepts" docs (#15444)
• Improve docstrings for various modules (#15047)
• Add documentation on database connection URI (#14124)
• Add Helm Chart logo to docs index (#14762)
• Create a new documentation package for Helm Chart (#14643)
• Add docs about supported logging levels (#14507)
• Update docs about tableau and salesforce provider (#14495)
• Replace deprecated doc links to the correct one (#14429)
• Refactor redundant doc url logic to use utility (#14080)
• docs: NOTICE: Updated 2016-2019 to 2016-now (#14248)
• Skip DAG perm sync during parsing if possible (#15464)
• Add picture and examples for Edge Labels (#15310)
• Add example DAG & how-to guide for sqlite (#13196)
• Add links to new modules for deprecated modules (#15316)
• Add note in Updating.md about FAB data model change (#14478)

Misc/Internal """""""""""""

• Fix logging.exception redundancy (#14823)
• Bump stylelint to remove vulnerable sub-dependency (#15784)
• Add resolution to force dependencies to use patched version of lodash (#15777)
• Update croniter to 1.0.x series (#15769)
• Get rid of Airflow 1.10 in Breeze (#15712)
• Run helm chart tests in parallel (#15706)
• Bump ssri from 6.0.1 to 6.0.2 in /airflow/www (#15437)
• Remove the limit on Gunicorn dependency (#15611)
• Better "dependency already registered" warning message for tasks #14613 (#14860)
• Pin pandas-gbq to <0.15.0 (#15114)
• Use Pip 21.* to install airflow officially (#15513)
• Bump mysqlclient to support the 1.4.x and 2.x series (#14978)
• Finish refactor of DAG resource name helper (#15511)
• Refactor/Cleanup Presentation of Graph Task and Path Highlighting (#15257)
• Standardize default fab perms (#14946)
• Remove datepicker for task instance detail view (#15284)
• Turn provider's import warnings into debug logs (#14903)
• Remove left-over fields from required in provider_info schema. (#14119)
• Deprecate tableau extra (#13595)
• Use built-in cached_property on Python 3.8 where possible (#14606)
• Clean-up JS code in UI templates (#14019)
• Bump elliptic from 6.5.3 to 6.5.4 in /airflow/www (#14668)
• Switch to f-strings using flynt. (#13732)
• use jquery ready instead of vanilla js (#15258)
• Migrate task instance log (ti_log) js (#15309)
• Migrate graph js (#15307)
• Migrate dags.html javascript (#14692)
• Removes unnecessary AzureContainerInstance connection type (#15514)
• Separate Kubernetes pod_launcher from core airflow (#15165)
• update remaining old import paths of operators (#15127)
• Remove broken and undocumented "demo mode" feature (#14601)
• Simplify configuration/legibility of Webpack entries (#14551)
• remove inline tree js (#14552)
• Js linting and inline migration for simple scripts (#14215)
• Remove use of repeated constant in AirflowConfigParser (#14023)
• Deprecate email credentials from environment variables. (#13601)
• Remove unused 'context' variable in task_instance.py (#14049)
• Disable suppress_logs_and_warning in cli when debugging (#13180)

The full changelog is about 3,000 lines long (already excluding everything backported to 1.10), so for now I’ll simply share some of the major features in 2.0.0 compared to 1.10.14:

A new way of writing dags: the TaskFlow API (AIP-31)

(Known in 2.0.0alphas as Functional DAGs.)

DAGs are now much much nicer to author especially when using PythonOperator. Dependencies are handled more clearly and XCom is nicer to use

A quick teaser of what DAGs can now look like:

from airflow.decorators import dag, task
from airflow.utils.dates import days_ago

@dag(default_args={'owner': 'airflow'}, schedule_interval=None, start_date=days_ago(2))
def tutorial_taskflow_api_etl():
   @task
   def extract():
       return {"1001": 301.27, "1002": 433.21, "1003": 502.22}

   @task
   def transform(order_data_dict: dict) -> dict:
       total_order_value = 0

       for value in order_data_dict.values():
           total_order_value += value

       return {"total_order_value": total_order_value}

   @task()
   def load(total_order_value: float):

       print("Total order value is: %.2f" % total_order_value)

   order_data = extract()
   order_summary = transform(order_data)
   load(order_summary["total_order_value"])

tutorial_etl_dag = tutorial_taskflow_api_etl()

Fully specified REST API (AIP-32)

We now have a fully supported, no-longer-experimental API with a comprehensive OpenAPI specification

Read more here:

REST API Documentation.

Massive Scheduler performance improvements

As part of AIP-15 (Scheduler HA+performance) and other work Kamil did, we significantly improved the performance of the Airflow Scheduler. It now starts tasks much, MUCH quicker.

Over at Astronomer.io we’ve benchmarked the scheduler—it’s fast (we had to triple check the numbers as we don’t quite believe them at first!)

Scheduler is now HA compatible (AIP-15)

It’s now possible and supported to run more than a single scheduler instance. This is super useful for both resiliency (in case a scheduler goes down) and scheduling performance.

To fully use this feature you need Postgres 9.6+ or MySQL 8+ (MySQL 5, and MariaDB won’t work with more than one scheduler I’m afraid).

There’s no config or other set up required to run more than one scheduler—just start up a scheduler somewhere else (ensuring it has access to the DAG files) and it will cooperate with your existing schedulers through the database.

For more information, read the Scheduler HA documentation.

Task Groups (AIP-34)

SubDAGs were commonly used for grouping tasks in the UI, but they had many drawbacks in their execution behaviour (primarirly that they only executed a single task in parallel!) To improve this experience, we’ve introduced “Task Groups”: a method for organizing tasks which provides the same grouping behaviour as a subdag without any of the execution-time drawbacks.

SubDAGs will still work for now, but we think that any previous use of SubDAGs can now be replaced with task groups. If you find an example where this isn’t the case, please let us know by opening an issue on GitHub

For more information, check out the Task Group documentation.

Refreshed UI

We’ve given the Airflow UI a visual refresh and updated some of the styling. Check out the UI section of the docs for screenshots.

We have also added an option to auto-refresh task states in Graph View so you no longer need to continuously press the refresh button :).

Smart Sensors for reduced load from sensors (AIP-17)

If you make heavy use of sensors in your Airflow cluster, you might find that sensor execution takes up a significant proportion of your cluster even with “reschedule” mode. To improve this, we’ve added a new mode called “Smart Sensors”.

This feature is in “early-access”: it’s been well-tested by AirBnB and is “stable”/usable, but we reserve the right to make backwards-incompatible changes to it in a future release (if we have to. We’ll try very hard not to!)

Simplified KubernetesExecutor

For Airflow 2.0, we have re-architected the KubernetesExecutor in a fashion that is simultaneously faster, easier to understand, and more flexible for Airflow users. Users will now be able to access the full Kubernetes API to create a .yaml pod_template_file instead of specifying parameters in their airflow.cfg.

We have also replaced the executor_config dictionary with the pod_override parameter, which takes a Kubernetes V1Pod object for a 1:1 setting override. These changes have removed over three thousand lines of code from the KubernetesExecutor, which makes it run faster and creates fewer potential errors.

Airflow core and providers: Splitting Airflow into 60+ packages

Airflow 2.0 is not a monolithic “one to rule them all” package. We’ve split Airflow into core and 61 (for now) provider packages. Each provider package is for either a particular external service (Google, Amazon, Microsoft, Snowflake), a database (Postgres, MySQL), or a protocol (HTTP/FTP). Now you can create a custom Airflow installation from “building” blocks and choose only what you need, plus add whatever other requirements you might have. Some of the common providers are installed automatically (ftp, http, imap, sqlite) as they are commonly used. Other providers are automatically installed when you choose appropriate extras when installing Airflow.

The provider architecture should make it much easier to get a fully customized, yet consistent runtime with the right set of Python dependencies.

But that’s not all: you can write your own custom providers and add things like custom connection types, customizations of the Connection Forms, and extra links to your operators in a manageable way. You can build your own provider and install it as a Python package and have your customizations visible right in the Airflow UI.

Security

As part of Airflow 2.0 effort, there has been a conscious focus on Security and reducing areas of exposure. This is represented across different functional areas in different forms. For example, in the new REST API, all operations now require authorization. Similarly, in the configuration settings, the Fernet key is now required to be specified.

Configuration

Configuration in the form of the airflow.cfg file has been rationalized further in distinct sections, specifically around “core”. Additionally, a significant amount of configuration options have been deprecated or moved to individual component-specific configuration files, such as the pod-template-file for Kubernetes execution-related configuration.

We’ve tried to make as few breaking changes as possible and to provide deprecation path in the code, especially in the case of anything called in the DAG. That said, please read through UPDATING.md to check what might affect you. For example: We re-organized the layout of operators (they now all live under airflow.providers.*) but the old names should continue to work - you’ll just notice a lot of DeprecationWarnings that need to be fixed up.

Release Date: 2024-11-18

Features

Add support for semicolon stripping to DbApiHook, PrestoHook, and TrinoHook (#41916)

Add ProcessingEngineRunFacet to OL DAG Start event (#43213)

Bug Fixes

serialize asset/dataset timetable conditions in OpenLineage info also for Airflow 2 (#43434)

OpenLineage: accept whole config when instantiating OpenLineageClient. (#43740)

Misc

Temporarily limit openlineage to <1.24.0 (#43732)

Move python operator to Standard provider (#42081)

Release Date: 2024-10-31

Features

feat(providers/openlineage): Use asset in common provider (#43111)

Misc

Ignore attr-defined for compat import (#43301)

nit: remove taskgroup's tooltip from OL's AirflowJobFacet (#43152)

require 1.2.1 common.compat for openlineage provider (#43039)

Release Date: 2024-10-14

Misc

Change imports to use Standard provider for BashOperator (#42252)

Drop python3.8 support core and providers (#42766)

Rename dataset related python variable names to asset (#41348)

Release Date: 2024-10-01

Bug Fixes

fix: OpenLineage dag start event not being emitted (#42448)

fix: typo in error stack trace formatting for clearer output (#42017)

Release Date: 2024-09-24

Features

feat: notify about potential serialization failures when sending DagRun, don't serialize unnecessary params, guard listener for exceptions (#41690)

Bug Fixes

fix: cast list to flattened string in openlineage InfoJsonEncodable (#41786)

Misc

chore: bump OL provider dependencies versions (#42059)

move to dag_run.logical_date from execution date in OpenLineage provider (#41889)

Unify DAG schedule args and change default to None (#41453)

Release Date: 2024-08-28

Note

This release of provider is only available for Airflow 2.8+ as explained in the Apache Airflow providers support policy .

Features

feat: add debug facet to all OpenLineage events (#41217)

feat: add fileloc to DAG info in AirflowRunFacet (#41311)

feat: remove openlineage client deprecated from_environment() method (#41310)

feat: openlineage listener captures hook-level lineage (#41482)

Bug Fixes

fix: get task dependencies without serializing task tree to string (#41494)

fix: return empty data instead of None when OpenLineage on_start method is missing (#41268)

fix: replace dagTree with downstream_task_ids (#41587)

Misc

Bump minimum Airflow version in providers to Airflow 2.8.0 (#41396)

chore: remove openlineage deprecation warnings (#41284)

Release Date: 2024-08-06

Features

Add AirflowRun on COMPLETE/FAIL events (#40996)

openlineage: extend custom_run_facets to also be executed on complete and fail (#40953)

openlineage: migrate OpenLineage provider to V2 facets. (#39530)

openlineage: Add AirflowRunFacet for dag runEvents (#40854)

[AIP-62] Translate AIP-60 URI to OpenLineage (#40173)

Ability to add custom facet in OpenLineage events (#38982)

openlineage: add method to common.compat to not force hooks to try/except every 2.10 hook lineage call (#40812)

openlineage: use airflow provided getters from conf (#40790)

openlineage: add config to include 'full' task info based on conf setting (#40589)

Add TaskInstance log_url to OpenLineage facet (#40797)

openlineage: add deferrable information to task info in airflow run facet (#40682)

Bug Fixes

Adjust default extractor's on_failure detection for airflow 2.10 fix (#41094)

openlineage: make value of slots in attrs.define consistent across all OL usages (#40992)

Set 'slots' to True for facets used in DagRun (#40972)

openlineage: fix / add some task attributes in AirflowRunFacet (#40725)

Misc

openlineage: replace dt.now with airflow.utils.timezone.utcnow (#40887)

openlineage: remove deprecated parentRun facet key (#40681)

Release Date: 2024-07-12

Bug Fixes

fix openlineage parsing dag tree with MappedOperator (#40621)

Release Date: 2024-07-04

Features

local task job: add timeout, to not kill on_task_instance_success listener prematurely (#39890)

openlineage: add some debug logging around sql parser call sites (#40200)

Add task SLA and queued datetime information to AirflowRunFacet (#40091)

Add error stacktrace to OpenLineage task event (#39813)

Introduce AirflowJobFacet and AirflowStateRunFacet (#39520)

Use UUIDv7 for OpenLineage runIds (#39889)

openlineage: execute extraction and message sending in separate process (#40078)

Add few removed Task properties in AirflowRunFacet (#40371)

Bug Fixes

openlineage, redshift: do not call DB for schemas below Airflow 2.10 (#40197)

fix: scheduler crashing with OL provider on airflow standalone (#40459)

nit: fix logging level (#40461)

fix: provide stack trace under proper key in OL facet (#40372)

Misc

fix: sqa deprecations for airflow providers (#39293)

Enable enforcing pydocstyle rule D213 in ruff. (#40448)

Release Date: 2024-05-30

Warning

In Airflow 2.10.0, we fix the way try_number works. For Airflow >= 2.10.0, use apache-airflow-providers-openlineage >= 1.8.0 . Older versions of Airflow are not affected, In case you run an incompatible version an exception will be raised asking you to upgrade provider version.

Features

Scheduler to handle incrementing of try_number (#39336)

Bug Fixes

fix: Prevent error when extractor can't be imported (#39736)

Re-configure ORM in spawned OpenLineage process in scheduler. (#39735)

Misc

chore: Update conf retrieval docstring and adjust pool_size (#39721)

Remove 'openlineage.common' dependencies in Google and Snowflake providers. (#39614)

Use 'ProcessPoolExecutor' over 'ThreadPoolExecutor'. (#39235)

misc: Add custom provider runtime checks (#39609)

Faster 'airflow_version' imports (#39552)

Simplify 'airflow_version' imports (#39497)

openlineage: notify that logged exception was caught (#39493)

chore: Add more OpenLineage logs to facilitate debugging (#39136)

Release Date: 2024-05-06

Misc

openlineage, snowflake: do not run external queries for Snowflake (#39113)

Raw changelog

for the record, generated with: ~/node_modules/github-changes/bin/index.js -o airbnb -r airflow --only-pulls --use-commit-body --between-tags 1.6.2...1.7.0rc1 --token {your_gh_token_here}

1.7.0

• #1147 Enhance CLI Test command to accept a JSON-formatted dictionary of par… (@r39132)
• #975 statuses column on /admin shows only active or most recent dag_runs (@jtschoonhoven)
• #1127 FAQ entry about start_date (@mistercrunch)
• #1129 Showing active dag runs as in (3/16) in tooltip (@mistercrunch)
• #1144 More explicit zindex on modals (@mistercrunch)
• #1143 small fixes to previous bq project inclusion pr. (@mtagle)
• #1139 Allow specificiation of project in BigQuery Hook methods (@mtagle)
• #1110 Add date support to MySQL to GCS operator (@criccomini)
• #1135 Added start_date initialization for DagRun creation within schedule_dag(self, dag_id) (@RvN76)
• #1140 License check (@bolkedebruin)
• #1138 Add support for BigQuery User Defined Functions in BigQuery operator (@LeBlanc)
• #1132 Add custom email backends. (@jmcarp)
• #1119 Add gcloud-based GCSHook (@jlowin)
• #1134 Parameterizing DagBag import timeouts (@mistercrunch)
• #1045 Add startup scripts for upstart based systems (@d-lee)
• #1131 Adding ssh connection type to webform (@hyperborea)
• #792 add error handling for slack api (@abridgett)
• #1115 This patch adds license checking for Airflow. (@bolkedebruin)
• #1116 Add WePay and committer list to README.md (@criccomini)
• #1118 Don't force installation of GCP API client dependencies (@jlowin)
• #1090 Adding template support in qbol operator (@msumit)
• #1108 new company + link to pitfalls (@kretes)
• #1104 Add INT24 (MEDIUMINT) support to MySQL to Google cloud storage operator (@criccomini)
• #1076 SubDAG docs and examples (@nicktrav)
• #1101 modify datastore hook so that authorization is maintained for the lif… (@mtagle)
• #1099 Adding @mention in Airbnb and links to airbnb open source projects (@artwr)
• #1096 Make SqlAlchemy pool_recycle and pool_size configurable (@amread)
• #1070 Proper sqlalchemy syntax for desc (@mistercrunch)
• #1079 SID Oracle DB connection support (@biln)
• #1094 Adding to inits (@artwr)
• #1093 Add MySQL to BQ support for TINYINT (@criccomini)
• #1088 Fix MySQL to Google cloud storage scoping. (@criccomini)
• #1075 Pig hook and operator stub (@artwr)
• #1084 Replace deprecated flask.ext.* with flask_* (@jeffwidman)
• #1082 Cleanup Contributing.md (@jeffwidman)
• #1086 Default to 0 if no rows loaded in GCS to BQ operator. (@criccomini)
• #1080 Add MySQL->GCS, GCS->BQ operators (@criccomini)
• #1073 Fixing small issue with qbol operator and hook (@msumit)
• #1068 Add a new hook for google datastore (@mtagle)
• #1065 dag pausing should pause queued tasks as well (@mistercrunch)
• #1067 Add two methods to BigQueryBaseCursor: (@mtagle)
• #1064 Add Thumbtack to organizations using Airflow (@natekupp)
• #1063 Documenting task details doc_ feature (@mistercrunch)
• #1049 Allow the use of the autoconfig client (@bolkedebruin)
• #1056 Add output encoding option to BashOperator (@Attsun1031)
• #996 Added the dst filename to template_fields GoogleCloudStorageDownloadOperator (@0xR)
• #1051 Running unit tests with local executor (@mistercrunch)
• #1008 Add FTPSHook class. (@geeknam)
• #1055 Improve [Hide/Show all series] perforamce (@lumengxi)
• #1054 CLI's trigger_dag now accepts --conf as json (@mistercrunch)
• #1053 Add Kogan.com to the list of users (@geeknam)
• #1046 Documenting the cluster policy feature (@mistercrunch)
• #1044 Updating the Readme with a link to the TriggerDagRunOperator post (@r39132)
• #1043 Adding an example to illustrate the TriggerDagRunOperator (@r39132)
• #1039 Only set headers and delimiters for CSV exports in Google BigQuery hook (@criccomini)
• #1026 Minor documentation tweaks to the FAQ under the fernet key section (@r39132)
• #1025 Add notes on connection password encryption (@d-lee)
• #1018 add SSL support for SMTP (@bbrumi)
• #1020 Add BigQuery copy operator. (@criccomini)
• #964 Clean test database out in between unit test runs (@nicktrav)
• #1019 Add direct dependencies for Google cloud contribs (@criccomini)
• #985 [airflow][presto] Gracefully handle 503 errors and avoid eval() (@airbnb)
• #1013 Fixed issue 1012: pool not used with celery executor (@rdavison)
• #1011 drop the tmp table after ingestion (@dayzzz)
• #995 Fix LDAP error messages when login fails. (@criccomini)
• #1006 Fix forgetting to expose yesterday_ds_nodash and tomorrow_ds_nodash (@0xR)
• #1003 Update to the README such as adding Hootsuite (@r39132)
• #998 Pedantic documentation tweaks (@joshmarlow)
• #940 Refresh border coloring in graph view without having to refresh the entire page (@DinoCow)
• #987 Add logout button to Airflow (@criccomini)
• #930 When either data_profiler_filter or superuser_filter aren't defined,… (@mtagle)
• #994 there could be dot in the key string, which is illegal in the hive ta… (@dayzzz)
• #993 Added yesterday_ds_nodash and tommorow_ds_nodash (@0xR)
• #992 Rename BashOperator.xcom_push member to avoid collision with parent class (@seregasheypak)
• #978 Fixing conflicting params in default_args (@mistercrunch, @nicktrav)
• #984 Add tests for params handling in Dag construction (@nicktrav)
• #986 [documentation] document that max_active_runs can prevent a DAG from running (@aoen)
• #983 Added destination_dataset_table to template_fieds of bigquery_operator (@0xR)
• #981 Add BigQuery PEP 249 support (@criccomini)
• #980 Support user-defined macros and params in dry-run backfills with task… (@r39132)
• #976 More verbose logging for are_dependencies_met when called from run (@mistercrunch)
• #944 Fix statuses in dag list for dags with dag_ids prefixed by a number (@wil5for)
• #942 Set effective user of (web)hdfs hooks using connection config, optionally overridden by constructor (@xadhoom)
• #965 Adding mock lib to devel extras_require (@mistercrunch)
• #963 Only schedule DagRuns between start and end dates (@nicktrav)
• #948 Fix encryption alert msg for env var, configuration->conf refactor (@mistercrunch)
• #960 Fix parsing file that contains multi byte char (@Attsun1031)
• #949 Allow for domain-wide delegation in google cloud apps (@mtagle)
• #958 Fix bug in reporting of attempt number when queuing tasks in a pool (@r39132)
• #946 Add dag_state to cli (@wil5for)
• #956 Support new configuration dags_are_paused_at_creation that, when True… (@r39132)
• #945 Add documentation to gcs_download_operator (@criccomini)
• #947 Encrypt Variables if Fernet key provided (@r39132)
• #943 Fix running custom pool task with mark-success is not marked success (@Attsun1031)
• #816 Implement SSH Execute Operator (@KeenS)
• #941 Fix process_subdir bug (@Attsun1031)
• #939 Show rendered templates from the CLI (@mistercrunch)
• #224 PrestoToMySqlTransfer (@mistercrunch)
• #801 allow slack attachments to be templated (@abridgett)
• #938 [hotfix] dag missing from dagbag (@mistercrunch)
• #862 handle default option for extra_options argument in HttpHook.run method (@JoergRittinger)
• #935 Fix and refactor the httphook (@mistercrunch)
• #919 Add support for three-legged OAuth for Google connections. Useful for… (@criccomini)
• #937 Statsd abstraction (@mistercrunch)
• #743 Get connection string containing username and password during runtime (@praveev)
• #926 Normalize plugin paths that include both slashes and dots (@jasonjho)
• #934 Support for encrypting the connection extra field (@r39132)
• #928 fix function name error: active_tasks (@flying5)
• #882 Docker operator (@asnir)
• #925 Small change to fix a crash when reading log files from remote workers (@kmevissen)
• #868 Encrypt logs (@vansivallab)
• #927 [fix] disregarding adhoc tasks when closing dag runs (@mistercrunch)
• #918 Reverting production issues from 876 and undead (@mistercrunch)
• #922 Add link to SmartNews in README (@takus)
• #875 SLA Miss Alert Callbacks : Allow DAGs to specify a callback function for SLA miss handling (@r39132)
• #917 Allowing for relative path and dot notation for -sd (@mistercrunch)
• #901 Make Google Cloud Storage download operator use a filename, not a fil… (@criccomini)
• #912 Add Clover Health to Airflow users (@vansivallab)
• #813 Add "search_scope" as a configuration variable for LDAP (@NeilHanlon)
• #910 Add similarweb airflow puppet module to the readme (@danielbenzvi)
• #909 Fixes typo (@rosner)
• #907 Call Session.remove after each run, to survive DB restart (@KMK-ONLINE)
• #906 Update some datetime column default args for consistent treatment (@t1m0thy)
• #904 Bugix for TriggerDagRunOperator (@t1m0thy)
• #905 Add SimilarWeb to the who uses airflow section (@danielbenzvi)
• #903 Fixing the tutorial. Removing an unnecessary import of MySqlOperator … (@cesararevalo)
• #797 Add super user and profiler to ldap (@criccomini)
• #898 Making force a task instance member, so it becomes available for (@iddoav)
• #896 Cherry picked commits out of recent rollback (@mistercrunch)
• #894 [hotfix] fixing infinite retries in prod (@mistercrunch)
• #893 Fix ISSUE 798: password printed to STDOUT when running initdb, resetdb, upgradedb (@DinoCow)
• #889 Merging airbnb_prod hotfixes into master (@mistercrunch)
• #887 Killing tasks that aren't in a running state (@mistercrunch)
• #888 Airbnb prod (@mistercrunch)
• #808 Adding support for QDS (Qubole Date Services) (@msumit)
• #876 Airbnb prod (@iivvaall)
• #884 Fixing issue where try_number isn't incremented (@mistercrunch)
• #878 state wasn't being saved (@abridgett)
• #879 Readme update (@thibault-ketterer)
• #835 Issue 832: creating run_id if not given while executing trigger_dag (@msumit)
• #867 Airbnb prod (@mistercrunch)
• #866 [hotfix] fixing subdag not refreshing properly (@mistercrunch)
• #865 Bigquery and Google Cloud Storage operators (@criccomini)
• #861 Bigquery operator (@criccomini)
• #779 Adding base date and run number form to Task Duration and Landing Times views (@MaximeKestemont)
• #854 [hotfix] subdag not showing up (@mistercrunch)
• #853 Fixing a bug where some dags can't be retrived from DagBag.get_dag (@mistercrunch)
• #848 Delete dags from dagbag (@mistercrunch)
• #851 add a parameter for number of shard in batch ingestion (@dayzzz)
• #850 Update models.py to increase password field length (@rahul342)
• #763 Added title attribute for logs (@JordyMoos)
• #817 Fix ISSUE-812 (@bolkedebruin)
• #840 Fixing SLA handling related bug (@mistercrunch)
• #841 Add a Gitter chat badge to README.md (@gitter-badger)
• #839 Adding visibility as to which dag is pickleable (@mistercrunch)
• #826 Feature/cli fixes (@abridgett)
• #830 Add Sidecar Interactive to list of companies using Airflow (@robottokauf3)
• #825 color alternate rows so it's easier to use (@abridgett)
• #802 Implemented GitHub Enterprise Authentication (@mtp401)
• #831 order dag run drop down in graph view (@abridgett)
• #828 Fix invalid syntax in SSHHook (@robottokauf3)
• #822 Merging prod hotfixes into master (@mistercrunch)
• #824 Upgrade flask admin (@mistercrunch)
• #823 Fixing bad ONE_FAILED in recent PR (@mistercrunch)

v1.6.0 brings:

• [scheduler] the notion of DAG runs allows for more parallelization, and controls around scheduling (max number of running task instance per DAG, max number of DAG runs to be evaluated for scheduling, ...)
• [scheduler] support for "externally triggered" DAG runs, or DAGs that run on demand as opposed to on a schedule
• [scheduler] support for cron-like syntax (as in: "0 0 * * *") and macros (as in "@montlhly", "@hourly", "@weekly", ...)
• UI changes related to new scheduler features
• LDAP authentication for the web UI, more extensible authentication backend
• UI activity logging
• WebHdfsSensor and Hook for HDFS interactions that are py3 compatible
• Continuous integration with Travis-UI and Coveralls
• ShortCircuitOperator
• python3 compatibility!
• Tons of bug fixes and incremental improvements
• + all the things I'm forgetting while browsing through an infinite list of commits!

Thanks to everyone in the community for all the PRs (stellar contributions!), comments and issue reporting.

v1.5.0 is a huge release. Tons of important features.

Make sure to run airflow upgradedb after you upgrade

Improvements:

• @jlowin landed a feature to communicate information across tasks called XCom, more information here
• @neovintage integrated Airflow with Alembic, making database migration easy, run airflow upgradedb to get your database up to date as you upgrade Airflow
• The dependency engine is now more flexible, allowing for trigger rules, before this update, tasks would only be triggered when all dependencies were successful (still the default), now you can set tasks to trigger when a single parent succeeds, when one fails, when they all fail, or to fire regardless of their dependencies:
• @neovintage added support for connection to be defined in environment variables, allowing to bypass or override the metadata database
• @jlowin improved the tree view to render a non expendable when trees go above 5k nodes
• The Druid hook and HiveToDruidTransfer are maturing and becoming production grade
• @kapil-malik added a UI feature where some users can only see their own DAGs while superusers can still see all, this feature is turned off by default
• Allowing all operators and some preoperators to receive list of sql statements to be execute as a chain
• Improved the task_instance table to log the operator name and queued timestamp
• Passwords in the metadata can now be encrypted
• Improvements to the unit tests (speed + coverage)
• Bugfixes + more

• Python3 compatibility improvements
• + TimeDeltaSensor
• + Slack related hooks and operators
• Fancy widget for pausing DAGs from the main dash
• Bugfixes and polish!

Release Date: 2023-12-27

Bug Fixes

Fix typo. (#36362)

Release Date: 2023-12-12

Features

feat: Add parent_run_id for COMPLETE and FAIL events (#36067)

Add basic metrics to stats collector. (#35368)

Bug Fixes

fix: Repair run_id for OpenLineage FAIL events (#36051)

Fix and reapply templates for provider documentation (#35686)

Misc

Remove ClassVar annotations. (#36084)

Release Date: 2023-11-12

Misc

Make schema filter uppercase in 'create_filter_clauses' (#35428)

Release Date: 2023-10-31

Features

Send column lineage from SQL operators. (#34843)

Release Date: 2023-10-17

Misc

Adjust log levels in OpenLineage provider (#34801)

Release Date: 2023-09-12

Features

Allow to disable openlineage at operator level (#33685)

Bug Fixes

Fix import in 'get_custom_facets'. (#34122)

Misc

Improve modules import in Airflow providers by some of them into a type-checking block (#33754)

Add OpenLineage support for DBT Cloud. (#33959)

Refactor unneeded jumps in providers (#33833)

Refactor: Replace lambdas with comprehensions in providers (#33771)

Release Date: 2023-08-29

Bug Fixes

openlineage: don't run task instance listener in executor (#33366)

openlineage: do not try to redact Proxy objects from deprecated config (#33393)

openlineage: defensively check for provided datetimes in listener (#33343)

Misc

Add OpenLineage support for Trino. (#32910)

Simplify conditions on len() in other providers (#33569)

Replace repr() with proper formatting (#33520)