apache-airflow-providers-common-sql

v1.6.0 brings:

• [scheduler] the notion of DAG runs allows for more parallelization, and controls around scheduling (max number of running task instance per DAG, max number of DAG runs to be evaluated for scheduling, ...)
• [scheduler] support for "externally triggered" DAG runs, or DAGs that run on demand as opposed to on a schedule
• [scheduler] support for cron-like syntax (as in: "0 0 * * *") and macros (as in "@montlhly", "@hourly", "@weekly", ...)
• UI changes related to new scheduler features
• LDAP authentication for the web UI, more extensible authentication backend
• UI activity logging
• WebHdfsSensor and Hook for HDFS interactions that are py3 compatible
• Continuous integration with Travis-UI and Coveralls
• ShortCircuitOperator
• python3 compatibility!
• Tons of bug fixes and incremental improvements
• + all the things I'm forgetting while browsing through an infinite list of commits!

Thanks to everyone in the community for all the PRs (stellar contributions!), comments and issue reporting.

This is probably missing some important items, but most of it should be captured

• Initial setup on Travis CI provides continuous Integration, automated testing, no Hadoop unit tests just yet, but that is coming up
• Unit tests coverage reports with coverals
• Better py3 compatibility, unit tests run against both 2.7 ad 3.4 and we're now using the from __future__ imports to prevent regression
• A MesosExecutor to run your tasks on Mesos
• Some Kerberos integration for Hive / Hadoop
• DAG's graph view legend for states are now toggles to highlight tasks in specific states
• Automated zombie task instance killing as part of the scheduler's routine. The process looks for running tasks that don't have a heartbeat and kills them
• MySqlHook bulk load option
• More options in the UI's Mark Success form
• MySQL uses mysqlclient lib instead of mysql-python
• Using gunicorn instead of tornado as the wsgi web server
• OracleHook
• FTPHook
• Much more! tons of bug fixes and usability improvements.

Bugfix around XCom table creation timestamp issue

v1.5.0 is a huge release. Tons of important features.

Make sure to run airflow upgradedb after you upgrade

Improvements:

• @jlowin landed a feature to communicate information across tasks called XCom, more information here
• @neovintage integrated Airflow with Alembic, making database migration easy, run airflow upgradedb to get your database up to date as you upgrade Airflow
• The dependency engine is now more flexible, allowing for trigger rules, before this update, tasks would only be triggered when all dependencies were successful (still the default), now you can set tasks to trigger when a single parent succeeds, when one fails, when they all fail, or to fire regardless of their dependencies:
• @neovintage added support for connection to be defined in environment variables, allowing to bypass or override the metadata database
• @jlowin improved the tree view to render a non expendable when trees go above 5k nodes
• The Druid hook and HiveToDruidTransfer are maturing and becoming production grade
• @kapil-malik added a UI feature where some users can only see their own DAGs while superusers can still see all, this feature is turned off by default
• Allowing all operators and some preoperators to receive list of sql statements to be execute as a chain
• Improved the task_instance table to log the operator name and queued timestamp
• Passwords in the metadata can now be encrypted
• Improvements to the unit tests (speed + coverage)
• Bugfixes + more

• Python3 compatibility improvements
• + TimeDeltaSensor
• + Slack related hooks and operators
• Fancy widget for pausing DAGs from the main dash
• Bugfixes and polish!

Release Date: 2023-03-06

Bug Fixes

Do not process output when do_xcom_push=False (#29599)

Release Date: 2023-01-17

Bug Fixes

Handle non-compliant behaviour of Exasol cursor (#28744)

Release Date: 2023-01-05

Bug Fixes

fIx isort problems introduced by recent isort release (#28434)

Fix template rendering for Common SQL operators (#28202)

Defer to hook setting for split_statements in SQLExecuteQueryOperator (#28635)

Misc

Clarify docstrings for updated DbApiHook (#27966)

Release Date: 2022-11-29

This release fixes a few errors that were introduced in common.sql operator while refactoring common parts:

_process_output method in SQLExecuteQueryOperator has now consistent semantics and typing, it can also modify the returned (and stored in XCom) values in the operators that derive from the SQLExecuteQueryOperator ).

descriptions of all returned results are stored as descriptions property in the DBApiHook

last description of the cursor whether to return single query results values are now exposed in DBApiHook via last_description property.

Lack of consistency in the operator caused 1.3.0 to be yanked - the 1.3.0 should not be used - if you have 1.3.0 installed, upgrade to 1.3.1 .

Bug Fixes

Restore removed (but used) methods in common.sql (#27843)

Fix errors in Databricks SQL operator introduced when refactoring (#27854)

Bump common.sql provider to 1.3.1 (#27888)

Fixing the behaviours of SQL Hooks and Operators finally (#27912)

Warning

This release has been yanked with a reason: Breaks Google 8.4.0 provider for SQLExecute

Note

This release of provider is only available for Airflow 2.3+ as explained in the Apache Airflow providers support policy .

Misc

Move min airflow version to 2.3.0 for all providers (#27196)

Features

Add SQLExecuteQueryOperator (#25717)

Use DbApiHook.run for DbApiHook.get_records and DbApiHook.get_first (#26944)

DbApiHook consistent insert_rows logging (#26758)

Bug Fixes

Common sql bugfixes and improvements (#26761)

Use unused SQLCheckOperator.parameters in SQLCheckOperator.execute. (#27599)

Release Date: 2022-09-08

Features

Make placeholder style configurable (#25939)

Better error message for pre-common-sql providers (#26051)

Bug Fixes

Fix (and test) SQLTableCheckOperator on postgresql (#25821)

Don't use Pandas for SQLTableCheckOperator (#25822)

Discard semicolon stripping in SQL hook (#25855)

Release Date: 2022-08-18

Features

Improve taskflow type hints with ParamSpec (#25173)

Move all "old" SQL operators to common.sql providers (#25350)

Deprecate hql parameters and synchronize DBApiHook method APIs (#25299)

Unify DbApiHook.run() method with the methods which override it (#23971)

Common SQLCheckOperators Various Functionality Update (#25164)

Bug Fixes

Allow Legacy SqlSensor to use the common.sql providers (#25293)

Fix fetch_all_handler & db-api tests for it (#25430)

Align Common SQL provider logo location (#25538)

Fix SQL split string to include ';-less' statements (#25713)

Release Date: 2022-07-16

Initial version of the provider. Adds SQLColumnCheckOperator and SQLTableCheckOperator . Moves DBApiHook , SQLSensor and ConnectorProtocol to the provider.

Previous Next

Was this entry helpful?

Suggest a change on this page