NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
PyPI · #5662 most downloaded on PyPI
Provider package apache-airflow-providers-apache-spark for Apache Airflow
Last release 1 months ago
23 Aug 2026
Ships fairly regularly
a new release about every 2 weeks
Nearly every release is documented
notes for 60 of the last 60 stable releases
Nothing withdrawn
no release was ever pulled
6 years old
135 releases · first in 2020
Make SparkSubmitOperator durable execution inert below Airflow 3.3
Make SparkSubmitOperator durable execution inert below Airflow 3.3 (#71534)
Make 'durable' reach 'default_args' and warn when set below Airflow 3.3 (#71531)
Document how clearing tasks works with task state store on durable operators (#71358)
Nothing published for this version
Anchor spark-submit log trail on actual exception instead of lines
One column per quarter.
Anchor spark-submit log trail on actual exception instead of lines (#70513)
Spark: Fix missing space in YARN waitAppCompletion error message (#71055)
Decouple SparkSubmitOperator resumable deployment backends (#68679)
Drop unused post_submit_commands copy from SparkSubmitOperator init (#70323)
Nothing published for this version
Nothing published for this version
Surface YARN diagnostics for spark on yarn clusters
Surface YARN diagnostics for spark on yarn clusters (#70183)
Include spark submit canonical logs in failure exceptions (#70178)
Better log when spark k8s driver remains in Unknown Phase (#70176)
Link task state store docs in durable execution across providers (#69851)
Document each provider's optional extras in its docs index (#69478)
Nothing published for this version
Add a standard toggle for resumability to ResumableJobMixin
Add a standard toggle for resumability to ResumableJobMixin (#68623)
Add crash recovery ability to SparkSubmitOperator against Kubernetes (#68067)
Crash recovery for YARN cluster mode in SparkSubmitOperator built on AIP-103 (#67473)
Nothing published for this version
Nothing published for this version
Add 'ResumableJobMixin' with 'SparkSubmitOperator' as a case study for surviving worker failures (standalone)
Add 'ResumableJobMixin' with 'SparkSubmitOperator' as a case study for surviving worker failures (standalone) (#67118)
Track Spark job status for YARN cluster mode via RM REST API to free JVM (#65991)
Use K8s API to track Spark on K8s instead of JVM based spark-submit (#67715)
Enhance 'ResumableJobMixin.get_job_status' with context for better job status tracking (#68009)
Rename resumablemixin file to match class name (#68137)
Auto-sync provider README.rst Requirements with pyproject.toml (#67669)
Document REST scheme and port connection fields for Spark (#67682)
Nothing published for this version
Nothing published for this version
Fix SparkPipelinesHook for Spark Connect (sc://) connections
Fix SparkPipelinesHook for Spark Connect (sc://) connections (#66498)
Nothing published for this version
Add post_submit_commands to SparkSubmitHook for sidecar lifecycle management
Add post_submit_commands to SparkSubmitHook for sidecar lifecycle management (#64391)
Load hook metadata from YAML without importing Hook class (#63826)
Nothing published for this version
Breaking changes ~~~~~~~~~~~~~~~~
Warning
The pyspark package is no longer included by default, so only the spark-connect connection will works by default. If you want to keep using other spark connection types you must install the airflow spark provider with the pyspark extra
example: apache-airflow-providers-apache-spark[pyspark]==X.Y.Z
Because the package pyspark is more than 400mb and is not necessary if only using spark-connect to trigger a job
The minimum pyspark and spark-connect version is now 4.0.0
[breaking] Make pyspark-client as default and pyspark package optional (#60031)
Fix unclear error when pyspark is not installed for JDBC script (#64174)
Add Python 3.14 Support (#63520)
Nothing published for this version
.. Below changes are excluded from the changelog. Move them to appropriate section above if needed. Do not delete the lines(!):
spark-pipelines operator (#61681)
Nothing published for this version
fix: resolve connection master URL construction for various protocols in SparkSubmitHook
fix: resolve connection master URL construction for various protocols in SparkSubmitHook (#61528)
Nothing published for this version
PysparkOperator and not only a decorator
PysparkOperator and not only a decorator (#60041)
New year means updated Copyright notices (#60344)
Source conf from 'airflow.sdk.configuration.conf' for apache providers (#59966)
Nothing published for this version
Cleanup of variables in settings.py
Cleanup of variables in settings.py (#59875)
Nothing published for this version
Add backcompat for exceptions in providers
Add backcompat for exceptions in providers (#58727)
Nothing published for this version
.. note:: This release of provider is only available for Airflow 2.11+ as explained in the Apache Airflow providers support policy _.
Note
This release of provider is only available for Airflow 2.11+ as explained in the Apache Airflow providers support policy <https://github.com/apache/airflow/blob/main/PROVIDERS.rst#minimum-supported-version-of-airflow-for-community-managed-providers>_.
Bump minimum Airflow version in providers to Airflow 2.11.0 (#58612)
Nothing published for this version
Convert all airflow distributions to be compliant with ASF requirements
Convert all airflow distributions to be compliant with ASF requirements (#58138)
Nothing published for this version
Use ValueError instead of RuntimeError when resolving SparkSubmitHook connection
Use ValueError instead of RuntimeError when resolving SparkSubmitHook connection (#56631)
Migrate Apache providers & Elasticsearch to ''common.compat'' (#57016)
Nothing published for this version
Add Python 3.13 support for Airflow.
Deprecate decorators from Core (#53629)
Add Python 3.13 support for Airflow. (#46891)
Cleanup type ignores (#53301)
Remove type ignore across codebase after mypy upgrade (#53243)
Remove upper-binding for "python-requires" (#52980)
Temporarily switch to use >=,< pattern instead of '~=' (#52967)
Replace BaseHook to Task SDK for apache/pyspark (#52842)
Replace 'BaseHook' to Task SDK for 'apache/spark' (#52683)
Nothing published for this version
Move 'BaseHook' implementation to task SDK
Move 'BaseHook' implementation to task SDK (#51873)
add: version_compat (#52448)
Drop support for Python 3.9 (#52072)
Replace 'models.BaseOperator' to Task SDK one for Standard Provider (#52292)
Cleanup unused args example_pyspark.py (#52492)
Nothing published for this version
.. note:: This release of provider is only available for Airflow 2.10+ as explained in the Apache Airflow providers support policy _.
Note
This release of provider is only available for Airflow 2.10+ as explained in the Apache Airflow providers support policy <https://github.com/apache/airflow/blob/main/PROVIDERS.rst#minimum-supported-version-of-airflow-for-community-managed-providers>_.
add root parent information to OpenLineage events (#49237)
Bump Pyspark to even higher version (#50308)
Lower bind pyspark and pydruid to relatively new versions (#50205)
Remove AIRFLOW_2_10_PLUS conditions (#49877)
Bump min Airflow version in providers to 2.10 (#49843)
Nothing published for this version
.. Below changes are excluded from the changelog. Move them to appropriate section above if needed. Do not delete the lines(!):
remove superfluous else block (#49199)
Nothing published for this version
Add openlineage as Extra dep for Spark provider
Add openlineage as Extra dep for Spark provider (#48972)
Make '@task' import from airflow.sdk (#48896)
Nothing published for this version
add OpenLineage configuration injection to SparkSubmitOperator
add OpenLineage configuration injection to SparkSubmitOperator (#47508)
Nothing published for this version
spark on kubernetes removes dependency on Spark Exit code
spark on kubernetes removes dependency on Spark Exit code (#46817)
Upgrade flit to 3.11.0 (#46938)
Include driver classpath in --jars cmd docstring in spark-submit hook and operator (#45210)
Nothing published for this version
Breaking changes ~~~~~~~~~~~~~~~~
Note
This release of provider is only available for Airflow 2.9+ as explained in the Apache Airflow providers support policy.
Warning
All deprecated classes, parameters and features have been removed from the Apache Spark provider package. The following breaking changes were introduced:
Operators
Removed _sql() support for SparkSqlOperator. Please use sql attribute instead. _sql was introduced in 2016 and since it was listed as templated field, which is no longer the case, we handled it as public api despite the _ prefix that marked it as private.
Remove deprecated code from apache spark provider (#44567)
Bump minimum Airflow version in providers to Airflow 2.9.0 (#44956)
Fix failing mypy check on 'main' (#44191)
spark-submit: replace 'principle' by 'principal' (#44150)
Update DAG example links in multiple providers documents (#44034)
Nothing published for this version
Nothing published for this version
Move python operator to Standard provider
Move python operator to Standard provider (#42081)
Nothing published for this version
Changed conf property from str to dict in SparkSqlOperator
Changed conf property from str to dict in SparkSqlOperator (#42835)
Nothing published for this version
Refactor function resolve_kerberos_principal
Refactor function resolve_kerberos_principal (#42777)
Nothing published for this version
Add kerberos related connection fields(principal, keytab) on SparkSubmitHook
Add kerberos related connection fields(principal, keytab) on SparkSubmitHook (#40757)
Nothing published for this version
Your coding agent can read these notes before it upgrades. Set up the MCP server →