apache-airflow-providers-amazon

Release Date: 2025-07-12

Features

Add support for S3 dag bundle (#46621)

Bug Fixes

Fix GlueJobOperator deferred waiting (#52314)

Handle exceptions when fetching status in GlueJobHook (#52262)

Handle 'S3KeySensor' in 'deferrable' mode splits 'bucket_key' into individual chars (#52983)

Pass the region_name from the GlueJobOperator / GlueJobSensor to the Trigger (#52904)

Misc

Move 'BaseHook' implementation to task SDK (#51873)

Replace 'models.BaseOperator' to Task SDK one for Amazon Provider (#52667)

Disable UP038 ruff rule and revert mandatory 'X | Y' in insintance checks (#52644)

Upgrade ruff to latest version (0.12.1) (#52562)

Replace usage of 'set_extra' with 'extra' for athena sql hook (#52340)

Drop support for Python 3.9 (#52072)

Replace 'models.BaseOperator' to Task SDK one for Standard Provider (#52292)

Replace occurences of 'get_password' with 'password' to ease migration (#52333)

Use BaseSensorOperator from task sdk in providers (#52296)

Use base AWS classes in Glue Trigger / Sensor and implement custom waiter (#52243)

Add Airflow 3.0+ Task SDK support to AWS Batch Executor (#52121)

Refactor operator_extra_links property in BatchOperator (#51385)

Remove unused batch methods from auth manager (#52883)

Add debug logging for endpoint_url in AWS Connection (#52856)

Remove 'MENU' from 'ResourceMethod' in auth manager (#52731)

Remove 'MAX_XCOM_SIZE' hardcoded constant from Airflow core (#52978)

More robust handling of 'BaseHook.get_connection''s 'CONNECTION_NOT_FOUND' Task SDK exception (#52838)

Move all BaseHook usages to version_compat in Amazon (#52796)

Remove upper-binding for "python-requires" (#52980)

Temporarily switch to use >=,< pattern instead of '~=' (#52967)

Doc-only

Clean some leftovers of Python 3.9 removal - All the rest (#52432)

Release Date: 2025-06-18

Features

Add 'MessageDeduplicationId' support to 'SnsPublishOperator' (#51383)

Add support for RequestPay=requester option in Amazon S3's Operators, Sensors and Triggers (#51098)

Add AWS Lambda Executor (#50516)

Bug Fixes

Removed unnecessary 'aws_conn_id' param from operators constructors (#51236)

Fix EcsRunTaskOperator reattach (#51412)

Fix EKS token generation (#51333)

Fix 'EksPodOperator' in deferrable mode (#51255)

Rds Operator pass custom conn_id to superclass (#51196)

Fix remote logging CloudWatch handler initialization and stream name assignment (#51022)

Check 'is_mapped' to prevent 'operator_extra_links' property failing for Airflow 3. #50932

Fix aws_conn_id defaulting after dag.test was updated to use TaskSDK. (#50515)

AWS ECS Executor. Assign public ip defaults false (#50713)

Misc

Remove unused entries from 'DagAccessEntity' (#51174)

Update Redshift cluster operator and sensor to inherit AwsBaseOperator (#51129)

Remove Airflow 2 code path in executors (#51009)

Move AWS auth dependencies to python3-saml extra (#50449)

Bump some provider dependencies for faster resolution (#51727)

Doc-only

docs: Add missing 'param' for waiter_max_attempts in EMR operator docstring (#51676)

Update comment in CloudWatchRemoteLogIO (#51092)

Use explicit directives instead of implicit syntax (#50870)

Release Date: 2025-05-18

Features

Add in-memory buffer and gzip support in SqlToS3Operator (#50287)

Bug Fixes

Add Key to default meta keys in S3KeySensor (#50122)

Misc

Update Sagemaker Operators and Sensors to inherit Base AWS classes (#50321)

Use non-deprecated context in tests for Airflow 3 (#50391)

Release Date: 2025-05-10

Note

This release of provider is only available for Airflow 2.10+ as explained in the Apache Airflow providers support policy .

Features

Move SQS messaging from common.messaging to Amazon provider (#50057)

Bug Fixes

Fix typo in emr sensor _hook_parameters (#49597)

Update 'prune_logs' in Amazon provider package to support 'RuntimeTaskInstance' (#49363)

Misc

Remove AIRFLOW_2_10_PLUS conditions (#49877)

Bump min Airflow version in providers to 2.10 (#49843)

Update GlueJobOperator to inherit AWS base class (#49750)

Amazon EMR Sensors/Operators inherit AWS Base Classes (#49486)

Introduce lower bind to lxml as 5.4.0 (#49612)

Remove Marshmallow from Core (#49388)

Migrate 'HiveToDynamoDBOperator' and 'SqlToS3Operator' to use 'get_df' (#50126)

add root parent information to OpenLineage events (#49237)

Remove limits for aiobotocore limiting boto3 (#50285)

Release Date: 2025-04-19

Bug Fixes

Fix 'EksClusterStateSensor'. Save 'region' as attribute (#49138)

Decrease default value of 'waiter_max_attempts' in 'MwaaTriggerDagRunOperator' (#49136)

Increase default value of 'waiter_max_attempts' in 'BedrockBatchInferenceOperator' (#49090)

Misc

Use contextlib.suppress(exception) instead of try-except-pass and add SIM105 ruff rule (#49251)

Add base_url fallback for aws auth_manager (#49305)

remove superfluous else block (#49199)

AWS Batch Operators/Sensors inherit AWS Base classes (#49172)

Help pip to find appropriate boto for aiobotocore (#49166)

Update EKS Operators and Sensors to inherit AWS base classes (#48192)

Release Date: 2025-04-14

Features

Add Bedrock Batch Inference Operator and accompanying parts (#48468)

Update ECS executor to support Task SDK (#48513)

Bug Fixes

Handle NoCredentialsError in waiter_with_logging.py (#48946)

Bedrock Batch Inference - Trying to stop a completed job is a successful result (#48964)

S3Hook: remove error return on inactivity period check (#48782)

Misc

Rename list_jobs method to describe_jobs in GlueJobHook (#48904)

Fix typo in docstring for MwaaHook (#48980)

Update Amazon RDS Operators and Sensors to inherit AWS Base classes (#48872)

Change provider-specific dependencies to refer to providers (#48843)

Release Date: 2025-04-09

Features

Add a backup implementation in AWS MwaaHook for calling the MWAA API (#47035)

Add AWS SageMaker Unified Studio Workflow Operator (#45726)

Add error statuses check in RdsExportTaskExistenceSensor (#46917)

Common Message Queue (#46694)

add startTime to paginator.paginate when fetching logs in GlueJobHook (#46950)

Add MwaaDagRunSensor to Amazon Provider Package (#46945)

Add wait/defer support - MwaaTriggerDagRunOperator (#47528)

Add deferrable support for MwaaDagRunSensor (#47527)

Bug Fixes

Fix aws trigger tests, use get_async_conn for mock object (#47515)

fix: don't use blocking property access for async purposes (#47326)

Fix and simplify 'get_permitted_dag_ids' in auth manager (#47458)

Log state for EMR Containers sensor on failure (#47125)

Don't expect default conns in S3ToRedshiftOperator (#48363)

Don't expect default connections to be present in RedshiftToS3Operator (#47968)

fix PosixPath not working with file create_asset in download_file of S3Hook (#47880)

Fix Cloudwatch remote logging (#48774)

Fix 'conf.get_boolean("api", "ssl_cert")' (#48465)

Fix signature of 'BatchWaitersHook.get_waiter' not matching parent class (#48581)

Misc

Relocate airflow.auth to airflow.api_fastapi.auth (#47492)

AIP-72: Moving BaseOperatorLink to task sdk (#47008)

Add some typing and require kwargs for auth manager (#47455)

AIP-84 - Add Auth for Assets (#47136)

Base AWS classes - S3 (#47321)

Remove unused methods from auth managers (#47316)

Move api-server to port 8080 (#47310)

Render structured logs in the new UI rather than showing raw JSON (#46827)

Remove old UI and webserver (#46942)

Don't remove log groups from example_glue.py (#47128)

Move 'fastapi-api' command to 'api-server' (#47076)

Remove '/webapp' prefix from new UI (#47041)

Restricting moto 5.1.0 to fix ci (#47005)

Bump minimum boto3 version to 1.37.0 (#48238)

Move BaseNotifier to Task SDK (#48008)

Updating EC2 Operators and Sensors with AWS Base classes (#47931)

Bump mypy-boto3-appflow>=1.37.0 (#47912)

Lower bind xmlsec dependency version (#47696)

Clarify the Redshift delete cluster operator messaging. (#48652)

Rework remote task log handling for the structlog era. (#48491)

Move 'BaseSensorOperator' to TaskSDK definitions (#48244)

Cookies in non TLS mode (#48453)

Release Date: 2025-02-26

Note

This version contains no code changes. It was released to replace a previous version that was yanked due to a packaging issue.

Release Date: 2025-02-21

Warning

The experimental AWS auth manager is no longer compatible with Airflow 2

Features

Add MwaaTriggerDagRunOperator and MwaaHook to Amazon Provider Package (#46579)

Adding extra links for EC2 (#46340)

Allow to pass container_name parameter to EcsRunTaskOperator (#46152)

Adding DataSync links (#46292)

Added extra links for Comprehend operators (#46031)

Add support for timeout to BatchOperator (#45660)

Adding SageMaker Transform extra link (#45677)

Add MessageDeduplicationId support to AWS SqsPublishOperator (#45051)

Bug Fixes

Rework the TriggererJobRunner to run triggers in a process without DB access (#46677)

Fix schema path in AWS auth manager system test due to restructure (#46625)

Increase retries in 'EmrContainerHook.create_emr_on_eks_cluster' (#46562)

Update 'create_emr_on_eks_cluster' method to try when "cluster is not reachable as its connection is currently being updated" (#46497)

Generate partition aware STS endpoints for EKS Hook (#45725)

Sagemaker Operator Character limit fix (#45551)

Fix 'fetch_access_token_for_cluster' in EKS hook (#45469)

The DMS waiter replication_terminal_status has been extended to proceed on 2 additional states: "created" and "deprovisioned" (#46684)

Misc

AIP-72: Improving Operator Links Interface to Prevent User Code Execution in Webserver (#46613)

Update 'example_sqs' to not use 'logical_date' (#46696)

Change improper AirflowProviderDeprecationWarning ignore to DeprecationWarning ignore for 3.12 tests (#46612)

Update AWS auth manager to use Fastapi instead of Flask (#46381)

AIP-72: Move Secrets Masker to task SDK (#46375)

Swap CeleryExecutor over to use TaskSDK for execution. (#46265)

Make parameter 'user' mandatory for all methods in the auth manager interface (#45986)

Add 'run_job_kwargs' as templated field in 'GlueJobOperator' (#45973)

Use Protocol for 'OutletEventAccessor' (#45762)

AIP-72: Support better type-hinting for Context dict in SDK (#45583)

Remove classes from 'typing_compat' that can be imported directly (#45589)

Move Literal alias into TYPE_CHECKING block (#45345)

Remove marshmallow version restriction; update deprecated usages (#45499)

Remove obsolete pandas specfication for pre-python 3.9 (#45399)

Add option in auth manager interface to define FastAPI api (#45009)

Release Date: 2024-12-26

Note

This release of provider is only available for Airflow 2.9+ as explained in the Apache Airflow providers support policy .

Features

Add DMS Serverless Operators (#43988)

Add fail_on_file_not_exist option to SFTPToS3Operator (#44320)

Add 'wait_policy' option to 'EmrCreateJobFlowOperator' (#44055)

Add meta_data_directive to 'S3CopyObjectOperator' (#44160)

Misc

Remove references to AIRFLOW_V_2_9_PLUS (#44987)

Bump minimum Airflow version in providers to Airflow 2.9.0 (#44956)

Consistent way of checking Airflow version in providers (#44686)

Remove unnecessary compatibility code in S3 asset import (#44714)

Remove AIP-44 from taskinstance (#44540)

Add do_xcom_push documentation in EcsRunTaskOperator (#44440)

Move Asset user facing components to task_sdk (#43773)

Set up JWT token authentication in Fast APIs (#42634)

Bump to mypy-boto3-appflow and pass without '# type: ignore[arg-type]' (#44115)

Update DAG example links in multiple providers documents (#44034)

Release Date: 2024-11-18

Features

feat: add OpenLineage support for RedshiftToS3Operator (#41632)

Add 'SageMakerProcessingSensor' (#43144)

Make 'RedshiftDataOperator' handle multiple queries (#42900)

Bug Fixes

fix(providers/amazon): alias is_authorized_dataset to is_authorized_asset (#43470)

Remove returns in final clause of athena hooks (#43426)

fix: replace \s with space in EksHook (#43849)

Fix 'HttpToS3Operator' throws exception if s3_bucket parameter is not passed (#43828)

Add 'container_name' and update 'awslogs_stream_prefix' pattern (#43138)

Check if awslogs_stream_prefix already ends with container_name (#43724)

bugfix description should be optional for openlineage integration with 'AthenaOperator' (#43576)

(bugfix): 'EcsRunTaskOperator' decouple 'volume_configurations' from 'capacity_provider_strategy' (#43047)

GlueJobOperator: add option to wait for cleanup before returning job status (#43688)

Resolve 'GlueJobTrigger' serialization bug causing verbose to always be True (#43622)

Remove returns in final clause of S3ToDynamoDBOperator (#43456)

Misc

Remove sqlalchemy-redshift dependency (#43271)

feat(providers/amazon): Use asset in common provider (#43110)

Restrict looker-sdk version 24.18.0 and microsoft-kiota-http 1.3.4 (#42954)

Limit mypy-boto3-appflow (#43436)

Move PythonOperator to Standard provider (#42081)

Add support for semicolon stripping to DbApiHook, PrestoHook, and TrinoHook (#41916)

Remove deprecations from cncf.kubernetes provider (#43689)

Fix docstring for AthenaTrigger (#43616)

Release Date: 2024-10-14

Breaking changes

Warning

In order to support session reuse in RedshiftData operators, the following breaking changes were introduced:

The database argument is now optional and as a result was moved after the sql argument which is a positional one. Update your DAGs accordingly if they rely on argument order. Applies to:

RedshiftDataHook ’s execute_query method

RedshiftDataOperator

RedshiftDataHook ’s execute_query method now returns a QueryExecutionOutput object instead of just the statement ID as a string.

RedshiftDataHook ’s parse_statement_resposne method was renamed to parse_statement_response .

S3ToRedshiftOperator ’s schema argument is now optional and was moved after the s3_key positional argument. Update your DAGs accordingly if they rely on argument order.

Warning

All deprecated classes, parameters and features have been removed from the Amazon provider package. The following breaking changes were introduced:

Hooks

Removed sleep_time parameter from AthenaHook . Use poll_query_status instead

Removed BaseAsyncSessionFactory

Removed AwsBaseAsyncHook

Removed start_from_head parameter from AwsLogsHook.get_log_events method

Removed sts_hook property from QuickSightHook

Removed RedshiftAsyncHook

Removed S3 connection type. Please use aws as conn_type instead, and specify bucket_name in service_config.s3 within extras

Removed wait_for_completion , check_interval and verbose parameters from SageMakerHook.start_pipeline method

Removed wait_for_completion , check_interval and verbose parameters from SageMakerHook.stop_pipeline method

Operators

Removed source parameter from AppflowRunOperator

Removed overrides parameter from BatchOperator . Use container_overrides instead

Removed status_retries parameter from BatchCreateComputeEnvironmentOperator

Removed get_hook method from DataSyncOperator . Use hook property instead

Removed wait_for_completion , waiter_delay and waiter_max_attempts parameters from EcsDeregisterTaskDefinitionOperator . Please use waiter_max_attempts and waiter_delay instead

Removed wait_for_completion , waiter_delay and waiter_max_attempts parameters from EcsRegisterTaskDefinitionOperator . Please use waiter_max_attempts and waiter_delay instead

Removed eks_hook property from EksCreateClusterOperator . Use hook property instead

Removed pod_context , pod_username and is_delete_operator_pod parameters from EksPodOperator

Removed waiter_countdown and waiter_check_interval_seconds parameters from EmrStartNotebookExecutionOperator . Please use waiter_max_attempts and waiter_delay instead

Removed waiter_countdown and waiter_check_interval_seconds parameters from EmrStopNotebookExecutionOperator . Please use waiter_max_attempts and waiter_delay instead

Removed max_tries parameter from EmrContainerOperator . Use max_polling_attempts instead

Removed waiter_countdown and waiter_check_interval_seconds parameters from EmrCreateJobFlowOperator . Please use waiter_max_attempts and waiter_delay instead

Removed waiter_countdown and waiter_check_interval_seconds parameters from EmrServerlessCreateApplicationOperator . Please use waiter_max_attempts and waiter_delay instead

Removed waiter_countdown and waiter_check_interval_seconds parameters from EmrServerlessStartJobOperator . Please use waiter_max_attempts and waiter_delay instead

Removed waiter_countdown and waiter_check_interval_seconds parameters from EmrServerlessStopApplicationOperator . Please use waiter_max_attempts and waiter_delay instead

Removed waiter_countdown and waiter_check_interval_seconds parameters from EmrServerlessDeleteApplicationOperator . Please use waiter_max_attempts and waiter_delay instead

Removed delay parameter from GlueDataBrewStartJobOperator . Use waiter_delay instead

Removed hook_params parameter from RdsBaseOperator

Removed increment as possible value from action_if_job_exists parameter from SageMakerProcessingOperator

Removed increment as possible value from action_if_job_exists parameter from SageMakerTransformOperator

Removed increment as possible value from action_if_job_exists parameter from SageMakerTrainingOperator

Secrets

Removed from full_url_mode and are_secret_values_urlencoded as possible key in kwargs from SecretsManagerBackend

Sensors

Removed get_hook method from BatchSensor . Use hook property instead

Removed get_hook method from DmsTaskBaseSensor . Use hook property instead

Removed get_hook method from EmrBaseSensor . Use hook property instead

Removed get_hook method from GlueCatalogPartitionSensor . Use hook property instead

Removed get_hook method from GlueCrawlerSensor . Use hook property instead

Removed quicksight_hook property from QuickSightSensor . Use QuickSightSensor.hook instead

Removed sts_hook property from QuickSightSensor

Removed get_hook method from RedshiftClusterSensor . Use hook property instead

Removed get_hook method from S3KeySensor . Use hook property instead

Removed get_hook method from SageMakerBaseSensor . Use hook property instead

Removed get_hook method from SqsSensor . Use hook property instead

Removed get_hook method from StepFunctionExecutionSensor . Use hook property instead

Transfers

Removed aws_conn_id parameter from AwsToAwsBaseOperator . Use source_aws_conn_id instead

Removed bucket and delimiter parameters from GCSToS3Operator . Use gcs_bucket instead of bucket

Triggers

Removed BatchOperatorTrigger . Use BatchJobTrigger instead

Removed BatchSensorTrigger . Use BatchJobTrigger instead

Removed region parameter from EksCreateFargateProfileTrigger . Use region_name instead

Removed region parameter from EksDeleteFargateProfileTrigger . Use region_name instead

Removed poll_interval and max_attempts parameters from EmrCreateJobFlowTrigger . Use waiter_delay and waiter_max_attempts instead

Removed poll_interval and max_attempts parameters from EmrTerminateJobFlowTrigger . Use waiter_delay and waiter_max_attempts instead

Removed poll_interval parameter from EmrContainerTrigger . Use waiter_delay instead

Removed poll_interval parameter from GlueCrawlerCompleteTrigger . Use waiter_delay instead

Removed delay and max_attempts parameters from GlueDataBrewJobCompleteTrigger . Use waiter_delay and waiter_max_attempts instead

Removed RdsDbInstanceTrigger . Use the other RDS triggers such as RdsDbDeletedTrigger , RdsDbStoppedTrigger or RdsDbAvailableTrigger

Removed poll_interval and max_attempts parameters from RedshiftCreateClusterTrigger . Use waiter_delay and waiter_max_attempts instead

Removed poll_interval and max_attempts parameters from RedshiftPauseClusterTrigger . Use waiter_delay and waiter_max_attempts instead

Removed poll_interval and max_attempts parameters from RedshiftCreateClusterSnapshotTrigger . Use waiter_delay and waiter_max_attempts instead

Removed poll_interval and max_attempts parameters from RedshiftResumeClusterTrigger . Use waiter_delay and waiter_max_attempts instead

Removed poll_interval and max_attempts parameters from RedshiftDeleteClusterTrigger . Use waiter_delay and waiter_max_attempts instead

Removed SageMakerTrainingPrintLogTrigger . Use SageMakerTrigger instead

Utils

Removed test_endpoint_url as possible key in extra_config from AwsConnectionWrapper . Please set endpoint_url in service_config.sts within extras

Removed s3 as possible value in conn_type from AwsConnectionWrapper . Please update your connection to have conn_type='aws'

Removed session_kwargs as key in connection extra config. Please specify arguments passed to boto3 session directly

Removed host from AWS connection, please set it in extra['endpoint_url'] instead

Removed region parameter from AwsHookParams . Use region_name instead

Remove deprecated stuff from Amazon provider package (#42450)

Support session reuse in 'RedshiftDataOperator' (#42218)

Features

Add STOPPED to the failure cases for Sagemaker Training Jobs (#42423)

Bug Fixes

'S3DeleteObjects' Operator: Handle dates passed as strings (#42464)

Small fix to AWS AVP cli init script (#42479)

Make the AWS logging faster by reducing the amount of sleep (#42449)

Fix logout in AWS auth manager (#42447)

fix(providers/amazon): handle ClientError raised after key is missing during table.get_item (#42408)

Misc

Drop python3.8 support core and providers (#42766)

Removed conditional check for task context logging in airflow version 2.8.0 and above (#42764)

Rename dataset related python variable names to asset (#41348)

Remove identity center auth manager cli (#42481)

Refactor AWS Auth manager user output (#42454)

Remove 'sqlalchemy-redshift' dependency from Amazon provider (#42830)

Revert "Remove 'sqlalchemy-redshift' dependency from Amazon provider" (#42864)

Release Date: 2024-09-24

Features

Adding support for volume configurations in ECSRunTaskOperator (#42087)

Openlineage s3 to redshift operator integration (#41575)

Bug Fixes

ECSExecutor: Drop params that aren't compatible with EC2 (#42228)

Fix 'GlueDataBrewStartJobOperator' template fields (#42073)

validate aws service exceptions in waiters (#41941)

Fix treatment of "#" in S3Hook.parse_s3_url() (#41796)

fix: remove part of openlineage extraction from S3ToRedshiftOperator (#41631)

filename template arg in providers file task handlers backward compatibility support (#41633)

fix: select_query should have precedence over default query in RedshiftToS3Operator (#41634)

Misc

Actually move saml to amazon provider (mistakenly added in papermill) (#42148)

Use base aws classes in AWS Glue DataBrew Operators/Triggers (#41848)

Move 'register_views' to auth manager interface (#41777)

airflow.models.taskinstance deprecations removed (#41784)

remove deprecated soft_fail from providers (#41710)

remove deprecated soft_fail from providers part2 (#41727)

Limit watchtower as depenendcy as 3.3.0 breaks moin. (#41612)

Remove deprecated log handler argument filename_template (#41552)

Release Date: 2024-08-22

Note

This release of provider is only available for Airflow 2.8+ as explained in the Apache Airflow providers support policy .

Warning

When deferrable mode was introduced for RedshiftDataOperator , in version 8.17.0, tasks configured with deferrable=True and wait_for_completion=True would not enter the deferred state. Instead, the task would occupy an executor slot until the statement was completed. A workaround may have been to set wait_for_completion=False . In this version, tasks set up with wait_for_completion=False will not wait anymore, regardless of the value of deferrable .

Features

Add incremental export and cross account export functionality in 'DynamoDBToS3Operator' (#41304)

EKS Overrides for AWS Batch submit_job (#40718)

Bug Fixes

Fix 'AwsTaskLogFetcher' missing logs (#41515)

Fix the Exception name and unpin dependency in 'RdsHook' (#41256)

Fix RedshiftDataOperator not running in deferred mode as expected (#41206)

Misc

Partial fix for example_dynamodb_to_s3.py (#41517)

Remove deprecated code is AWS provider (#41407)

Bump minimum Airflow version in providers to Airflow 2.8.0 (#41396)

Limit moto temporarily - 5.0.12 is breaking our tests (#41244)

Release Date: 2024-08-06

Features

Add RedriveExecution support to 'StepFunctionStartExecutionOperator' (#40976)

openlineage: add support for hook lineage for S3Hook (#40819)

Introduce Amazon Kinesis Analytics V2 (Managed Service for Apache Flink application) (#40765)

Bug Fixes

Make EMR Container Trigger max attempts retries match the Operator (#41008)

Fix 'RdsStopDbOperator' operator in deferrable mode (#41059)

Fix 'RedshiftCreateClusterOperator' to always specify 'PubliclyAccessible' (#40872)

Fix Redshift cluster operators and sensors using deferrable mode (#41191)

Fix 'EmrServerlessStartJobOperator' with deferrable mode (#41103)

Misc

Update 'example_redshift' and 'example_redshift_s3_transfers' to use 'RedshiftDataHook' instead of 'RedshiftSQLHook' (#40970)

openlineage: migrate OpenLineage provider to V2 facets. (#39530)

[AIP-62] Translate AIP-60 URI to OpenLineage (#40173)

Move AWS Managed Service for Apache Flink sensor states to Hook (#40896)

Replace usages of task context logger with the log table (#40867)

Deprecate 'SageMakerTrainingPrintLogTrigger' (#41158)

Release Date: 2024-07-15

Note

Reduce memory footprint of S3KeyTrigger Decorator provide_bucket_name_async is removed. Async does not require a separated decorator. The old one is removed and users can use provide_bucket_name for coroutine functions, async iterators, and normal synchronous functions. Hook method get_file_metadata_async is now an async iterator Previously, the metadata objects were accumulated in a list. Now the objects are yielded as we page through the results. To get a list you may use async for in a list comprehension. S3KeyTrigger avoids loading all positive matches into memory in some circumstances

Note

This release contains significant resources utilization improvements for async sessions

Features

Do not dynamically determine op links for emr serverless (#40627)

Be able to remove ACL in S3 hook's copy_object (#40518)

feat(aws): provide the context to check_fn in S3 sensor (#40686)

Bug Fixes

fix OpenLineage extraction for AthenaOperator (#40545)

Reduce memory footprint of s3 key trigger (#40473)

Adding cluster to ecs trigger event to avoid defer error (#40482)

Fix deferrable AWS SageMaker operators (#40706)

Make 'AwsAuthManager' compatible with only Airflow >= 2.9 (#40690)

Add serialization opt to s3 operator (#40659)

Misc

Use base aws classes in AWS Glue Data Catalog Sensors (#40492)

Use base aws classes in AWS Glue Crawlers Operators/Sensors/Triggers (#40504)

Share data loader to across asyncio boto sessions (#40658)

Send executor logs to task logs in 'EcsExecutor' (#40468)

Send executor logs to task logs in 'AwsBatchExecutor' (#40698)

Release Date: 2024-06-27

Features

Add Amazon Comprehend Document Classifier (#40287)

Bug Fixes

Fix 'importlib_metadata' import in aws utils (#40134)

openlineage, redshift: do not call DB for schemas below Airflow 2.10 (#40197)

Lazy match escaped quotes in 'RedshiftToS3Operator' (#40206)

Use stdlib 'importlib.metadata' for retrieve 'botocore' package version (#40137)

Misc

Update pandas minimum requirement for Python 3.12 (#40272)

Release Date: 2024-06-09

Features

ECS Overrides for AWS Batch submit_job (#39903)

Add transfer operator S3ToDynamoDBOperator (#39654)

Adding Glue Data Quality Rule Recommendation Run (#40014)

Allow user-specified object attributes to be used in check_fn for S3KeySensor (#39950)

Adding Amazon Glue Data Quality Service (#39923)

Bug Fixes

Deduplicate model name in SageMakerTransformOperator (#39956)

Fix: remove process_func from templated_fields (#39948)

Fix aws assume role session creation when deferrable (#40051)

Misc

Resolving ECS fargate deprecated warnings (#39834)

Resolving EMR notebook deprecated warnings (#39829)

Bump boto min versions (#40052)

docs: mention minimum boto3 1.34.52 for AWS provider when using Batch 'ecs_properties_override' (#39983)

Release Date: 2024-05-30

Features

Amazon Bedrock - Retrieve and RetrieveAndGenerate (#39500)

Introduce Amazon Comprehend Service (#39592)

Bug Fixes

fix: empty openlineage dataset name for AthenaExtractor (#39677)

Fix default value for aws batch operator retry strategy (#39608)

Sagemaker trigger: pass the job name as part of the event (#39671)

Handle task adoption for batch executor (#39590)

bugfix: handle invalid cluster states in NeptuneStopDbClusterOperator (#38287)

Fix automatic termination issue in 'EmrOperator' by ensuring 'waiter_max_attempts' is set for deferrable triggers (#38658)

Misc

Resolving EMR deprecated warnings (#39743)

misc: add comment about remove unused code (#39748)

Release Date: 2024-05-17

Features

'S3DeleteObjectsOperator' Added ability to filter keys by last modified time (#39151)

Amazon Bedrock - Add Knowledge Bases and Data Sources integration (#39245)

Bug Fixes

EcsExcecutor Scheduler to handle incrementing of try_number (#39336)

ECS Executor: Set tasks to RUNNING state once active (#39212)

Misc

Add 'jmespath' as an explicit dependency (#39350)

Drop 'xmlsec' dependency (#39534)

Reapply templates for all providers (#39554)

Faster 'airflow_version' imports (#39552)

enh(amazon_hook): raise not found exception instead of general exception when download file (#39509)

Simplify 'airflow_version' imports (#39497)

Release Date: 2024-05-06

Note

This release of provider is only available for Airflow 2.7+ as explained in the Apache Airflow providers support policy .

Features

Added logging_config,snapstart,ephemeral_storage parameters to aws lambdacreatefunctionoperator (#39300)

Bug Fixes

Fix bug in GlueJobOperator where consecutive runs fail when a local script file is used (#38960)

Update 'is_authorized_custom_view' from auth manager to handle custom actions (#39167)

Update logic to allow retries in AWS Batch Client hook to be effective (#38998)

Amazon Bedrock - Model Throughput Provisioning (#38850)

Misc

Adding MSGraphOperator in Microsoft Azure provider (#38111)

Bump minimum Airflow version in providers to Airflow 2.7.0 (#39240)

Allow importing the aws executors with a shorter path (#39093)

Remove flag from AWS auth manager to use it (#39033)

Limit xmlsec<1.3.14 (#39104)

Release Date: 2024-04-13

Features

AWS Batch Executor (#37618)

Add fallback 'region_name' value to AWS Executors (#38704)

Introduce Amazon Bedrock service (#38602)

Amazon Bedrock - Model Customization Jobs (#38693)

ECS Executor - add support to adopt orphaned tasks. (#37786)

Update AWS auth manager CLI command to not disable AVP schema validation (#38301)

Bug Fixes

Reduce 's3hook' memory usage (#37886)

Add check in AWS auth manager to check if the Amazon Verified Permissions schema is up to date (#38333)

fix: EmrServerlessStartJobOperator not serializing DAGs correctly when partial/expand is used. (#38022)

fix(amazon): add return statement to yield within a while loop in triggers (#38396)

Fix set deprecated amazon operators arguments in 'MappedOperator' (#38346)

'ECSExecutor' API Retry bug fix (#38118)

Fix 'region' argument in 'MappedOperator' based on 'AwsBaseOperator' / 'AwsBaseSensor' (#38178)

Fix bug for ECS Executor where tasks were being skipped if one task failed. (#37979)

Fix init checks for aws redshift to s3 operator (#37861)

Misc

Make the method 'BaseAuthManager.is_authorized_custom_view' abstract (#37915)

Replace "Role" by "Group" in AWS auth manager (#38078)

Avoid use of 'assert' outside of the tests (#37718)

Use 'AwsLogsHook' when fetching Glue job logs (#38010)

Implement 'filter_permitted_dag_ids' in AWS auth manager (#37666)

AWS auth manager CLI: persist the policy store description when doing updates (#37946)

Change f-string to formatting into the logging messages for Batch Executor (#37929)

Release Date: 2024-03-08

Features

Implement 'filter_permitted_menu_items' in AWS auth manager (#37627)

Implement 'batch_is_authorized_*' APIs in AWS auth manager (#37430)

Bug Fixes

Fix init checks for aws 'eks' (#37674)

Fix init checks for aws gcs_to_s3 (#37662)

Misc

Use named loggers instead of root logger (#37801)

Avoid non-recommended usage of logging (#37792)

Unify 'aws_conn_id' type to always be 'str | None' (#37768)

Limit 'pandas' to '<2.2' (#37748)

Implement AIP-60 Dataset URI formats (#37005)

Bump min versions of openapi validators (#37691)

Update action names in AWS auth manager (#37572)

Release Date: 2024-02-18

Features

ECS Executor - Add backoff on failed task retry (#37109)

SqlToS3Operator: feat/ add max_rows_per_file parameter (#37055)

Adding Amazon Neptune Hook and Operators (#37000)

Add retry configuration in 'EmrContainerOperator' (#37426)

Create CLI commands for AWS auth manager to create AWS Identity Center related resources (#37407)

Add extra operator links for EMR Serverless (#34225)

Bug Fixes

Fix 'log_query' to format SQL statement correctly in 'AthenaOperator' (#36962)

check sagemaker training job status before deferring 'SageMakerTrainingOperator' (#36685)

Misc

Merge all ECS executor configs following recursive python dict update (#37137)

Update default value for 'BatchSensor' (#37234)

remove info log from download_file (#37211)

S3ToRedshiftOperator templating aws_conn_id (#37195)

Updates to ECS Docs (#37125)

feat: Switch all class, functions, methods deprecations to decorators (#36876)

Replace usage of 'datetime.utcnow' and 'datetime.utcfromtimestamp' in providers (#37138)

add type annotations to Amazon provider "execute_coplete" methods (#36330)