apache-airflow-providers-amazon

Release Date: 2024-01-27

Features

add deferrable mode to RedshiftDataOperator (#36586)

Adds support for capacity providers to ECS Executor (#36722)

Add use_regex argument for allowing 'S3KeySensor' to check s3 keys with regular expression (#36578)

Add deferrable mode to RedshiftClusterSensor (#36550)

AthenaSqlHook implementation (#36171)

Create CLI commands for AWS auth manager to create Amazon Verified Permissions related resources (#36799)

Implement 'is_authorized_dag' in AWS auth manager (#36619)

Bug Fixes

Fix stacklevel in warnings.warn into the providers (#36831)

EC2 'CreateInstance': terminate instances in on_kill (#36828)

Fallback to default value if '[aws] cloudwatch_task_handler_json_serializer' not set (#36851)

AWS auth manager: raise AirflowOptionalProviderfeature exception for AVP command (#36824)

check transform job status before deferring SageMakerTransformOperator (#36680)

check sagemaker processing job status before deferring (#36658)

check job_status before BatchOperator execute in deferrable mode (#36523)

Update the redshift hostname check to avoid possible bugs (#36703)

Refresh credentials in 'AwsEcsExecutor' (#36179)

Misc

Fix docstring for apply_wildcard parameter in 'S3ListOperator'. Changed the order of docstring for fix (#36679)

Use base aws classes in AWS DMS Operators/Sensors (#36772)

Use base aws classes in AWS Redshift Data API Operators (#36764)

Use base aws classes in Amazon EventBridge Operators (#36765)

Use base aws classes in Amazon QuickSight Operators/Sensors (#36776)

Use base aws classes in AWS Datasync Operators (#36766)

Use base aws classes in Amazon DynamoDB Sensors (#36770)

Use base aws classes in AWS CloudFormation Operators/Sensors (#36771)

Set min pandas dependency to 1.2.5 for all providers and airflow (#36698)

Bump min version of amazon-provider related dependencies (#36660)

Release Date: 2024-01-10

Features

Add AWS Step Functions links (#36599)

Add OpenLineage support for Redshift SQL (#35794)

Bug Fixes

Fix assignment of template field in 'init' in 'AwsToAwsBaseOperator' (#36604)

Fix assignment of template field in 'init' in 'DataSyncOperator' (#36605)

Check redshift cluster state before deferring to triggerer (#36416)

Misc

Use base aws classes in Amazon SQS Operators/Sensors/Triggers (#36613)

Use base aws classes in Amazon SNS Operators (#36615)

Use base aws classes in AWS Step Functions Operators/Sensors/Triggers (#36468)

Release Date: 2023-12-31

Features

Add Amazon Athena query results extra link (#36447)

Bug Fixes

fix(providers/amazon): remove event['message'] call in EmrContainerOperator.execute_complete|as the key message no longer exists (#36417)

handle tzinfo in S3Hook.is_keys_unchanged_async (#36363)

Misc

Use base aws classes in Amazon ECS Operators/Sensors/Triggers (#36393)

Release Date: 2023-12-27

Features

Add 'jsonpath_ng.ext.parse' support for 'SqsSensor' (#36170)

Increase ConflictException retries to 4 total (#36337)

Increase width of execution_date input in trigger.html (#36278) (#36304)

Allow storage options to be passed (#35820)

Bug Fixes

Remove 'is_authorized_cluster_activity' from auth manager (#36175)

Follow BaseHook connection fields method signature in child classes (#36086)

Misc

Add code snippet formatting in docstrings via Ruff (#36262)

Remove remaining Airflow 2.6 backcompat code from Amazon Provider (#36324)

Release Date: 2023-12-12

Note

This release of provider is only available for Airflow 2.6+ as explained in the Apache Airflow providers support policy .

Features

Support IAM authentication for Redshift serverless (#35897)

Implement 'is_authorized_variable' in AWS auth manager (#35804)

Enhance 'attribute_value' in 'DynamoDBValueSensor' to accept list (#35831)

Bug Fixes

Fix handling of single quotes in 'RedshiftToS3Operator' (#35986)

Fix a bug in get_iam_token for Redshift Serverless (#36001)

Fix reraise outside of try block in 'AthenaHook.get_output_location' (#36008)

Fix a bug with accessing hooks in EKS trigger (#35989)

Fix a bug in method name used in 'GlacierToGCSOperator' (#35978)

Fix EC2Hook get_instance for client_type api (#35960)

Avoid creating the hook in the EmrServerlessCancelJobsTrigger init (#35992)

Stop getting message from event after migrating 'EmrContainerTrigger' to 'AwsBaseWaiterTrigger' (#35892)

Fix for 'EksCreateClusterOperator' deferrable mode (#36079)

Misc

Bump minimum Airflow version in providers to Airflow 2.6.0 (#36017)

Update 'boto3' and 'botocore' versions notes (#36073)

Improve typing hints for only_client_type decorator (#35997)

Refactor some methods in EmrContainerHook (#35999)

Refactor get_output_location in AthenaHook (#35996)

Move RDS hook to a cached property in RDS trigger (#35990)

Replace default empty dict value by None in AzureBlobStorageToS3Operator (#35977)

Update 'set_context' signature to match superclass one and stop setting the instance attribute in CloudwatchTaskHandler (#35975)

Use S3 hook instead of AwsGenericHook in AWS S3 FS (#35973)

AWS auth manager: implement all 'is_authorized_*' methods (but 'is_authorized_dag') (#35928)

Remove setting a non-existing object param and use local var instead in S3Hook (#35950)

Release Date: 2023-11-29

Features

Add 'EC2HibernateInstanceOperator' and 'EC2RebootInstanceOperator' (#35790)

Add OpenLineage support to 'S3FileTransformOperator' (#35819)

Add OpenLineage support to S3Operators - Copy, Delete and Create Object (#35796)

Added retry strategy parameter to Amazon AWS provider Batch Operator to allow dynamic Batch retry strategies (#35789)

Added name field to template_fields in EmrServerlessStartJobOperator (#35648)

openlineage, aws: Add OpenLineage support for AthenaOperator. (#35090)

Implement login and logout in AWS auth manager (#35488)

Bug Fixes

Fix Batch operator's retry_strategy (#35808)

Fix and reapply templates for provider documentation (#35686)

Make EksPodOperator exec config not rely on log level (#35771)

Fix 'configuration_overrides' parameter in 'EmrServerlessStartJobOperator' (#35787)

Misc

Updated docstring: 'check_key_async' is now in line with description of '_check_key_async' (#35799)

Check attr on parent not self re TaskContextLogger set_context (#35780)

Allow a wider range of watchtower versions (#35713)

Extend task context logging support for remote logging using AWS S3 (#32950)

Log failure reason for containers if a task fails for ECS Executor (#35496)

Release Date: 2023-11-12

Breaking changes

Features

Add support for anonymous access to s3 buckets for objectstorage (#35273)

ECS Executor Health Check (#35412)

Bug Fixes

Fix AWS RDS hook's DB instance state check (#34773)

Fix parameter syntax in Amazon docstrings (#35349)

Improve error handling in AWS Links (#35518)

Update ECS executor healthcheck with a catchall except (#35512)

Misc

Move ECS Executor to its own file (#35418)

Clarify "task" in ECS Executor log messages (#35304)

Make optional 'output_location' attribute in 'AthenaOperator' (#35265)

Release Date: 2023-10-31

Note

This release introduce experimental feature: AWS ECS Executor.

Features

Add AWS ECS Executor (#34381)

AIP-58: Add Airflow ObjectStore (AFS) (#34729)

Add Http to s3 operator (#35176)

Bug Fixes

Enable encryption in S3 download_files() hook. (#35037)

Misc

Use base aws classes in Amazon AppFlow Operators (#35082)

Use base aws classes in Amazon Athena Operators/Sensors/Triggers (#35133)

Use base aws classes in Amazon Lambda Operators/Sensors (#34890)

Use base aws classes in Amazon S3 Glacier Operators/Sensors (#35108)

Expose catalog parameter in 'AthenaOperator' (#35103)

Release Date: 2023-10-21

Features

Add Glue 'DataBrew' operator (#34807)

Add 'check_interval' and 'max_attempts' as parameter of 'DynamoDBToS3Operator' (#34972)

Bug Fixes

Set 'EcsRunTaskOperator' default waiter duration to 70 days (#34928)

Release Date: 2023-10-17

Note

This release of provider is only available for Airflow 2.5+ as explained in the Apache Airflow providers support policy .

Features

Implements 'AwsBaseOperator' and 'AwsBaseSensor' (#34784)

Extend hooks arguments into 'AwsBaseWaiterTrigger' (#34884)

Allow setup 'endpoint_url' per-service in AWS Connection (#34593)

Include AWS Lambda execution logs to task logs (#34692)

Bug Fixes

fix(providers/amazon): respect soft_fail argument when exception is raised (#34134)

do not fail operator if we cannot find logs (#34570)

Respect 'soft_fail' argument when running 'BatchSensors' (#34592)

Respect 'soft_fail' argument when running 'SqsSensor' (#34569)

Respect 'soft_fail' argument when running 'EcsBaseSensor' (#34596)

Respect 'soft_fail' argument when running 'SageMakerBaseSensor' (#34565)

Respect 'soft_fail' parameter in 'S3KeysUnchangedSensor' and 'S3KeySensor' (#34550)

Respect 'soft_fail' parameter in 'LambdaFunctionStateSensor' (#34551)

Respect 'soft_fail' parameter in 'AthenaSensor' (#34553)

Respect 'soft_fail' parameter in 'QuickSightSensor' (#34555)

Respect 'soft_fail' parameter in 'GlacierJobOperationSensor' (#34557)

Respect 'soft_fail' parameter in 'GlueJobSensor', 'GlueCatalogPartitionSensor' and 'GlueCrawlerSensor' (#34559)

Respect 'soft_fail' parameter in 'StepFunctionExecutionSensor' (#34560)

Misc

Refactor consolidate import from io in providers (#34378)

Upgrade watchtower to 3.0.1 (#25019) (#34747)

Bump min airflow version of providers (#34728)

Refactor: consolidate import time in providers (#34402)

Refactor usage of str() in providers (#34320)

Refactor import from collections (#34406)

Clarify Amazon Lambda invocation and sensing (#34653)

Refactor multiple equals to contains in providers (#34441)

Rename 'bucket' to 'gcs_bucket' in 'GCSToS3Operator' (#33031)

Remove duplicate 'asgiref' dependency in Amazon Provider (#34580)

Update 'BatchOperator' operator_extra_links property (#34506)

sagemaker.py spell error fix (#34445)

Use 'airflow.exceptions.AirflowException' in providers (#34511)

Use 'AirflowProviderDeprecationWarning' in the deprecated decorator in Amazon provider (#34488)

Use 'AirflowProviderDeprecationWarning' in EMR Operators (#34453)

Deprecate get_hook in DataSyncOperator and use hook instead (#34427)

Refactor shorter defaults in providers (#34347)

Release Date: 2023-09-18

Bug Fixes

Bugfix: Fix RDS triggers parameters so that they handle serialization/deserialization (#34222)

Use a AwsBaseWaiterTrigger-based trigger in EmrAddStepsOperator deferred mode (#34216)

Misc

Refactor: Think positively in providers (#34279)

Remove unused parameter 'cluster_role_arn' from 'EksPodOperator''s docstring (#34300)

Correct parameter names in docstring for 'S3CreateObjectOperator' (#34263)

Refactor: Simplify comparisons (#34181)

Simplify to bool(...) (#34258)

Release Date: 2023-09-12

Warning

A bug introduced in version 8.0.0 caused all EcsRunTaskOperator tasks to detach from the ECS task and fail after 10 minutes, even if the ECS task was still running. In this version we are fixing it by returning the default waiter_max_attempts value to sys.maxsize .

Features

Add Amazon SQS Notifier (#33962)

Add Amazon SNS Notifier (#33828)

Bug Fixes

Increase 'waiter_max_attempts' default value in 'EcsRunTaskOperator' (#33712)

Fix AWS 'EmrStepSensor' ignoring the specified 'aws_conn_id' in deferred mode (#33952)

Fix type annotation in AppflowHook (#33881)

Make Amazon Chime connection lazy loaded and consistent with docs (#34000)

respect "soft_fail" argument when running BatchSensor in deferrable mode (#33405)

Misc

Refactor: Consolidate import and usage of random (#34108)

Consolidate importing of os.path.* (#34060)

Refactor regex in providers (#33898)

Refactor: Simplify loop in aws/triggers/batch.py (#34052)

Combine similar if logics in providers (#33987)

Replace single quotes by double quotes in tests (#33864)

Remove useless string join from providers (#33968)

Make 'aws.session_factory' part of Amazon provider configuration documentation (#33960)

Refactor unneeded jumps in providers (#33833)

Replace try - except pass by contextlib.suppress in providers (#33980)

Remove some useless try/except from providers code (#33967)

Refactor: Replace lambdas with comprehensions in providers (#33771)

Replace sequence concatenation by unpacking in Airflow providers (#33933)

Reorganize devel_only extra in airflow's setup.py (#33907)

Remove explicit str concat from Airflow providers package and tests (#33860)

Improve modules import in AWS provider by move some of them into a type-checking block (#33780)

Always use 'Literal' from 'typing_extensions' (#33794)

Use literal dict instead of calling dict() in providers (#33761)

remove unnecessary and rewrite it using list in providers (#33763)

Release Date: 2023-08-29

Features

Added Amazon SageMaker Notebook hook and operators (#33219)

Add 'deferrable' option to 'LambdaCreateFunctionOperator' (#33327)

Add Deferrable mode to GlueCatalogPartitionSensor (#33239)

Add 'sql_hook_params' parameter to 'S3ToSqlOperator' (#33427)

Add 'sql_hook_params' parameter to 'SqlToS3Operator' (#33425)

Add parameter to pass role ARN to 'GlueJobOperator ' (#33408)

Add new RdsStartExportTaskOperator parameters (#33251)

Bug Fixes

Fix bug in task logs when using AWS CloudWatch. Do not set 'start_time' (#33673)

Fix AWS Batch waiter failure state (#33656)

Fix AWS appflow waiter (#33613)

Fix striping tags when falling back to update in 'SageMakerEndpointOperator' (#33487)

Misc

Simplify conditions on len() in providers/amazon (#33565)

Remove non-public interface usage in EcsRunTaskOperator (#29447)

Upgrade botocore/aiobotocore minimum requirements (#33649)

Consolidate import and usage of itertools (#33479)

Consolidate import and usage of pandas (#33480)

always push ECS task ARN to xcom in 'EcsRunTaskOperator' (#33703)

Use 'boto3.client' linked to resource meta instead of create new one for waiters (#33552)

Release Date: 2023-08-14

Bug Fixes

Get failure information on EMR job failure (#32151)

Fix get_log_events() in AWS logs hook (#33290)

Misc

Improve fetching logs from AWS (#33231)

Refactor: Simplify code in providers/amazon (#33222)

Implement EventBridge enable and disable rule operators (#33226)

Update mypy-boto3-appflow dependency (#32930)

use 'cached_property' from functools in 'RdsBaseOperator' (#33133)

Use set for 'template_fields' of 'EcsDeregisterTaskDefinitionOperator' (#33129)

Release Date: 2023-08-08

Features

openlineage, sagemaker: add OpenLineage support for SageMaker's Processing, Transform and Training operators (#31816)

Add Amazon EventBridge PutRule hook and operator (#32869)

Add GCS Requester Pays bucket support to GCSToS3Operator (#32760)

Bug Fixes

Check google provider version in GCSToS3Operator before provide match_glob param (#32925)

Set longer default 'waiter_max_attempts' for deferred BatchJobOperator (#33045)

Misc

openlineage, sagemaker: add missing OpenLineage type signature (#33114)

Add S3Bucket for mypy (#33028)

Release Date: 2023-08-01

Features

Add endpoint_url in test_connection (#32664)

Add support for querying Redshift Serverless clusters (#32785)

Add Deferrable mode to StepFunctionStartExecutionOperator (#32563)

Add Deferrable mode for EMR Serverless Start Job Operator (#32534)

Add Eventbridge PutEvents operator and hook (#32498)

add deferrable mode to rds start & stop DB (#32437)

EMR serverless Create/Start/Stop/Delete Application deferrable mode (#32513)

Make Start and Stop SageMaker Pipelines operators deferrable (#32683)

Deferrable mode for EKS Create/Delete Operator (#32355)

Bug Fixes

FIX AWS deferrable operators by using AioCredentials when using 'assume_role' (#32733)

[bugfix] fix AWS triggers where deserialization would crash if region was not specified (#32729)

Fix bug in prune_dict where empty dict and list would be removed even in strict mode (#32573)

Fix S3ToRedshiftOperator does not support default values on UPSERT (#32558)

Do not return success from AWS ECS trigger after max_attempts (#32589)

Misc

Move all k8S classes to cncf.kubernetes provider (#32767)

Limit Appflow mypy to 1.28.12 as it introduces strange typing issue (#32901)

Further limit mypy-boto3-appflow as the fix is not in sight (#32927)

Release Date: 2023-07-16

Bug Fixes

Append region info to S3ToRedshitOperator if present (#32328)

Release Date: 2023-07-10

Features

Add 'ChimeWebhookHook' (#31939)

Add 'ChimeNotifier' (#32222)

Add deferrable mode to S3KeysUnchangedSensor (#31940)

Add deferrable mode to 'RdsCreateDbInstanceOperator' and 'RdsDeleteDbInstanceOperator' (#32171)

Add deferrable mode for 'AthenaOperator' (#32186)

Add a deferrable mode to 'BatchCreateComputeEnvironmentOperator' (#32036)

Add deferrable mode in EMR operator and sensor (#32029)

add async wait method to the "with logging" aws utils (#32055)

Add custom waiters to EMR Serverless (#30463)

Add an option to 'GlueJobOperator' to stop the job run when the TI is killed (#32155)

deferrable mode for 'SageMakerTuningOperator' and 'SageMakerEndpointOperator' (#32112)

EKS Create/Delete Nodegroup Deferrable mode (#32165)

Deferrable mode for ECS operators (#31881)

feature: AWS - GlueJobOperator - job_poll_interval (#32147)

Added 'AzureBlobStorageToS3Operator' transfer operator (#32270)

Introduce a base class for aws triggers (#32274)

Bug Fixes

bugfix: break down run+wait method in ECS operator (#32104)

Handle 'UnboundLocalError' while parsing invalid 's3_url' (#32120)

Fix 'LambdaInvokeFunctionOperator' payload parameter type (#32259)

Bug fix GCSToS3Operator: avoid 'ValueError' when 'replace=False' with files already in S3 (#32322)

Misc

Deprecate 'delimiter' param and source object's wildcards in GCS, introduce 'match_glob' param. (#31261)

aws waiter util: log status info with error level on waiter error (#32247)

rewrite method used in ecs to fetch less logs (#31786)

Refactor Eks Create Cluster Operator code (#31960)

Use a waiter in 'AthenaHook' (#31942)

Add 'on_finish_action' to 'KubernetesPodOperator' (#30718)

Add default_deferrable config (#31712)

deprecate arbitrary parameter passing to RDS hook (#32352)

quick fix on RDS operator to prevent parameter collision (#32436)

Remove ability to specify arbitrary hook params in AWS RDS trigger (#32386)

Only update crawler tags if present in config dict (#32331)

Release Date: 2023-06-23

Note

This release dropped support for Python 3.7

Features

Add deferrable option to EmrTerminateJobFlowOperator (#31646)

Add Deferrable option to EmrCreateJobFlowOperator (#31641)

Add deferrable mode to 'BatchSensor' (#30279)

Add deferrable mode for S3KeySensor (#31018)

Add Deferrable mode to Emr Add Steps operator (#30928)

Add deferrable mode in Redshift delete cluster (#30244)

Add deferrable mode to AWS glue operators (Job & Crawl) (#30948)

Add deferrable param in BatchOperator (#30865)

Add Deferrable Mode to RedshiftCreateClusterSnapshotOperator (#30856)

Deferrable mode for EksCreateFargateProfileOperator and EksDeleteFargateProfileOperator (#31657)

allow anonymous AWS access (#31659)

Support of wildcard in S3ListOperator and S3ToGCSOperator (#31640)

Add 'deferrable' param in 'EmrContainerSensor' (#30945)

Add realtime container execution logs for BatchOperator (#31837)

Bug Fixes

Various fixes on ECS run task operator (#31838)

fix return values on glue operators deferrable mode (#31694)

Add back missing AsyncIterator import (#31710)

Use a continuation token to get logs in ecs (#31824)

Fetch status in while loop so as to not exit too early (#31804)

[AWS hook] use provided client to get the official waiter on fallback (#31748)

handle missing LogUri in emr 'describe_cluster' API response (#31482)

Misc

Add Python 3.11 support (#27264)

Added config template field to EmrServerlessStartJobOperator (#31746)

Add null check for host in Amazon Redshift connection (#31567)

add workgroup to templated fields (#31574)

Add docstring and signature for _read_remote_logs (#31623)

Deprecate 'wait_for_completion' from 'EcsRegisterTaskDefinitionOperator' and 'EcsDeregisterTaskDefinitionOperator' (#31884)

Remove Python 3.7 support (#30963)

Change Deferrable implementation for RedshiftResumeClusterOperator to follow standard (#30864)

Change Deferrable implementation for RedshiftPauseClusterOperator to follow standard (#30853)

Release Date: 2023-05-22

Note

This release of provider is only available for Airflow 2.4+ as explained in the Apache Airflow providers support policy .

Features

DynamoDBToS3Operator - Add a feature to export the table to a point in time. (#31142)

Add deferrable param in SageMakerTransformOperator (#31063)

Add deferrable param in SageMakerTrainingOperator (#31042)

Add deferrable param in SageMakerProcessingOperator (#31062)

Add IAM authentication to Amazon Redshift Connection by AWS Connection (#28187)

'StepFunctionStartExecutionOperator': get logs in case of failure (#31072)

Add on_kill to EMR Serverless Job Operator (#31169)

Add Deferrable Mode for EC2StateSensor (#31130)

Bug Fixes

bigfix: EMRHook Loop through paginated response to check for cluster id (#29732)

Misc

Bump minimum Airflow version in providers (#30917)

Add template field to S3ToRedshiftOperator (#30781)

Add extras links to some more EMR Operators and Sensors (#31032)

Add retries to S3 delete_bucket (#31192)

Add tags param in RedshiftCreateClusterSnapshotOperator (#31006)

improve/fix glue job logs printing (#30886)

Import aiobotocore only if deferrable is true (#31094)

Update return types of 'get_key' methods on 'S3Hook' (#30923)

Support 'shareIdentifier' in BatchOperator (#30829)

BaseAWS - Override client when resource_type is user to get custom waiters (#30897)

Add future-compatible mongo Hook typing (#31289)

Handle temporary credentials when resource_type is used to get custom waiters (#31333)

Release Date: 2023-04-27

Breaking changes

Warning

In this version of the provider, deprecated GCS hook’s parameter delegate_to is removed from the following operators: GCSToS3Operator , GlacierToGCSOperator and GoogleApiToS3Operator . Impersonation can be achieved instead by utilizing the impersonation_chain param.

Removed deprecated parameter google_cloud_storage_conn_id from GCSToS3Operator , gcp_conn_id should be used instead.

Removed deprecated parameter max_tries from the Athena & EMR hook & operators in favor of max_polling_attempts .

Removed deprecated method waiter from emr hook in favor of the more generic airflow.providers.amazon.aws.utils.waiter.waiter

Removed deprecated unused parameter cluster_identifier from Redshift Cluster’s hook method get_cluster_snapshot_status

Removed deprecated method find_processing_job_by_name from Sagemaker hook, use count_processing_jobs_by_name instead.

Removed deprecated module airflow.providers.amazon.aws.operators.aws_lambda in favor of airflow.providers.amazon.aws.operators.lambda_function

Removed EcsOperator in favor of EcsRunTaskOperator. EcsTaskLogFetcher and EcsProtocol should be imported from the hook.

Removed AwsLambdaInvokeFunctionOperator in favor of LambdaInvokeFunctionOperator.

Removed deprecated param await_result from RedshiftDataOperator in favor of wait_for_completion . Some methods from this operator should be imported from the hook instead.

Removed deprecated RedshiftSQLOperator in favor of the generic SQLExecuteQueryOperator . The parameter that was passed as redshift_conn_id needs to be changed to conn_id , and the behavior should stay the same.

Removed deprecated method get_conn_uri from secrets manager in favor of get_conn_value Also removed deprecated method get_conn_uri from systems manager. deserialize_connection(...).get_uri() should be used instead.

Removed deprecated and unused param s3_conn_id from ImapAttachmentToS3Operator , MongoToS3Operator and S3ToSFTPOperator .

remove delegate_to from GCP operators and hooks (#30748)

Remove deprecated code from Amazon provider (#30755)

Features

add a stop operator to emr serverless (#30720)

SqlToS3Operator - Add feature to partition SQL table (#30460)

New AWS sensor — DynamoDBValueSensor (#28338)

Add a "force" option to emr serverless stop/delete operator (#30757)

Add support for deferrable operators in AMPP (#30032)

Bug Fixes

Fixed logging issue (#30703)

DynamoDBHook - waiter_path() to consider 'resource_type' or 'client_type' (#30595)

Add ability to override waiter delay in EcsRunTaskOperator (#30586)

Add support in AWS Batch Operator for multinode jobs (#29522)

AWS logs. Exit fast when 3 consecutive responses are returned from AWS Cloudwatch logs (#30756)

Fix async conn for none aws_session_token (#30868)

Misc

Remove @poke_mode_only from EmrStepSensor (#30774)

Organize Amazon providers docs index (#30541)

Remove duplicate param docstring in EksPodOperator (#30634)

Update AWS EMR Cluster Link to use the new dashboard (#30844)

Restore aiobotocore as optional dependency of amazon provider (#30874)

Release Date: 2023-04-12

Bug Fixes

Fix 'RedshiftResumeClusterOperator' deferrable implementation (#30370)

Misc

Add more info to quicksight error messages (#30466)

add template field for s3 bucket (#30472)

Add s3_bucket to template fields in SFTP to S3 operator (#30444)

Release Date: 2023-04-06

Features

Add deferrable mode to 'RedshiftResumeClusterOperator' (#30090)

Add 'AwsToAwsBaseOperator' (#30044)

Add deferrable mode in RedshiftPauseClusterOperator (#28850)

Add support of a different AWS connection for DynamoDB (#29452)

Add 'EC2CreateInstanceOperator', 'EC2TerminateInstanceOperator' (#29548)

Make update config behavior optional in GlueJobOperator (#30162)

custom waiters with dynamic values, applied to appflow (#29911)

Support deleting the local log files when using remote logging (#29772)

Misc

Move string enum class to utils module + add test (#29906)

Align cncf provider file names with AIP-21 (#29905)

rewrite polling code for appflow hook (#28869)

Release Date: 2023-03-06

Features

add num rows affected to Redshift Data API hook (#29797)

Add 'wait_for_completion' param in 'RedshiftCreateClusterOperator' (#29657)

Add Amazon Redshift-data to S3<>RS Transfer Operators (#27947)

Allow to specify which connection, variable or config are being looked up in the backend using *_lookup_pattern parameters (#29580)

Implement file credentials provider for AWS hook AssumeRoleWithWebIdentity (#29623)

Implement custom boto waiters for some EMR operators (#29822)

Bug Fixes

fix code checking job names in sagemaker (#29245)

Avoid emitting fallback message for S3TaskHandler if streaming logs (#29708)

Use waiters in ECS Operators instead of inner sensors (#29761)

Misc

Improvements for RedshiftDataOperator: better error reporting and an ability to return SQL results (#29434)

Standardize AWS lambda naming (#29749)

AWS Glue job hook: Make s3_bucket parameter optional (#29659)

'RedshiftDataOperator' replace 'await_result' with 'wait_for_completion' (#29633)

Release Date: 2023-02-21

Bug Fixes

Explicitly handle exceptions raised by config parsing in AWS provider (#29587)

Misc

Fix docstring for EcsRunTaskOperator region_name -> region (#29562)