awswrangler

New Functionalities

• DynamoDB support #448
• SQLServer support (Driver must be installed separately) #356
• Excel files support #419 #509
• Amazon S3 Access Point support #393
• Amazon Chime initial support #494
• Write compressed CSV and JSON files on S3 #308 #359 #412

Enhancements

• Add query parameters for Athena #432
• Add metadata caching for Athena #461
• Add suffix filters for s3.read_parquet_table() #495

Bug Fix

• Fix keep_files behavior for failed Redshift COPY executions #505

Thanks

We thank the following contributors/users for their work on this release:

@maxispeicher, @danielwo, @jiteshsoni, @gvermillion, @rodalarcon, @imanebosch, @dwbelliston, @tochandrashekhar, @kylepierce, @njdanielsen, @jasadams, @gtossou, @JasonSanchez, @kokes, @hanan-vian @igorborgest.

---

P.S. The AWS Lambda Layer file (.zip) and the AWS Glue file (.whl) are available below. Just upload it and run!

New Functionalities

• Add aws_access_key_id, aws_secret_access_key, aws_session_token and boto3_session for Redshift copy/unload #484

Bug Fix

• Remove dtype print statement #487

Thanks

We thank the following contributors/users for their work on this release:

@danielwo, @thetimbecker, @njdanielsen, @igorborgest.

---

P.S. Lambda Layer zip file and Glue wheel/egg files are available below. Just upload it and run!

New Functionalities

• Add secretmanager module and support for databases connections #402

con = wr.redshift.connect(secret_id="my-secret", dbname="my-db")
df = wr.redshift.read_sql_query("SELECT ...", con=con)
con.close()

Bug Fix

• Fix connection attributes quoting for wr.*.connect() #481
• Fix parquet table append for nested struct columns #480

Thanks

We thank the following contributors/users for their work on this release:

@danielwo, @nmduarteus, @nivf33, @kinghuang, @igorborgest.

---

P.S. Lambda Layer zip file and Glue wheel/egg files are available below. Just upload it and run!

New Functionalities

• New wr.timestream.create_database() function
• New wr.timestream.create_table() function
• New wr.timestream.delete_database() function
• New wr.timestream.delete_table() function
• New ignore_empty argument to ignore 0 bytes files for:
  • wr.s3.merge_datasets()
  • wr.s3.list_objects()
  • wr.s3.read_parquet()
  • wr.s3.read_parquet_metadata()
  • wr.s3.read_csv()
  • wr.s3.read_fwf()
  • wr.s3.read_json()
  • wr.s3.store_parquet_metadata()

Enhancements

• Automatically rollback in case of failed queries for:
  • wr.redshift.read_sql_query()
  • wr.postgresql.read_sql_query()
  • wr.mysql.read_sql_query()

Thanks

We thank the following contributors/users for their work on this release:

@danielwo, @igorborgest.

---

P.S. Lambda Layer zip file and Glue wheel/egg files are available below. Just upload it and run!

Breaking changes

• sqlalchemy and psycopg2 dependencies replaced by redshift_connector and pg8000
• All wr.db.* functions was distributed into wr.redshift.*, wr.postgresql.* and wr.mysql.* (Tutorial)
• Redshift COPY and UNLOAD function was refactored into wr.redshift.* (Tutorial)
• wr.catalog.get_engine() was replaced by wr.redshift.connect(), wr.postgresql.connect(), wr.mysql.connect() (Tutorial)

New Functionalities

• Amazon Timestream support (Tutorial)

Enhancements

• General performance improved for s3 I/O removing eventual consistency guardrails (Reference)
• Add retry with decorrelated jitter for Athena and Glue Catalog calls to overcome throttling in high concurrency scenarios.

Docs

• Updates regarding all new functionalities
• Add Amazon Timestream tutorial
• Add Amazon Timestream tutorial 2

AWS re:Invent related news

• AWS Lambda now supports up to 10 GB of memory and 6 vCPU cores
• Amazon S3 now delivers strong read-after-write consistency
• AWS Lambda now supports container images as a packaging format
• Serverless Batch Scheduling with AWS Batch and AWS Fargate

Thanks

We thank the following contributors/users for their work on this release:

@Brooke-white, @danielwo, @sapientderek, @pmleveque, @igorborgest.

---

P.S. Lambda Layer zip file and Glue wheel/egg files are available below. Just upload it and run!

New Functionalities

• catalog.add_column() #451
• catalog.delete_column() #451

Enhancements

• Deterministic result for s3.read_parquet_metadata() #449
• ~30% faster package import time #460

Bug Fix

• Fix Athena read with ctas_approach=False and chunksize=True #458
• Fix overwriting for not enforced configs #450

Docs

• Small fixes #462 #458 #446

Thanks

We thank the following contributors/users for their work on this release:

@tuannguyen0901, @bryanyang0528, @czagoni, @jesusch, @danielwo, @DonghanYang, @eric-valente, @igorborgest.

---

P.S. Lambda Layer zip file and Glue wheel/egg files are available below. Just upload it and run!

New Functionalities

• Add configurable Endpoint URL for AWS services #418
• Add global environment configuration for Athena workgroups #437

Enhancements

• Support for Apache Arrow 2.0.0 #436
• Allow Decimal to float casting for wr.db.read_sql_query() #431
• Allow unsafe conversions for wr.db.read_sql_query() #427

Bug Fix

• QuickSight functions now allow usernames with "/" #434
• Fix duplicated carriage return for wr.s3.to_csv() running on Windows platform.

Thanks

We thank the following contributors/users for their work on this release:

@martinSpears-ECS, @imanebosch, @Eric-He-98, @brombach, @Thomas-Hirsch, @vuchetichbalint, @igorborgest.

---

P.S. Lambda Layer zip file and Glue wheel/egg files are available below. Just upload it and run!

Enhancements

• Add encrypted glue connection management #413

Bug Fix

• Double carriage return when using \r\n as line terminator (s3.to_csv()) #415
• s3.read_parquet failing with some timezone aware columns #417

Thanks

We thank the following contributors/users for their work on this release:

@jeanbaptistepriez, @mike-at-upside, @Thiago-Dantas, @igorborgest.

---

P.S. Lambda Layer zip file and Glue wheel/egg files are available below. Just upload it and run!

Enhancements

• General exceptions handling improvements #409
• General error messages improvements #409

Bug Fix

• [Parquet Read] Fix index recovery combined with columns filter #408

Docs

• Handling and documenting ctas_approach for custom data sources #392

Thanks

We thank the following contributors/users for their work on this release:

@tasq-inc, @igorborgest.

---

P.S. Lambda Layer zip file and Glue wheel/egg files are available below. Just upload it and run!

Enhancements

• Add s3_additional_kwargs for wr.s3.copy_objects() and wr.s3.merge_datasets() #388
• Add data_source argument for Athena queries #392
• Handling parquet tinyint columns on Redshift loads #400

Bug Fix

• Fix issue with Hive partitions compatibility. #397
• Fix missing catalog_id arguments in partitioned wr.s3.to_parquet() calls #399
• Remove adaptive retry for boto3 resource. #403

Docs

• Few updates.

Thanks

We thank the following contributors/users for their work on this release:

@timgates42, @bvsubhash, @DonghanYang, @sl-antoinelaborde, @Xiangyu-C, @tuannguyen0901, @JPFrancoia, @sapientderek, @igorborgest.

---

P.S. Lambda Layer zip file and Glue wheel/egg files are available below. Just upload it and run!

Bug Fix

• Fix bug for wr.s3.read_parquet() with timezone offset. #385

Thanks

We thank the following contributors/users for their work on this release:

@chrisrana, @igorborgest.

---

P.S. Lambda Layer zip file and Glue wheel/egg files are available below. Just upload it and run!

Bug Fix

• Fix issues in reading Parquet files with timestamp (timezone aware) columns. #382 #383

Thanks

We thank the following contributors/users for their work on this release:

@tasq-inc, @chrisrana, @igorborgest.

---

P.S. Lambda Layer zip file and Glue wheel/egg files are available below. Just upload it and run!

Enhancements

• Significant Amazon S3 I/O speed up for big files #377
• Create Parquet Datasets with columns with CamelCase names #380

Bug Fix

• Read Parquet error for some files created by DMS #376

Docs

• Few updates.

Thanks

We thank the following contributors/users for their work on this release:

@jarretg, @chrisrana, @vikramshitole, @igorborgest.

---

P.S. Lambda Layer zip file and Glue wheel/egg files are available below. Just upload it and run!

Breaking changes

• Global configuration s3fs_block_size was replaced by s3_block_size #370

New Functionalities

• Automatic recovery of Pandas indexes from Parquet files. #366
• Automatic recovery of Pandas time zones from Parquet files. #366
• Optional schema evolution disabling through the new schema_evolution argument. #353

Enhancements

• s3fs dependency was replaced by builtin code. #370
• Significant Amazon S3 I/O speed up for high latency environments (e.g. local, on-premises). #370

Bug Fix

• Improve NaN handling. #362
• Sanitise table name for partitions insertion #360

Docs

• Few updates.

Thanks

We thank the following contributors/users for their work on this release:

@isrsal, @bppont, @weishao-aws, @alexifm, @Digma, @samcon, @TerrellV, @msantino, @alvaropc, @luigift, @igorborgest.

---

P.S. Lambda Layer zip file and Glue wheel/egg files are available below. Just upload it and run!

Bug Fix

• Fix NaN values handling for wr.athena.read_sql_*(). #351

Docs

• Instructions for installation in AWS Glue PySpark Jobs. #46

Thanks

We thank the following contributors/users for their work on this release:

@czagoni, @josecw, @igorborgest.

---

P.S. Lambda Layer zip file and Glue wheel file are available below. Just upload it and run!

New Functionalities

• wr.s3.to_parquet() now has max_rows_by_file argument. #283
• Support for Unix path pattern matching (*, ?, [seq], [!seq]) for any list/read/delete/copy function on S3. #322

Enhancements

• Mypy applied with strict mode.

Bug Fix

• Fix unnecessary table versioning (glue catalog) creation for wr.s3.to_parquet() during appends. #342
• Lack of sanitisation in indexes names for wr.s3.to_parquet/csv(). #343

Docs

• New Who uses AWS Data Wrangler? section!!!

Thanks

We thank the following contributors/users for their work on this release:

@Thiago-Dantas, @andre-marcos-perez, @ericct, @marcelo-vilela, @edvorkin, @nicholas-miles, @chrispruitt, @rparthas ,@igorborgest.

---

P.S. Lambda Layer zip file and Glue wheel file are available below. Just upload it and run!

Breaking changes

• The partitioned parquet reading now has a different approach for pushdown filters. For details check the tutorial

New Functionalities

• Global configuration module - TUTORIAL
• Concurrently partitions write - TUTORIAL
• Flexible Partitions Filter (PUSH-DOWN) - TUTORIAL
• Add Athena query metadata to Pandas DataFrames returned by wr.athane.read_sql_*() - TUTORIAL #331
• wr.athena.describe_table() #329
• wr.athena.show_create_table() #334
• Add path_ignore_suffix argument to all read functions #326

Enhancements

• Support for PyArrow 1.0.0 #337
• Support for Pandas 1.1.0
• Support writing encrypted redshift copy manifest to S3 #327
• wr.athane.read_sql_*() now accepts empty results #299
• Allow connect_args to be passed when creating an SQL engine from a glue connection #309
• Add skip_header_line_count argument to wr.catalog.create_csv_table() #338

Bug Fix

• Add missing type annotations and fix types in docstrings. #321
• KeyError: 'StatementType' with Athena using max_cache_seconds #323
• wr.s3.read_csv() slow with chunksize #324
• wr.s3.read_csv() with "chunksize" does not forward pandas_kwargs "encoding" #330
• Ensure DataFrame mutability for wr.athane.read_sql_*() w/ ctas_approach=True #335

Docs

• Several small updates.

Thanks

We thank the following contributors/users for their work on this release:

@kylepierce, @davidszotten, @meganburger, @erikcw, @JPFrancoia, @zacharycarter, @DavideBossoli88, @c-line, @anand086, @jasadams, @mrtns, @schot, @koiker, @flaviomax, @bryanyang0528, @igorborgest.

---

P.S. Lambda Layer zip file and Glue wheel file are available below. Just upload it and run!

New Functionalities

• Add wr.catalog.get_partitions(). #305

Enhancements

• Improving Decimal casting.

Bug Fix

• Fix support for support for boto3 >= 1.14.18. 🐞 #315

Docs

• Add Spark Table Interoperability tutorial.
• General small updates.

Thanks

We thank the following contributors/users for their work on this release:

@jasadams, @bryanyang0528, @qemtek, @igorborgest.

---

P.S. Lambda Layer zip file and Glue wheel file are available below. Just upload it and run!

Enhancements

• Now casting columns before append on an existing table only if necessary (wr.s3.to_parquet()).
• Add retry mechanism for InternalError on s3 object deletion.
• Add handling of immutable numpy arrays. (flag.writeable==False)

---

P.S. Lambda Layer's zip-file and Glue's wheel/egg are available below. Just upload it and run!

P.P.S. AWS Data Wrangler counts on compiled dependencies (C/C++) so there is no support for Glue PySpark by now (Only Glue Python Shell).

Enhancements

• Casting support for any column type to string using dtype argument on wr.s3.to_parquet()

Bug Fix

• General bugs related to Athena Cache. 🐞

Docs

• General small updates.

---

P.S. Lambda Layer's zip-file and Glue's wheel/egg are available below. Just upload it and run!

P.P.S. AWS Data Wrangler counts on compiled dependencies (C/C++) so there is no support for Glue PySpark by now (Only Glue Python Shell).

New Functionalities

• Amazon Athena CACHE 🚀 #285
• Initial AWS STS module

Enhancements

• Numpy 1.19.0
• Add auto_create and db_groups arguments to get_redshift_temp_engine #288
• Add validate_schema arguments to wr.s3.read_parquet_table
• Add safe argument to read_parquet #296
• Refactor naming of pandas kwargs #291
• Allow providing suffix to s3.store_parquet_metadata #295
• Add last_modified_begin and last_modified_begin to list_objects, read_csv, read_json, read_fwf and read_parquet

Bug Fix

• Fix bug on get_table_description on tables w/o description #294

Docs

• Add Athena cache tutorial.

Thanks

We thank the following contributors/users for their work on this release:

@koiker, @patrick-muller, @flaviomax, @acere, @jarretg, @bryanyang0528, @schrobot, @kinghuang, @igorborgest.

---

P.S. Lambda Layer's zip-file and Glue's wheel/egg are available below. Just upload it and run!

P.P.S. AWS Data Wrangler counts on compiled dependencies (C/C++) so there is no support for Glue PySpark by now (Only Glue Python Shell).

New Functionalities

• Amazon QuickSight support! 🎉
• Add create/delete database on wr.glue

Enhancements

• General improvements in the tutorials
• New Amazon S3 path check
• Add sanitize_columns arg for s3.to_parquet and s3.to_csv #278 #279
• Remove memory copy of DataFrame for to_parquet and to_csv

Bug Fix

• Force index=False for wr.db.to_sql() with redshift

Thanks

We thank the following contributors/users for their work on this release:

@ywang103, @patrick-muller, @tuliocasagrande, @sarojdongol, @sdknij, @ilyanoskov, @igorborgest.

---

P.S. Lambda Layer's zip-file and Glue's wheel/egg are available below. Just upload it and run!

P.P.S. AWS Data Wrangler counts on compiled dependencies (C/C++) so there is no support for Glue PySpark by now (Only Glue Python Shell).

New Functionalities

• Add support for reading CSV, JSON and FWF partitions. #265

Enhancements

• General improvement of moto tests

Bug Fix

• Fix encoding arg support for reading CSV, JSON and FWF. #271

Thanks

We thank the following contributors/users for their work on this release:

@bryanyang0528, @dwbelliston, @patrick-muller, @sdknij, @igorborgest.

---

P.S. Lambda Layer's zip-file and Glue's wheel/egg are available below. Just upload it and run!

P.P.S. AWS Data Wrangler counts on compiled dependencies (C/C++) so there is no support for Glue PySpark by now (Only Glue Python Shell).

New Functionalities

• Support for Athena Partition Projection [TUTORIAL]

Enhancements

• Bumping SQLAlchemy version to 1.3.15 #259
• General improvement of moto tests #254

Bug Fix

• Fix dtype (cast) on wr.s3.to_parquet with nested types #263
• Fix EMR utilities for others region different than us-east-1 #252
• Fix wr.s3.to_parquet for partitions in reverse order #264

Thanks

We thank the following contributors/users for their work on this release:

@bryanyang0528, @zachmoshe, @buseynehannes, @jiajie999, @igorborgest.

---

P.S. Lambda Layer's zip-file and Glue's wheel/egg are available below. Just upload it and run!

P.P.S. AWS Data Wrangler counts on compiled dependencies (C/C++) so there is no support for Glue PySpark by now (Only Glue Python Shell).

New Functionalities

• Infer mixed Parquet schemas on wr.s3.read_parquet_metadata and wr.s3.store_parquet_metadata #195
• Support to add new columns on wr.s3.to_parquet and wr.s3.store_parquet_metadata [TUTORIAL] #232

Enhancements

• Now wr.s3.delete_objects raises exception for not deleted objects #237
• User-friendly exceptions on wr.athena.read_sql_query and wr.athena.read_sql_table #239

Bug Fix

• Fix issue to use wr.s3.store_parquet_metadata on non-partitioned datasets #231
• Fix bug on wr.s3.read_json using chunksize #235
• s3fs version bumped #236
• wr.s3.to_parquet single file does not sanitize column names fixed #240

Thanks

We thank the following contributors/users for their work on this release:

@mrshu, @bryanyang0528, @JPFrancoia, @jaidisido, @qemtek, @dwbelliston, @mbiemann, @parasml, @BrainMonkey, @hyperloglog, @igorborgest.

---

P.S. Lambda Layer's zip-file and Glue's wheel/egg are available below. Just upload it and run!

P.P.S. AWS Data Wrangler counts on compiled dependencies (C/C++) so there is no support for Glue PySpark by now (Only Glue Python Shell).

New Functionalities

• Add support for uint8, uint16, uint32 and uint64 on Parquet. #76
• Add get_table_parameters, upsert_table_parameters and upsert_table_parameters on wr.catalog. #224

Enhancements

• Add readahead cache for s3fs.

Bug Fix

• Fixing type hints for sortkey. #226
• Fix s3.to_parquet overwriting with different partition schema.

Thanks

We thank the following contributors/users for their work on this release:

@robertaves ,@jar-no1, @JPFrancoia, @igorborgest.

---

P.S. Lambda Layer's zip-file and Glue's wheel/egg are available below. Just upload it and run!

P.P.S. AWS Data Wrangler counts on compiled dependencies (C/C++) so there is no support for Glue PySpark by now (Only Glue Python Shell).

Bug Fix

• Removing objects ending with "/" from wr.s3.list_objects()

---

P.S. Lambda Layer's zip-file and Glue's wheel/egg are available below. Just upload it and run!

P.P.S. AWS Data Wrangler counts on compiled dependencies (C/C++) so there is no support for Glue PySpark by now (Only Glue Python Shell).

New Functionalities

• Support for nested arrays and structs on wr.s3.to_parquet() #206
• Support for Read Parquet/Athena/Redshift chunked by number of rows #192
• Add custom_classifications to wr.emr.create_cluster() #193
• Support for Docker on EMR #193
• Add kms_key_id, max_file_size, region arguments to wr.db.unload_redshift() #197
• Add catalog_versioning argument to wr.s3.to_csv() and wr.s3.to_parquet() #198
• Add keep_files and ctas_temp_table_name arguments to wr.athena.read_sql_*() #203
• Add replace_filenames argument to wr.s3.copy_objects() #215

Enhancements

• wr.s3.to_csv() and wr.s3.to_parquet() no longer need delete table permission to overwrite catalog table #198
• Added support for UUID on wr.db.read_sql_query()(PostgreSQL) #200
• Refactoring of Athena encryption and workgroup support #212

Bug Fix

• Support for read full NULL columns from PostgreSQL, MySQL, and Redshift #218

Thanks

We thank the following contributors/users for their work on this release:

@robkano ,@luigift, @parasml, @OElesin, @jar-no1, @keatmin, @pmleveque, @sapientderek, @jadayn, @igorborgest.

---

P.S. Lambda Layer's zip-file and Glue's wheel/egg are available below. Just upload it and run!

P.P.S. AWS Data Wrangler counts on compiled dependencies (C/C++) so there is no support for Glue PySpark by now (Only Glue Python Shell).

New Functionalities

• Add wr.s3.copy_objects and wr.s3.merge_datasets #186
• Registering module's type annotations #194

Enhancements

• Support append mode for wr.catalog.create_parquet_table and wr.catalog.create_csv_table #188

Docs

• Adding a note about collisions for wr.catalog.sanitize_dataframe_columns_names #185

Thanks

We thank the following contributors/users for their work on this release:

@JPFrancoia, @deathrowe, @igorborgest.

---

P.S. Lambda Layer's bundle and Glue's wheel/egg are available below. Just upload it and run!

P.P.S. AWS Data Wrangler counts on compiled dependencies (C/C++) so there is no support for Glue PySpark by now (Only Glue Python Shell).

New Functionalities

• Add moto support for S3 and EMR (partially) #109

Enhancements

• Add CSV tutorials #181

Bug Fix

• Fix cast for char and varchar lengths #182
• Fix Athena issues with boto3 session #179

Thanks

We thank the following contributors/users for their work on this release:

@russellbrooks, @vincentclaes, @JPFrancoia, @igorborgest.

---

P.S. Lambda Layer's bundle and Glue's wheel/egg are available below. Just upload it and run!

P.P.S. AWS Data Wrangler counts on compiled dependencies (C/C++) so there is no support for Glue PySpark by now (Only Glue Python Shell).

New Functionalities

• Add new wr.catalog.extract_athena_types() #141

Enhancements

• Add validate_schema to wr.s3.to_parquet() #167

Bug Fix

• Add CSV Dataset utilities to wr.s3.to_csv #170
• Fix CSV decompression #175
• Fix missing boto3_session #172

Thanks

We thank the following contributors/users for their work on this release:

@vfrank66, @JPFrancoia, @jewelltp, @hjuhel-cdpq, @jar-no1, @rmlove, @josecw, @igorborgest.

---

P.S. Lambda Layer's bundle and Glue's wheel/egg are available below. Just upload it and run!

P.P.S. AWS Data Wrangler counts on compiled dependencies (C/C++) so there is no support for Glue PySpark by now (Only Glue Python Shell).

New Functionalities

• categories arg in s3.read_parquet, db.unload_redshift, athena.read_sql_query [#160]

Enhancements

• Athena's table and columns names sanitisation revisited [#161]

Bug Fix

• Add support for Athena queries on workgroups without encryption [#159]

Thanks

We thank the following contributors/users for their work on this release:

@vfrank66, @nitin-kakkar, @sapientderek, @nagomiso, @igorborgest.

---

P.S. Lambda Layer's bundle and Glue's wheel/egg are available below. Just upload it and run!

P.P.S. AWS Data Wrangler counts on compiled dependencies (C/C++) so there is no support for Glue PySpark by now (Only Glue Python Shell).

1.0.0 🎉

Check out the brand new documentation page!

---

P.S. Lambda Layer's bundle and Glue's wheel/egg are available below. Just upload it and run!

P.P.S. AWS Data Wrangler counts on compiled dependencies (C/C++) so there is no support for Glue PySpark by now (Only Glue Python Shell).

New Functionalities

• Add header and filename arguments to Pandas.to_csv()

Enhancements

• Pandas.read_parquet() will return Int64 for integers with null values mixed #132
• Pandas.to_redshift() now is able to cast Int64 for integers with null values mixed #132

Bug Fixies

• s3.head_object_with_retry() public again #133

---

P.S. Lambda Layer's bundle and Glue's wheel/egg are available below. Just upload it and run!

P.P.S. Have you never used Layers? Check the step-by-step guide.

P.P.P.S. AWS Data Wrangler counts on compiled dependencies (C/C++) so there is no support for Glue PySpark by now (Only Glue Python Shell).

New Functionalities

• Add pandas.read_fwf(), read_fwf_list(), read_fwf_prefix() for fixed-width files #131
• Support for compressed files for pandas.read_csv(), read_csv_list() and read_csv_prefix() #129
• Support for consistent view on emr.create_cluste() #130

Enhancements

• Support for Python 3.8
• Bumping Pandas version to 1.0.1
• Bumping PyArrow version to 0.16.0

Docs

• New documentation page

---

P.S. Lambda Layer's bundle and Glue's wheel/egg are available below. Just upload it and run!

P.P.S. Have you never used Layers? Check the step-by-step guide.

P.P.P.S. AWS Data Wrangler counts on compiled dependencies (C/C++) so there is no support for Glue PySpark by now (Only Glue Python Shell).

Enhancements

• Support for Pandas 1.0.0
• Support for all pandas.read_csv() arguments
• Support for custom VARCHAR length for Aurora and Redshift

---

P.S. Lambda Layer's bundle and Glue's wheel/egg are available below. Just upload it and run!

P.P.S. Have you never used Layers? Check the step-by-step guide.

P.P.P.S. AWS Data Wrangler counts on compiled dependencies (C/C++) so there is no support for Glue PySpark by now (Only Glue Python Shell).

Enhancements

• Smaller Lambda layers #113
• Support for categorical partitions for Pandas.to_parquet() #115
• Support for RangeIndex for Pandas.to_parquet() #111
• Add columns parameter for Pandas.to_csv() #110
• Add columns parameter for Pandas.to_aurora() #110
• Improving NaN handling during Pandas.read_sql_athena()
• Small performance improvements

Bugfixes

• Fixing bug to unload null values from Aurora #114

---

P.S. Lambda Layer's bundle and Glue's wheel/egg are available below. Just upload it and run!

P.P.S. Have you never used Layers? Check the step-by-step guide.

P.P.P.S. AWS Data Wrangler counts on compiled dependencies (C/C++) so there is no support for Glue PySpark by now (Only Glue Python Shell).

Enhancements

• Pandas.to_aurora() improvements
• Pandas.to_redshift() improvements
• Pandas.read_sql_athena(ctas_approach=True) improvements
• Pandas.read_parquet() improvements

---

P.S. Lambda Layer's bundle and Glue's wheel/egg are available below. Just upload it and run!

P.P.S. Have you never used Layers? Check the step-by-step guide.

P.P.P.S. AWS Data Wrangler counts on compiled dependencies (C/C++) so there is no support for Glue PySpark by now (Only Glue Python Shell).

Enhancements

• Support for empty dataframe for Pandas.read_sql_athena(ctas_approach=True)
• Cleaning temp S3 files for Pandas.read_sql_athena(ctas_approach=True)
• Inverting file format and file compression extensions (key suffix) (Hadoop/Spark/Hive compatibility)
• Aurora ingestion revisited
• Bumping dependencies version
• Add Pandas.read_csv_prefix()
• Improve Athena._normalize_name() rules
• Improving autocomplete support
• Simplifying everything on Sagemaker
• Adding Glue.get_connection()
• Adapt read_sql_athena(ctas_approach=True) for eventual consistency caveats.

Bugfixes

• Fixing bug to fetch Glue tables comments
• Fixing Spark for default Session

Docs

• Add athena_nested.ipynb tutorial
• Add catalog_and_metadata.ipynb tutorial

---

P.S. Lambda Layer's bundle and Glue's wheel/egg are available below. Just upload it and run!

P.P.S. Have you never used Layers? Check the step-by-step guide.

P.P.P.S. AWS Data Wrangler counts on compiled dependencies (C/C++) so there is no support for Glue PySpark by now (Only Glue Python Shell).

Enhancements

• Add description, parameters and column's comments as arguments to all methods that creates any Glue tables (METADATA).
• Add several methods to explore the Glue Catalog.

---

P.S. Lambda Layer's bundle and Glue's wheel/egg are available below. Just upload it and run!

P.P.S. Have you never used Layers? Check the step-by-step guide.

P.P.P.S. AWS Data Wrangler counts on compiled dependencies (C/C++) so there is no support for Glue PySpark by now (Only Glue Python Shell).

Enhancements

• Pandas -> Aurora (MySQL/PostgreSQL) (Append/Overwrite) (Via S3)
• Aurora -> Pandas (MySQL) (Via S3)
• Aurora -> CSV (S3) (MySQL)
• Smaller lambda layers

---

P.S. Lambda Layer's bundle and Glue's wheel/egg are available below. Just upload it and run!

P.P.S. Have you never used Layers? Check the step-by-step guide.

P.P.P.S. AWS Data Wrangler counts on compiled dependencies (C/C++) so there is no support for Glue PySpark by now (Only Glue Python Shell).

Bugfixes

• Fix Default Session bug for environments without credentials

---

P.S. Lambda Layer's bundle and Glue's wheel/egg are available below. Just upload it and run!

Enhancements

• Pandas to Redshift with upsert mode
• Load SageMaker Job outputs
• Default Session

P.S. Lambda Layer's bundle and Glue's wheel/egg are available below. Just upload it and run!

Enhancements

• Read Parquet tables from Glue Catalog directly to Pandas DataFrame
• Read Athena's results to Pandas DataFrame via CTAS (Blazing fast 🚀)
• Redshift's results to S3 as Parquet
• Read Redshift's results to Pandas DataFrame via Parquet export (Blazing fast 🚀)

P.S. Lambda Layer's bundle and Glue's wheel/egg are available below. Just upload it and run!

Enhancements

• Read parquet data from s3 directly to Pandas DataFrame #73

Bugfixes

• Fix Pandas.read_sql_athena() usage with the Session() default s3_output

P.S. Lambda Layer's bundle and Glue's wheel/egg are available below. Just upload it and run!

Enhancements

• Add support for Decimal data type #58
• Add more Athena's settings in Session() (defaults)
• Add PyArrow's toggle option for EMR.create_cluster()

Bugfixes

• Fix Pandas.read_sql_athena() issues with arrays data types #72

P.S. Lambda Layer's bundle and Glue's wheel/egg are available below. It's just upload and run!

Enhancements

• Improving cast for date columns

P.S. Lambda Layer's bundle and Glue's wheel/egg are available below. It's just upload and run!

Bugfixes

• Setting null date values as None for pandas.read_sql_athena() #69

P.S. Lambda Layer's bundle and Glue's wheel/egg are available below. It's just upload and run!

Bugfixes

• Fix bug for boolean type on spark.to_redshift()

P.S. Lambda Layer's bundle and Glue's wheel/egg are available below. It's just upload and run!

Enhancements

• Updating all dependencies #65

Bugfixes

• Fix cast for nan values #66

P.S. Lambda Layer's bundle and Glue's wheel/egg are available below. It's just upload and run!

Bugfixes

• Fix issues for partitions with a single raw #62

P.S. Lambda Layer's bundle and Glue's wheel/egg are available below. It's just upload and run!

Bugfixes

• Fix cast issues for partition columns #59 #60

P.S. Lambda Layer's bundle and Glue's wheel/egg are available below. It's just upload and run!

Enhancements

• Pandas.read_sql_athena() now also accept bool columns with null values.

P.S. Lambda Layer's bundle and Glue's wheel/egg are available below. It's just upload and run!

Enhancements

• Add tags parameter to EMR.create_cluster()

P.S. Lambda Layer's bundle and Glue's wheel/egg are available below. It's just upload and run!