s3pathlib

Bugfixes

• Relax the requirements of func_args>=0.1.1,<1.0.0 to func_args>=0.1.1,<2.0.0 to allow the latest version of func_args to be used.

Miscellaneous

• Use pywf_open_source code skeleton.

Minor Improvements

• Allow user to use context.attach_s3_client(your_custom_s3_client) and context.attach_sts_client(your_custom_sts_client) to set a custom S3 client, so that you can work with any S3 compatible storage by customizing your own s3 client object.

Miscellaneous

• Upgrade moto to 5.X.Y

• Add support to use smart_open 7.X.Y

• Stop testing on Python 3.7

Features and Improvements

• Add the following public APIs:

  • S3Path.get_regional_console_url

  • S3Path.get_regional_us_gov_cloud_console_url

  • S3Path.get_regional_s3_select_console_url

  • S3Path.get_regional_s3_select_us_gov_cloud_console_url

Features and Improvements

• Add s3pathlib.core.metadata.MetadataAPIMixin.response property method to allow developer to access the underlying head_object API response.

• Add support for Python 3.12.

Minor Improvements

• now most of the S3Path methods can take both boto3.client("s3") or boto_session_manager.BotoSesManager(...) object for bsm argument.

Features and Improvements

• ~s3pathlib.core.opener.OpenerAPIMixin.open method now takes full list of s3_client.get_object, s3_client.put_object, s3_client.create_multipart_upload arguments.

New APIs:

• add ~s3pathlib.core.bucket.BucketAPIMixin.from_bucket

• add ~s3pathlib.core.bucket.BucketAPIMixin.create_bucket

• add ~s3pathlib.core.bucket.BucketAPIMixin.delete_bucket

• add ~s3pathlib.core.bucket.BucketAPIMixin.get_bucket_versioning

• add ~s3pathlib.core.bucket.BucketAPIMixin.is_versioning_enabled

• add ~s3pathlib.core.bucket.BucketAPIMixin.is_versioning_suspended

• add ~s3pathlib.core.bucket.BucketAPIMixin.put_bucket_versioning

• add ~s3pathlib.core.bucket.BucketAPIMixin.list_buckets

• add ~s3pathlib.content_type.ContentTypeEnum

• add ~s3pathlib.validate.validate_s3_bucket

• add ~s3pathlib.validate.validate_s3_key

• add ~s3pathlib.validate.validate_s3_uri

• add ~s3pathlib.validate.validate_s3_arn

Bugfixes

• ~s3pathlib.core.delete.DeleteAPIMixin.delete method now can delete cached metadata correctly.

• fix a bug that ~s3pathlib.core.opener.OpenerAPIMixin.open method cannot explicitly set client kwargs.

Features and Improvements

Changes:

• most of methods now supports version_id argument for S3 versioning enabled bucket.

• s3pathlib.core.rw.ReadAndWriteAPIMixin.read_bytes and s3pathlib.core.rw.ReadAndWriteAPIMixin.read_text now support version_id and full list of get_object boto3 arguments.

• s3pathlib.core.uri.UriAPIMixin.console_url now take the version id into consideration.

• add s3pathlib.core.is_test.IsTestAPIMixin.is_delete_marker method.

• s3pathlib.core.base.BaseS3Path.__new__ now can take S3 URI and ARN directly without using the from_s3_uri and from_s3_arn method.

New APIs:

• add s3pathlib.better_client to the public API, it is a collection of more user friendly boto3 API, with type hint.

• add ~s3pathlib.core.iter_object_versions.IterObjectVersionsAPIMixin.list_object_versions method. It will return all objects, all versions, all delete markers in reverse chronological order.

• The ~s3pathlib.core.is_test.IsTestAPIMixin.is_delete_marker method can be used to check if the version is a delete marker.

• ~s3pathlib.core.delete.DeleteAPIMixin.delete: delete object, directory (recursively), specific object version, object versions, all objects all versions in directory. This API will replace the old ~s3pathlib.core.delete.DeleteAPIMixin.delete_if_exists API in 3.X.Y.

Minor Improvements

• use moto for unit test.

• add tons of more unit test for edge cases.

• prompt to confirm when delete everything in a bucket.

Bugfixes

• fix a bug that should not allow user to set value to attributes like bucket, key.

Miscellaneous

• completely rewrite the documentation in Jupyter notebook to provide interactive reading experience.

Imcompatible Change

• removed ~s3pathlib.core.joinpath.JoinPathAPIMixin.join_path.

Features and Improvements

• now the following API supports full list of native boto3 argument:

  • S3Path.write_bytes

  • S3Path.write_text

  • S3Path.copy_file

  • S3Path.copy_dir

  • S3Path.copy_to

  • S3Path.move_to

• add many optional argument to the S3Path.sync() method.

Features and Improvements

• allow and_, or_, not_ in iterproxy filter.

• allow update metadata and taggings in ~s3pathlib.core.S3Path.copy.CopyAPIMixin.copy_file, ~s3pathlib.core.S3Path.copy.CopyAPIMixin.copy_to and ~s3pathlib.core.S3Path.copy.CopyAPIMixin.move_to method.

Miscellaneous

• the iterproxy.py module is taken out and released as a independent project.

Features and Improvements

• ~s3pathlib.core.opener.OpenerAPIMixin.open method now takes metadata and tags arguments.

• ~s3pathlib.core.rw.ReadAndWriteAPIMixin.write_bytes and ~s3pathlib.core.rw.ReadAndWriteAPIMixin.write_text, ~s3pathlib.core.rw.ReadAndWriteAPIMixin.touch method now takes metadata and tags arguments.

• add ~s3pathlib.core.sync.SyncAPIMixin.sync, ~s3pathlib.core.sync.SyncAPIMixin.sync_from, ~s3pathlib.core.sync.SyncAPIMixin.sync_to method that execute aws s3 sync command

Minor Improvements

• raise a warning if there is upper case key used in user defined metadata.

Minor Improvements

• Refactor core.py module, split the 2.5k line script into 10+ 100 line module.

• Made the metadata pull more intelligent.

Bugfixes

• Fix a bug that the S3Path constructor should not take concrete S3 object / dir path as the second / third / fourth / ... arguments

• Fix a bug that cannot get the metadata value of the S3Path is created by _from_content_dict method.

Features and Improvements

• add tagging management feature

  • add ~s3pathlib.core.S3Path.get_tags method

  • add ~s3pathlib.core.S3Path.put_tags method

  • add ~s3pathlib.core.S3Path.update_tags method

• allow update metadata and taggings in ~s3pathlib.core.S3Path.write_text and ~s3pathlib.core.S3Path.write_bytes method.

Features and Improvements

• add s3pathlib.core.S3Path.joinpath method to mimick pathlib.Path.joinpath behavior

• add s3pathlib.core.S3Path.s3_select_console_url property

• add s3pathlib.core.S3Path.s3_select_us_gov_cloud_console_url property

Bugfixes

• made s3pathlib.core.S3Path.console_url and s3pathlib.core.S3Path.us_gov_cloud_console_url regular property, they should not be FilterableProperty

Miscellaneous

• mark s3pathlib.core.S3Path.join_path as deprecated

Features and Improvements

• add the __truediv__ operator override. it is a s3path / part1 / part2 syntax sugar.

• add the __sub__ operator override. it is a S3Path("bucket/folder") - S3Path("bucket") syntax sugar.

Minor Improvements

Bugfixes

Miscellaneous

Miscellaneous

• remove the boto session module, now it depends on boto_session_manager library.

• add compatibility support for smart_open >= 6.0 due to the ignore_ext arg is removed.

• for s3 IO feature, you need smart_open>=5.1.x

Features and Improvements

• add s3pathlib.core.S3Path.boto_ses.BotoSesManager.get_client method

Bugfixes

• fix import bug in s3pathlib.aws module

Bugfixes

• fix import bug in s3pathlib.boto_ses module

Features and Improvements

• add s3pathlib.core.S3Path.to_file method.

• add s3pathlib.core.S3Path.to_dir method.

• add s3pathlib.core.S3Path.parents method.

• add s3pathlib.core.S3Path.iterdir method.

• add s3pathlib.core.S3Path.touch method.

• add s3pathlib.core.S3Path.mkdir method.

• add s3pathlib.core.S3Path.boto_ses.BotoSesManager class.

Minor Improvements

• add bsm boto session manager parameter for all method using s3 api.

Bugfixes

Miscellaneous

Features and Improvements

• add s3pathlib.core.S3Path.us_gov_cloud_console_url property

• add s3pathlib.utils.parse_data_size method

Minor Improvements

Bugfixes

Miscellaneous

Features and Improvements

• add s3pathlib.core.S3PathIterProxy.equal_to

• add s3pathlib.core.S3PathIterProxy.not_equal_to

• add s3pathlib.core.S3PathIterProxy.greater

• add s3pathlib.core.S3PathIterProxy.greater_equal

• add s3pathlib.core.S3PathIterProxy.less

• add s3pathlib.core.S3PathIterProxy.less_equal

• add recursive = True | False argument for s3pathlib.util.iter_objects, so you can ignore files in nested folders

• add recursive = True | False argument for s3pathlib.core.S3Path.iter_objects, so you can ignore files in nested folders

Minor Improvements

Bugfixes

• fix a bug that s3pathlib.core.S3Path.fname was a regular property and not filterable

Miscellaneous

• Add "S3 Object filter" doc

• Add "File Liked Object IO Object filter" doc

Features and Improvements

• make s3pathlib.core.S3Path a file-like object that support open, read, write.

• add s3pathlib.core.S3PathIterProxy that greatly simplify S3 object filtering.

• add s3pathlib.core.S3Path.open method, makes S3Path a file-like object

• add s3pathlib.core.S3Path.write_text

• add s3pathlib.core.S3Path.read_text

• add s3pathlib.core.S3Path.write_bytes

• add s3pathlib.core.S3Path.read_bytes

Features and Improvements

• add s3pathlib.core.S3Path.from_s3_uri method.

• add s3pathlib.core.S3Path.from_s3_arn method.

• add s3pathlib.core.S3Path.change method.

• add s3pathlib.core.S3Path.is_parent_of method.

• add s3pathlib.core.S3Path.is_prefix_of method.

• add s3pathlib.core.S3Path.dirpath property.

• add better support to handle auto-created "empty folder" object, add include_folder=True parameter for s3pathlib.core.S3Path.list_objects, s3pathlib.core.S3Path.count_objects, s3pathlib.core.S3Path.calculate_total_size method.

Bugfixes

• fix a bug that AWS S3 will create an invisible object when creating a folder, it should not counts as a valid object for s3pathlib.core.S3Path.count_objects

Miscellaneous

• A lot doc improvement.

Features and Improvements

• s3pathlib.S3Path API becomes stable

• s3pathlib.utils API becomes stable

• s3pathlib.context API becomes stable

Miscellaneous

• First stable release.

• First release, a placeholder release.