cloudpathlib

• Security fix (GHSA-r4f8-3xc4-c8vw): local path traversal via .. in cloud object keys.
  Cloud object keys are opaque strings and some backends (e.g. Google Cloud Storage) accept ..
  segments. cloudpathlib mapped keys onto local paths with plain path arithmetic and no containment
  check, so a key such as ../../evil could escape the local cache directory (on any
  read/write/open/os.fspath of a CloudPath) or a download destination
  (download_to/copytree/copy/move to a local path), writing attacker-controlled bytes
  outside the intended directory. cloudpathlib now confirms each computed local target stays within
  its base directory and raises CloudPathLocalPathTraversalError otherwise. This guards the cache
  mapping (CloudPath._local and its S3/HTTP overrides), both branches of CloudPath.download_to
  and CloudPath.copytree (including joins that can escape via \ or drive letters in keys on
  Windows), and the cloudpathlib.local mock's storage-side mapping. Reported by mohammad adnan
  (cystack.ps redteam).
• Added a lazy keyword argument to CloudPath.walk. By default (lazy=False) the existing fast behavior is preserved: the whole subtree is fetched up front with a single recursive listing. Passing lazy=True lists each directory on demand so that, when top_down=True, callers can prune subdirectories by modifying dirnames in-place (à la os.walk / Path.walk) to skip fetching the contents of those subtrees entirely — dramatically reducing API calls for large, sparsely-traversed trees. (Issue #518, PR #574)
• Added presigned URL support to the mock clients in cloudpathlib.local, so as_url(presign=True) on a LocalS3Path, LocalGSPath, or LocalAzureBlobPath returns a deterministic URL with expires and signature query parameters instead of raising NotImplementedError. This lets code that generates presigned URLs be exercised in tests that use the mock classes. (PR #572)
• Fixed S3 copy and move operations to forward copy-specific extra args such as CopySourceSSECustomerKey, and added addressing_style="virtual" support for S3Client (Issues #500, #527, PR #576)
• Changed S3Client._get_metadata to read object metadata with HeadObject instead of GetObject, so stat, etag, and size no longer open the object body. Also fixes a KeyError on ContentLength against S3-compatible gateways that drop Content-Length from GetObject responses. (Issue #564, PR #565)
• Fixed KeyError on ETag against S3-compatible gateways that omit ETag from HeadObject responses. (Issue #582, PR #583)
• Fixed mypy 2.x type errors in Client and CloudPath that caused CI lint failures (Issue #563, PR #566)
• Added AGENTS.md with repository-specific guidance for coding agents covering contributor
  workflow, compatibility expectations, test rig and mock usage, live backend validation, and PR
  hygiene. (PR #573)

• Added support for S3 Multi-Region Access Point (MRAP) URLs in S3Path (Issue #556, PR #557)
• Added support for Pydantic serialization (Issue #537, PR #538)

• Added support for Python 3.14 (Issue #529, PR #530)
  • Changed CloudPath.copy to have the first parameter named target instead of destination and added new follow_symlinks and preserve_metadata kwargs. Breaking change for users that relied on the first parameter being named destination instead of target.
  • Added CloudPath.copy_into to copy a file or directory into another file or directory.
  • Added CloudPath.move to move a file or directory to another location.
  • Added CloudPath.move_into to move a file or directory into another file or directory.
  • Added CloudPathInfo and CloudPath.info to get information about a file or directory.
  • Added additional no-op kwargs to mkdir, touch, glob, rglob, stat to match pathlib.

• Fixed issue with GS credentials, using default auth enables a wider set of authentication methods in GS (Issue #390, PR #514, thanks @ljyanesm)
• Added support for http(s) urls with HttpClient, HttpPath, HttpsClient, and HttpsPath. (Issue #455, PR #468)
• Added experimental support for patching the builtins open, os, os.path, and glob to work with CloudPath objects. It is off by default; see the new "Compatibility" section in the docs for more information. (Issue #128, PR #322)
• Added support for CloudPath(*parts) to create a CloudPath object from a list of parts (to match pathlib.Path). This is a potentially breaking change for users that relied on the second arg being the client instead of making it an explicit kwarg. (PR #322)

• Fixed rmtree fail on Azure with no hns and more than 256 blobs to drop (Issue #509, PR #508, thanks @alikefia)

• Removed support for deprecated env var that had a typo (CLOUPATHLIB_FILE_CACHE_MODE; you should use CLOUDPATHLIB_FILE_CACHE_MODE).
• Added support for timeout and retry kwargs for GSClient. (Issue #484, PR #485, thanks @Mchristos)
• Fixed CloudPath(...) / other to correctly attempt to fall back on other's __rtruediv__ implementation, in order to support classes that explicitly support the / with a CloudPath instance. Previously, this would always raise a TypeError if other were not a str or PurePosixPath. (PR #479)
• Added md5 property to GSPath, updated LocalGSPath to include md5 property, updated mock_gs.MockBlob to include md5_hash property. (Issue #482, PR #483)
• Fixed an uncaught exception on Azure Gen2 storage accounts with HNS enabled when used with DefaultAzureCredential. (Issue #486, PR #487)
• Removed support for Python 3.8, which was EOL in October 2024. (PR #502)

• Added support for custom schemes in CloudPath and Client subclases. (Issue #466, PR #467)
• Fixed ResourceNotFoundError on Azure gen2 storage accounts with HNS enabled and issue that some Azure credentials do not have account_name. (Issue #470, Issue #476, PR #478)
• Added support for Python 3.13 (Issue #472, PR #474):
  • .full_match added
  • .from_uri added
  • follow_symlinks kwarg added to is_file added as no-op
  • follow_symlinks kwarg added to is_dir added as no-op
  • newline kwarg added to read_text
  • recurse_symlinks kwarg added to glob added as no-op
  • pattern parameter for glob can be PathLike
  • recurse_symlinks kwarg added to rglob added as no-op
  • pattern parameter for rglob can be PathLike
  • .parser property added

• Fixed an error that occurred when loading and dumping CloudPath objects using pickle multiple times. (Issue #450, PR #454, thanks to @kujenga)
• Fixed typo in FileCacheMode where values were being filled by environment variable CLOUPATHLIB_FILE_CACHE_MODE instead of CLOUDPATHLIB_FILE_CACHE_MODE. (PR #424, thanks to @mynameisfiber)
• Fixed CloudPath cleanup via CloudPath.__del__ when Client encounters an exception during initialization and does not create a file_cache_mode attribute. (Issue #372, thanks to @bryanwweber)
• Removed support for Python 3.7 and pinned minimal boto3 version to Python 3.8+ versions. (PR #407)
• Changed GSClient to use the native exists() method from the Google Cloud Storage SDK. (PR #420, thanks to @bachya)
• Changed default clients to be lazily instantiated (Issue #428, PR #432)
• Fixed download_to to check for the existence of the cloud file (Issue #430, PR #433)
• Added env vars CLOUDPATHLIB_FORCE_OVERWRITE_FROM_CLOUD and CLOUDPATHLIB_FORCE_OVERWRITE_TO_CLOUD. (Issue #393, PR #437)
• Fixed glob for cloudpathlib.local.LocalPath and subclass implementations to match behavior of cloud versions for parity in testing. (Issue #415, PR #436)
• Changed how cloudpathlib.local.LocalClient and subclass implementations track the default local storage directory (used to simulate the cloud) used when no local storage directory is explicitly provided. (PR #436, PR #462)
  • Changed LocalClient so that client instances using the default storage access the default local storage directory through the get_default_storage_dir rather than having an explicit reference to the path set at instantiation. This means that calling get_default_storage_dir will reset the local storage for all clients using the default local storage, whether the client has already been instantiated or is instantiated after resetting. This fixes unintuitive behavior where reset_local_storage did not reset local storage when using the default client. (Issue #414)
  • Added a new local_storage_dir property to LocalClient. This will return the current local storage directory used by that client instance.
    by reference through the `get_default_ rather than with an explicit.
• Refined the return type annotations for CloudPath.open() to match the behavior of pathlib.Path.open(). The method now returns specific types (TextIOWrapper, FileIO, BufferedRandom, BufferedWriter, BufferedReader, BinaryIO, IO[Any]) based on the provided mode, buffering, and encoding arguments. (Issue #465, PR #464)
• Added Azure Data Lake Storage Gen2 support (Issue #161, PR #450), thanks to @M0dEx for PR #447 and PR #449

• Fixed import error due to incompatible google-cloud-storage by not using transfer_manager if it is not available. (Issue #408, PR #410)

Includes all changes from v0.18.0.

Note: This is the last planned Python 3.7 compatible release version.

• Implement sliced downloads in GSClient. (Issue #387, PR #389)
• Implement as_url with presigned parameter for all backends. (Issue #235, PR #236)
• Stream to and from Azure Blob Storage. (PR #403)
• Implement file: URI scheme support for AnyPath. (Issue #401, PR #404)

Note: This version was yanked due to incompatibility with google-cloud-storage <2.7.0 that causes an import error.

• Fix S3Client cleanup via Client.__del__ when S3Client encounters an exception during initialization. (Issue #372, PR #373, thanks to @bryanwweber)
• Skip mtime checks during upload when force_overwrite_to_cloud is set to improve upload performance. (Issue #379, PR #380, thanks to @Gilthans)

• Add "CloudPath" as return type on __init__ for mypy issues. (Issue #179, PR #342)
• Add with_stem to all path types when python version supports it (>=3.9). (Issue #287, PR #290, thanks to @Gilthans)
• Add newline parameter to the write_text method to align to pathlib functionality as of Python 3.10. PR #362, thanks to @pricemg.
• Add support for Python 3.12 (PR #364)
• Add CLOUDPATHLIB_LOCAL_CACHE_DIR env var for setting local_cache_dir default for clients (Issue #352, PR #357)
• Add CONTRIBUTING.md instructions for contributors (Issue #213, PR #367)

• Compatibility with pydantic >= 2.0.0. (PR #349)

• Changed return type for CloudPathMeta.__call__ to fix problems with pyright/pylance (PR #330)
• Make CloudPath.is_valid_cloudpath a TypeGuard so that type checkers can know the subclass if is_valid_cloudpath is called (PR #337)
• Added follow_symlinks to stat for 3.11.4 compatibility (see bpo 39906)
• Add follow_symlinks to is_dir implementation for CPython glob compatibility (see CPython PR #104512)

• Changed to pyproject.toml-based build.
• Changed type hints from custom type variable DerivedCloudPath to typing.Self (PEP 673). This adds a dependency on the typing-extensions backport package from Python versions lower than 3.11.
• Fixed a runtime key error when an S3 object does not have the Content-Type metadata set. (Issue #331, PR #332)

• Implement file_cache_modes to give users finer-grained control over when and how the cache is cleared. (Issue #10, PR #314)
• Speed up listing directories for Google Cloud Storage. (PR #318)
• Add compatibility for Python 3.11 (PR #317)

• Fix glob logic for buckets; add regression test; add error on globbing all buckets (Issue #311, PR #312)

• API Change: S3Client supports an extra_args kwarg now to pass extra args down to boto3 functions; this enables Requester Pays bucket access and bucket encryption. (Issues #254, #180; PR #307)
• Speed up glob! (Issue #274, PR #304)
• Ability to list buckets/containers a user has access to. (Issue #48, PR #307)
• Remove overly specific status check and assert in production code on remove. (Issue #212, PR #307)
• Update docs, including accessing public buckets. (Issue #271, PR #307)

• API change: Add ignore parameter to CloudPath.copytree in order to match shutil API. (Issue #145, PR #272)
• Use the V2 version for listing objects list_objects_v2 in S3Client. (Issue #155, PR #302)
• Add abilty to use .exists to check for a raw bucket/container (no additional path components). (Issue #291, PR #302)
• Prevent data loss when renaming by skipping files that would be renamed to the same thing. (Issue #277, PR #278)
• Speed up common glob/rglob patterns. (Issue #274, PR #276)

• API change: Make stat on base class method instead of property to follow pathlib (Issue #234, PR #250)
• Fixed "S3Path.exists() returns True on partial matches." (Issue #208, PR #244)
• Make AnyPath subclass of AnyPath (Issue #246, PR #251)
• Skip docstrings if not present to avoid failing under -00 (Issue #238, PR #249)
• Add py.typed file so mypy runs (Issue #243, PR #248)

• Added absolute to CloudPath (does nothing as CloudPath is always absolute) (PR #230)
• Added resolve to CloudPath (does nothing as CloudPath is resolved in advance) (Issue #151, PR #230)
• Added relative_to to CloudPath which returns a PurePosixPath (Issue #149, PR #230)
• Added is_relative_to to CloudPath (Issue #149, PR #230)
• Added is_absolute to CloudPath (always true as CloudPath is always absolute) (PR #230)
• Accept and delegate read_text parameters to cached file (PR #230)
• Added exist_ok parameter to touch (PR #230)
• Added missing_ok parameter to unlink, which defaults to True. This diverges from pathlib to maintain backward compatibility (PR #230)
• Fixed missing root object entries in documentation's Intersphinx inventory (Issue #211, PR #237)

• Fixed pickling of CloudPath objects not working. (Issue #223, PR #224)
• Added functionality to push the MIME (media) type to the content type property on cloud providers by default. (Issue #222, PR #226)

• Fixed inadvertent inclusion of tests module in package. (Issue #173, PR #219)

• Fixed glob and rglob functions by using pathlib's globbing logic rather than fnmatch. (Issue #154)
• Fixed iterdir to not include self. (Issue #15)
• Fixed error when calling suffix and suffixes on a cloud path with no suffix. (Issue #120)
• Changed parents return type from list to tuple, to better match pathlib's tuple-like _PathParents return type.
• Remove support for Python 3.6. Issue #186

• Fixed error when "directories" created on AWS S3 were reported as files. (Issue #148, PR #190)
• Fixed bug where GCE machines can instantiate default client, but we don't attempt it. (Issue #191)
• Support AWS_ENDPOINT_URL environment variable to set the endpoint_url for S3Client. (PR #193)

• Fixed error where BlobProperties type hint causes import error if Azure dependencies not installed.

• Fixed error when using rmtree on nested directories for Google Cloud Storage and Azure Blob Storage. (Issue #184, PR #185)
• Fixed broken builds due mypy errors in azure dependency (PR #177)
• Fixed dev tools for building and serving documentation locally (PR #178)

• Fixed error when importing cloudpathlib for missing botocore dependency when not installed with S3 dependencies. (PR #168)

• Fixed absolute documentation URLs to point to the new versioned documentation pages.
• Fixed bug for S3 where no_sign_request couldn't be used to download files since our code required list permissions to the bucket to do so. (Issue #169, PR #168).

• Added no_sign_request parameter to S3Client instantiation for anonymous requests for public resources on S3. See documentation for more details. (#164)

• Added boto3_transfer_config parameter to S3Client instantiation, which allows passing a boto3.s3.transfer.TransferConfig object and is useful for controlling multipart and thread use in uploads and downloads. See documentation for more details. (#150)

• Added support for custom S3-compatible object stores. This functionality is available via the endpoint_url keyword argument when instantiating an S3Client instance. See documentation for more details. (#138 thanks to @YevheniiSemendiak)
• Added CloudPath.upload_from which uploads the passed path to this CloudPath (issuse #58)
• Added support for common file transfer functions based on shutil. Issue #108. PR #142.
  • CloudPath.copy copy a file from one location to another. Can be cloud -> local or cloud -> cloud. If client is not the same, the file transits through the local machine.
  • CloudPath.copytree reucrsively copy a directory from one location to another. Can be cloud -> local or cloud -> cloud. Uses CloudPath.copy so if client is not the same, the file transits through the local machine.

• Added rich comparison operator support to cloud paths, which means you can now use them with sorted. (#129)
• Added polymorphic class AnyPath which creates a cloud path or pathlib.Path instance appropriately for an input filepath. See new documentation for details and example usage. (#130)
• Added integration with Pydantic. See new documentation for details and example usage. (#130)
• Exceptions: (#131)
  • Changed all custom cloudpathlib exceptions to be located in new cloudpathlib.exceptions module.
  • Changed all custom cloudpathlib exceptions to subclass from new base CloudPathException. This allows for easy catching of any custom exception from cloudpathlib.
  • Changed all custom exceptions names to end with Error as recommended by PEP 8.
  • Changed various functions to throw new CloudPathFileExistsError, CloudPathIsADirectoryError or CloudPathNotADirectoryError exceptions instead of a generic ValueError.
  • Removed exception exports from the root cloudpathlib package namespace. Import from cloudpathlib.exceptions instead if needed.
• Fixed download_to method to handle case when source is a file and destination is a directory. (#121 thanks to @genziano)
• Fixed bug where hash(...) of a cloud path was not consistent with the equality operator. (#129)
• Fixed AzureBlobClient instantiation to throw new error MissingCredentialsError when no credentials are provided, instead of AttributeError. LocalAzureBlobClient has also been changed to accordingly error under those conditions. (#131)
• Fixed GSClient to instantiate as anonymous with public access only when instantiated with no credentials, instead of erroring. (#131)

• Added a new module cloudpathlib.local with utilities for mocking cloud paths in tests. The module has "Local" substitute classes that use the local filesystem in place of cloud storage. See the new documentation article "Testing code that uses cloudpathlib" to learn more about how to use them. (#107)

• Fixed bug where a NameError was raised if the Google Storage dependencies were not installed (even if using a different storage provider). (#115)

• Added support for Google Cloud Storage. Instantiate with URIs prefixed by gs:// or explicitly using the GSPath class. (#113 thanks to @wolfgangwazzlestrauss)
• Changed backend logic to reduce number of network calls to cloud. This should result in faster cloud path operations, especially when dealing with many small files. (#110, #111)

• Fixed CloudPath instantiation so that reinstantiating with an existing CloudPath instance will reuse the same client, if a new client is not explicitly passed. This addresses the edge case of non-idempotency when reinstantiating a CloudPath instance with a non-default client. (#104)

• Fixed a character-encoding bug when building from source on Windows. (#98)

Initial release of cloudpathlib with support for Amazon S3 and Azure Blob Storage! 🎉