fsspec

Features:

Add dbfs:// support (#504, #514)

Enhancements

don’t import pyarrow (#503)

update entry points syntax (#515)

ci precommit hooks (#534)

Fixes:

random appending of a directory within the filesystems find() method (#507, 537)

fix git tests (#501)

fix recursive memfs operations (#502)

fix recursive/maxdepth for cp (#508)

fix listings cache timeout (#513)

big endian bytes tests (#519)

docs syntax (#535, 524, 520, 542)

transactions and reads (#533)

Features:

config system

libarchive implementation

add reference file system implementation

Features:

function can_be_local to see whether URL is compatible with open_local

concurrent cat with filecaches, if backend supports it

jupyter FS

Fixes:

dircache expiry after transaction

blockcache garbage collection

close for HDFS

windows tests

glob depth with “**”

Features:

error options for cat

memory fs created time in detailed ls`

Fixes:

duplicate directories could appear in MemoryFileSystem

Added support for hat dollar lbrace rbrace regex character escapes in glob

Fix blockcache (was doing unnecessary work)

handle multibyte dtypes in readinto

Fix missing kwargs in call to _copy in asyn

Other:

Stop inheriting from pyarrow.filesystem for pyarrow>=2.0

Raise low-level program friendly OSError.

Guard against instance reuse in new processes

Make hash_name a method on CachingFileSystem to make it easier to change.

Use get_event_loop for py3.6 compatibility

Fixes:

More careful strip for caching

Features:

add sign to base class

Allow calling of coroutines from normal code when running async

Implement writing for cached many files

Allow concurrent caching of remote files

Add gdrive:// protocol

Fixes:

Fix memfs with exact ls

HTTPFileSystem requires requests and aiohttp in registry

Other:

Allow http kwargs to clientSession

Use extras_require in setup.py for optional dependencies

Replacing md5 with sha256 for hash (CVE req)

Test against Python 3.8, drop 3.5 testing

add az alias for abfs

Major release allowing async implementations with concurrent batch operations.

Features:

async filesystem spec, first applied to HTTP

OpenFiles cContext for multiple files

Document async, and ensure docstrings

Make LocalFileOpener iterable

handle smb:// protocol using smbprotocol package

allow Path object in open

simplecache write mode

Fixes:

test_local: fix username not in home path

Tighten cacheFS if dir deleted

Fix race condition of lzma import when using threads

properly rewind MemoryFile

OpenFile newline in reduce

Other:

Add aiobotocore to deps for s3fs check

Set default clobber=True on impl register

Use _get_kwargs_from_url when unchaining

Add cache_type and cache_options to HTTPFileSystem constructor

panel-based GUI

added git and github interfaces

added chained syntax for open, open_files and get_mapper

adapt webHDFS for HttpFS

added open_local

added simplecache , and compression to both file caches

Added adl and abfs protocols to the known implementations registry ( GH#209 )

Fixed issue with whole-file caching and implementations providing multiple protocols ( GH#219 )

LocalFileSystem is now considered a filestore by pyarrow ( GH#211 )

Fixed bug in HDFS filesystem with cache_options ( GH#202 )

Fixed instance caching bug with multiple instances ( GH#203 )

Fixed issues with filesystem instance caching. This was causing authorization errors in downstream libraries like gcsfs and s3fs in multi-threaded code ( GH#155 , GH#181 )

Changed the default file caching strategy to fsspec.caching.ReadAheadCache ( GH#193 )

Moved file caches to the new fsspec.caching module. They’re still available from their old location in fsspec.core , but we recommend using the new location for new code ( GH#195 )

Added a new file caching strategy, fsspec.caching.BlockCache for fetching and caching file reads in blocks ( GH#191 ).

Fixed equality checks for file system instance to return False when compared to objects other than file systems ( GH#192 )

Fixed a bug in fsspec.FSMap.keys returning a generator, which was consumed upon iteration ( GH#189 ).

Removed the magic addition of aliases in AbstractFileSystem.init . Now alias methods are always present ( GH#177 )

Deprecated passing trim to fsspec.spec.AbstractBufferedFile . Pass it in storage_options instead ( GH#188 )

Improved handling of requests for fsspec.implementations.http.HTTPFileSystem when the HTTP server responds with an (incorrect) content-length of 0 ( GH#163 )

Added a detail=True parameter to fsspec.spec.AbstractFileSystem.ls() ( GH#168 )

Fixed handling of UNC/DFS paths ( GH#154 )