Changelog

2022.11.0

Enhancements

  • Speed up FSMap._key_to_str (#1101)

  • Add modified/created to Memory and Arrow (#1096)

  • Clear expired cache method (#1092)

  • Allow seekable arrow file (#1091)

  • Allow append for arrow (#1089)

  • recursive for sftp.get (#1082)

  • topdown arg to walk() (#1081)

Fixes

  • fix doc warnings (#1106, #1084)

  • Fix HDFS _strip_protocol (#1103)

  • Allow URLs with protocol for HDFS (#1099)

  • yarl in doc deps (#1095)

  • missing await in genericFS.cp (#1094)

  • explicit IPv4 for test HTTP server (#1088)

  • sort when merging ranges for referenceFS (#1087)

Other

  • Check that snappy is snappy (#1079)

2022.10.0

Enhancements

  • referenceFS consolidates reads in the same remote file (#1063)

  • localfs: add link/symlink/islink (#1059)

  • asyncfs: make mirroring methods optional (#1054)

  • local: info: provide st_ino and st_nlink from stat (#1053)

  • arrow_hdfs replaces hdfs (#1051)

  • Add read/write_text (#1047)

  • Add pipe/cat to genericFS (#1038)

Fixes

  • SSH write doesn’t return number of bytes (#1072)

  • wrap flush method for LocalFileOpened (#1070)

  • localfs: fix support for pathlib/os.PathLike objects in rm (#1058)

  • don’t get_file remote FTP directory (#1056)

  • fix zip write to remote (#1046)

  • fix zip del following failed init (#1040)

Other

  • add asynclocalfs to the registry (#1060)

  • add DVCFileSystem to the registry (#1049)

  • add downstream tests (#1037)

  • Don’t auto-close OpenFiles (#1035)

2022.8.2

  • don’t close OpenFile on del (#1035)

2022.8.1

  • revert #1024 (#1029), with strciter requirements on OpenFile usage

2022.8.0

Enhancements

  • writable ZipFileSystem (#1017)

  • make OpenFile behave like files and remove dynamic closer in .open() (#1024)

  • use isal gunzip (#1008)

Fixes

  • remove strip from _parent (#1022)

  • disallow aiohttp prereleases (#1018)

  • be sure to close cached file (#1016)

  • async rm in reverse order (#1014)

  • expose fileno in LocalFileOpener (#1010, #1005)

  • remove temp files with simplecache writing (#1006)

  • azure paths (#1003)

  • copy dircache keys before iter

2022.7.1

Fixes

  • Remove fspath from LocalFileOpener (#1005)

  • Revert 988 (#1003)

2022.7.0

Enhancements

  • added fsspec-xrootd implementation to registry (#1000)

  • memory file not to copy bytes (#999)

  • Filie details passed to FUSE (#972)

Fixes

  • Return info for root path of archives (#996)

  • arbitrary kwargs passed through in pipe_file (#993)

  • special cases for host in URLs for azure (#988)

  • unstrip protocol criterion (#980)

  • HTTPFile serialisation (#973)

Other

  • Show erroring path in FileNotFounds (#989)

  • Reference file info without searching directory tree (#985)

  • Truncate for local files (#975)

2022.5.0

Enhancements

  • mutable ReferenceFS (#958)

Fixes

  • Make archive FSs not cachable (#966)

  • glob fixes (#961)

  • generic copy with unknown size (#959)

  • zstd open (#956)

  • empty archive file (#954)

  • tar chaining (#950, 947)

  • missing exceptions in mapper (#940)

Other

  • update registry (#852)

  • allow None cache (#942)

  • mappers to remember init arguments (#939)

  • cache docstrings (#933)

2022.03.0

Enhancements

  • tqdm example callback with simple methods (#931, 902)

  • Allow empty root in get_mapper (#930)

  • implement real info for reference FS (#919)

  • list known implementations and compressions (#913)

Fixes

  • git branch for testing git backend (#929)

  • maintain mem FS’s root (#926)

  • kargs to FS in parquet module (#921)

  • fix on_error in references (#917)

  • tar ls consistency (#9114)

  • pyarrow: don’t decompress twice (#911)

  • fix FUSE tests (#905)

2022.02.0

Enhancements

  • reference FS performance (#892, 900)

  • directory/prefix FS (#745)

Fixes

  • FUSE (#905, 891)

  • iteration in threads (#893)

  • OpenFiles slicing (#887)

Other

  • drop py36 (#889, 901)

2022.01.0

Fixes

  • blocks cache metadata (#746)

  • default SMB port (#853)

  • caching fixes (#856, 855)

  • explicit close for http files (#866)

  • put_file to continue when no bytes (#869, 870)

Other

  • temporary files location (#851, 871)

  • async abstract methods (#858, 859, 860)

  • md5 for FIPS (#872)

  • remove deprecated pyarrow/distutils (#880, 881)

2021.11.1

Enhancements

  • allow compression for fs.open (#826)

  • batch more async operations (#824)

  • allow github backend for alternate URL (#815)

  • speec up reference filesystem (#811)

Fixes

  • fixes for parquet functionality (#821, 817)

  • typos and docs (#839, 833, 816)

  • local root (#829)

Other

  • remove BlockSizeError for http (#830)

2021.11.0

Enhancement

  • parquet-specific module and cache type (#813, #806)

Fixes

  • empty ranges (#802, 801, 803)

  • doc typos (#791, 808)

  • entrypoints processing (#784)

  • cat in ZIP (#789)

Other

  • move to fsspec org

  • doc deps (#786, 791)

2021.10.1

Fixes

  • Removed inaccurate ZipFileSystem.cat() override so that the base class’ version is used (#789)

  • fix entrypoint processing (#784)

  • case where no blocks of a block-cache have yet been loaded (#801)

  • don’t fetch empty ranges (#802, 803)

Other

  • simplify doc deps (#786, 791)

2021.10.0

Fixes

  • only close http connector if present (#779)

  • hdfs strip protocol (#778)

  • fix filecache with check_files (#772)

  • put_file to use _parent (#771)

Other

  • add kedro link (#781)

2021.09.0

Enhancement

  • http put from file-like (#764)

  • implement webhdfs cp/rm_file (#762)

  • multiple (and concurrent) cat_ranges (#744)

Fixes

  • sphinx warnings (#769)

  • lexists for links (#757)

  • update versioneer (#750)

  • hdfs updates (#749)

  • propagate async timeout error (#746)

  • fix local file seekable (#743)

  • fix http isdir when does not exist (#741)

Other

  • ocifs, arrow added (#754, #765)

  • promote url_to_fs to top level (#753)

2021.08.1

Enhancements

  • HTTP get_file/put_file APIs now support callbacks (#731)

  • New HTTP put_file method for transferring data to the remote server (chunked) (#731)

  • Customizable HTTP client initializers (through passing get_client argument) (#731, #701)

  • Support for various checksum / fingerprint headers in HTTP info() (#731)

  • local implementation of rm_file (#736)

  • local speed improvements (#711)

  • sharing options in SMB (#706)

  • streaming cat/get for ftp (#700)

Fixes

  • check for remote directory when putting (#737)

  • storage_option update handling (#734(

  • await HTTP call before checking status (#726)

  • ftp connect (#722)

  • bytes conversion of times in mapper (#721)

  • variable overwrite in WholeFileCache cat (#719)

  • http file size again (#718)

  • rm and create directories in ftp (#716, #703)

  • list of files in async put (#713)

  • bytes to dict in cat (#710)

2021.07.0

Enhancements

  • callbacks (#697)

2021.06.1

Enhancements

  • Introduce fsspec.asyn.fsspec_loop to temporarily switch to the fsspec loop. (#671)

  • support list for local rm (#678)

Fixes

  • error when local mkdir twice (#679)

  • fix local info regression for pathlike (#667)

Other

  • link to wandbfs (#664)

2021.06.0

Enhancements

  • Better testing and folder handling for Memory (#654)

  • Negative indexes for cat_file (#653)

  • optimize local file listing (#647)

Fixes

  • FileNoteFound in http and range exception subclass (#649, 646)

  • async timeouts (#643, 645)

  • stringify path for pyarrow legacy (#630)

Other

  • The fsspec.asyn.get_loop() will always return a loop of a selector policy (#658)

  • add helper to construct Range headers for cat_file (#655)

2021.05.0

Enhancements

  • Enable listings cache for HTTP filesystem (#560)

  • Fold ZipFileSystem and LibArchiveFileSystem into a generic implementation and add new TarFileSystem (#561)

  • Use throttling for the get/put methods of AsyncFileSystem (#629)

  • rewrite for archive filesystems (#624)

  • HTTP listings caching (#623)

Fixes

  • gcsfs tests (#638)

  • stringify_path for arrow (#630)

Other

  • s3a:// alias

2021.04.0

Major changes

  • calendar versioning

Enhancements

  • better link and size finding for HTTP (#610, %99)

  • link following in Local (#608)

  • ReferenceFileSystem dev (#606, #604, #602)

Fixes

  • drop metadata dep (#605)

0.9.0

Major Changes:

  • avoid nested sync calls by copying code (#581, #586, docs #593)

  • release again for py36 (#564, #575)

Enhancements:

  • logging in mmap cacher, explicitly close files (#559)

  • make LocalFileOpener an IOBase (#589)

  • better reference file system (#568, #582, #584, #585)

  • first-chunk cache (#580)

  • sftp listdir (#571)

  • http logging and fetch all (#551, #558)

  • doc: entry points (#548)

Fixes:

  • get_mapper for caching filesystems (#559)

  • fix cross-device file move (#547)

  • store paths without trailing “/” for DBFS (#557)

  • errors that happen on _initiate_upload when closing the AbstractBufferedFile will now be propagated (#587)

  • infer_compressions with upper case suffix ($595)

  • file initialiser errors (#587)

  • CI fix (#563)

  • local file commit cross-device (#547)

Version 0.8.7

Fixes:

  • fix error with pyarrow metadata for some pythons (#546)

Version 0.8.6

Features:

  • Add dbfs:// support (#504, #514)

Enhancements

  • don’t import pyarrow (#503)

  • update entry points syntax (#515)

  • ci precommit hooks (#534)

Fixes:

  • random appending of a directory within the filesystems find() method (#507, 537)

  • fix git tests (#501)

  • fix recursive memfs operations (#502)

  • fix recorsive/maxdepth for cp (#508)

  • fix listings cache timeout (#513)

  • big endian bytes tests (#519)

  • docs syntax (#535, 524, 520, 542)

  • transactions and reads (#533)

Version 0.8.5

Features:

  • config system

  • libarchive implementation

  • add reference file system implementation

Version 0.8.4

Features:

  • function can_be_local to see whether URL is compatible with open_local

  • concurrent cat with filecaches, if backend supports it

  • jupyter FS

Fixes:

  • dircache expiry after transaction

  • blockcache garbage collection

  • close for HDFS

  • windows tests

  • glob depth with “**”

Version 0.8.3

Features:

  • error options for cat

  • memory fs created time in detailed ls`

Fixes:

  • duplicate directories could appear in MemoryFileSystem

  • Added support for hat dollar lbrace rbrace regex character escapes in glob

  • Fix blockcache (was doing unnecessary work)

  • handle multibyte dtypes in readinto

  • Fix missing kwargs in call to _copy in asyn

Other:

  • Stop inheriting from pyarrow.filesystem for pyarrow>=2.0

  • Raise low-level program friendly OSError.

  • Guard against instance reuse in new processes

  • Make hash_name a method on CachingFileSystem to make it easier to change.

  • Use get_event_loop for py3.6 compatibility

Version 0.8.2

Fixes:

  • More careful strip for caching

Version 0.8.1

Features:

  • add sign to base class

  • Allow calling of coroutines from normal code when running async

  • Implement writing for cached many files

  • Allow concurrent caching of remote files

  • Add gdrive:// protocol

Fixes:

  • Fix memfs with exact ls

  • HTTPFileSystem requires requests and aiohttp in registry

Other:

  • Allow http kwargs to clientSession

  • Use extras_require in setup.py for optional dependencies

  • Replacing md5 with sha256 for hash (CVE req)

  • Test against Python 3.8, drop 3.5 testing

  • add az alias for abfs

Version 0.8.0

Major release allowing async implementations with concurrent batch operations.

Features:

  • async filesystem spec, first applied to HTTP

  • OpenFiles cContext for multiple files

  • Document async, and ensure docstrings

  • Make LocalFileOpener iterable

  • handle smb:// protocol using smbprotocol package

  • allow Path object in open

  • simplecache write mode

Fixes:

  • test_local: fix username not in home path

  • Tighten cacheFS if dir deleted

  • Fix race condition of lzma import when using threads

  • properly rewind MemoryFile

  • OpenFile newline in reduce

Other:

  • Add aiobotocore to deps for s3fs check

  • Set default clobber=True on impl register

  • Use _get_kwargs_from_url when unchaining

  • Add cache_type and cache_options to HTTPFileSystem constructor

Version 0.7.5

  • async implemented for HTTP as prototype (read-only)

  • write for simplecache

  • added SMB (Samba, protocol >=2) implementation

Version 0.7.4

  • panel-based GUI

0.7.3 series

  • added git and github interfaces

  • added chained syntax for open, open_files and get_mapper

  • adapt webHDFS for HttpFS

  • added open_local

  • added simplecache, and compression to both file caches

Version 0.6.2

  • Added adl and abfs protocols to the known implementations registry (GH#209)

  • Fixed issue with whole-file caching and implementations providing multiple protocols (GH#219)

Version 0.6.1

  • LocalFileSystem is now considered a filestore by pyarrow (GH#211)

  • Fixed bug in HDFS filesystem with cache_options (GH#202)

  • Fixed instance caching bug with multiple instances (GH#203)

Version 0.6.0

  • Fixed issues with filesystem instance caching. This was causing authorization errors in downstream libraries like gcsfs and s3fs in multi-threaded code (GH#155, GH#181)

  • Changed the default file caching strategy to fsspec.caching.ReadAheadCache (GH#193)

  • Moved file caches to the new fsspec.caching module. They’re still available from their old location in fsspec.core, but we recommend using the new location for new code (GH#195)

  • Added a new file caching strategy, fsspec.caching.BlockCache for fetching and caching file reads in blocks (GH#191).

  • Fixed equality checks for file system instance to return False when compared to objects other than file systems (GH#192)

  • Fixed a bug in fsspec.FSMap.keys returning a generator, which was consumed upon iteration (GH#189).

  • Removed the magic addition of aliases in AbstractFileSystem.__init__. Now alias methods are always present (GH#177)

  • Deprecated passing trim to fsspec.spec.AbstractBufferedFile. Pass it in storage_options instead (GH#188)

  • Improved handling of requests for fsspec.implementations.http.HTTPFileSystem when the HTTP server responds with an (incorrect) content-length of 0 (GH#163)

  • Added a detail=True parameter to fsspec.spec.AbstractFileSystem.ls() (GH#168)

  • Fixed handling of UNC/DFS paths (GH#154)