remove_dir_all

Windows dep and code tidyup (#72)

New features

• RemoveDir trait is now public. (#59)
• Added remove-dir-all CLI binary (opt-in via the cli Cargo feature). (#61)

Bug fixes

• Windows: use fs_at's POSIX-deletion support to correctly handle in-use files on Windows, fixing #34. (#64)

Other changes

• Improved crate documentation. (#58)

Other changes

• Fix use of fcntl, missing undocumented extra argument.

Security changes

• Fix TOCTOU race conditions both inside the implementation of functions and the contract: functions now only operate on directories. Callers wanting to process the contents of a symlink (e.g. for remove_dir_contents) should resolve the symlink themselves. This is an API break from 0.7.0, but the previous behaviour was insecure.

  This is due to the same code pattern as caused CVE-2022-21658 in Rust itself: it was possible to trick a privileged process doing a recursive delete in an attacker controlled directory into deleting privileged files, on all operating systems.

  For instance, consider deleting a tree called 'etc' in a parent directory called 'p'. Between calling remove_dir_all("a") and remove_dir_all("a") actually starting its work, the attacker can move 'p' to 'p-prime', and replace 'p' with a symlink to '/'. Then the privileged process deletes 'p/etc' which is actually /etc, and now your system is broken. There are some mitigations for this exact scenario, such as CWD relative file lookup, but they are not guaranteed - any code using absolute paths will not have that protection in place.

  The same attack could be performed at any point in the directory tree being deleted: if 'a' contains a child directory called 'etc', attacking the deletion by replacing 'a' with a link is possible.

  The new code in this release mitigates the attack within the directory tree being deleted by using file-handle relative operations: to open 'a/etc', the path 'etc' relative to 'a' is opened, where 'a' is represented by a file descriptor (Unix) or handle (Windows). With the exception of the entry points into the directory deletion logic, this is robust against manipulation of the directory hierarchy, and remove_dir_all will only delete files and directories contained in the tree it is deleting.

  The entry path however is a challenge - as described above, there are some potential mitigations, but since using them must be done by the calling code, it is hard to be confident about the security properties of the path based interface.

  The new extension trait RemoveDir provides an interface where it is much harder to get it wrong.

  somedir.remove_dir_contents("name-of-child").

  Callers can then make their own security evaluation about how to securely get a directory handle. That is still not particularly obvious, and we're going to follow up with a helper of some sort (probably in the fs_at crate). Once that is available, the path based entry points will get deprecated.

  In the interim, processes that might run with elevated privileges should figure out how to securely identify the directory they are going to delete, to avoid the initial race. Pragmatically, other processes should be fine with the path based entry points : this is the same interface std::fs::remove_dir_all offers, and an unprivileged process running in an attacker controlled directory can't do anything that the attacker can't already do.

  tl;dr: state shared with threat actors makes things dangerous; library functions cannot assume anything about the particular threat model of a program and must err on the side of caution.

Other changes

• Made feature to control use of rayon off-by-default for easier integration by other crates.

(cargo-release) remove_dir_all version 0.7.0

• update author
• update README.md

Introduction

Welcome to the 0.6 release of remove_dir_all! remove_dir_all is a fast and reliable std::fs::remove_dir_all implementation for Windows, for other platforms it simply re-exports std. This release brings a lot internal updates and improvements with a complete internal refactor, switching to the 2018 edition and moving to GitHub Actions for handling CI. You can update it simply by updating your Cargo.toml.

remove_dir_all = "0.6"

Removing Directories In Parallel

The biggest change in this release is a refactor of the current the serial implementation with a Rayon based parallel implementation that is substantially faster due to the Windows IO model.

On Windows it is not enough to just recursively remove the contents of a directory and then the directory itself. Deleting does not happen instantaneously, but is delayed by IO being completed in the fs stack. Further, typical Windows machines can handle many more concurrent IOs than a single threaded application is capable of submitting: the overlapped (async) calls available do not cover the operations needed to perform directory removal.

To work around this, we use a work stealing scheduler and submit deletions concurrently with directory scanning, and delete sibling directories in parallel. This allows the slight latency of STATUS_DELETE_PENDING to only have logarithmic effect: a very deep tree will pay wall clock time for that overhead per level as the tree traverse completes, but not for every interior not as a simple recursive deletion would result in.

Earlier versions of this crate moved the contents of the directory being deleted to become siblings of base_dir, which required write access to the parent directory under all circumstances; this is no longer required - though it may be re-instated if in-use files turn out to be handled very poorly with this new threaded implementation.

There is a single small race condition where external side effects may be left: when deleting a hard linked readonly file, the syscalls required are: open, set rw, unlink (SetFileDispositionDelete), and finally set ro. A crash or power failure could lead to the loss of the readonly bit on the hardlinked inode.

Thank you so much to @rbtcollins for working on this effort, and providing the updated implementation!

• lints and doc fixes

• Added support for aarch64-pc-windows-msvc.

• Fixed deletion of readonly items.

• Upgraded to winapi 0.3.