lmdb

The significant updates in v2.4 include:

• Testing and fixes for compatibility with and optimization for bun (coming soon with some bug fixes in bun, but major parts for optimized use of bun are done).
• Databases are opened with a write transaction (partly to address #164 and possibly #153). In v2.3, databases were opened with a read txn when possible, but it turns out this has a number of problems.
• Switching to scoped names for the platform binary packages (like @lmdb/lmdb-linux-x64)

V2.3 represents a major code refactor/upgrade switching to using NAPI instead of NAN as the interface to Node/V8. This has two primary benefits:

• A single binary (per platform) can be used across all (supported) versions of NodeJS, past and future.
• Deno is adopting NAPI (https://github.com/denoland/deno/pull/13633), and this will make it much easier and more efficient to support Deno (than FFI).

In addition, lmdb-js is now using a set of platform-specific sub-packages to install the appropriate binary for the platform (and only the appropriate binary). Together, this should dramatically decrease the downloaded/installed size of lmdb-js (from probably 15 MB to about 1 MB).

NAPI is a slightly slower than NAN (because it is an extra layer of abstraction). However, lmdb-js also ships binaries with extra version-specific NAN/V8 functions for performance sensitive functions. The distribution includes binary functions for Node V16 and also includes fast-api-calls, which should actually be faster than previous versions for V16 users. lmdb-js can be compiled to different Node versions if an extra boost in performance is needed.

Also note that with the transition to using NAPI as the interface for Deno support, FFI support is being dropped. That means that lmdb-js V2.3 will (probably) require Deno 1.22+, lmdb-js will need to be installed via package manager. For earlier versions of Deno, you would need to use lmdb-js V2.2.

Some other new features/improvements in V2.3:

• Multiple versions of lmdb-js (V2.3+) can now be simultaneously installed and should properly work on the same database #136.
• Added support for caching in combination with (abortable) child transactions #137.

V2.2.0 has enabled the overlappingSync functionality by default and improved the scheduling of sync operations to achieve better performance.

This release also introduces support for temporary databases. By simply omitting the path when opening a database, lmdb-js will auto-generate a temporary database (in the system temp directory), and delete it on close (it will also be opened with sync disabled for better performance). This can be useful for testing or creating intermediary tables.

The release also includes support for coordinating with the weak referencing cache to clear "kept" objects for better garbage collection in long-running synchronous jobs.

And v2.2.0 includes better performance for retrieving large binary entries by employing a reduced memory copying technique (affects entries over > 64KB).

The most significant new capability in this release is support for Deno. Deno functionality still has a number of limitations (no async transactions), but core functionality is now available. This release also includes new functionality for supporting asynchronous prefetching for gets, in situations where page faults/disk-I/O are likely. Also includes bug fixes for closing databases. Of particular note is that the close() method is now asynchronous (returns a promise).

Producer-Consumer writes

The most central change in v2.0 is a rewrite of how data is batched and written to LMDB, implementing a true multi-thread producer-consumer protocol such that all writes are directly and immediately serialized to a binary format that the writer thread can consume in real-time. This has multiple implications (and hopefully benefits):

• The writer thread can begin writing pending puts/deletes to LMDB (and async-txns can be queued) immediately after they have been called (in parallel to the main thread), rather than having to wait for a future event to submit the writes and start the transaction. And as more writes occur, they can continue to be queued and consumed by the writer thread as long as it is still processing a transaction. This should provide better write latency and flexibility in batching.
• This implementation improves memory management, allowing pending writes to written and be cleared/collected from memory without having to wait for transactions to complete and events to be queued, which should be more robust for heavy writing situations, preventing overloading of the queue of operations.
• All keys are immediately serialized using JavaScript. The (async) put/delete operations don't need to depend on a C++ for writing keys, so any errors in serializing keys are immediately triggered and since key (de)-serialization happens in JS, lmdb-js can now support custom (user-provided) key encodings (so for example, if you want a custom UUID encoder that can efficiently encode UUIDs in 16 bytes).
• Some additional put flags can now be supported (like "append" puts) (Expose MDB_APPEND #43) in performant way.
• The V8 "fast api" calls can be used for synchronous puts/deletes (used within async-txns).

Overlapping Sync

In lmdb-js, LMDB now supports "overlapping" syncs. This allows transactions to safely commit prior to be flushed to disk, and then the data can be flushed (and database metadata updated) after the transaction is visible and unlocked, allowing most of the disk I/O to happen in parallel with future write transactions, which can potentially provide some significant performance benefits in some heavy-write situations.

Other Performance Improvements

• A new binary key comparison function is used that does (32-bit) word by word comparison, instead of byte by byte comparison, for faster b-tree traversals.
• Compression is now performed in separate thread/tasks from the writes for more parallelism and speed.

Name Change

V.2+ is now published to https://www.npmjs.com/package/lmdb as its canonical NPM package. The 1.x line will continue to be available in the lmdb-store package. The repository has also been renamed to lmdb-js, to reflect the cross-platform goals of this project with Deno support coming soon (and significant internal upgrades in v2 to support this).

Other Upgrades

• The codebase has been upgraded to ESM (and refactored to be more modular), so lmdb-js can be used with native ESM loading (of course, built CJS modules are included too). This is also one of the necessary steps in hopefully supporting Deno (#83) at some point (once their foreign function interface is complete enough to be useable).
• Added functions for explicit batching.
• Improved compatibility with LevelUp.

Legacy Compatibility/Changes

There aren't any intentionally substantive breaking changes to the (documented) API itself, just cleanup and removal of previously deprecated APIs. But the underlying changes are large enough that testing should be done with an update.