NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
crates.io
crates.io, #4147 most downloaded on crates.io
A fully typed LMDB (mdb.master) wrapper with minimum overhead
Latest stable
0.23.1
Last release 27 days ago
26 Oct 2026
Release timing varies
gaps range from 8 days to 7 months
Some releases are documented
notes for 15 of 36 stable releases
1 version withdrawn
withdrawn after publishing
7 years old
54 releases · first in 2019
Drag across the ruler to compare, or tap a release to open it.
Make sure we cannot use a transaction that is not the one intended to open/create a database by @Kerollmops in #388
<p align="center"><img width="280px" src="https://raw.githubusercontent.com/meilisearch/heed/main/assets/heed-pigeon-logo.png"></a></p> <h1 align="center" >heed & heed3</h1>
Full Changelog: https://github.com/meilisearch/heed/compare/v0.23.0...v0.23.1
This release is breaking regarding the way we open databases. It is no longer possible to produce undefined behaviors when opening or creating databas
<p align="center"><img width="280px" src="https://raw.githubusercontent.com/meilisearch/heed/main/assets/heed-pigeon-logo.png"></a></p> <h1 align="center" >heed & heed3</h1>
This release is breaking regarding the way we open databases. It is no longer possible to produce undefined behaviors when opening or creating databases. The LMDB documentation specifies that a transaction that opened or created a database must commit or abort before any other transaction can open or create a database. This wasn't enforced in previous versions of heed nor heed3.
To ensure a safe usage of LMDB we broke the database opening/creating API.
let mut wtxn = env.write_txn()?;
let database = env.create_database(&mut wtxn, Some("my-db"))?;
// At this point you can use the database everywhere in the program
// even though you haven't committed it, yet.
wtxn.commit()?;
The former API let you use uncommitted databases outside the transaction scope resulting in EINVAL errors and was allowing multiple transactions to open/create databases concurrently, resulting in undefined behavior. The new API disallow both behaviors but introduces a more complex API.
let wtxn = env.write_txn()?;
let database = env
.create_databases_and_commit(wtxn, |dbo, wtxn| {
let db = dbo.create_database(wtxn, Some("my-db"))?;
// At this point you can use the database but it is tied to the wtxn
// transaction and therefore, can't be used outside this closure.
// The only way to use it outside of the current transaction
// is to commit it by returning it from this very closure.
Ok(CommitOrAbort::<_, ()>::Commit(db))
})?
.unwrap_commit();
// At this point you can use the database everywhere in the program.
Your coding agent can read these notes before it upgrades. Set up the MCP server →
This release brings a new functionality: the possibility to create multiple nested read transactions from an uncommitted write transaction. You can al
<p align="center"><img width="280px" src="https://raw.githubusercontent.com/meilisearch/heed/main/assets/heed-pigeon-logo.png"></a></p> <h1 align="center" >heed & heed3</h1>
This release brings a new functionality: the possibility to create multiple nested read transactions from an uncommitted write transaction. You can also examine another usage in the dedicated example file.
We finally merged the changes to upstream LMDB (mdb.master3), and the patch has been thoroughly reviewed by Howard Chu.
// opening a write transaction
let mut wtxn = env.write_txn()?;
// [use the RwTxn to write into the env]
// opening multiple read-only transactions without committing beforehand
let rtxns = (0..1000).map(|_| env.nested_read_txn(&wtxn)).collect::<heed::Result<Vec<_>>>()?;
// you can also spawn nested read transactions from the write txn too
let rtxns = (0..1000).map(|_| wtxn.nested_read_txn()).collect::<heed::Result<Vec<_>>>()?;
// [use the RoTxns and move them onto different threads]
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
This release brings a new experimental functionality: the possibility to create multiple nested read transactions from an uncommitted write transactio
<p align="center"><img width="280px" src="https://raw.githubusercontent.com/meilisearch/heed/main/assets/heed-pigeon-logo.png"></a></p> <h1 align="center" >heed & heed3</h1>
This release brings a new experimental functionality: the possibility to create multiple nested read transactions from an uncommitted write transaction. You can also examine another usage in the dedicated example file or in this Meilisearch pull request.
We are attempting to merge the necessary changes to upstream LMDB, but it will require additional work to ensure it produces correct C code. You can review the dedicated OpenLDAP tracking issue to see how it progresses.
// opening a write transaction
let mut wtxn = env.write_txn()?;
// [use the RwTxn to write into the env]
// opening multiple read-only transactions without committing beforehand
let rtxns = (0..1000).map(|_| env.nested_read_txn(&wtxn)).collect::<heed::Result<Vec<_>>>()?;
// [use the RoTxns and move them onto different threads]
This release brings better-polished support for the heed/heed3 duality. It also improves the documentation, adds support for custom duplicate sort com
<p align="center"><img width="280px" src="https://raw.githubusercontent.com/meilisearch/heed/main/assets/heed-pigeon-logo.png"></a></p> <h1 align="center" >heed & heed3</h1>
This release brings better-polished support for the heed/heed3 duality. It also improves the documentation, adds support for custom duplicate sort comparators, supports the PREV_SNAPSHOT flag, better support for Thread Local Storage (TLS)-backed of transactions, and bumps LMDB to the latest patch.
MDB_NOTLS documentation by @Kerollmops in https://github.com/meilisearch/heed/pull/302Env::copy_to_file method by @Kerollmops in https://github.com/meilisearch/heed/pull/312PREV_SNAPSHOT flag by @irevoire in https://github.com/meilisearch/heed/pull/296FIXED_MAP flag name by @Kerollmops in https://github.com/meilisearch/heed/pull/313USE_VALGRIND compilation flag as use-valgrind feature by @dureuill in https://github.com/meilisearch/heed/pull/288AnyTls by @dureuill in https://github.com/meilisearch/heed/pull/315Env parameter accessors by @Kerollmops in https://github.com/meilisearch/heed/pull/321NDEBUG on release builds by @Kerollmops in https://github.com/meilisearch/heed/pull/324-sys crates and heed/heed3 versions by @Kerollmops in https://github.com/meilisearch/heed/pull/325Env::copy_to_path method by @Kerollmops in https://github.com/meilisearch/heed/pull/326Env<WithTls> the default by @Kerollmops in https://github.com/meilisearch/heed/pull/327Full Changelog: https://github.com/meilisearch/heed/compare/v0.21.0...v0.22.0
This release introduces the first wrapper around the mdb.master3 LMDB branch, which features encryption at rest. You can use the new heed3 crate to re
This release introduces the first wrapper around the mdb.master3 LMDB branch, which features encryption at rest. You can use the new heed3 crate to read and write data encrypted on disk and decrypted on demand.
RoTxn covariant and keep it !Sync by @Kerollmops in #285HeedDecode bound on Lazy::remap by @irevoire in #287Full Changelog: v0.20.5...v0.21.0
<p align="center"><img width="280px" src="https://raw.githubusercontent.com/meilisearch/heed/main/assets/heed-pigeon-logo.png"></a></p> <h1 align="center" >heed & heed3</h1>
This release introduces the first wrapper around the mdb.master3 LMDB branch, which features encryption at rest. You can use the new heed3 crate to read and write data encrypted on disk and decrypted on demand.
RoTxn covariant and keep it !Sync by @Kerollmops in https://github.com/meilisearch/heed/pull/285HeedDecode bound on Lazy::remap by @irevoire in https://github.com/meilisearch/heed/pull/287Full Changelog: https://github.com/meilisearch/heed/compare/v0.20.5...v0.21.0
fix function docs (clippy warnings) by @antonilol in https://github.com/meilisearch/heed/pull/273
<p align="center"><img width="280px" src="https://raw.githubusercontent.com/meilisearch/heed/main/assets/heed-pigeon-logo.png"></a></p> <h1 align="center" >heed</h1>
Use features that actually exists by @Kerollmops in https://github.com/meilisearch/heed/pull/270
<p align="center"><img width="280px" src="https://raw.githubusercontent.com/meilisearch/heed/main/assets/heed-pigeon-logo.png"></a></p> <h1 align="center" >heed</h1>
Update dependencies by @irevoire in https://github.com/meilisearch/heed/pull/265
<p align="center"><img width="280px" src="https://raw.githubusercontent.com/meilisearch/heed/main/assets/heed-pigeon-logo.png"></a></p> <h1 align="center" >heed</h1>
Introduce the longer-keys feature which sets -DMDB_MAXKEYSIZE=0 by @tpunder in https://github.com/meilisearch/heed/pull/263
<p align="center"><img width="280px" src="https://raw.githubusercontent.com/meilisearch/heed/main/assets/heed-pigeon-logo.png"></a></p> <h1 align="center" >heed</h1>
longer-keys feature which sets -DMDB_MAXKEYSIZE=0 by @tpunder in https://github.com/meilisearch/heed/pull/263Close the p tag by @Kerollmops in https://github.com/meilisearch/heed/pull/250
<p align="center"><img width="280px" src="https://raw.githubusercontent.com/meilisearch/heed/main/assets/heed-pigeon-logo.png"></a></p> <h1 align="center" >heed</h1>
p tag by @Kerollmops in https://github.com/meilisearch/heed/pull/250FnMut with FnOnce in put_reserve_* by @nolanderc in https://github.com/meilisearch/heed/pull/251Database::get_or_put: insert if value does not exist, otherwise return previous value by @nolanderc in https://github.com/meilisearch/heed/pull/252ReservedSpace more flexible by @nolanderc in https://github.com/meilisearch/heed/pull/256ReservedSpace (closes #258) by @nolanderc in https://github.com/meilisearch/heed/pull/259ReservedSpace by @nolanderc in https://github.com/meilisearch/heed/pull/260Heed is a fully typed LMDB wrapper with minimum overhead. It is also the most maintained Rust wrapper on top of LMDB and is used by meilisearch/meilis
<p align="center"><img width="280px" src="https://raw.githubusercontent.com/meilisearch/heed/main/assets/heed-pigeon-logo.png"></a></p> <h1 align="center" >heed</h1>
Heed is a fully typed LMDB wrapper with minimum overhead. It is also the most maintained Rust wrapper on top of LMDB and is used by meilisearch/meilisearch. LMDB is a memory-mapped key-value store that has been battle-tested for a long time.
This release is about more than 80 closed PRs and issues ✨. All of them to improve the safety of the library and the usage simplicity. We now even have a great cookbook to help library users do tricky or advanced operations with Heed and LMDB.
sync-read-txn FeatureWe removed the unsound sync-read-txn feature that was making the RoTxn: Sync when it mustn't as it is not safe. We replaced this feature with the read-txn-no-tls, which makes the RoTxn: Send, usable from different threads using a Mutex.
We apologize for this and are discussing with the RustSec advisory team how best to advise people not to use this unsound feature.
Thanks to @hinto-janai, opening an environment is now unsafe. There has been a lot of discussion about memory mapping and its safety, so I decided to follow the general mood around that. The EnvOpenOptions::open() is unsafe now, and we have a safety paragraph explaining why.
We exposed nearly every LMDB features: DUPSORT, INTERGER_KEY/DUP, REVERSE_KEY/DUP... You'll also be able to iterate over duplicate items or skip them. That's your choice.
You'll also be able to use the new Database::put_with_flags and <Iterator>::put_current_with_flags methods that support the NO_DUP_DATA, NO_OVERWRITE, APPEND, and APPEND_DUP flags. Allowing you to append data faster on keys or duplicate data.
We improved the iterators' put_current and put_current_reserved methods to accept flags and the codec to serialize the data. It is easier to do custom encoding operations on databases.
Thanks to @xiaoyawei, you can use the LMDB key comparison custom functions and not only rely on the default lexicographic comparison. You can read more about this key-value feature in the LMDB source code.
use std::cmp::Ordering;
use heed_traits::Comparator;
enum StringAsIntCmp {}
impl Comparator for StringAsIntCmp {
fn compare(a: &[u8], b: &[u8]) -> Ordering {
let a: i32 = str::from_utf8(a).unwrap().parse().unwrap();
let b: i32 = str::from_utf8(b).unwrap().parse().unwrap();
a.cmp(&b)
}
}
let mut wtxn = env.write_txn()?;
let db = env.database_options().types::<Str, Unit>().key_comparator::<StringAsIntCmp>().create(&mut wtxn)?;
db.put(&mut wtxn, "-1000", &())?;
db.put(&mut wtxn, "-100", &())?;
db.put(&mut wtxn, "100", &())?;
let mut iter = db.iter(&wtxn)?;
assert_eq!(iter.next().transpose()?, Some(("-1000", ())));
assert_eq!(iter.next().transpose()?, Some(("-100", ())));
assert_eq!(iter.next().transpose()?, Some(("100", ())));
assert_eq!(iter.next().transpose()?, None);
We now have our own update-to-date lmdb-master-sys crate. It represents the bindgen-generated bindings to the LMDB library, and heed is directly plugged into it.
It will be easier for Meilisearch to bump the engine's LMDB version now. We previously used a fork of the outdated lmdb-rkv-sys crate of Mozilla, but it was cumbersome to bump three repositories, i.e., our fork, meilisearch/lmdb-rs and finally heed.
Now we can make all the changes in the heed repository to bump the LMDB version :tada:
Thanks to @GregoryConrad, we now have a posix-sem feature. This change allows iOS and macOS build to comply with Apple's App Sandbox (necessary for distribution in the App Store) and possible speed improvements brought upon by the POSIX semaphores.
You can now declare a heed Database with a number as the key or the value straightforwardly. Just specify the endianness of it, and that's it.
use heed::byteorder::BE;
use heed::types::*;
type BEI64 = I64<BE>;
let mut wtxn = env.write_txn()?;
let db: Database<BEI64, Unit> = env.create_database(&mut wtxn, Some("big-endian-iter"))?;
let ret = db.put(&mut wtxn, &0, &())?;
let ret = db.put(&mut wtxn, &68, &())?;
let ret = db.put(&mut wtxn, &35, &())?;
let ret = db.put(&mut wtxn, &42, &())?;
wtxn.commit()?;
@irevoire added some new Env methods to get the size of a database:
Env::map_size returns the size of the original memory map.Env::real_disk_size returns the size on the disk as seen by the file system.Env::non_free_pages_size returns the size of the non-free pages of the current transaction.Env::resize unsafe method to resize the environment.Database::stat method to get detailed information about the internal BTree pages of a database.You'll also be able to get the number of entries in a database in a snap. We no longer .iter().count() internally and directly ask LMDB about this count.
Sometimes, it is possible to directly write into your database without first serializing your data into an intermediary buffer. For example, it can be true for many data structures like RoaringBitmaps.
use roaring::RoaringBitmap;
type BEI64 = I64<BE>;
let mut wtxn = env.write_txn()?;
let db = env.create_database::<BEI32, ByteSlice>(&mut wtxn, Some("number-string"))?;
let bitmap = RoaringBitmap::from_iter([1, 2, 3, 4]);
// Instead of serializing the data into a buffer, as you know the length of it,
//You can directly write the data in the LMDB value reserved space.
db.put_reserved(&mut wtxn, &42, bitmap.serialize_size(), |reserved| {
bitmap.serialize_into(reserved)
})?;
The support for custom encoding/decoding errors has been added. Weren't you frustrated when heed triggered an error in one of the encoding/decoding traits, and you could not understand why? It is no longer an issue as the BytesEncode/BytesDecode traits can return a BoxedError to keep the information and let you decide what to do with it.
We introduced the BadOpenOptions heed error when a database is already opened in the same program, but you tried to open it with different options. This behavior will also be improved in v0.20.0 to simplify the usage of the lib and make it more correct towards LMDB behaviors around the map size.
Debug for most StructsA lot more types implement the Debug trait. It will be easier to embed an Env, a Database, or even an iterator in a struct that already implements Debug.
Thanks to @AureliaDolo and @darnuria, we have a much better documentation covering and added examples to nearly everything that could look complex. On the other hand, @wackbyte improved the general documentation and quality of the sentences.
The principle of least astonishment applies to user interface and software design. It proposes that a system component should behave how most users expect it to behave. The behavior should not astonish or surprise users.
Since the early days of heed, it has automatically linked to the already available libmdb library installed on the system. We saw a lot of strange issues, non-reproducible on our side, and later discovered that the system-LMDB of Arch Linux was used by heed instead of the vendored one!
It is no longer an issue as we removed this behavior for the build.rs. The vendored version is always used. We no longer use an unknown version of LMDB.
Thanks to @darnuria again, read-only transactions sometimes need to commit to making databases globally usable in the program. We now have tests to open and commit databases in read-only environments. However, this change is subtle. We must commit to making a just-opened database global and not just local.
let rtxn = env.read_txn()?;
let db = env.open_poly_database(&rtxn, Some("my-database"))?;
rtxn.commit()?;
// We can store and use `db` here if the database is alive.
This detail raised an issue in heed. It is currently not safe to use a Database. We must redefine how we open and create databases to make them safe. The new API should be released for v0.20.0.
In this release, the RwTxn::abort method no longer returns a heed::Result as LMDB can't fail. It was introduced when we were supporting MDBX.
We simplified the signature of the RoTxn and RwTxn types by removing one lifetime and only keeping a single one. The new signature only has a single 'p lifetime, the environment lifetime, or the parent transaction. The simplification was possible as the parent transaction must already live longer than the environment.
// Previous Signature
struct RwTxn<'env, 'parent, T = ()>;
// New Signature
struct RwTxn<'p>;
We also removed the types of transactions. Those types were first introduced to avoid using a transaction opened with one environment with another one. Unfortunately, as the T type was optional, it wasn't used much. We decided that a runtime check would be better and added a bunch of assert_eq! to be sure that transactions and environments weren't mixed.
The previous version of Heed used nested transactions when opening or creating databases. The operation did it this way to simplify internal methods. Unfortunately, LMDB has some limitations: using nested transactions with the MDB_WRITEMAP option is impossible.
It is now possible to use LMDB with MDB_WRITEMAP and open databases freely :blush:
MDB_IDL_LOGN parameterYou can change the MDB_IDL_LOGN parameter to change the internal behavior of LMDB. This parameter is mainly used to control memory usage and the size of the dirty pages pool, which affects the transaction size. You can read more on this on the tracking issue, why Mozilla used it, and how.
Nothing published for this version
Nothing published for this version
Nothing published for this version
Heed is a fully typed LMDB wrapper with minimum overhead. It is also the most maintained Rust wrapper on top of LMDB and is used by meilisearch/meilis
Heed is a fully typed LMDB wrapper with minimum overhead. It is also the most maintained Rust wrapper on top of LMDB and is used by meilisearch/meilisearch. LMDB is a memory-mapped key-value store battle-tested for a long time.
sync-read-txn featureWe removed the unsound sync-read-txn feature that was making the RoTxn: Sync when it mustn't as it is not safe. We replaced this feature with the read-txn-no-tls, which makes the RoTxn: Send, usable from different threads using a Mutex.
We apologize for this and are discussing with the RustSec advisory team how best to advise people not to use this unsound feature.
We exposed nearly every LMDB features: DUPSORT, INTERGER_KEY/DUP, REVERSE_KEY/DUP... You'll also be able to iterate over duplicate items or skip them. That's your choice.
You'll also be able to use the new Database::put_with_flags and <Iterator>::put_current_with_flags methods that support the NO_DUP_DATA, NO_OVERWRITE, APPEND, and APPEND_DUP flags. Allowing you to append data faster on keys or duplicate data.
Thanks to @xiaoyawei, you can use the LMDB key comparison custom functions and not only rely on the default lexicographic comparison. You can read more about this key-value feature in the LMDB source code.
use std::cmp::Ordering;
use heed_traits::Comparator;
enum StringAsIntCmp {}
impl Comparator for StringAsIntCmp {
fn compare(a: &[u8], b: &[u8]) -> Ordering {
let a: i32 = str::from_utf8(a).unwrap().parse().unwrap();
let b: i32 = str::from_utf8(b).unwrap().parse().unwrap();
a.cmp(&b)
}
}
let mut wtxn = env.write_txn()?;
let db = env.database_options().types::<Str, Unit>().key_comparator::<StringAsIntCmp>().create(&mut wtxn)?;
db.put(&mut wtxn, "-1000", &())?;
db.put(&mut wtxn, "-100", &())?;
db.put(&mut wtxn, "100", &())?;
let mut iter = db.iter(&wtxn)?;
assert_eq!(iter.next().transpose()?, Some(("-1000", ())));
assert_eq!(iter.next().transpose()?, Some(("-100", ())));
assert_eq!(iter.next().transpose()?, Some(("100", ())));
assert_eq!(iter.next().transpose()?, None);
We now have our own update-to-date lmdb-master-sys crate. It represents the bindgen-generated bindings to the LMDB library, and heed is directly plugged into it.
It will be easier for Meilisearch to bump the engine's LMDB version now. We previously used a fork of the outdated lmdb-rkv-sys crate of Mozilla, but it was cumbersome to bump three repositories, i.e., our fork, meilisearch/lmdb-rs and finally heed.
Now we can make all the changes in the heed repository to bump the LMDB version :tada:
Thanks to @GregoryConrad, we now have a posix-sem feature. This change allows iOS and macOS build to comply with Apple's App Sandbox (necessary for distribution in the App Store) and possible speed improvements brought upon by the POSIX semaphores.
You can now declare a heed Database with a number as the key or the value in a straightforward way. Just specify the endianness of it, and that's it.
use heed::byteorder::BE;
use heed::types::*;
type BEI64 = I64<BE>;
let mut wtxn = env.write_txn()?;
let db: Database<BEI64, Unit> = env.create_database(&mut wtxn, Some("big-endian-iter"))?;
let ret = db.put(&mut wtxn, &0, &())?;
let ret = db.put(&mut wtxn, &68, &())?;
let ret = db.put(&mut wtxn, &35, &())?;
let ret = db.put(&mut wtxn, &42, &())?;
wtxn.commit()?;
@irevoire added some new Env methods to get the size of a database:
Env::map_size returns the size of the original memory map.Env::real_disk_size returns the size on the disk as seen by the file system.Env::non_free_pages_size returns the size of the non-free pages of the current transaction.Env::resize unsafe method to resize the environment.You'll also be able to get the number of entries in a database in a snap. We no longer .iter().count() internally and directly ask LMDB about this count.
Sometimes, it is possible to directly write into your database without first serializing your data into an intermediary buffer. For example, it can be true for many data structures like RoaringBitmaps.
use roaring::RoaringBitmap;
type BEI64 = I64<BE>;
let mut wtxn = env.write_txn()?;
let db = env.create_database::<BEI32, ByteSlice>(&mut wtxn, Some("number-string"))?;
let bitmap = RoaringBitmap::from_iter([1, 2, 3, 4]);
// Instead of serializing the data into a buffer, as you know the length of it,
// you can directly write the data into the LMDB value reserved space.
db.put_reserved(&mut wtxn, &42, bitmap.serialize_size(), |reserved| {
bitmap.serialize_into(reserved)
})?;
zerocopy with the more popular bytemuckThe new version of heed now uses bytemuck to replace zerocopy. The bytemuck library seems much easier to contribute to; it seems much more popular than the former (710k downloads by month compared to 109k). It brings a better API, at least for heed, as it can return information on which kind of problem happens when a cast fails. It would simplify some codecs.
Support custom encoding/decoding errors has been added. Weren't you frustrated when heed triggered an error in one of the encoding/decoding traits, and you could not understand why? It is no longer an issue as the BytesEncode/BytesDecode trait can return a BoxedError that can be displayed.
We introduced the BadOpenOptions heed error when a database is already opened in the same program, but you tried to open it with different options. This behavior will also be improved in v0.20.0 to simplify the usage of the lib and make it more correct towards LMDB behaviors around the map size.
Debug for most structsA lot more types implement the Debug trait. It will be easier to embed an Env, a Database, or even an iterator in a struct that already implements Debug.
Thanks to @AureliaDolo and @darnuria, we have a much better documentation covering and added examples to nearly everything that could look complex.
The principle of least astonishment applies to user interface and software design. It proposes that a system component should behave how most users expect it to behave. The behavior should not astonish or surprise users.
Since the early days of heed, it would automatically link to the already available libmdb library installed on the system. We saw a lot of strange issues, non-reproducible on our side, and later discovered that the system-LMDB of Arch Linux was used by heed instead of the vendored one!
It is no longer an issue as we removed this behavior for the build.rs. The vendored version is always used. We no longer use an unknown version of LMDB.
Thanks to @darnuria again, read-only transactions sometimes need to commit to making databases globally usable in the program. We now have tests to ensure we can open and commit databases in read-only environments. However, this change is subtle. We must commit to making a just-opened database global and not just local.
let rtxn = env.read_txn()?;
let db = env.open_poly_database(&rtxn, Some("my-database"))?;
rtxn.commit()?;
// We can store and use `db` here as long as the database is alive.
This detail raised an issue in heed. It is currently not safe to use a Database. We must redefine how we open and create databases to make them safe. The new API should be released for v0.20.0.
In this release, the RwTxn::abort method no longer returns a heed::Result as LMDB can't fail. It was introduced when we were supporting MDBX.
We simplified the signature of the RoTxn and RwTxn types by removing one lifetime and only keeping a single one. The new signature only has a single 'p lifetime, the environment lifetime, or the parent transaction. The simplification was possible as the parent transaction must already live longer than the environment.
// Previous signature
struct RwTxn<'env, 'parent, T = ()>;
// New signature
struct RwTxn<'p>;
We also removed the types of transactions. Those types were first introduced to avoid using a transaction opened with one environment with another one. Unfortunately, as the T type was optional, it wasn't used much. We decided that a runtime check would be better and added a bunch of assert_eq! to be sure that transactions and environments weren't mixed.
The previous version of heed used nested transactions when opening or creating databases. The operation did it this way to simplify internal methods. Unfortunately, LMDB has some limitations: using nested transactions with the MDB_WRITEMAP option is impossible.
It is now possible to use LMDB with MDB_WRITEMAP and open databases freely :blush:
Nothing published for this version
Heed is a fully typed LMDB wrapper with minimum overhead. It is also the most maintained Rust wrapper on top of LMDB and is used by meilisearch/meilis
Heed is a fully typed LMDB wrapper with minimum overhead. It is also the most maintained Rust wrapper on top of LMDB and is used by meilisearch/meilisearch. LMDB is a memory-mapped key-value store battle-tested for a long time.
sync-read-txn featureWe removed the unsound sync-read-txn feature that was making the RoTxn: Sync when it mustn't as it is not safe. We replaced this feature with the read-txn-no-tls, which makes the RoTxn: Send, usable from different threads using a Mutex.
We apologize for this and are discussing with the RustSec advisory team how best to advise people not to use this unsound feature.
We exposed nearly every LMDB features: DUPSORT, INTERGER_KEY/DUP, REVERSE_KEY/DUP... You'll also be able to iterate over duplicate items or skip them. That's your choice.
You'll also be able to use the new Database::put_with_flags and <Iterator>::put_current_with_flags methods that support the NO_DUP_DATA, NO_OVERWRITE, APPEND, and APPEND_DUP flags. Allowing you to append data faster on keys or duplicate data.
We now have our own update-to-date lmdb-master-sys crate. It represents the bindgen-generated bindings to the LMDB library, and heed is directly plugged into it.
It will be easier for Meilisearch to bump the engine's LMDB version now. We previously used a fork of the outdated lmdb-rkv-sys crate of Mozilla, but it was cumbersome to bump three repositories, i.e., our fork, meilisearch/lmdb-rs and finally heed.
Now we can make all the changes in the heed repository to bump the LMDB version :tada:
Thanks to @GregoryConrad, we now have a posix-sem feature. This change allows iOS and macOS build to comply with Apple's App Sandbox (necessary for distribution in the App Store) and possible speed improvements brought upon by the POSIX semaphores.
You can now declare a heed Database with a number as the key or the value in a straightforward way. Just specify the endianness of it, and that's it.
use heed::byteorder::BE;
use heed::types::*;
type BEI64 = I64<BE>;
let mut wtxn = env.write_txn()?;
let db: Database<BEI64, Unit> = env.create_database(&mut wtxn, Some("big-endian-iter"))?;
let ret = db.put(&mut wtxn, &0, &())?;
let ret = db.put(&mut wtxn, &68, &())?;
let ret = db.put(&mut wtxn, &35, &())?;
let ret = db.put(&mut wtxn, &42, &())?;
wtxn.commit()?;
@irevoire added some new Env methods to get the size of a database:
Env::map_size returns the size of the original memory map.Env::real_disk_size returns the size on the disk as seen by the file system.Env::non_free_pages_size returns the size of the non-free pages of the current transaction.Env::resize unsafe method to resize the environment.You'll also be able to get the number of entries in a database in a snap. We no longer .iter().count() internally and directly ask LMDB about this count.
Sometimes, it is possible to directly write into your database without first serializing your data into an intermediary buffer. For example, it can be true for many data structures like RoaringBitmaps.
use roaring::RoaringBitmap;
type BEI64 = I64<BE>;
let mut wtxn = env.write_txn()?;
let db = env.create_database::<BEI32, ByteSlice>(&mut wtxn, Some("number-string"))?;
let bitmap = RoaringBitmap::from_iter([1, 2, 3, 4]);
// Instead of serializing the data into a buffer, as you know the length of it,
// you can directly write the data into the LMDB value reserved space.
db.put_reserved(&mut wtxn, &42, bitmap.serialize_size(), |reserved| {
bitmap.serialize_into(reserved)
})?;
zerocopy with the more popular bytemuckThe new version of heed now uses bytemuck to replace zerocopy. The bytemuck library seems much easier to contribute to; it seems much more popular than the former (710k downloads by month compared to 109k). It brings a better API, at least for heed, as it can return information on which kind of problem happens when a cast fails. It would simplify some codecs.
Support custom encoding/decoding errors has been added. Weren't you frustrated when heed triggered an error in one of the encoding/decoding traits, and you could not understand why? It is no longer an issue as the BytesEncode/BytesDecode trait can return a BoxedError that can be displayed.
We introduced the BadOpenOptions heed error when a database is already opened in the same program, but you tried to open it with different options. This behavior will also be improved in v0.20.0 to simplify the usage of the lib and make it more correct towards LMDB behaviors around the map size.
Debug for most structsA lot more types implement the Debug trait. It will be easier to embed an Env, a Database, or even an iterator in a struct that already implements Debug.
Thanks to @AureliaDolo and @darnuria, we have a much better documentation covering and added examples to nearly everything that could look complex.
The principle of least astonishment applies to user interface and software design. It proposes that a system component should behave how most users expect it to behave. The behavior should not astonish or surprise users.
Since the early days of heed, it would automatically link to the already available libmdb library installed on the system. We saw a lot of strange issues, non-reproducible on our side, and later discovered that the system-LMDB of Arch Linux was used by heed instead of the vendored one!
It is no longer an issue as we removed this behavior for the build.rs. The vendored version is always used. We no longer use an unknown version of LMDB.
Thanks to @darnuria again, read-only transactions sometimes need to commit to making databases globally usable in the program. We now have tests to ensure we can open and commit databases in read-only environments. However, this change is subtle. We must commit to making a just-opened database global and not just local.
let rtxn = env.read_txn()?;
let db = env.open_poly_database(&rtxn, Some("my-database"))?;
rtxn.commit()?;
// We can store and use `db` here as long as the database is alive.
This detail raised an issue in heed. It is currently not safe to use a Database. We must redefine how we open and create databases to make them safe. The new API should be released for v0.20.0.
In this release, the RwTxn::abort method no longer returns a heed::Result as LMDB can't fail. It was introduced when we were supporting MDBX.
We simplified the signature of the RoTxn and RwTxn types by removing one lifetime and only keeping a single one. The new signature only has a single 'p lifetime, the environment lifetime, or the parent transaction. The simplification was possible as the parent transaction must already live longer than the environment.
// Previous signature
struct RwTxn<'env, 'parent, T = ()>;
// New signature
struct RwTxn<'p>;
We also removed the types of transactions. Those types were first introduced to avoid using a transaction opened with one environment with another one. Unfortunately, as the T type was optional, it wasn't used much. We decided that a runtime check would be better and added a bunch of assert_eq! to be sure that transactions and environments weren't mixed.
The previous version of heed used nested transactions when opening or creating databases. The operation did it this way to simplify internal methods. Unfortunately, LMDB has some limitations: using nested transactions with the MDB_WRITEMAP option is impossible.
It is now possible to use LMDB with MDB_WRITEMAP and open databases freely :blush:
Nothing published for this version
Nothing published for this version
Nothing published for this version
Heed is a fully typed LMDB wrapper with minimum overhead. It is also the most maintained Rust wrapper on top of LMDB and is used by meilisearch/meilis
Heed is a fully typed LMDB wrapper with minimum overhead. It is also the most maintained Rust wrapper on top of LMDB and is used by meilisearch/meilisearch. LMDB is a memory-mapped key-value store battle-tested for a long time.
We now have our own update-to-date lmdb-master-sys crate. It represents the bindgen-generated bindings to the LMDB library, and heed is directly plugged into it.
It will be easier for Meilisearch to bump the engine's LMDB version now. We previously used a fork of the outdated lmdb-rkv-sys crate of Mozilla, but it was cumbersome to bump three repositories, i.e., our fork, meilisearch/lmdb-rs and finally heed.
Now we can make all the changes in the heed repository to bump the LMDB version :tada:
Thanks to @GregoryConrad, we now have a posix-sem feature. This change allows iOS and macOS build to comply with Apple's App Sandbox (necessary for distribution in the App Store), in addition to possible speed improvements brought upon by the POSIX semaphores.
You will now be able to declare a heed Database with a number as the key or the value in a straightforward way. Just specify the endianness of it, and that's it.
use heed::byteorder::BE;
use heed::types::*;
type BEI64 = I64<BE>;
let mut wtxn = env.write_txn()?;
let db: Database<BEI64, Unit> = env.create_database(&mut wtxn, Some("big-endian-iter"))?;
let ret = db.put(&mut wtxn, &0, &())?;
let ret = db.put(&mut wtxn, &68, &())?;
let ret = db.put(&mut wtxn, &35, &())?;
let ret = db.put(&mut wtxn, &42, &())?;
wtxn.commit()?;
@irevoire added some new Env methods to get the size of a database:
Env::map_size returns the size of the original memory map.Env::real_disk_size returns the size on the disk as seen by the file system.Env::non_free_pages_size returns the size of the non-free pages of the current transaction.You'll also be able to get the number of entries in a database in a snap. We no longer .iter().count() internally and directly ask LMDB about this count.
Sometimes it is possible to directly write into your database without first serializing your data into an intermediary buffer. It can be the case for many data-structure like RoaringBitmaps, for example.
use roaring::RoaringBitmap;
type BEI64 = I64<BE>;
let mut wtxn = env.write_txn()?;
let db = env.create_database::<BEI32, ByteSlice>(&mut wtxn, Some("number-string"))?;
let bitmap = RoaringBitmap::from_iter([1, 2, 3, 4]);
// Instead of serializing the data into a buffer, as you know the length of it,
// you can directly write the data into the LMDB value reserved space.
db.put_reserved(&mut wtxn, &42, bitmap.serialize_size(), |reserved| {
bitmap.serialize_into(reserved)
})?;
zerocopy with the more popular bytemuckThe new version of heed now uses bytemuck to replace zerocopy. The bytemuck library seems much easier to contribute to; it seems much more popular than the former (710k downloads by month compared to 109k). It brings a better API, at least for heed, as it can return information on which kind of problem happens when a cast fails, it would simplify some codecs.
Support custom encoding/decoding errors has been added. Weren't you frustrated when heed triggered an error in one of the encoding/decoding traits, and you were unable to understand the reason? It is no more an issue as the BytesEncode/BytesDecode trait can return a BoxedError that can be displayed.
We introduced the BadOpenOptions heed error for when a database is already opened in the same program, but you tried to open it with different options. This behavior will also be improved in v0.20.0 to simplify the usage of the lib and make it more correct towards LMDB behaviors around the map size.
Debug for most structsA lot more types implement the Debug trait. It will be easier to embed an Env or a Database in a struct that already implements Debug.
The principle of least astonishment applies to user interface and software design. It proposes that a system component should behave how most users expect it to behave. The behavior should not astonish or surprise users.
Since the early days of heed would automatically link to the already available libmdb library installed on the system. We saw a lot of strange issues, non-reproducible on our side, and later discovered that the system-LMDB of Arch Linux was used by heed instead of the vendored one!
It is no longer an issue as we removed this behavior for the build.rs. The vendored version is always used. We no more use an unknown version of LMDB.
The read-only transaction is immutable and therefore doesn't require the commit nor abort methods as it cannot make changes. We removed those two methods from the RoTxn type.
However, this change is subtle. Making opened (not created) databases global is no longer possible without committing the transaction and, therefore can only be possible with a write transaction. We must commit to making a just-opened database global and not just local.
let rtxn = env.read_txn()?;
let db = env.open_poly_database(&rtxn, Some("my-database"))?;
rtxn.commit()?;
// We can store and use `db` here as long as the database is alive.
This detail raised an issue in heed. It is currently not safe to use a Database. We must redefine how we open and create databases to make them safe. The new API should be released for v0.20.0.
Note that in this release, the RwTxn::abort method no more returns a heed::Result as LMDB can't fail.
We simplified the signature of the RoTxn and RwTxn types by removing one lifetime and only keeping a single one. The new signature only has a single 'p lifetime, either the environment lifetime or the parent transaction. The simplification was possible as the parent transaction must already live longer than the environment.
// Previous signature
struct RwTxn<'env, 'parent, T = ()>;
// New signature
struct RwTxn<'p>;
We also removed the types of transactions. Those types were first introduced to avoid using a transaction opened with one environment with another one. Unfortunately, as the T type was optional, it wasn't used much. We decided that a runtime check would be better and added a bunch of assert_eq! to be sure that transactions and environments weren't mixed.
The previous version of heed used nested transactions when opening or creating databases. The operation did it this way to simplify internal methods. Unfortunately, LMDB has some limitations: it is impossible to use nested transactions with the MDB_WRITEMAP option.
It is now possible to use LMDB with MDB_WRITEMAP and open databases freely :blush:
This release is breaking. Here is what has been updated compared to the v0.8 releases.
This release is breaking. Here is what has been updated compared to the v0.8 releases.
PolyDatabaseThe PolyDatabase was a non-typed heed database that was forcing library users to always specify the en/decoder to store/read entries from the database. It was sometimes confusing and made the codebase harder to maintain due to the Database that was mirroring the PolyDatabase method but with already known en/decoders.
The PolyDatabase can now be replaced by the UntypedDatabe combined with the newly introduced remap_key/data_type or remap_types methods.
BytesDecode and BytesEncode traitsThe BytesDecode and BytesEncode traits have been reworked to allow users to return error messages now. It was a big source of frustration for users only to know that an en/decoding was failing without knowing anything about the error happening.
zerocopy by bytemuckWe used the zerocopy crate since the first version of heed, the crate was named zerocopy-lmdb at first!
This is now the past as we switch from it, we found out that proposing changes to the zerocopy crate was more complex than proposing a change to the bytemuck crate hosted on Github. I am really impressed by the work done by both crate maintainers and grateful to people working on the zerocopy crate, nothing political. You can read more about that on the related issue.
Database::len methodThanks to @Keats this is now O(1) to get the number of entries in a database. The previous heed version was O(N) as it was iterating through the whole tree to count the length.
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Thanks to @maroider and @gentoid for your help!
We now compile on Windows 🎉
Thanks to @maroider and @gentoid for your help!
We support typed transactions, it is useful when you need to work with multiple environments and you don't want to shuffle the transactions between th
We support typed transactions, it is useful when you need to work with multiple environments and you don't want to shuffle the transactions between them.
We now support infinite nested write transactions.
We now support infinite nested write transactions.
It is a feature that LMDB provides, it allows to create a write transaction inside of another one, when a nested transaction is committed, nothing is saved consistently until the main write transaction (the parent) is committed.
Nested transactions can be aborted without aborting the main one, it can be useful to create nested jobs that can fail.
You can see an example usage of nested transactions in the examples folder.
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
One column per quarter.