vsdb

Fixed

• DagMap prune writes a durable clearing marker before it tears down the mainline. A retry returns the genesis and does not re-fold a half-cleared head or destroy children already re-parented onto it. Existing handles stay readable; the marker is an extra key old readers ignore.
• get_mut and iter_mut no longer rewrite storage when a value encoding is unstable and the caller did not change the value, so a non-mutating guard does not panic in read-only mode. Interior edits of round-tripping values still persist.
• A retried destroy unlinks the node from its parent after a crash that already nulled the parent slot.
• Engine benchmarks isolate their data under /tmp instead of writing the default dataset.

Changed

• DagMapRawKey::shadow documents the same structural-exclusion rule as DagMapRaw::shadow.
• load_instance_meta no longer claims to enforce the typed-handle magic gate. from_meta still rejects a legacy prefix payload.

Fixed

• Document the single-active-instance requirement for restored SlotDex/VecDex handles; independent aliases still do not share runtime caches.

• Measure warm MPT/SMT root lookups without cloning and dropping the whole tree inside the timed operation.

• Give read benchmarks independent fixtures and replenish removal fixtures outside timing, so filtered and long-running workloads keep measuring hits.

• Keep cosine distance finite and scale invariant for finite extreme-magnitude f32/f64 vectors while retaining the ordinary fast path.

• Allow read-only access to fully initialized datasets with a stale initialization sentinel, without changing any files.

• Surface streaming storage read errors in both scan directions instead of returning truncated results as successful exhaustion.

• Fence VerMap shard WALs before publishing references or reclaiming nodes; reject incomplete reachable history before recovery deletes data.

• DagMap publishes a whole mainline merge atomically, so an interrupted prune and its retries cannot expose intermediate ancestor values through genesis metadata or already re-parented children. Staging uses memory proportional to the distinct keys changed by the consumed mainline; APIs and stored formats remain compatible.

Added

• Process-wide read-only mode. Call vsdb_configure(VsdbOptions::read_only(path)) before any other VSDB API, then restore existing handles with from_meta or serde. The default and all non-default namespaces open with the same capability. Point/range/history reads, namespace inspection, vector search, Merkle proof/root computation, and in-memory recovery of committed residual WAL records are supported without changing the database tree.
• OpenMode, VsdbOptions, vsdb_configure, vsdb_open_mode, Namespace::is_read_only, and the structured VsdbError::ReadOnly capability error are available from both crates' primary exports.

Changed

• mmdb upgraded to v4.3.0. VSDB uses its native read-only open path, which takes shared shard locks on Unix, recovers WAL state in memory, starts no compaction workers, and leaves MANIFEST/WAL/SST files untouched.
• Fallible write APIs reject read-only use before side effects. Historical infallible collection mutations remain fail-fast and panic; maintenance-only flush, deferred-delete, VerMap::gc, and automatic trie-cache writes are no-ops, while explicit MPT/SMT cache saves return the capability error. Creating a new collection now fails immediately instead of producing a lazily unusable handle, and clone_in returns VsdbError::ReadOnly even for an empty source.
• VerMap restore no longer attempts durable branch/ref-count repair in read-only mode; authoritative branch-table lookup and in-memory B+ tree runtime reconstruction keep queries available without writes.
• The minimum supported Rust version is now 1.89, matching mmdb 4.3.0.

Compatibility

• The API change is additive and the VSDB on-disk format is unchanged. Existing supported datasets remain writable as before. Read-only opens do not persist format/allocator/lifecycle migrations or VSDB repair metadata; datasets that require such maintenance should be opened writable on a backup or staging copy before taking the immutable reader snapshot.
• Added a complete read-only deployment guide covering handle preparation, operation behavior, namespace scope, Unix locking, filesystem snapshots, permissions, WAL recovery, and upgrade constraints.

Changed

• The default namespace now sizes from a fixed 2 GiB memory budget; non-default namespaces keep their fixed 512 MB default. The library no longer sizes itself from the host's available RAM or its cgroup limits — a moment-in-time host reading baked deployment noise into engine sizing and made footprints unpredictable across machines. Every budget is now a binding limit, so write-buffer sizing always scales with it (the budget_limited distinction is gone). Applications that can afford more memory should raise VSDB_MEM_BUDGET_MB (applied verbatim, as before): a larger budget enlarges the block cache and write buffers, which directly improves performance.
• The historical dynamic detection (host MemAvailable min-folded with a ¾-derated cgroup limit) moved into the benchmark support code (benches/units/legacy_budget.rs in both crates): benches export the computed value through VSDB_MEM_BUDGET_MB at startup, so results stay comparable with releases that auto-sized from the host. An operator-provided VSDB_MEM_BUDGET_MB still wins there too.

Full-repository audit fixes across engine lifecycle, persistent collections, DagMap, SlotDex, VecDex, and trie caches. Valid existing formats remain compatible; malformed or incomplete state now fails loudly instead of being silently adopted.

Fixed

• Restored B+ tree / VerMap aliases share runtime allocator and node reference state, preventing sequential safe restores from reusing a live NodeId or deleting another alias's committed snapshot. Standalone restored deletes also stop flushing discarded temporary nodes.
• Completed namespace roots are validated strictly: exact shard set, supported format marker, and every MMDB CURRENT anchor. A new additive pending/established sidecar prevents an established missing root from being recreated as an empty database; valid legacy registry records migrate on first open.
• from_meta binds payload identity to the requested canonical InstanceId, including the documented Some(DEFAULT_NS_ID) spelling.
• DagMap migration and teardown hardening: the authentic pre-ceiling id_num wire format is max-folded into dag_id_ceiling; destructive clears are flushed before registries unlink them; mixed-namespace serde payloads are rejected.
• VecDex preserves HNSW reachability and atomic replacement: pruning keeps a reciprocal edge, entry-point repair selects/reconnects a true-max-layer node, and remove+insert replacements commit in one transaction/chunk.
• SlotDex bulk insertion follows exact serial tier-growth cadence, including multiple unique keys in the slot that crosses a growth boundary.
• Trie cache loading revalidates the complete trust boundary: canonical shape, cached hashes, handle state, trailing bytes, checked varints/ranges, and a 512 MiB pre-allocation limit.
• Parallel staged-row tests no longer mutate the process environment.

Added

• VerMapWithProof<_, _, SmtCalc>::prove_key / verify_key_proof encode typed keys exactly as the committed VerMap.
• DagMapRaw and DagMapRawKey expose child_ids / child_id for selective pruning.

Documentation and tooling

• Added concrete legacy export/reimport, backup, validation, and rollback guidance.
• Corrected namespace examples, review invariants, shared-cache proposal status, crate license links, and benchmark workload semantics.
• CI now enforces cargo fmt --all -- --check.

Post-v16.3.1 review fixes (the reviewed commit was never published): the new SlotDex tier-capacity gate is now O(1) and bulk-load-aware, plus style/doc alignment. mmdb upgraded to v4.2. No on-disk format changes.

Changed

• mmdb upgraded to v4.2.

Fixed

• SlotDex bulk loads grow tiers mid-batch: v16.3.1's "no tiers yet" capacity gate counted only committed level-0 rows, which do not advance inside a single insert_batch — one bulk load of N ≫ tier_capacity distinct slots into a fresh index built zero tier levels, silently degrading every subsequent paged/count query to the O(N) level-0 walk (persisting across reopen, never healing on a read-mostly index). The gate now also counts rows staged earlier in the same batch and, on promotion, builds the new level from the merged committed ⊕ staged level-0 stream — restoring exact serial-insert cadence, as the batch-equivalence test now verifies with a single whole-workload batch.
• SlotDex growth gate is O(1): the same v16.3.1 gate re-scanned (and materialized) every committed level-0 row on each insert while the index was tier-less. The gate now reads slot_rows, an in-memory mirror of the committed level-0 row count, maintained on each mutation's 0↔1 slot-row transition and re-derived from committed rows whenever the index (re)enters the tier-less state — at open (hydrate) and after remove's tier truncation, both provably bounded scans.

Style

• Hoisted repeated inline paths to imports (Commit in versioned/handle.rs, VSDB in engine/mmdb.rs); added the missing // SAFETY: comment on DagMapRaw::shadow()'s inner unsafe block; updated dagmap.md INV-DG6 to the namespace-scoped flush barriers.

Per-engine block-cache pool + cache telemetry (shared-mem-pool RFC tier (i), steps 0 + 4; mechanism shipped in mmdb v4.1.0).

Added

• Per-engine block-cache pool: every engine (default namespace and every non-default namespace alike) now shares ONE BlockCachePool across its shards instead of statically splitting the cache slice per shard. Since routing is prefix % shards, a collection lives entirely inside one shard — under the old split a single hot map could only ever use 1/shards of the engine's cache. The pool's capacity is exactly the sum of the former per-shard capacities, so engine memory totals are unchanged; this is a pure reallocation. Measured (Q1 gate benchmark, 64 MB budget, SST-backed random reads): skewed load (one hot map) −72% (1 thread) / −81% (8 threads) read latency; uniform all-shard load: parity within noise (after the pool's LRU store was 64-way segmented in mmdb — see mmdb v4.1.0). No isolation trade: the shards of one engine are one tenant (same dataset, same budget).
• Per-shard property passthrough (Namespace::shard_properties): one engine-property reading per shard in shard order (mmdb DB::get_property names) — the observability tier the RFC's trigger conditions require. Per-shard cache hit/miss counters stay per-shard under the pool (counted at each shard's read site); "block-cache-usage" reports the engine-wide pool total plus the shard's own pins.
• cache_pool criterion bench (core/benches/cache_pool.rs): the Q1 gate — skew and uniform read scenarios at 1/8 threads with a documented A/B protocol against the private-split baseline.

Post-release audit fixes (registry: docs/audit.md).

Fixed

• vsdb_ns_relocate dataset check hardened: the target must now hold mmdb's CURRENT manifest anchor in every expected shard dir, not merely the shard dirs themselves. CURRENT is exactly mmdb's recover-vs-create test at open (absent ⇒ a shard is silently recreated fresh), so the guard now asks the semantically right question — "would every shard take the recover path?" — refusing bare skeletons (a "prepared" volume, or a copy interrupted before any shard content landed) in addition to empty dirs. Which dataset lives there still cannot be verified (roots carry no namespace id); moving the right data remains the operator's documented contract.
• Relocate validates the registered shard count: the 1..=64 registry-damage guard (previously only in the open path, now shared via validated_shards) runs before the dataset probe — a corrupt shards == 0 fails loudly instead of vacuously passing the per-shard checks and degrading the guard to marker-only.

In-process namespace close() — the ownership-inverted engine lifecycle (RFC: docs/proposals/ns-close.md).

Added

• vsdb_ns_close(id): closes an open namespace, releasing all of its resources — engine memory, compaction threads, fds, and mmdb LOCK files — without a process restart. Active memtables are flushed and WALs synced first (errors surface, unlike a plain drop). Refuse-don't-poison: it succeeds only when every handle (collections, iterators, Namespace clones) is gone, otherwise it returns an error naming the live-handle count; a live handle is never invalidated. The registry entry survives: re-open via Namespace::open (restart-equivalent recovery) or reclaim via vsdb_ns_destroy — create → fill → close → destroy is the in-process epoch-rotation loop.

Changed

• Engines are owned, not leaked: both Box::leak sites are gone. NsInner owns its Engine, MmDB owns its shard DBs (Box<[DB]>), and every engine reference is a plain borrow bounded by a live handle — the soundness invariant moved from review discipline into the type system. The default engine is owned by the default Namespace (a static, so it still lives for the whole process); the public VSDB singleton delegates to it.

Post-release audit fixes (all findings from the v16.0.0 deep review; registry: docs/audit.md).

Fixed

• Namespace lifecycle TOCTOU: the not-open check of vsdb_ns_destroy/vsdb_ns_relocate now runs under REGISTRY_LOCK (the same lock open holds while caching a live engine), so a racing open can no longer have its root deleted or repointed underneath it.
• Base-dir freeze gap: Namespace::open (and the admin fns) freeze the base dir before reading the registry — a later vsdb_set_base_dir fails loudly instead of moving the global allocator's backing store to another universe under live namespaces.
• Explicit-root validation hardened: ./.. components rejected; overlap checks run on physically normalized paths (symlinked spellings of the base or another root are caught); adopting an existing non-empty dir is refused (foreign prefix provenance; importing/attaching stays an explicit non-goal — RFC §9). Empty dirs (fresh mount points) remain accepted.
• Crash-resumable namespace creation: shard-layout validation is completion-aware (keyed on the format marker, which is written only after every shard exists): a create that crashed mid-way is completed on the next open instead of bricking the root; genuine mismatches stay rejected.
• DagMap cross-namespace split: parented construction always inherits the parent's namespace (ambient scope overridden; mismatched explicit new_in is debug-asserted) — one DAG never spans namespaces.
• DagMap destroy ownership order (INV-DG5): the ownership check runs before duplicate suppression, so a stale foreign registry entry can no longer make destroy skip a genuinely owned child.
• InstanceId canonical form: "42@0" and wire-level Some(DEFAULT_NS_ID) fold to ns: None at parse/deserialize time — Eq/Hash are reliable for API-obtained tokens; new canonical constructor InstanceId::new(map_id, ns).
• Shard-layout lifecycle hardening (follow-up review of the initial completion-aware rule): brand-new roots raise a durable __SYSTEM__/__initializing__ sentinel before the first shard dir and retire it after the format marker — resumability now requires proof. A marker-present root missing shard dirs (even all of them) is damage and refused, never silently reinitialized; a partial set with neither sentinel nor marker (e.g. a corrupted legacy base) is refused instead of "resumed" into silent data loss; the scan checks the exact expected shard set (misnamed/non-dir shard_* entries are rejected, not counted).
• Create rollback also clears the root: a failed create leaves an explicit path immediately retryable (pre-existing empty dirs are emptied, never deleted) and no unregistered residue under derived roots.
• Registry robustness: an out-of-range shard count in a (damaged) registry entry is refused cleanly instead of reaching prefix % 0; vsdb_ns_list freezes the base dir like every other registry reader.
• flush_all_open no longer holds the namespace table lock across engine flushes; Namespace::meta_path is public and is the single source of truth for instance-meta naming (strata reuses it).

Documentation

• RFC save_as_meta → the implemented save_meta; namespace subsystem added to CLAUDE.md, review-core mapping, and the engine pattern guide (per-namespace engines, global allocator, marker semantics, REGISTRY_LOCK rule).

Design: docs/proposals/namespaces.md (rev 10).

Added (P1 — namespaces)

• Namespaces: anonymous placement groups. A Namespace is an independently-rooted engine instance (own dir tree — placeable on its own volume, own shards/WALs/compactions/memtable budget). Users never name one, never pass a path on the normal tier; the everyday primitive is co-location: existing.namespace() + new_in/ns.scope(..).
  • Namespace::{create, create_with, open, default_ns, current, scope, id, path, flush, system_dir, meta_dir}; NamespaceOpts { path, shards, mem_budget_mb } (everything defaulted; explicit roots validated against nesting).
  • Admin tier: vsdb_ns_list / vsdb_ns_destroy / vsdb_ns_relocate (not-open-only; destroy = registry removal + rm -rf — O(1) bulk reclaim; relocate updates the registry pointer only).
  • Registry: {base}/__SYSTEM__/__namespaces__ (postcard, durable atomic writes); derived roots under {base}/__NAMESPACES__/{ns_id:016x} recorded base-relative so the whole universe stays movable as one tree. Ids are never reused; DEFAULT_NS_ID = 0 is a fixed constant — never registered, never looked up.
  • Every collection (MapxRaw → typed wrappers → Orphan, PersistentBTree, VerMap, SlotDex, VecDex, DagMapRaw/DagMapRawKey) gains new_in(&ns, ..) + namespace(); plain new() places into Namespace::current() (ambient scope, creation-time only — never routing). A composite and all its internal maps live in exactly one namespace. The one exception is VerMapWithProof, which is placed via from_map(VerMap::new_in(&ns)) and exposes its namespace via .map().namespace().
  • One global prefix allocator serves all namespaces: prefixes (= map_ids) stay unique across the whole registry by construction.
  • Per-namespace __SYSTEM__ tree: instance metas and MPT/SMT cache files live beside their data (destroy reclaims them together); the TrieCalc trait now takes the cache dir explicitly.
• InstanceId { map_id, ns: Option<NsId> } — the complete public identity, mirroring the persisted meta bytes (ns: None ⇔ default namespace ⇔ the 16-byte pre-v16 meta form). Display/FromStr as "42" / "42@7". instance_id()/save_meta() now return it; from_meta(impl Into<InstanceId>) still accepts bare u64 ids (⇒ default namespace) — stored pre-v16 tokens keep working, and resolution is deterministic (never a search).
• Meta wire format: optional ns_id suffix. Default-namespace handles serialize byte-identically to v15 ("VSMAPX01" ‖ prefix, 16 B); non-default handles append ns_id_le (24 B). One magic, no format versions; v15 data decodes as None verbatim.

Changed (P0 — allocator persistence relocation)

• The prefix-allocator ceiling now lives outside the shard DBs, in {base_dir}/__SYSTEM__/__prefix_ceiling__ (8-byte LE u64, written durably: tmp + fsync + rename + parent-dir fsync). This decouples prefix allocation from the default engine — the prerequisite for namespaces sharing one global allocator.
  • Upgrade is automatic and idempotent: at every open the ceiling is take-max-folded from the file, the legacy shard-0 key, and the allocation floor, then persisted; the legacy key is never written again. A pre-tripwire v15 binary that advanced the legacy key after migration is re-absorbed by the max-fold instead of causing prefix reuse.
  • The dataset is marked __SYSTEM__/format_version = 16 durably at open, before the file-based allocator can issue anything, so v15.0.2+ binaries refuse the dataset cleanly (downgrade is unsupported by policy; see v15.0.2 notes).
  • SUPPORTED_FORMAT_VERSION = 16; datasets marked newer are refused.

Added

• On-disk format-version tripwire. Opening a dataset now checks {base_dir}/__SYSTEM__/format_version (ASCII decimal, written by v16+; v15 itself writes nothing — absence is the v15 signature) and refuses to open anything newer than format 15 with a descriptive error. Rationale: a future layout (v16 relocates the prefix-allocator ceiling out of shard 0) would leave the legacy shard-0 ceiling stale; a v15 binary reading it would re-issue already-used prefixes — silent data corruption. Downgrade stays unsupported by policy; this makes the violation fail loudly instead of silently, and makes v15.0.2+ the safe landing point for out-of-contract rollbacks. (Design: docs/proposals/namespaces.md §7.)

Changed

• Doc terminology: per-prefix key ranges are now called "prefix ranges" (previously "prefix namespaces"), freeing the term namespace for the upcoming v16 feature.

Fixed

• clear() is now crash-atomic everywhere — the v15 single-batch contract holds for every mutation, wipes included. The engine-level clear() (all collection types) previously deleted rows in chunked batches; a hard crash mid-clear could leave a partially-cleared namespace that hydration silently trusted (stale SlotDex totals/tiers, stale VecDex graph state over missing rows). It is now one write batch containing a single range tombstone covering the whole prefix: all-or-nothing (even across a crash) and O(1) instead of O(n).
  • New primitive: MapxRaw::batch_entry_wiped() — a batch pre-staged with the whole-range wipe; operations added afterwards apply on top of it and the whole set commits atomically.
  • StagedRows::wipe() — a wiped overlay reads the committed store as empty and commits the tombstone plus all staged rows in one batch.
• VecDex::compact() is atomic. The rebuild is staged through one wiped transaction and commits in a single engine write batch: a crash (or error) leaves either the old graph or the new one, never anything in between. The former crash window (wipe applied, re-inserts pending) is gone.
• VecDex::clear() persists the preserved ef_search. The reset graph state row (carrying the live ef_search) commits in the same atomic batch as the wipe, so a post-clear restore no longer silently reverts to the creation-time value.

Changed

• VecDex Txn decoded-vector cache and search-path cache now use an imported RefCell; staged/vecdex import groups tidied.
• Docs: slotdex pattern guide file list and stale multi-handle bug pattern refreshed; vecdex pattern guide compact section matches the atomic rebuild; prune_and_detach doc describes its actual return value.

Changed

• B+ tree node writes are batched per operation. PersistentBTree now stages the COW node group of each insert/remove/bulk_load in a write buffer and lands it through ONE engine write batch at the end of the operation, instead of one engine put per node — per-node shard-lock/WAL overhead is paid once per operation, and the node group becomes all-or-nothing (a torn, partially-written path-copy can no longer appear on disk; previously it was benign but had to be swept by recovery). Intra-operation churn (split/borrow/merge intermediates) is discarded from the buffer and never reaches the engine at all. bulk_load flushes in bounded chunks, so its buffer cannot grow with the dataset. No API or crash-ordering change: the buffer is always drained before a root escapes to the caller, so branch state still lands strictly after the nodes it references. Versioned write benches improve ~10-20% (insert −20%, merge −12%); read paths are untouched apart from one branch on an empty map.

Added

• SlotDex::insert_batch — bulk insertion that is observationally identical to per-pair insert but amortizes engine writes: keys are grouped per slot, each touched container is loaded/persisted once, and container records plus per-tier counters are flushed through one write batch per collection. Intended for imports and index rebuilds.
• Missing invariant tests from the full audit: prefix-allocator uniqueness/monotonicity/ceiling-persistence (INV-E1), MapxRaw prefix isolation (INV-E3), VerMap rollback isolation (INV-V4), and a ground-truth ref-count recount (INV-V1).

Fixed

• MPT cache loader rejects mixed Cached/InMemory trees. A checksum-valid but crafted cache file could place an unhashed (InMemory) child under a Cached parent; commit_rec skips Cached subtrees, so the child was never re-hashed and prove() panicked on its missing hash. Rejected at the load trust boundary, like the other cache-shape validations.
• rollback_to targeting the current head with a clean working state is now an early-return no-op instead of a full rewrite cycle that set gc_dirty and re-wrote identical state.
• merge_empty_source_fails test used a branch ID from the wrong VerMap instance (worked only by coincidence of ID layouts).

Changed

• Orphan arithmetic/bit/negation operator impls no longer require Ord + Eq, so f64 (and other PartialOrd-only types) can use +, -, *, /, unary -, etc.
• Serde deserialization of collection handles is now documented as an aliasing operation (same SWMR obligations as shadow()); the fast-forward precondition of merge and the net-zero ref-count convention of commit() are documented at the code site.
• rand moved from vsdb_core's [dependencies] to [dev-dependencies] (only tests/benches use it); unused hex dev-dependency removed workspace-wide.

Fixed

• Detected cgroup limits are derated to 3/4 before sizing engine memory. memory.high is a throttle line, not a quota: an engine budgeted exactly to it reaches steady state pinned AT the line, where every allocation pays reclaim-stall latency. Observed in production: a follower under a 9626M MemoryHigh peaked at exactly 9.6G, its ingest pipeline stalled for most of an hour, and a SIGTERM drain could not complete inside the unit's stop timeout (SIGKILL -> dirty store -> minutes-long derived-state rebuild on next boot). The explicit VSDB_MEM_BUDGET_MB override is still applied verbatim -- the operator asked for that exact number.

Fixed

• Write buffers now scale with a detected memory limit (follow-up to v14.0.9). v14.0.9 clamped the sizing INPUT to the cgroup/env budget, but the write-buffer branch for budgets <= 16 GB is a fixed 1 GB / NUM_SHARDS floor, and each shard holds one active memtable plus up to max_immutable_memtables (4) frozen ones -- a worst-case memtable footprint of ~5 GB regardless of a 2-3 GB ceiling. An ingest burst under such a limit pinned anonymous memory at the throttle line (memory.high), and the resulting reclaim pressure slowed the very flush threads that are the only way out: the process wedged at the limit with tens of thousands of memory.events: high entries (reproduced empirically; the v14.0.9 clamp alone shrank the block cache but not this). When (and only when) a limit tightened the budget, the per-shard write buffer is now additionally capped at budget / 8 / NUM_SHARDS (floor 4 MB), bounding the worst-case memtable footprint at ~5/8 of budget alongside the block cache's 1/8. Unconstrained hosts keep the legacy sizing byte-for-byte.

Fixed

• Engine cache sizing now respects the process's cgroup memory limit. mmdb_open sized write buffers and the block cache from host-wide MemAvailable alone, so a process running under a systemd MemoryHigh/MemoryMax drop-in or a container memory limit computed budgets from memory it is not allowed to use -- on a 32 GB host with a 12.8 GB cgroup ceiling, engine caches alone (~7.5 GB) grew the process to the OOM-kill line during bulk ingest (observed in production as unbounded-looking anonymous-memory growth of a service holding a 1.2 GB store). The budget is now min(host MemAvailable, cgroup limit, VSDB_MEM_BUDGET_MB): the cgroup walk covers v2 (memory.max + memory.high, unified hierarchy) and v1 (memory.limit_in_bytes), takes the tightest ancestor limit (limits are hierarchical), and treats max/PAGE_COUNTER_MAX-style sentinels as unlimited; the new VSDB_MEM_BUDGET_MB env var is an explicit highest-precedence bound for operators who want engine memory below any detected limit.

Fixed

• Safety comments relaxed to per-key granularity. The shadow() SAFETY comments and docs incorrectly claimed a global single-writer constraint (SWMR / "all shadows must be dropped before the next write"). The actual contract is per-key: concurrent writers on disjoint keys are safe — the engine provides snapshot isolation and per-key shard routing.
• from_bytes() no longer requires "same code version". The doc comments incorrectly required the same code version for deserialization. from_prefix_slice performs no memory-unsafe operation itself; the real requirement is unique ownership of the prefix bytes.

Fixes for the two findings the v14.0.6 post-release review surfaced (both residuals of the bug class that release addressed).

Fixed

• VerMapWithProof no longer serves a silently wrong Merkle root after a failed sync. batch_update is documented non-atomic, so a diff whose op is rejected partway (concretely: a committed or uncommitted key over MAX_MPT_KEY_LEN with T = MptCalc — the VerMap layer imposes no key-length limit) left the trie holding a partially applied diff while the sync bookkeeping still claimed the previously synced commit. A later merkle_root_at_commit(C1) (or merkle_root after rolling the branch back to C1) short-circuited on that stale claim and returned a root over C1-plus-partial-diff data — no error, wrong root (empirically confirmed via both the committed-diff and dirty-overlay paths; v14.0.6 had fixed only the emptied-trie variant of this desync). sync_to_commit now poisons the sync state (default trie, no synced commit) on a failed incremental application, forcing a full rebuild on the next sync; sync_to_branch restores the clean HEAD snapshot taken just before the dirty overlay, so the trie keeps matching sync_commit exactly. Regression tests cover both paths, including re-syncing successfully after the failure.
• Trie cache deserializers now validate whole-tree structure, closing the crafted-cache gap in v14.0.6's root-preservation guarantee. Per-node checks (path ≤ 256 bits, file checksum) could not see cross-node violations, so a checksum-valid but malformed cache file could load trees the walkers can't handle: an SMT whose cumulative descent depth exceeds 256 bits made a later insert fail after consuming the working tree — silently emptying the whole tree one layer below v14.0.6's SmtCalc-level restore (its "a rejected insert never loses tree data" guarantee) — and mispositioned leaves could drive path arithmetic out of range (a release-mode panic); an MPT cache bypassed the insertion-time MAX_MPT_KEY_LEN stack-depth cap entirely, and out-of-range nibble values (> 0x0F) panicked on branch-child indexing. The SMT deserializer now threads the routing prefix down the tree and rejects any leaf whose position+path doesn't reconstruct its own key hash exactly, any internal node pushing cumulative depth past 256 bits, and any cached hash that isn't 32 bytes; the MPT deserializer enforces the cumulative nibble budget (2 * MAX_MPT_KEY_LEN), rejects empty extension paths (organic tries never produce them; they were the only zero-progress construct, so the nibble budget is now a real recursion bound), out-of-range nibble values, and non-32-byte cached hashes. Accepted trees are exactly the canonically-positioned ones organic mutation builds; valid caches round-trip unchanged (no format/version change). The commit() doc comments on both tries were also corrected — they claimed the root is "restored" on a commit_rec failure, but that error arm drops the consumed working tree; it is now genuinely unreachable (MPT commit_rec is total; the SMT's only failure input — a bad cached hash length — is rejected at load).

Full-codebase audit sweep (9 parallel subsystem reviews) with every finding fixed except one documented, deliberate exception.

Breaking

• DagMapRaw/DagMapRawKey<V> no longer implement Default. The derived impl silently performed real disk I/O (an eager write through Orphan::new()'s parent slot) on every call, so generic code (mem::take, Option::unwrap_or_default(), HashMap::entry().or_default()) could create orphaned, unreclaimable on-disk state without any visible indication. Use DagMapRaw::new(None) / DagMapRawKey::new(None) explicitly. Migration: replace any Default::default()/mem::take/.or_default() usage on these types with an explicit new(None) call.
• TrieCalc/MptCalc/SmtCalc now return vsdb::Result<T> (VsdbError), not the internal TrieError. This closes a gap in the "single error type" invariant (vsdb_core::common::error::VsdbError is documented as the only error type across both crates' public APIs). TrieError is still exported (vsdb::trie::TrieError) for downstream matching via VsdbError::Trie { detail }, but is no longer the error type of the trie trait/struct methods themselves. Migration: replace Result<T, vsdb::trie::TrieError> bounds/matches with vsdb::Result<T> / VsdbError::Trie.
• vsdb_core::common::atomic_write_file (and its vsdb::common re-export) now returns Result<()> (VsdbError) instead of std::io::Result<()>. The only other raw-error-type leak found in either crate's public surface. Migration: handle VsdbError::Io instead of std::io::Error.

Fixed

• MptCalc/SmtCalc (insert/remove/root_hash/batch_update) no longer silently empty the trie/tree on a rejected mutation. All four methods mem::take the root into a local working value before the fallible operation; previously, on Err (concretely reachable via MptCalc::insert/batch_update when a key exceeds MAX_MPT_KEY_LEN, a public 1024-byte constant), the function returned early without restoring self.root, permanently replacing the entire trie with an empty one. All eight methods now unconditionally restore self.root/self.trie from the working value before propagating any error — a rejected batch_update still applies operations before the failing one (documented as non-atomic) but never discards unrelated prior state. This also fixes VerMapWithProof::merkle_root, which could otherwise silently return the empty-trie root hash for legitimate, unmodified committed data after such a rejection desynced its incremental-sync bookkeeping from the trie's actual content.
• MmDB::new() no longer leaks already-opened shard handles (and their background compaction threads) if a later shard or the meta-init step fails to open. Shards are now opened into an owned, non-'static Vec<DB> first — so a mid-loop failure drops (and cleanly closes) every already-opened shard via normal Drop glue — and only Box::leak'd after every fallible step has succeeded.
• SlotDex crash recovery now eagerly rebuilds the tier-acceleration stack instead of leaving it empty until the next insert(). Previously, ensure_count() correctly discarded the (potentially skewed) tier stack on unclean-shutdown detection but deferred rebuilding it, silently degrading every pagination query to an O(N) raw scan (measured ~950–2600× slower at 200k entries) for as long as the process stayed idle or read-only after the crash.
• SmtMut::remove no longer discards cached ancestor hashes on a no-op removal (removing a key that shares a path prefix with real data but isn't actually present). Mirrors MPT's existing rewrap-based no-change path: remove_rec now threads a changed flag and restores the prior Cached hash when neither child subtree actually changed, instead of unconditionally reconstructing (and later re-hashing) the node via compact.
• DagMapRaw::is_dead() now recognizes tombstoned entries. remove() writes an empty-value tombstone rather than deleting outright (existing, documented convention also used by get/get_mut); is_dead() previously checked only for a literally-empty backing store, so a node whose sole key was removed incorrectly reported is_dead() == false.
• Mapx::keys()/MapxOrd::keys() no longer decode values. Both previously routed through iter().map(|(k, _)| k), which unconditionally decoded V per entry (an engine::Mapx::deserialize call, including lock acquisition, for nested-VSDB-collection value types) before discarding it. A new MapxOrdRawKey::keys() decodes only the raw key bytes; Mapx/MapxOrd::keys() now build on it, decoding only K.
• core/src/common/engine/mmdb.rs's PENDING_WINDOWS registry no longer grows unboundedly for the life of the process. A thread-per-task workload (e.g. a thread-per-request server) previously accumulated one entry per historical thread (since ThreadIds are never reused and no cleanup path existed). A thread_local! guard now removes a thread's entry via Drop when that thread exits — always safe, since a dead thread's un-issued batch tail can never be issued by any other thread.

Added

• from_prefix_slice (core engine, unsafe fn) now has a # Safety doc comment at its definition, matching every other unsafe fn in the crate.
• Orphan/Mapx/MapxOrd/MapxOrdRawKey's from_meta() now documents the aliasing hazard it shares with shadow() (restoring while the original handle is still live creates a second handle to the same storage) — previously this was undocumented on the one restore path that isn't unsafe.
• Typed collections' batch_entry() doc comments now state the raw layer's existing "failed commit is not retryable" caveat.
• Regression tests: rejected-mutation trie root preservation (MPT insert/batch_update), VsdbError-typed trie errors, SMT no-op-remove cache preservation, SlotDex crash-recovery tier rebuild (both the dirty-flag and invalid-empty-tier detection paths), DagMap tombstone-aware is_dead(), keys() never decoding values (typed collections), and a PENDING_WINDOWS thread-exit cleanup test.

Won't Fix

• VecDex::compact() remains non-atomic across a hard crash (documented in docs/audit.md): a true fix requires a prefix-swap/version-indirection redesign, and a naive version would silently desync any earlier save_meta/parent-collection reference to the index — a worse failure mode (silent staleness) than the current rare-crash-window data loss it would trade away.

Breaking

• SMT hash domain switched to the Diem/JMT leaf-shortcut construction. A subtree holding exactly one leaf now commits to Keccak256(0x01 || key_hash || value) directly — independent of depth — instead of folding the leaf hash through its ~246 residual path levels; internal nodes are unchanged (Keccak256(0x00 || left || right), compressed internal prefixes still wrap through empty siblings). All SMT root hashes change. The SMT disk-cache format is now v3; v2 caches are rejected cleanly and the trie rebuilds from authoritative data. MPT is unaffected. This removes the dominant O(N × 256) hashing term: whole-tree hashing is now O(N) hash operations.
• SmtProof is now compact (variable-length). siblings holds hashes only from the root down to the terminal lone-leaf/empty subtree on the key's path (O(log N) entries instead of a fixed 256), and the value: Option<Vec<u8>> field is replaced by leaf: Option<([u8; 32], Vec<u8>)> — the lone leaf occupying the terminal subtree (leaf.0 == key_hash ⇒ membership; a different leaf.0 ⇒ conflicting-leaf non-membership, checked for path-prefix consistency during verification; None ⇒ empty-slot non-membership). Use the new SmtProof::value() accessor for the proven value. Proof size drops from a fixed 8 KiB to typically well under 1 KiB, and verification folds O(log N) hashes instead of 256.

Fixed

• SlotDex reverse paging restored to tier-accelerated complexity (commit 9758b70, folded into this release): the v13.4.7 correctness fix had degraded get_entries_by_page(.., reverse=true) to a linear reverse scan (~17 ms vs ~10 µs forward at 100k entries). Reverse paging now mirrors the forward path via locate_page_rstart — a rightmost-distance offset plus a descending tier-cache locate — returning reverse pages to the 10–35 µs range while preserving the corrected slot-descending / within-slot-ascending semantics.
• SMT tree walks no longer materialize a path slice per level. insert/remove/get/prove compared the remaining key path by allocating full_path.slice(depth, 256) (a bit-by-bit copy) at every internal node; they now use allocation-free offset-based comparison (BitPath::common_prefix_from / starts_with_from, byte-wise with unaligned assembly). BitPath itself is now a zero-allocation inline [u8; 32] (paths never exceed 256 bits — a type invariant), with slice/concat rewritten from per-bit loops to byte-wise shifts, and the cache deserializer now rejects bit lengths over 256 instead of allocating attacker-controlled buffers. Combined with the hash-domain change: 1000-key insert 4.6 ms → 0.77 ms, remove 9.0 ms → 1.5 ms, get 3.9 ms → 0.41 ms, cold root hash 76.6 ms → 0.93 ms, verify 79 µs → 3.2 µs per proof (reference box).
• mapx / sequential / iter (5k entries) bench measured an unbounded dataset. The iterated map had accumulated entries from all preceding timed write benches (hundreds of thousands and growing), so the reported number was meaningless and unreproducible; the bench now iterates a dedicated 5000-entry map.

Added

• SMT adversarial proof tests: sibling-list depth extension, sibling truncation, conflicting-leaf prefix substitution, proof compactness, plus the existing tamper/wrong-root suite adapted to the new format.
• Bench symmetry: smt_batch_update_{100,1000} and mpt_prove_100 / mpt_verify_100 cases in trie_bench, with black_box hygiene on discarded results.

Breaking

• Replaced CBOR codec with postcard — serde_cbor_2 has been removed and replaced with postcard as the sole serialization codec. Existing data serialized with CBOR is incompatible; export it with v11.0.0 and follow Legacy persisted-data migration.

Breaking

• Removed RocksDB backend — MMDB is now the sole storage engine. The backend_rocksdb and backend_mmdb feature flags have been removed. No C/C++ toolchain required.
• Commit reference counting — Commit gains a ref_count: u32 field. delete_branch and rollback_to immediately hard-delete orphaned commits via cascading ref-count decrement. No manual gc() call needed for commit cleanup.
• VerMapWithProof: automatic cache lifecycle — save_cache() and load_cache_and_sync() have been removed from the public API. The trie cache is now eagerly saved after each sync_to_commit and auto-loaded on construction. No manual calls required.

Added

• Commit ref counting — each commit tracks the number of branch HEADs and child parent-links pointing to it. commit(), create_branch(), delete_branch(), merge(), and rollback_to() all maintain ref counts automatically.
• B+ tree in-memory node ref counting — PersistentBTree maintains a HashMap<NodeId, NodeRef> for zero-overhead lifecycle tracking. Dead nodes are cascade-released in memory; disk reclamation happens on gc() / startup.
• VerMapWithProof auto-cache — auto-load in new()/from_map(), eager save after each sync_to_commit. A cache_dirty flag avoids redundant serialization in read-only scenarios.

Removed

• backend_rocksdb feature flag and all RocksDB-related code, Makefile targets, and documentation.
• strata/docs/engine-comparison.md (no longer applicable).
• pending_gc, next_gc_seq, process_pending_gc(), recover_pending_gc() — replaced by commit ref counting.

Breaking

• Removed msgpack codec — CBOR (serde_cbor_2) is now the only serde encoding. Existing data serialized with msgpack is incompatible; export it with v9.1.0 and follow Legacy persisted-data migration.
• Default backend for vsdb crate — backend_mmdb is now enabled by default so that vsdb = "10.0.0" works out of the box without a C/C++ toolchain (previously required explicit feature selection).

Added

• MMDB backend (backend_mmdb) — a pure-Rust LSM-Tree alternative to RocksDB. No C/C++ dependency; suitable for cross-compilation and WASM targets.
• Engine comparison guide — strata/docs/engine-comparison.md with detailed benchmarks of MMDB vs RocksDB.
• make all-rocksdb target and RocksDB-specific lint/test/bench targets in Makefile (default targets use MMDB).

Changed

• SlotDex performance — tier data backed by an in-memory BTreeMap cache (auto-hydrated via RefCell), reducing page query latency from ~1 ms to ~8 us.
• Aligned MMDB DB options with RocksDB configuration for consistent behavior.
• Fixed mmdb 2.2 API: replaced removed prefix_iterator with iter_with_prefix.
• Replaced unwrap/panic with c(d!()) error chains; hardened decode bounds.
• Expanded benchmark coverage and fixed methodological issues.
• Bumped dependencies.

Removed

• Removed lint-codecs CI target (no longer needed with single codec).
• Removed RocksDB pre-built static lib cache from Makefile.

Changed

• Merged vsdb_trie_db and vsdb_slot_db into vsdb — they are now modules (trie and slotdex) instead of separate crates. The workspace is reduced to two crates: vsdb_core and vsdb.
• Renamed trie_db -> trie, inner trie/trie -> trie/mpt, slot_db -> slotdex.
• Moved VerMapWithProof from versioned::proof to trie::proof, alongside MptCalc and SmtCalc.
• Removed merkle feature gate — the trie module (including sha3 and thiserror) is always compiled.
• All public types re-exported from crate root: MptCalc, SmtCalc, SmtProof, VerMapWithProof, SlotDex.

Added

• SmtCalc — Sparse Merkle Tree with 256-level proofs (prove / verify_proof).
• SMT cache — save_cache / load_cache for SmtCalc (disposable on-disk persistence).
• Comprehensive SMT test suite (24 tests) and benchmarks.
• Architecture diagram in trie module docs.

Added

• VerMap convenience APIs: branch_id, branch_name, has_uncommitted, range, iter_at_commit, get_commit — small, high-value methods for common caller patterns.
• Versioned benchmarks: Criterion benchmarks covering single-branch CRUD, commit/rollback, branching, iteration/range, historical reads, three-way merge, and GC.
• Comprehensive test coverage: 65 new tests for the new APIs; 136 total versioned tests.

Fixed

• Fixed all Clippy warnings for Rust 1.93+ (collapsible if-let chains, ptr_arg, needless_borrows_for_generic_args, type_complexity).

Changed

• License changed from GPL-3.0 to MIT. The entire project is now licensed under the MIT license, allowing for more permissive use and integration.