blosc2

Python-Blosc2 4.14.1 is a security and feature release introducing safe persisted-object deserialization by default, Caterva2 remote stores and tables, compressed slicing for RemoteArray, and optimized expression gather performance.

Safe persisted-object deserialization

• Persisted-input APIs now default to deserialize="safe". This includes open(), load(), from_cframe(), CFrame constructors (CFrameNDArray, CFrameCTable), store traversal (TreeStore, DictStore, EmbedStore), and CTable variable-length and batch columns. Safe mode rejects embedded objects, references, remote/source descriptors, proxies, and lazy recipes before reconstruction.
• Trusted callers that intentionally use rich persisted values must pass deserialize="full" explicitly. Blocked values raise the new public UnsafeDeserializationError. This default change is intentionally behavior-breaking for authority-bearing serialized values.

Caterva2 remote stores and tables

• blosc2.open() now opens Caterva2 groups as RemoteStore and Caterva2 URLPath tables as RemoteCTable.
• RemoteCTable backed by Caterva2 supports on-demand reads, bounded row slices, column projection (select()), iteration, and materialization.
• Caterva2 tables fetch large row slices and materializations in bounded batches, preventing out-of-memory issues on large datasets. Materializing to urlpath= writes batches directly to disk and publishes only after the entire read succeeds.
• Caterva2 stores bind the active authentication token when opened and isolate disk cache entries per token, ensuring isolation across accounts.

Compressed slicing for RemoteArray

• Added RemoteArray.slice(item=(), **kwargs) to extract a requested selection directly as an independent compressed NDArray without assembling the complete selection as an uncompressed NumPy array.
• Output-construction keyword arguments such as cparams are forwarded directly to NDArray.slice.
• Fetching, compressed result assembly, and cache eviction are serialized into a single atomic operation, allowing selections to exceed the cache memory budget without cached chunks disappearing mid-construction.

Performance improvements

• Avoided zeroing full NumPy input blocks in aux_miniexpr during expression gather operations (#728, PR #728). Full blocks are overwritten directly by raw NumPy gather, keeping zero-padding only for partial edge blocks. Thanks to @Johnny-Kao.
• Sped up remote Caterva2 table verification in tests with column-wise comparisons and boundary row sampling.

Python-Blosc2 4.14.0 adds remote tables, shared caches, native HDF5 range reads, and indexed queries over remote data. It bundles C-Blosc2 3.3.5.

Remote arrays, tables, and hierarchies

• blosc2.open() returns RemoteArray, RemoteCTable, or RemoteStore according to the selected object. path= selects a node inside a container; dataset= remains an alias. Remote Zarr v2/v3 groups now open as stores.
• RemoteCTable reads B2Z CTables, single-file Parquet, and PyTables tables on demand. Column projection avoids unrelated data; Parquet reads and caches individual physical fields by row group. lazy=False imports Parquet eagerly.
• Multi-column reads overlap independent requests, bounded by max_concurrency and transport-buffer budgets. Decoding and cache publication stay serialized.
• RemoteObject provides the common source, attributes, traffic, cache, reference-saving, and lifetime API. RemoteStore, TreeStore, and DictStore provide .info summaries of their contents without opening leaf readers.
• cache_dir= retains metadata and accessed data across sessions. Add shared_cache=True for simultaneous processes; all users of that cache directory must enable sharing. Persistent caches also support local sources.
• save() writes a reference with retained cache data, without fetching missing data. include_cache=False creates a cold reference. materialize() produces independent local data; table copy(), to_b2z(), and to_b2d() do likewise.
• TreeStore can persist RemoteStore mounts. Reference exports preserve them; materialization expands them, preserves group attributes, and rebuilds indexes.
• Standalone arrays/tables and root stores support refresh(). Successful refresh invalidates previously obtained child handles; callers must retrieve them again.

HDF5, PyTables, and indexes

• Native HDF5 metadata indexes replace Kerchunk translation. Remote reads no longer require Kerchunk, Zarr, or numcodecs. Uncompressed, deflate/gzip, shuffle, and Blosc2 chunks are decoded directly; other filters use h5py. Cached metadata avoids repeated scans on warm opens.
• Local and remote PyTables tables open as RemoteCTable without requiring PyTables installed. Supported persisted PyTables indexes are imported as Blosc2 OPSI indexes and cached for subsequent queries.
• Remote CTable queries use persisted SUMMARY, FULL, PARTIAL, OPSI, BUCKET, and list-membership indexes. Unsupported query shapes fall back to scans. Index construction remains a local operation.

Lists and Arrow interchange

• ListArray supports nullable items and recursive nesting, up to 32 list levels. contains() and overlaps() provide row predicates; optional kind="membership" indexes accelerate flat scalar-list queries.
• New list schemas default to batch_rows=2048. Pass None for caller-managed batching. Existing schemas retain their stored batching behavior.
• Arrow/Parquet exports default to 65,536 rows per batch, improving throughput at the cost of larger temporary allocations and default Parquet row groups. Imports and persisted list batches retain the 2,048-row default.
• Parquet list imports now default to MessagePack, matching Arrow imports. Use list_serializer="arrow" or CLI --list-serializer arrow to opt in. Dense local Arrow-backed list exports keep their Arrow buffers where possible.

Bug fixes

• Fixed asarray() data corruption for arrays larger than 16 MB with block partitions that divide chunks but are not C-contiguous (#723, PR #724). Thanks to @jeandet.
• Fixed concurrent NDArray reads sharing mutable decompression state, including views, and released the GIL while waiting for read/frame locks (#555, PR #713). Thanks to @Johnny-Kao.
• Fixed stale remote index handles, cache cleanup, failed-open resource leaks, double-closing fsspec sessions, and Windows path/refresh handling.
• Hardened HDF5 metadata validation and bounded deflate decoding. Preserved query strings in dataset URLs and NumPy attributes in metadata round trips.
• Fixed Parquet list-cell shape preservation, list predicates, and cached-read exception handling. Arrow imports preserve nested item nullability.
• Preserved metadata-only groups during tree traversal and materialization; handled root entries in store summaries without looping.
• Added tested CTable query and sorting examples (#650, PR #711). Thanks to @armutlutost.

Compatibility notes

• hdf5_index replaces refs in blosc2.open() and RemoteArray. It accepts a native index dictionary or JSON index path; legacy Kerchunk maps are rejected. The hdf5 extra now requires only h5py and hdf5plugin.
• RemoteCTable.save() now saves a reference and returns its path. Use materialize() for the former full-copy behavior. Local CTable.save() is unchanged. Replacing a reference destination requires overwrite=True.
• Shared cache factories default to a 256 MiB aggregate compressed-payload budget, matching ordinary opens. Pass max_cache_bytes=None for unlimited retention. This does not limit peak RAM or total disk use.
• Shared caches require lazy access; explicit lazy=False is rejected. Ordinary table/store disk caches remain exclusive to one owner at a time.

A maintenance and performance follow-up to 4.13.0 focused on remote data access and expanded platform support. Lazy access is now the default for remote arrays and container datasets, local HDF5 files are read directly via h5py without auxiliary dependencies, warm opens replay cached bootstrap metadata to eliminate redundant network discovery, disk caches use human-readable directory structures, and native Windows ARM64 wheels are now built and tested.

Improvements

Remote access

• Lazy access is the default for remote arrays and datasets. blosc2.open() on remote .b2nd files or container datasets (HDF5, Zarr, B2Z) now returns a lazy RemoteArray without requiring an explicit lazy=True. Dataset paths require lazy access and reject lazy=False with NotImplementedError rather than attempting a silent eager download. For standalone array files (such as .b2nd), cache_dir= keeps the default lazy access and persists fetched chunks in a RemoteArray carrier. Pass lazy=False to download the complete container there instead. mmap_mode= or a nonzero offset= forces the eager path.
• Direct local HDF5 reads via h5py. Local .h5/.hdf5 datasets are now read directly through h5py without requiring kerchunk, zarr, or fsspec. Chunked datasets retain their native HDF5 chunk layout, while contiguous datasets receive automatically chosen Blosc2 cache chunks. Dataset attributes are loaded from HDF5 metadata, and file handles are safely closed when the source object is garbage-collected. Explicit refs= arguments continue to select the kerchunk reference reader.
• Faster HTTP discovery and instant warm opens.
  • Cold HTTP B2Z discovery: Opening a remote B2Z archive over HTTP now retrieves the ZIP directory tail and remote object identity in a single bounded range request, halving network round-trips during initial open.
  • Warm reopens: Reopening cached B2Z, Zarr, or HDF5 sources replays persisted bootstrap metadata directly from the carrier file, completely bypassing remote discovery. Sibling HDF5 leaves from the same container under a shared cache_dir= reuse a single on-disk reference snapshot.
  • Whole-member prefetch for small B2Z arrays: Members up to 64 KiB are fetched in full on open (frame header, chunks, and trailing metadata in a single request). Prefetched chunks live in the standard chunk cache, count toward cache_bytes, respect max_cache_bytes, and can be evicted with trim_cache().

Readable disk caches

• Readable source directories for cache_dir=. Persistent disk caches now mirror the source filename and internal dataset hierarchy (e.g., hierarchy.b2z--03cc6a2f9314/d0/a1.b2nd) with a 12-character identity fingerprint computed from the URL and storage options. Unsafe characters are escaped and long components are shortened. RemoteStore caches follow the same naming convention. Complete containers localized with lazy=False use it too (e.g., temperatures.b2nd--4908c5e5602a/temperatures.b2nd).
• Legacy hash-only cache entries are ignored rather than overwritten or deleted; the first open under 4.13.1 creates a new cache. Explicit cache_path= targets and direct opens of existing carrier files continue to work. Legacy cache directories can be safely deleted to reclaim disk space.

Packaging and platform support

• Windows ARM64 builds and wheels. Added native Windows on ARM64 (win_arm64) wheel building and testing via GitHub Actions runners, including build configuration adjustments for Clang and runtime fallback for JIT on ARM64.
• CI test stability and doctest isolation. Enhanced test suite stability with logical core utilization in pytest-xdist, per-test workspace isolation for doctests in conftest.py, and test deadlock diagnostics.
• Remote lazy expression examples. Added examples/remote/lazy-expr.py and documentation illustrating how to evaluate lazy expressions across remote B2Z, Zarr, and HDF5 arrays.

Bug fixes

• Embedded frames with an offset. Opening a local file whose name looks like a container (.h5, .hdf5, .b2z, .zarr) with a nonzero offset now opens the embedded Blosc2 frame directly instead of attempting container parsing.
• file:// HDF5 URLs on Windows. Dataset path separators (::) in file:// URLs survive URL-to-path conversion on Windows platforms (e.g., file:///C:/path/x.h5::/d0/a2).
• Credential masking. RemoteArray.info and str() mask sensitive query parameters and credentials in signed URLs. RemoteArray.info now also works reliably for local B2Z sources without raising an error.
• load_tensor() eager access. Explicitly requests eager access so it no longer inadvertently returns a lazy RemoteArray intermediate when loading known remote paths.
• Blosc2-filtered HDF5 decoding. Fixed decoding of HDF5 datasets compressed with the Blosc2 filter (filter ID 32026, as written by hdf5plugin) when read via kerchunk. Chunks containing multi-chunk super-chunk frames decoded via blosc2.from_cframe() now handle the resulting bytes correctly, avoiding an AttributeError on .tobytes().
• HDF5 reference snapshot publishing. Guarded reference snapshot export so it is only published when the leaf carrier actually holds a kerchunk map, fixing a crash when opening local h5py sources with a disk cache on Windows drive-letter paths.
• Read-only portable carriers. Attaching a sparse runtime cache to a read-only legacy B2Z carrier no longer fails when refreshing bootstrap metadata; the session safely uses the rebuilt bootstrap in memory.

This release turns Python-Blosc2's remote capabilities into a comprehensive, portable remote data access layer. It introduces RemoteArray as the unified lazy array interface with bounded in-memory and persistent disk caching, RemoteStore for discovering and navigating multi-dataset hierarchies across B2Z, Zarr, and HDF5 containers over fsspec, portable reference exports and snapshots, the .attrs user metadata interface, interactive remote browsing in b2view, and the b2nd-to-zarr CLI converter.

Note: The remote cache protocol should be considered somewhat experimental until it has seen more real S3/HTTP usage; feedback is very welcome!

Improvements

• Unified RemoteArray with bounded caching: blosc2.open(..., lazy=True) now returns a RemoteArray object that provides a consistent slicing, fetching, and caching interface across standalone .b2nd files, container datasets, and Caterva2 endpoints.

  • In-memory cache by default (CachePolicy.MEMORY): Fetched chunks are retained in RAM bounded by max_cache_bytes (defaults to 256 MiB) with automatic LRU eviction after operations.
  • Persistent disk caching (CachePolicy.DISK): Specify cache_dir (or cache_path) to retain fetched chunks and frame layout metadata on disk across sessions. cache_storage is deprecated in favor of cache_dir.
  • Stateless streaming (CachePolicy.NONE): Disables chunk retention entirely for memory-constrained streaming reads.
  • Cache partitioning: Disk caches for fsspec URLs are partitioned by storage_options so distinct credentials or custom endpoints do not clash.
  • Explicit cache pre-fetching: fetch() and afetch() preheat specified slices synchronously or asynchronously into the cache.
  • materialize(item) returns an independent in-memory NDArray.

• RemoteStore hierarchy discovery and shared caching: Multi-dataset containers (.b2z, .zarr, and .h5/.hdf5) can now be explored and sliced lazily without downloading the entire container:

  • Discovery and navigation: Discover child groups and datasets via store.keys() or iteration (for name in store:). Access nodes using slash paths (store["group/dataset"]) or chained indexing (store["group"]["dataset"]).
  • Lightweight inspection: store.get_info(name) returns a RemoteNode (path, kind, attrs, diagnostic) without initializing leaf readers or allocating cache memory. Unsupported nodes degrade gracefully.
  • Shared cache budget: All leaves accessed through a RemoteStore share a single cache coordinator (256 MiB RAM by default, or persistent disk via cache_dir). When the budget is reached, cross-leaf LRU eviction frees chunks across any leaf in the store. Closing a leaf handle preserves its warm cache in the active store session.
  • Atomic refresh: store.refresh() atomically checks and reloads remote discovery, safely invalidating previously opened leaf handles (RuntimeError on reuse) to prevent reading inconsistent state.

• Portable references and snapshots:

  • RemoteArray.save("ref.b2nd") exports portable array descriptors with optional persistent caches.
  • RemoteStore.save("snapshot.b2z") exports an entire remote hierarchy (discovered keys, attributes, and source locators) into a portable .b2z archive without exposing credentials or secrets.
  • include_cache=True (default) bundles warm cached chunks for offline or zero-traffic reuse; include_cache=False exports lightweight streaming references.
  • mutable=False (default) produces immutable snapshots safe on read-only media; mutable=True allows local cache expansion upon reopening.
  • Exported references can be reopened transparently using blosc2.open() (blosc2.open("snapshot.b2z") returns a RemoteStore).

• Remote container adapters (B2ZNDSource, ZarrNDSource, HDF5NDSource):

  • B2ZNDSource: Direct byte-range access to external uncompressed ZIP_STORED NDArray members within remote .b2z archives without downloading or unzipping.
  • ZarrNDSource: Reads remote Zarr v2 and v3 arrays lazily on demand, caching converted Blosc2 chunks. Supports scalar and empty arrays, as well as fixed-size dtypes.
  • HDF5NDSource: Accesses remote HDF5 datasets lazily via kerchunk reference indexing. Unifies dataset syntax across slashes (file.h5/group/data), double colons (file.h5::group/data), and dataset="group/data". Manifests are indexed once per container and shared across leaves.

• New b2nd-to-zarr CLI utility: Converts local Blosc2 NDArray containers into Zarr v2 (with numcodecs blosc/blosc2 compressor) or Zarr v3 datasets.

• Recommended .attrs metadata interface: Arrays, containers, and proxy sources now expose .attrs as the recommended interface for user-defined metadata (aliased to .vlmeta). C2Array.attrs selects server user attributes while C2Array.vlmeta retains Caterva2 protocol metadata. Internal HDF5 dimension metadata is filtered out of user .attrs. .vlmeta remains fully supported and is not deprecated.

• Interactive remote browsing in b2view:

  • The terminal browser can open remote arrays and remote container hierarchies (.b2z, .zarr, .h5, fsspec URLs, S3, HTTPS).
  • Added --cache-dir option to persist discovery metadata and fetched chunks between viewer sessions.
  • Improved TUI responsiveness: fixed column capping when paging backward to start, bounded wall-clock wait budgets, and robust async shutdown.

• Enhanced LazyExpr AST shape inferencer:

  • Shape inference now handles bracket subscripts and slices (arr[10:20, ...]).
  • Recognizes slice and len builtins, common array methods (astype, copy, flatten, ravel, squeeze), and attributes (shape, size, ndim, itemsize).
  • Supports negative slice indices and unary operations in AST slice evaluation.

• Bundled C-Blosc2 3.3.4: Updated to the latest C-Blosc2 release.

• Test suite and CI reliability: Added pytest watchdog deadman timer (PYTEST_DEADMAN_SECONDS, PYTEST_DEADMAN_SESSION_SECONDS) and thread stack dumps to catch and diagnose worker hangs. Fixed file descriptor exhaustion by clearing index handle caches between tests, and resolved Windows-specific in-process HTTP server aborts and file locking races.

This release focuses on efficient remote arrays. Blosc2 containers can now be read and written through fsspec URLs, while lazy proxies fetch only the required blocks, overlap requests and reuse validated caches. Caterva2 arrays gain block-range reads and concurrent chunk writers. UTF-8 FULL-index lookups also use substantially less memory by bisecting their vocabulary on disk.

Improvements

• New blosc2[fsspec] extra: blosc2.open(), save_array() and save_tensor() accept any fsspec URL — s3://, gs://, https://, zip:// and chained URLs. open() can download the container, keep a validated local copy with cache_storage=, or fetch only requested slices with lazy=True and the new blosc2.FsspecNDSource; lazy reads can also use a persistent cache. Saves upload one complete object, but URL-backed mutable containers are not supported. Protocol drivers and credentials remain the caller's responsibility.

• A C2Array can be written to a chunk at a time, which is how several processes fill one remote array at once: update_chunk() (and its async aupdate_chunk()) posts one compressed chunk into a slot of a pre-sized array, and written_chunks() says which slots hold anything yet. The array is laid out with blosc2.uninit() and uploaded -- a couple of hundred bytes whatever its size -- and each slot is written once: a second write raises blosc2.ChunkAlreadyWritten, which is the whole of the coordination between writers. Writing into an empty slot appends to the frame and moves no other chunk, so a fill is cheap and a concurrent reader's cached offsets stay good. Needs a Caterva2 subscriber that serves the endpoint.

• C2Array.stamp, which is what a Proxy checks its cache against, now names which array it is as well as whether it has changed. A subscriber writes a nonce into a filled array's vlmeta, so a cache is no longer served against a different array that came to sit at the same path with the same size and mtime; and a complete array — every chunk written, so every further write refused — is stamped without its mtime, so a cache of it survives a republish or a copy instead of being thrown away. Arrays that were never filled a chunk at a time are stamped exactly as before.

• Proxy.fetch() takes a max_concurrency= argument, and reads it from the source when the source has one, so blosc2.open(url, lazy=True, max_concurrency=...) overlaps its chunk fetches in a thread pool. Ordinary slicing benefits, not just the async afetch(). A lazy fsspec proxy defaults to 8, matching what afetch() already used for remote sources; pass 1 for a protocol with no latency to hide, where the pool costs ~10 µs per chunk and saves nothing. Other sources stay serial unless asked, since this is only safe for a thread-safe get_chunk.

• blosc2.open(url, lazy=True) fetches blocks rather than whole chunks when a slice lands in a small part of a large one. With the partitions blosc2.asarray() picks by default a chunk holds a hundred-odd blocks, so a point or window read costs about 1% of what it used to: measured against S3, 5-17x faster on arrays with multi-megabyte chunks and 2-5x on 1 MB ones. It is never a loss, because the two thresholds that decide it — a chunk under a megabyte is one cheap request anyway, and wanting more than half of a chunk's blocks is wanting the chunk — need nothing read to answer. Peak memory drops with the traffic, since a fetch in flight is now a block rather than a chunk. bench/ndarray/fsspec-block-granularity.py measures both on any array.

• A Proxy over a C2Array now reads only the required compressed blocks from file-backed datasets using concurrent, batched HTTP byte ranges. On cat2.cloud's kevlar-tomo.b2nd, this reduced a corner slice from 2.723 MB to 0.031 MB and a ten-chunk slice from 1.01 s to 0.14 s. Computed datasets fall back to whole-chunk reads. blosc2.ByteRangeNDSource provides the same frame reader to FsspecNDSource, C2NDSource and custom transports, with lazy, persistent caching of frame layout metadata.

• DictStore.member_window(key) says where a leaf's frame lies inside a .b2z, as (offset, nbytes). A zip store keeps each external leaf uncompressed, so those bytes are the frame that leaf would have been written as on its own -- which lets a reader take the window instead of the leaf: Caterva2 now serves a container leaf from it, so a Proxy over @public/tree.b2z/leaf reads blocks exactly as it does over a .b2nd (2.8x on a point read, 20x fewer bytes). None where there is no window: a directory-backed store, an embedded leaf, a C2Array reference, and a member some other tool repacked compressed -- which is now also left out of the store's keys rather than read as the deflate output it is, and said plainly when it is the store's own super-chunk.

• blosc2.Proxy(src, urlpath=..., mode="a") now adopts the cache left by an earlier run, including partially fetched chunks. It validates the cache's shape, dtype and source stamp, refetching stale data if the remote array was replaced and raising for incompatible caches. Caches from pre-release 4.12.0 builds require one fresh open with mode="w".

• The source protocol moved to its own module, blosc2.proxy_source: ProxySource, ProxyNDSource, ByteRangeNDSource, FsspecNDSource and the frame reading behind them. They are still blosc2.X and still reachable as blosc2.proxy.X, so nothing outside need change; what moved for good are the block-granularity knobs, blosc2.proxy_source.BLOCK_MIN_CBYTES and its neighbours. This is what lets proxy.py be imported after schunk and indexing, rather than being dragged in ahead of them by every module that wants a source.

• Querying a utf8() column through its FULL index no longer materializes the index vocabulary. The query literal is turned into an alphabetical rank by bisecting the vocabulary sidecar instead, so a lookup reads a few blocks rather than one fixed-width entry per distinct value. On a 1 Mrow column of near-unique free text, the first lookup goes from ~62 ms and 739 MiB of peak memory to ~12 ms and 5.5 MiB.

Nullability in CTable is rebuilt on Arrow's own model. A nullable column now keeps its nulls in a per-column validity sidecar instead of reserving a value from its own range, which is what makes it lossless — an int8 column can hold -128, a utf8 one can hold "", and a float64 one can tell NaN from missing. That is the new default for columns created from now on; nothing on disk changes, and sentinel storage remains supported indefinitely, one keyword away. Built on top of it: predicates follow three-valued (Kleene) logic, so ~(t.price > 10) no longer returns the null rows, and column indexes are null-aware, which makes min/max over a nullable column answer from the index instead of scanning.

On the packaging side, wheels are now a single Stable ABI (abi3) build per platform covering CPython 3.11+, with free-threaded 3.14 and 3.15 shipping alongside.

New features

Mask-based nullable columns for CTable, and they are now the default

• A nullable column keeps its nulls in a per-column validity sidecar — Arrow's own model — instead of reserving a value from its own range. This is what a bare nullable=True resolves to, and what every nullable column inferred from Arrow, Parquet or CSV gets: blosc2.bool(nullable=True) no longer reserves 255 and keeps np.bool_, blosc2.int8(nullable=True) has all 256 values usable, blosc2.utf8(nullable=True) accepts any string including "" and "\x00", and blosc2.complex128(nullable=True) is nullable at all for the first time.
• This is what makes nullability lossless. A sentinel steals a value from the dtype, so a nullable int8 could not hold -128, a free-text utf8 column had no safe sentinel at all, and Arrow columns whose type had no value to spare could not be imported. to_arrow(from_arrow(x)) now returns x for nullable bool, full-range int8/uint8, float64 containing nan/±inf/-0.0 as values, utf8 containing "" and "__BLOSC2_NULL__", and timestamp with int64.min as a value — none of which round-trip through a sentinel.
• None is how you write a null under mask storage (t.append((None,)), t["price"][3] = None), which a fixed-width sentinel column cannot accept at all. is_null() is unchanged and remains the uniform API across every kind.
• Nothing on disk changes. The new default governs creation only: opening a stored table never re-resolves anything, so every existing table keeps the storage, dtype and sentinel it was written with, and every rewrite rule for the reserved 255 stays permanently in place.
• Sentinel storage is supported indefinitely and is one keyword away, per column (null_storage="sentinel", or any explicit null_value=) or globally through NullPolicy. Setting a type-wide NullPolicy sentinel field still implies sentinel storage for the kinds it covers, so existing NullPolicy(float_value=...) code is unaffected — with one unavoidable exception: 255 is the only value a nullable bool may reserve, so it is also bool_value's default, and NullPolicy(bool_value=255) carries no information to act on. A bool column that wants a sentinel has to say so with null_storage or column_null_values.
• A table containing a mask column records schema version 3. Only such tables do: a table with no nullable column still records version 1, exactly as before, and a sentinel one does too. Readers older than 4.11.0 refuse a version-3 table rather than misreading it, but their message is a bare ValueError: Unsupported schema version 3 — the hint naming convert_nulls(to='sentinel') ships in 4.11.0, so only readers that can already open the file will print it.
• If some of your data has to stay readable by an earlier release, pin the storage rather than discovering this downstream. Per column as above, or process-wide — including for schemas inferred from Arrow, Parquet and CSV — with blosc2.null_policy(blosc2.NullPolicy(null_storage="sentinel")). That reinstates the sentinel's lossiness (a float column's nulls become NaN again, and a type with no value to spare still cannot be imported), which is the trade being made.
• Column.null_storage reports where a column keeps its nulls and info tags each column (int64 nullable[mask]), so CTable.convert_nulls() can move columns between the two in either direction — never implicitly, and refusing rather than silently relabelling data when a sentinel is unavailable.
• One deliberate semantic difference: in a mask column NaN is a value, following Arrow, and only the sidecar marks a null. Sentinel float columns keep NaN-as-null. See "Where nulls are stored" in the CTable reference.

Column indexes are null-aware

• Per-segment min/max are taken over the rows that carry a value. A column's nulls are read from its validity channel and left out, and a segment with no value at all is flagged rather than summarised. This applies to both storages — an INT64_MIN sentinel is exactly as invisible to a summary as a mask column's fill.
• Column.min/Column.max answer from the index for a nullable column (236x on a 20M-row int64, measured), where before every nullable column but a NaN-sentinel float had to scan.
• where() with an OR over a nullable indexed column uses the index instead of falling back to a full scan (1.6x on a 20M-row two-column probe). The fallback existed because the only null filtering available was global, and a global filter drops a row that is null in one branch but matches the other; the segment path never needed it, because it evaluates the predicate, which has been null-aware per leaf since the string-predicate fix below.
• Indexes written by an earlier release are read as not null-aware and keep the old fallback, so nothing silently changes meaning; rebuild_index() promotes them.
• Building an index over a nullable column that holds nulls costs one decompression pass (33 ms for a 20M-row int64 column), because the incremental per-block summaries folded during writes carry no validity; a nullable column with no nulls keeps that fast path untouched.

Predicates over nulls follow three-valued (Kleene) logic

• A comparison against a null is now unknown, the third value SQL and Arrow both use, and &, |, ^ and ~ combine it by Kleene's rules instead of collapsing it to False at the leaf. t.where(t.price > 10) gives the rows definitely above 10, t.where(~(t.price > 10)) the rows definitely not above 10 — nulls in neither.
• where() keeps what a predicate is true for, so the rows it returns for a plain comparison are unchanged. What this fixes is everything built on top of one: ~(t.price > 10) used to invert a null that had already been collapsed to False and so returned every null row — the exact opposite of the intent — and ~((a > 10) & (b == 999)) dropped rows that qualify, because unknown & false is false, not unknown, and only a real third value can express that. Both query forms are covered and both now agree with SQL: the string form carries the second channel through an AST rewrite under negation.
• A predicate can be asked about its unknown rows rather than only filtered with: p.is_null() gives the rows it cannot answer for, p.null_count() counts them, and t.where(p.fillna(True)) keeps what cannot be ruled out. fillna(False) is the other reading, and is what where() applies implicitly.
• Predicates over non-nullable columns are untouched and cost nothing extra; the result of a nullable comparison is still a blosc2.LazyExpr, so it computes, indexes and plans exactly as before. Measured cost of the exact answer: a negated two-column conjunction over a 20M-row nullable table runs 1.15x slower than the wrong answer it replaces; every other predicate shape is unchanged.
• Two consequences worth knowing. t.where(dict_col != "x") no longer returns the rows where dict_col is null (its reserved code differs from every value's, so it used to match); dict_col == None remains how to ask for them. And Column.isin() stays deliberately two-valued — it returns a materialized array and has its own spelling for nulls (None among the values).

Packaging and other changes

• Wheels are a single Stable ABI (abi3) build per platform. Instead of one wheel per CPython minor version, each platform ships one cp311-abi3 wheel that serves CPython 3.11 and every later version, including ones released after this one. Nothing changes for pip install blosc2; what changes is that a new CPython no longer has to wait for a blosc2 release to be installable from a wheel. CI installs that one wheel on 3.11 through 3.15 and runs a slice of the suite on each, since cibuildwheel only tests a wheel on the interpreter that built it.
• Free-threaded builds are shipped too, as version-specific cp314t and cp315t wheels — the free-threaded stable ABI (abi3t, PEP 803) starts at 3.15 and Cython cannot emit it yet. The limited API costs nothing measurable here: across the Linux and Windows cells of a throwaway benchmark matrix the worst ratio over every benchmark was 1.075x and 1.054x respectively, including the call-heavy ones where an ABI cost would show up first.
• CSV reads and writes take an encoding=. from_csv() defaults to utf-8-sig (plain UTF-8, absorbing a byte-order mark if present) and to_csv() to utf-8, but either can be given another codec, which is what an existing file written in a platform codec needs.
• Three new optimization tips: utf8 versus fixed-width string columns, when a dictionary column pays off, and what the @jit decorator actually buys you (with a worked Mandelbrot benchmark, including the compilation cost).

Bug fixes

• asarray() corrupted arrays whose chunks overhang the shape. Above 16 MB the fill goes chunk by chunk through SChunk.update_data(), and its guard only rejected partitions smaller than their container — so a chunk sticking out of the shape (e.g. shape (146, 23802) with chunks (147, 23802)) passed and got a slice one row short, after which reading the array back failed with "Error while getting the buffer". Thanks to @Zentrik.
• group_by returned the wrong min for a bool value column. The per-group accumulator was seeded from the dtype's opposite identity, and bool had none, so an all-True group reduced to False. Reachable with any plain non-nullable bool column on the generic aggregation path.
• Descending sort_by on a bool column raised, and on a signed-integer column holding its dtype's minimum (-128 for int8) that row sorted as if it were the largest. The descending key negated in the column's own dtype, where bool has no unary minus and a narrow signed type wraps.
• add_column() after copy() backfilled one row short, and raised for a variable-length column: the copy recorded its write watermark one below the convention every other writer follows.
• A string predicate over a nullable column returned its nulls as matches. t.where("a > 10") compared the stored sentinel, so any sentinel satisfying the predicate (null_value=999 against > 10) came back as a match. The operator form (t.where(t.a > 10)) was always correct. Fixed for both storages.
• A nullable uint8 ndarray column came back as bool. The bool → uint8 widening that sentinel storage needs was undone by dtype rather than by whether it had been applied, so a column declared uint8 was truncated to flags.
• ~ on a nullable bool column selected its nulls. SQL WHERE semantics say a null satisfies neither a predicate nor its negation; the mask path inverted the stored False fill instead. (The sentinel path was already correct, via its == 0 rewrite.)
• CSV import and export ignored a validity sidecar. to_csv compared against the sentinel to find nulls, so a mask column wrote its fill as if it were data, and from_csv had nothing to put in an empty field and raised. Both go through the sidecar now: an empty CSV field is a null in either direction. Sentinel columns keep writing their sentinel, unchanged.
• Reductions on a sorted view of a mask column read the wrong rows. The null flags were gathered in the view's order and the values in physical order, so sum() on a sorted view could return NaN from a column whose nulls are not NaN, and unique() could report the fill as data while dropping a real value. Sentinel columns were unaffected.
• convert_nulls() flattened a nested column, even when it had nothing to convert: the schema copy it makes dropped both the table metadata and the logical parent of a nested group, so a struct column came back as its leaves. An in-place conversion on a persistent table wrote that flattened schema to disk. Saving and reopening a nested table dropped the same parent, which is fixed alongside it.
• Descending sort_by mis-ordered the widest integers. The key was built by negating, and negation has a fixed point: int64's minimum sorted as if it were the largest, and a uint64 above 2**63 wrapped negative and sorted below small values.
• CSV was written and read in the platform's locale encoding, so text a column can hold but cp1252 cannot encode — anything outside Latin-1 — raised UnicodeEncodeError on Windows. Both directions are UTF-8 now, and reading absorbs a byte-order mark if one is present.
• A timestamp column could not be written through col[key] = value. A datetime was never encoded to the stored int64, so every key form failed; extend() was unaffected. ISO strings and datetime64 are accepted too.
• Assigning one value to many rows raised on a mask-storage column. col[0:2] = 7 — and the same write through a boolean mask or an index list — failed with TypeError: iteration over a 0-d array, because the write path looked for nulls inside a value that was a single cell rather than a batch. Scalar broadcast works again, and col[0:2] = None now makes every selected row null.
• extend() from another table lost nulls between storages. Copying rows from a mask-backed column into a sentinel-backed one (or the reverse, or between two sentinels reserving different values) wrote whatever stood in for the null as real data. Nullity is translated now.
• A row read showed a mask column's fill instead of None. t[i], iteration and repr surfaced the placeholder that occupies a null slot, which is not part of the format contract — and disagreed with a vlstring column in the same row, which already read None. Sentinel columns still show their sentinel, which is the value you chose.
• to_numpy(masked=True) and dropna() raised on a nullable dictionary column, which reported its nulls per physical slot rather than per live row.
• A null in a complex column reached pandas as nan+0j rather than as missing.
• A nullable ndarray of bool forgot it had been widened when reopened, so converting it to mask storage left it uint8, and the guard against a dtype-changing in-place conversion on a persistent table stopped firing.
• convert_nulls(to="sentinel") refused over a value in a deleted row. The collision check scanned physical slots, so a proposed sentinel present only in a row already deleted — unreadable, and dropped by the next compact() — blocked the conversion. Only live rows are consulted now, and the row named in the refusal is the logical one the caller can index rather than a physical slot.

A correctness release: lazy indexing and reductions now follow NumPy in a batch of cases where they quietly did not, the stores close several cross-process read races, and wheels finally ship usable C-Blosc2 development files. Bundled C-Blosc2 moves to 3.3.2.

Bug fixes

Lazy expressions and indexing

• Indexing a lazy expression with an integer squeezed too much. expr[0] dropped every length-1 axis of the result, including ones the index kept, so a (1, 4) expression indexed at [0] came back as (4,) where NumPy gives (4,) only for the consumed axis and keeps the rest. Only the dimensions the integer indices actually consumed are dropped now. where() results, whose length is data-dependent, are left alone. Closes #319.
• LazyUDF and broadcast operands mis-indexed on None. A None in the key inserts an axis in the result but consumes none in the operand; aligning it as if it did shifted every axis to its left by one, so expr[None, 2] read the wrong operand region. Closes #403, #688.
• Indexing a full reduction materialized the whole operand. (a + b).sum() [key] evaluated the reduction over everything and then indexed; the operands are sliced first now. Closes #457.
• Datetime comparisons in lazy expressions raised. numexpr has no datetime type, so t1 < t2 on datetime64/timedelta64 died with unknown type datetime64[s] and the error was re-raised rather than letting the NumPy fallback try. They are compared as their underlying int64 counts, which is exact, keeping the fast path. Mixed units and NaT decline that route and fall back to NumPy, since raw counts would silently lie about both. Closes #409.
• NDArray.nbytes reported the padded size. It now returns the logical size * itemsize, matching NumPy, whenever the shape does not fill the chunk grid exactly. cratio still measures the stored (padded) data, so nbytes / cbytes need not equal cratio; use .schunk.nbytes for the padded figure. Closes #544.

Tables and stores

• CTable.where() applied a short boolean mask to the wrong rows. A mask shorter than the live-row count was padded out to the physical length, which aligned it with the underlying column and selected rows outside the view. A mask no longer than the live-row count is now treated as logical — entry i selects the i-th live row — with a short one simply leaving the trailing rows unselected. Closes #607.
• Cross-process read races in EmbedStore and DictStore. Both resolved a key under the store lock but read the data after releasing it, so a concurrent writer could be caught mid-mutation: EmbedStore.__getitem__ returned bytes from a stale offset, and DictStore.__getitem__ opened an external leaf that a concurrent overwrite had just removed or half-rewritten (KeyError, or RuntimeError: Error while getting the buffer). The resolve, the existence check and the open now share one lock; non-shared stores skip it entirely. Fixes #691, #692.
• Overwriting an external DictStore leaf is now atomic. __setitem__ removed the old leaf and rebuilt it at the same path, and the handle __getitem__ returns holds no file descriptor — the C layer re-opens the leaf by path for every chunk it decompresses — so a read already in flight could open a truncated file. The new leaf is built beside its final name and moved in with a single os.replace(), so every such re-open sees one complete cframe or the other. A crash mid-write now leaves a stray .tmp staging file rather than a partial leaf.
• TreeStore.close() swallowed inline handle failures. A CTable that failed to flush left an archive whose row count disagreed with a varlen column, reported only on read, long after close() said it succeeded. Every handle still gets a chance to close and the store is still packed; the failure is then re-raised.

Other

• unpack_tensor() turned padding into a phantom column. np.dtype() renames the empty-named padding fields of a structured descr ('' -> f2), so a packed tensor with padding came back with an extra field. Closes #287.

Packaging

• Wheels ship usable C-Blosc2 development files. Install paths are now relative to CMAKE_INSTALL_PREFIX; the absolute ones made C-Blosc2 generate a blosc2.pc and exported targets pointing into the build tempdir, so pkg-config and find_package(Blosc2) both failed against an installed wheel. Verified by building and linking a C program against a wheel three ways: pkg-config, Blosc2::blosc2_shared and Blosc2::blosc2_static. miniexpr's license texts are mirrored into .dist-info/licenses, where PEP 639 tooling looks. Closes #627.
• Wheels carry two copies of libblosc2 instead of three, ~1.5 MB smaller. C-Blosc2 set both VERSION and SOVERSION, and scikit-build-core follows symlinks, so the fully versioned file — referenced by nothing but the symlinks pointing at it — was shipped as a third full copy.

Development

• The test suite runs in parallel by default (-n auto --dist loadfile in pytest.ini), 130s -> 35s locally, falling back to a serial run when pytest-xdist is absent. The heavy tests, 58% of everything collected and excluded from every push-time job, now run in a nightly workflow. The network tests run once per push on a single Linux job instead of five times.
• The ruff rule set is spelled out with select rather than extend-select, so a ruff release widening its defaults no longer redefines what CI enforces.

This release is the string-support milestone: string expressions and DSL kernels now run on miniexpr, utf8() and dictionary() columns gain full indexing, comparisons, and a documented conversion pair, and NumPy's StringDType is understood by the array constructors. Alongside, slicing with plain keys is up to 1.7x faster, a new blosc2.random module provides chunk-parallel NumPy-quality random constructors, and the optimization-tips guide gained two new tips and refreshed figures.

New features

• String-valued expressions and DSL kernels over fixed-width <Un arrays now run on miniexpr instead of falling back to NumPy. Concatenation (arr + "suffix", "prefix=" + arr) plus lower, upper, strip/lstrip/rstrip, removeprefix, removesuffix, replace, substr and split_part all produce string results, and @blosc2.dsl_kernel accepts method syntax (name.lower()) and tuple unpacking (before, after = desc.split(sep, 1)), which are rewritten to the DSL grammar. The output width is inferred by miniexpr and the container is allocated from it, so nothing truncates — .dtype may be wider than NumPy's exact answer, never narrower.

• Bytes (S) arrays go through the same engine, with NumPy's S semantics rather than <U's: ASCII-only case mapping (so upper/lower keep the width instead of growing) and ASCII-only stripping. S and <U operands do not mix in one expression, which is what NumPy does too. Variable-width utf8() columns still use the NumPy path.

• String expressions now work on utf8() columns. t.where("name == 'x'"), startswith/endswith/contains and mixed predicates such as t.where("(name == 'b') | (x > 2)") used to raise NotImplementedError; only the operator form t[t.name == "x"] was available. A variable-length column cannot be an expression operand (its offsets and data have independent chunk grids, so the prefilter contract does not apply), so these are evaluated span by span, each span materialized to a fixed-width array whose width is rounded up to a power of two and handed to miniexpr. Nulls are materialized to "" before any kernel sees them and re-masked afterwards, so a null never satisfies a predicate — the same answer the operator form gives.

• Scalar comparisons on utf8() columns are 5-6x faster in expression form. t.where("name == 'x'") (and !=, <, <=, >, >=, either operand order) is now answered by the same raw-byte scan the operator form t[t.name == "x"] uses, instead of decoding the column to fixed-width first: 156 -> 28 ms over 1M short values, 268 -> 56 ms over 1M ~31-byte values. Mixed expressions get whatever they can -- in startswith(name, 'x') | (name == 'zz') the comparison takes the fast path and startswith still decodes.

• New blosc2.utf8_array(seq, spec=None) builds a UTF8Array from an iterable of strings; UTF8Array is exported too. Previously the only construction path was UTF8Array(spec) + .extend() + .flush(), which was not exported at all.

• df.apply(f, axis=1, engine=blosc2.jit) now runs row["colname"] kernels that contain an if. Neither dispatch route could before: tracing evaluated the branch over a whole column (truth value ... is ambiguous) and the DSL parser rejected the subscript. Such references are now rewritten into named parameters, so the function is compiled and every branch runs. This is not string-specific — numeric row kernels with a branch were equally blocked. String columns reach this route too, which makes the pandas-3 "format room info" kernel run unmodified. Nulls in a string column are rejected rather than substituted, since a row-wise kernel over a null raises in pandas as well.

• New blosc2.random module: seedable, NumPy-quality random NDArray constructors. Each chunk gets its own independent SeedSequence-spawned stream and is generated concurrently in a thread pool, giving full PCG64 quality with genuinely parallel generation (measured ~3x faster than asarray(np.random.default_rng(...).random(...)) on a 100M-element array). Covers 42 of numpy.random.Generator's 43 public methods (full compatibility table in doc/reference/random.rst):

  • Core: random, integers, normal, uniform, choice (replace=True only).
  • 30 scalar distributions: beta, binomial, chisquare, exponential, f, gamma, geometric, gumbel, hypergeometric, laplace, logistic, lognormal, logseries, negative_binomial, noncentral_chisquare, noncentral_f, pareto, poisson, power, rayleigh, standard_cauchy, standard_exponential, standard_gamma, standard_normal, standard_t, triangular, vonmises, wald, weibull, zipf.
  • 4 vector-valued distributions (output shape shape + (k,), one draw per trailing vector): dirichlet, multinomial, multivariate_hypergeometric, multivariate_normal.
  • permutation, permuted, shuffle: unlike the rest of the module, these are not chunk-parallel — whole-array shuffling is inherently sequential, so they materialize the full array and shuffle it single-threaded. shuffle additionally requires its argument to already be an NDArray, since it mutates in place and returns None, matching numpy.
  • Not implemented: bytes (returns raw bytes, not an NDArray).

• create_index() now works on utf8() columns, the last string flavour without one. Both utf8 and dictionary are indexed by the alphabetical rank of each value: sorting by rank is sorting by the decoded string, so an int32 rank column drives the same machinery a numeric column uses. At 1M rows / cardinality 20k: sort_by 424 ms -> 7.2 ms, sorted_slice 458 ms -> 43 ms, and the index is the cheapest of the three flavours to build (277 ms against 867 ms for <U). Scalar comparisons are served from it too — utf8 == 29.0 ms -> 5.5 ms, < 34.6 ms -> 5.5 ms, and the dictionary operator form t[t.c == v] 329.6 ms -> 8.4 ms. startswith/substring searches are not accelerated (no index covers them), and ranks are frozen at build time, so a value inserted ahead of existing ones sends the index stale until it is rebuilt.

• CTable.add_column() accepts values=, a sequence with one entry per live row, as an alternative to backfilling from a declared default. This is the supported way to land a result computed outside the table back into it, which matters most for utf8() columns: string-returning expressions are evaluated on fixed-width arrays, and the result previously had to be written through the private t._cols[name].set_all(...). A declared default is still honoured for rows appended later, so the two can be combined. values= is checked against the constraints declared on the spec, like the constructor and extend() are: without that, coercion to a fixed-width dtype would truncate an over-long string to max_length instead of complaining.

• blosc2.from_utf8() / blosc2.to_utf8() and UTF8Array.astype() make the conversion between variable-length and fixed-width text an explicit, documented pair. utf8 columns store and filter text compactly, but string-returning expressions need miniexpr's compile-time output width, so they run on fixed-width arrays; the rule is now written down (see "Computing strings on a utf8 column" in the CTable reference) rather than left for callers to discover. from_utf8() sizes the result to the longest value in codepoints, counted from the raw bytes without decoding a row, so nothing truncates and non-ASCII text does not over-allocate the 3-4x a byte-length bound would.

• The array constructors dispatch on NumPy's StringDType. blosc2.asarray(np.array([...], dtype=StringDType())) used to raise TypeError: data type 'StringDType()' not understood, and blosc2.zeros(n, dtype=StringDType()) a malformed node ValueError; both now return a UTF8Array, as do empty, ones and full, with the same fill values NumPy uses ('', '', '1', str(fill_value)). The dispatch is on the target dtype, so asarray(utf8_source, dtype="<U8") still gives a fixed-width NDArray. StringDType still cannot back an NDArray — it keeps each row's payload outside the array buffer and offers no buffer protocol, so compressing that buffer would persist pointers — which is why the variable-length container is what comes back.

• UTF8Array gained .shape, .ndim, .size and __array__, so it now satisfies the blosc2.Array protocol (.shape was the only member it lacked) and np.asarray(arr) returns StringDType instead of silently widening to a fixed-width <Un — for 200-character values that was 1600 bytes where the payload is 203, and a different dtype than arr[:] reported for the same object.

• Column.assign() works on utf8, vlstring, vlbytes, struct and object columns. It previously raised TypeError: UTF8Array assignment index must be int, leaving no public way to overwrite a variable-length column's values. These are now rewritten whole (one write per backing batch) rather than row by row, which for the batched varlen columns would have rewritten a whole batch per row.

• @blosc2.jit dispatches control flow to the DSL, and the DSL engine widened its operand and function coverage. miniexpr's prefilter now gathers blocks directly from raw NumPy buffers instead of converting operands with asarray(), and Series are accepted as DSL operands. np.foo(...) calls inside a jitted function are rewritten to the bare names miniexpr recognizes, np.sign is supported, and np.square, np.negative, np.positive and np.reciprocal are dispatched. A configured blosc2.jit(...) is accepted as a pandas engine.

• C-Blosc2 bumped to 3.3.1 (bundled; min version 3.3.0).

Improvements

• Dictionary columns decode once per read, not once per row. Each dict_store[code] decompresses a whole msgpack batch, so reads and lexsort-based sort_by cost O(N) decompressions. At 1M rows an unindexed sort_by drops from 236 s to 713 ms, and a full column read from 44 s (at 200k rows) to 193 ms.
• sort_by on a dictionary column sorts int32 ranks, not decoded strings. A row's alphabetical rank orders exactly as its value does — the trick the FULL index already used — so the sort key needs neither the decode nor lexsort's string comparisons. Key construction drops from 106 ms to 21 ms per 200k rows (cardinality 5000) and sort_by(view=True) from 247 ms to 157 ms. The filtered small-copy path, which had its own copy of the key builder, now shares this one and picks up the same speedup.
• kind=BUCKET indexes no longer cost more than the scan they replace. Scattered matches were read one bucket run at a time, re-decompressing the same blocks many times, and the planner measured selectivity in buckets while the cost is paid in blocks — a mask selecting 21% of buckets could touch 96% of them. Affected every indexable dtype; the relative penalty was worst on numerics (float64 6.3 ms -> 77.9 ms before, 6.6 ms after).
• The utf8 compute refusals now route instead of merely refusing. Every path that cannot take a utf8 column — add_computed_column, add_generated_column, assign, apply, lazyudf, with a string expression or a DSL kernel — raises NotImplementedError naming the column and printing the three-line conversion, echoing the user's own expression where there is one. Two of those paths previously failed with a raw NumPy DTypePromotionError and a ValueError: malformed node or string ... StringDType(), neither of which named the column or the fix.
• Slicing with plain slice/int keys is up to 1.7x faster. process_key() used to route every key through ndindex's general index machinery, ~50x slower than needed for the common arr[2:7, :50]-style tuple: in the scattered-read benchmark it accounted for 43% of the loop. Plain tuples (and bare scalars) of slices and ints are now normalized directly — padding short tuples, converting ints to slice(k, k+1) with negative-index and out-of-bounds handling identical to ndindex's — and everything else (strided or negative steps, ellipsis, newaxis, fancy arrays) still falls through to ndindex. The optimization-tips guide's scattered-read tip went from 0.627 s to 0.363 s on the plain-open variant.

Bug fixes

• @blosc2.jit raised when a storage kwarg and an execution-tuning kwarg were combined and the decorated function returned a NumPy array — @blosc2.jit(jit=False, cparams=...) ended in blosc2.asarray(retval, jit=False, ...), which rejects the tuning kwargs. Only storage kwargs reach asarray() now; the function has already run, so there is nothing left to tune.
• A DSL kernel over a utf8 column registered as a computed column, then broke the table. add_computed_column(name, kernel, inputs=["utf8_col"]) was accepted, after which every read of that column and str(table) raised ValueError: malformed node or string. The kernel is now refused at registration, where the table is still untouched.
• min()/max() read from a column index returned the wrong value. Two independent causes, both affecting every indexable dtype. The block summaries cover the column's physical extent, so the capacity padding (zeros, empty strings) was reduced along with the data and min() reported it — wrong on any table whose row count is not exactly its slot capacity. And delete() bumps a visibility epoch that nothing recorded, so deleted rows kept contributing their values to the block they sat in. Whole blocks below the live row count are still read from the sidecar; the block straddling the boundary is now rescanned, and a deletion since the index was built makes the shortcut stand down.
• create_index on utf8() and dictionary() columns accepted any index kind and built one over the alphabetical ranks that no query would ever consult — only IndexKind.FULL reaches a rank index. kind now defaults to FULL for these two column kinds (BUCKET elsewhere, unchanged) and raises ValueError when another kind is requested explicitly. Previously create_index("category") on a dictionary column built an unused BUCKET index by default.
• Comparison operators on UTF8Array, dictionary and varlen scalar columns returned a plain False: none defined them, so column == "value" fell through to object identity. Silently wrong rather than an error. All now return boolean masks; UTF8Array and DictionaryColumn answer a scalar without decoding any row.
• dictcol != value raised IndexError on any table with capacity padding: the negation was applied after the live-row intersection, turning every dead slot True.
• Expressions over a bare UTF8Array (blosc2.lazyexpr("'x=' + a", {"a": arr})) produced correct values down the wrong path — widened to fixed-width <Un and evaluated by the NumPy fallback, never reaching miniexpr, ignoring the span budget and losing the utf8 container. They now run through the span driver and return a UTF8Array.
• add_column() on a varlen column left it short on tables with deleted rows. vlstring/vlbytes/utf8/struct/object columns are indexed by physical position but the new column was filled with only as many entries as there were live rows, so the first read after a delete() raised IndexError. The dead slots are now filled too. add_column() on a dictionary() column raised AttributeError from deep inside the fixed-width path; it now raises TypeError naming the limitation.
• blosc2.utf8(null_value="\x00") is rejected. NumPy does not match a lone NUL against a StringDType array ("\x00x" and "a\x00b" compare fine), so every null mask would silently stop marking nulls. The default sentinel '__BLOSC2_NULL__' was never affected.
• Nested (dotted) utf8() leaves can be filtered. t.where("trip.name == 'x'") raised NotImplementedError on a utf8 leaf, while the same query on a <Un, bytes() or dictionary() leaf worked — utf8 was the only flavour where a dotted name could not be queried at all. Dotted names are aliased to safe identifiers before evaluation, but utf8 columns are outside the operand namespace, so they never reached that rewrite; they are now aliased by the utf8 driver itself. Covers scalar comparisons, startswith/upper and friends, mixed numeric predicates and sum(where=).
• SChunk slices were broken for typesizes above 255 bytes. c-blosc2's blosc2_schunk_get_slice_buffer() derives the getitem for a partially covered chunk by dividing byte offsets by schunk->typesize, which the chunk header contradicts once the typesize is capped: the unit changes silently with the data, so blocks past the first could come back as uninitialised memory. Reachable from ordinary data — an <U64 NDArray is a 256-byte typesize, so arr.schunk[1:4] hit it. Partial reads now count in bytes via blosc2_getitem_bytes_ctx() (upstream c-blosc2 fix for Blosc/c-blosc2#796, included in the bundled 3.3.1), which is unambiguous at any typesize.
• create_index() on a string column made every query on it return zero rows. Silently — adding an index, an optimization, changed the answer. A segment summary is a (min, max, flags) record, so a <Un column makes it 8n + 1 bytes: 257 for max_length=32, which is the default width for a plain str annotation. Past 255 bytes c-blosc2 records the chunk typesize as 1, and the sidecar reader asked for spans in element units, so summaries decoded to garbage and pruned every candidate away. The boundary is exactly 8*max_length + 1 > 255 (31 works, 32 does not); summary, bucket, partial and full indexes were affected, opsi was not. A short span read now raises instead of leaving the destination partly uninitialised.
• Scalar bools inside tuple keys now match NumPy. a[(True, :)] raised ValueError from the fancy-index path, which cannot handle the 0-d bool array ndindex expands tuple bools to. NumPy treats them as a single np.newaxis-like dimension of length 1 (all True) or 0 (any False), placed at the first bool's position with multiple bools collapsing; __getitem__ now rewrites the first bool to None and drops the rest, covering bool, np.bool_ and 0-d bool arrays. Bare a[True] was already correct.
• Slices with start > stop crashed with ValueError: negative dimensions are not allowed. The slicing fast path normalized slices with slice.indices(), which leaves a[100:50] as (100, 50); ndindex clamps any empty positive-step slice to (0, 0), and without that clamp the result shape went negative. The fast path now clamps empty slices exactly like ndindex.
• blosc2.pack_tensor() failed on 0-dim arrays (e.g. a scalar np.array(17)); 0-dim inputs are now packed correctly.
• SChunk.meta.get(key, default) recursed forever when the key was absent; it now returns the default.
• Pinned miniexpr fixes: complex division with a scalar operand returned wrong results, expressions whose evaluation block was exactly 4096 elements could be mis-evaluated, and prefilter reads for operands wider than 255 bytes read the wrong data. String expression results also keep the code-unit shuffle width (the UCS4 shuffle for <Un, instead of silently falling back to byte shuffle), upper/lower preserve the width, and a string result containing the utf8 null sentinel is refused rather than corrupting null masks.
• blosc2.random constructors with NumPy integer scalar arguments (e.g. rng.integers(np.int64(3))) no longer produce wrong shapes.
• engine=blosc2.jit with Series operands no longer crashes on axis=1, and an already-jitted function is not jitted a second time.

Documentation

• Two new optimization tips joined doc/guides/optimization_tips.md: generating arrays with DSL kernels, and broadcasting small operands into large on-disk arrays; the guide's figures were regenerated against the reference machine and the benchmark harness now checks c-blosc2's pread/handle-cache work.
• The pandas engine guide was rewritten around the readable-UDF pitch (with a Kepler plot), the pandas guide around one question, and blosc2.argsort is now documented.

Changes from 4.8.1 to 4.9.0

This release is about cooperation: CTable now speaks the tabular
ecosystem's own protocols instead of asking it to speak blosc2's. Arrow
tools (pyarrow, DuckDB, Polars, and pandas >= 3.0 via DataFrame.from_arrow())
can consume or produce a CTable directly through the Arrow PyCapsule
interface; a new utf8() string column stores text in Arrow's own
offsets+bytes layout and reads back as NumPy StringDType; and
engine=blosc2.jit now runs correctly inside pandas 3 itself. Alongside
that, CTable gained a proper missing-data story (fillna/dropna,
null-safe arithmetic and comparisons) and a pandas-3-style chaining API
(assign()/col(), UDF aggregations, CTable.apply()).

New features

Ecosystem interop

• Arrow PyCapsule interchange: CTable.__arrow_c_stream__ lets
  pyarrow, DuckDB, and Polars consume a CTable directly as a stream of
  record batches, with bounded memory — no to_arrow()/copy step
  required. pandas >= 3.0 can do the same via the new
  pandas.DataFrame.from_arrow() classmethod (the plain pd.DataFrame(t)
  constructor does not use this protocol). CTable.from_arrow() now
  accepts any object implementing the same protocol on ingest
  (single-argument form), in addition to the existing (schema, batches)
  form — including Arrow's string_view layout (Polars' default string
  export type), which raised TypeError before this release.
• blosc2.utf8(): a new column type for high-cardinality/free-text
  strings, storing each column as two companion NDArrays — int64 row
  offsets plus a UTF-8 byte blob — the same layout Arrow uses for
  large_string. A row costs exactly its encoded byte length (7-13x
  smaller uncompressed than fixed-width string() on high-cardinality
  text), and reads materialize as NumPy StringDType arrays (NumPy >=
  2.0 required; older NumPy falls back to vlstring on Arrow/Parquet
  import with a clear message). Full query surface: comparisons,
  where(), sort_by, group_by keys, fillna, and Arrow export
  (large_string, sentinel-null mask) / import (Arrow/Parquet string
  columns now default to utf8 instead of vlstring). Measured on the
  1e7-row NYC-taxi company column: ingest 3622 ms -> 598.6 ms
  (6.05x faster, only 1.22x slower than string() and 2.63x faster
  than vlstring()), full-column read 2472.6 ms -> 165.3 ms (14.96x
  faster), equality filter ~1900 ms -> 162 ms (~11.7x faster), and
  groupby-key factorization 3304 ms -> 558 ms (2.83x, down from a
  16.9x-slower initial fallback, now faster than fixed-width string()
  keys). See the new "Choosing a string column type" guide in the
  CTable reference and examples/ctable/utf8_strings.py. Known gaps:
  string-expression filters (t.where("name == 'x'")) and
  create_index on utf8 columns still raise a clear
  NotImplementedError; use fixed-width string() if you need those.
• pandas engine, pandas 3 compatible: engine=blosc2.jit for
  DataFrame.apply now returns a properly indexed DataFrame/Series
  under pandas 3.0.3's default raw=False (previously a raw NumPy
  array, so results only matched by value, never by type).
  Series.map(func, engine=blosc2.jit) is now implemented (it
  previously always raised NotImplementedError). Non-numeric columns
  now raise a clear ValueError instead of a deep numexpr error. See
  the guide "Using Blosc2 as a pandas engine" and
  bench/bench_pandas_engine.py.

pandas-style CTable API

• CTable.assign(**named_exprs): return a view with additional computed
  columns, without mutating the table or copying column data. Pairs with
  the new blosc2.col(name) — an unbound column expression that defers
  operator replay until it's bound to a table (assign(), t[...],
  where()) — to write pandas-3-style chains:
  t.assign(profit=col("revenue") - col("cost"))[col("profit") > 0].sort_by("profit", ascending=False).head(10).
• Missing data: Column.fillna(value) replaces sentinel/None
  values for scalar, dictionary, and varlen-scalar columns;
  CTable.dropna(subset=None) returns a view excluding rows where any
  nullable column (or a chosen subset) is null. Column arithmetic
  (+ - * / // % **) and comparisons (< <= > >= == !=) on nullable
  int/timestamp/bool columns now propagate nulls instead of operating on
  the raw sentinel: arithmetic promotes to float64/NaN, comparisons
  follow SQL WHERE semantics (a null operand never satisfies any
  comparison) — fixing filters like t[t.x < 0] wrongly matching null
  rows. Also fixes null detection for timestamp columns
  (is_null()/null_count()/dropna() previously missed every
  NaT).
• UDF aggregations: group_by().agg() accepts a custom callable as
  the op via the named form (output_name=(column, callable[, dtype]));
  it receives each group's live, non-null values as a 1-D NumPy array.
  CTable.apply(func, columns=None, dtype=None, engine="auto") applies
  a UDF across the table's live rows, sugar over blosc2.lazyudf().
  group_by(engine=...) now accepts "auto"/"numpy" explicitly
  alongside the existing default.
• CTable views are now read-only for value writes: Column.__setitem__
  and Column.assign() raise ValueError on a view, pointing at
  take()/copy() as the escape hatch (structural mutations already
  raised; this closes the one unguarded cell-write path).
• NDArray.iter_sorted()/argsort() on a FULL-indexed array now reads
  the sidecar range directly instead of building the full permutation —
  ~52x faster and ~193x less memory on a 20M-element array for
  iter_sorted(start=-k)-style tail queries. See the new optimization
  tip.

Improvements

• String-key group_by() (fixed-width string() keys) now factorizes
  via an exact hash of each row's raw bytes instead of a NumPy
  UTF-32 argsort, with a vectorized collision-checked verify pass
  keeping the result bit-identical: 1157 ms -> 737 ms on a 1e7-row
  benchmark. Every caller benefits automatically; no engine= switch
  involved.
• An example showing ctables and ndarrays bundled together in the same
  TreeStore.
• C-Blosc2 bumped to 3.2.3.

Bug fixes

• Fixed a @blosc2.dsl_kernel-decorated function crashing unconditionally
  when passed as a groupby UDF aggregation (g.agg(name=(col, dsl_kernel_fn))): it now runs like the equivalent undecorated callable.
• Fixed CTable.head()/tail() silently discarding row order when called
  on a lazily-sorted view (e.g. t.sort_by("col", ascending=False) on a
  view, or any .sort_by() result chained off a prior filter): they
  ignored _cached_live_positions and built a plain physical-order mask
  instead, so t.where(...).sort_by("x", ascending=False).head(10) came
  back in the wrong order.
• Fixed a NameError when nan/inf scalars appeared in lazy
  expressions; ShapeInferencer no longer ignores user-provided shapes
  for nan/inf.
• Fixed CTable.from_arrow() raising TypeError: No blosc2 spec for Arrow type DataType(string_view) on any Arrow string_view/binary_view
  column — the layout Polars exports by default through the PyCapsule
  protocol. string_view/binary_view now import exactly like
  string/large_string/binary/large_binary everywhere a column's
  Arrow type is inspected (schema inference, null-sentinel selection,
  list/struct/dictionary value types).

A small hot-fix release for the Arrow interop work in 4.9.0: a real performance regression in dictionary-column export, and a clearer error message when opening a nonexistent CTable in append mode.

Improvements

• CTable.iter_arrow_batches() (and therefore to_arrow() and the Arrow PyCapsule interchange, __arrow_c_stream__) no longer recomputes the full live-row-position array from scratch on every batch, for every dictionary column — an O(n_rows) scan that was repeated O(n_rows / batch_size) times. The position array is now computed once per export call instead. Measured 6-14x faster export for dictionary-encoded string columns (e.g. company) on a 1M-row table.
• Reminder for anyone consuming a CTable through the Arrow PyCapsule protocol (DuckDB, pyarrow, Polars, pandas): the raw Arrow C Stream interface has no column-projection pushdown, so a consumer that only needs a few columns still triggers export of every column in the table. Use CTable.select([...]) to project down to the columns you actually need before handing the table to the consumer, particularly if any column is an expensive nested/list type.

Bug fixes

• SChunk slices were broken for typesizes above 255 bytes. c-blosc2's blosc2_schunk_get_slice_buffer() derives the getitem for a partially covered chunk by dividing byte offsets by schunk->typesize, which the chunk header contradicts once the typesize is capped. Across 153 slice shapes at typesize 256, 150 raised "Error while getting the slice" and the 3 single-element ones returned the wrong bytes with no error at all. Reachable from ordinary data -- an <U64 NDArray is a 256-byte typesize, so arr.schunk[1:4] hit it. Fixed upstream in Blosc/c-blosc2#796, so this release requires a c-blosc2 that carries that fix.
• create_index() on a string column made every query on it return zero rows. Silently -- adding an index, an optimization, changed the answer. A segment summary is a (min, max, flags) record, so a <Un column makes it 8n + 1 bytes: 257 for max_length=32, which is the default width for a plain str annotation. Past 255 bytes c-blosc2 records the chunk typesize as 1, and the sidecar reader asked for spans in element units, so summaries decoded to garbage and pruned every candidate away. The boundary is exactly 8*max_length + 1 > 255 (31 works, 32 does not); summary, bucket, partial and full indexes were affected, opsi was not. A short span read now raises instead of leaving the destination partly uninitialised.
• Expressions over operands wider than 255 bytes returned wrong results. c-blosc2 caps a typesize above 255 to 1 in the chunk header so its split machinery keeps working, and the miniexpr prefilter was asking blosc2_getitem_ctx() for operand blocks in element units, which the chunk then read as a byte range: every block past the first was uninitialised memory. arr == "hello" over 1200 rows of <U64 (256 bytes) matched 1 row instead of 400, non-deterministically and with no error raised. Affects any dtype whose itemsize exceeds 255 bytes; <U64 is the first fixed-width string that hits it.
• Opening a CTable with mode="a" at a path that doesn't exist yet now raises a clear FileNotFoundError ("mode='a' opens an existing table; use mode='w' to create a new one") instead of silently falling through and creating a new, empty table.

This release is about cooperation: CTable now speaks the tabular ecosystem's own protocols instead of asking it to speak blosc2's. Arrow tools (pyarrow, DuckDB, Polars, and pandas >= 3.0 via DataFrame.from_arrow()) can consume or produce a CTable directly through the Arrow PyCapsule interface; a new utf8() string column stores text in Arrow's own offsets+bytes layout and reads back as NumPy StringDType; and engine=blosc2.jit now runs correctly inside pandas 3 itself. Alongside that, CTable gained a proper missing-data story (fillna/dropna, null-safe arithmetic and comparisons) and a pandas-3-style chaining API (assign()/col(), UDF aggregations, CTable.apply()).

New features

Ecosystem interop

• Arrow PyCapsule interchange: CTable.__arrow_c_stream__ lets pyarrow, DuckDB, and Polars consume a CTable directly as a stream of record batches, with bounded memory — no to_arrow()/copy step required. pandas >= 3.0 can do the same via the new pandas.DataFrame.from_arrow() classmethod (the plain pd.DataFrame(t) constructor does not use this protocol). CTable.from_arrow() now accepts any object implementing the same protocol on ingest (single-argument form), in addition to the existing (schema, batches) form — including Arrow's string_view layout (Polars' default string export type), which raised TypeError before this release.
• blosc2.utf8(): a new column type for high-cardinality/free-text strings, storing each column as two companion NDArrays — int64 row offsets plus a UTF-8 byte blob — the same layout Arrow uses for large_string. A row costs exactly its encoded byte length (7-13x smaller uncompressed than fixed-width string() on high-cardinality text), and reads materialize as NumPy StringDType arrays (NumPy >= 2.0 required; older NumPy falls back to vlstring on Arrow/Parquet import with a clear message). Full query surface: comparisons, where(), sort_by, group_by keys, fillna, and Arrow export (large_string, sentinel-null mask) / import (Arrow/Parquet string columns now default to utf8 instead of vlstring). Measured on the 1e7-row NYC-taxi company column: ingest 3622 ms -> 598.6 ms (6.05x faster, only 1.22x slower than string() and 2.63x faster than vlstring()), full-column read 2472.6 ms -> 165.3 ms (14.96x faster), equality filter ~1900 ms -> 162 ms (~11.7x faster), and groupby-key factorization 3304 ms -> 558 ms (2.83x, down from a 16.9x-slower initial fallback, now faster than fixed-width string() keys). See the new "Choosing a string column type" guide in the CTable reference and examples/ctable/utf8_strings.py. Known gaps: string-expression filters (t.where("name == 'x'")) and create_index on utf8 columns still raise a clear NotImplementedError; use fixed-width string() if you need those.
• pandas engine, pandas 3 compatible: engine=blosc2.jit for DataFrame.apply now returns a properly indexed DataFrame/Series under pandas 3.0.3's default raw=False (previously a raw NumPy array, so results only matched by value, never by type). Series.map(func, engine=blosc2.jit) is now implemented (it previously always raised NotImplementedError). Non-numeric columns now raise a clear ValueError instead of a deep numexpr error. See the guide "Using Blosc2 as a pandas engine" and bench/bench_pandas_engine.py.

pandas-style CTable API

• CTable.assign(**named_exprs): return a view with additional computed columns, without mutating the table or copying column data. Pairs with the new blosc2.col(name) — an unbound column expression that defers operator replay until it's bound to a table (assign(), t[...], where()) — to write pandas-3-style chains: t.assign(profit=col("revenue") - col("cost"))[col("profit") > 0].sort_by("profit", ascending=False).head(10).
• Missing data: Column.fillna(value) replaces sentinel/None values for scalar, dictionary, and varlen-scalar columns; CTable.dropna(subset=None) returns a view excluding rows where any nullable column (or a chosen subset) is null. Column arithmetic (+ - * / // % **) and comparisons (< <= > >= == !=) on nullable int/timestamp/bool columns now propagate nulls instead of operating on the raw sentinel: arithmetic promotes to float64/NaN, comparisons follow SQL WHERE semantics (a null operand never satisfies any comparison) — fixing filters like t[t.x < 0] wrongly matching null rows. Also fixes null detection for timestamp columns (is_null()/null_count()/dropna() previously missed every NaT).
• UDF aggregations: group_by().agg() accepts a custom callable as the op via the named form (output_name=(column, callable[, dtype])); it receives each group's live, non-null values as a 1-D NumPy array. CTable.apply(func, columns=None, dtype=None, engine="auto") applies a UDF across the table's live rows, sugar over blosc2.lazyudf(). group_by(engine=...) now accepts "auto"/"numpy" explicitly alongside the existing default.
• CTable views are now read-only for value writes: Column.__setitem__ and Column.assign() raise ValueError on a view, pointing at take()/copy() as the escape hatch (structural mutations already raised; this closes the one unguarded cell-write path).
• NDArray.iter_sorted()/argsort() on a FULL-indexed array now reads the sidecar range directly instead of building the full permutation — ~52x faster and ~193x less memory on a 20M-element array for iter_sorted(start=-k)-style tail queries. See the new optimization tip.

Improvements

• String-key group_by() (fixed-width string() keys) now factorizes via an exact hash of each row's raw bytes instead of a NumPy UTF-32 argsort, with a vectorized collision-checked verify pass keeping the result bit-identical: 1157 ms -> 737 ms on a 1e7-row benchmark. Every caller benefits automatically; no engine= switch involved.
• An example showing ctables and ndarrays bundled together in the same TreeStore.
• C-Blosc2 bumped to 3.2.3.

Bug fixes

• Fixed a @blosc2.dsl_kernel-decorated function crashing unconditionally when passed as a groupby UDF aggregation (g.agg(name=(col, dsl_kernel_fn))): it now runs like the equivalent undecorated callable.
• Fixed CTable.head()/tail() silently discarding row order when called on a lazily-sorted view (e.g. t.sort_by("col", ascending=False) on a view, or any .sort_by() result chained off a prior filter): they ignored _cached_live_positions and built a plain physical-order mask instead, so t.where(...).sort_by("x", ascending=False).head(10) came back in the wrong order.
• Fixed a NameError when nan/inf scalars appeared in lazy expressions; ShapeInferencer no longer ignores user-provided shapes for nan/inf.
• Fixed CTable.from_arrow() raising TypeError: No blosc2 spec for Arrow type DataType(string_view) on any Arrow string_view/binary_view column — the layout Polars exports by default through the PyCapsule protocol. string_view/binary_view now import exactly like string/large_string/binary/large_binary everywhere a column's Arrow type is inspected (schema inference, null-sentinel selection, list/struct/dictionary value types).

Improvements

• Read-only memory mapping for CTable stores: CTable.open() (and FileTableStorage) gain an mmap_mode="r" parameter, mirroring blosc2.open(). All members of a read-only store — scalar, list, varlen and dictionary columns alike — are then read from mapped pages; for .b2z archives, in place at their offsets inside the single mapped container file. With several concurrent readers on one file this pays off quickly: 2.5x/4.4x/4.5x faster wall time for 1/4/8 readers in our benchmark (bench/optim_tips/tip_10_mmap_many_readers.py).
• Reduced memory consumption in CTable.extend() when passed an NDArray.
• C-Blosc2 bumped to 3.2.2.

Bug fixes

• Fixed lifetime/use-after-free hazards around zero-copy cframes and vlmeta: schunk_from_cframe()/ndarray_from_cframe() with copy=False (the default) returned objects pointing into the caller's bytes buffer without keeping it alive, so a temporary cframe (e.g. ndarray_from_cframe(response.content)) could be reclaimed under the live object, corrupting reads. The buffer is now pinned on the returned object. Also, vlmeta read paths (__getitem__/__len__/__iter__) now raise ReferenceError on an orphaned owner instead of segfaulting, matching the write paths.
• BatchArray.delete() / ObjectArray.delete(): negative-step slices (e.g. del arr[3:0:-1]) deleted chunks in ascending order, shifting the indices of chunks still to be deleted and removing the wrong ones (or raising RuntimeError).
• ListArray.extend_arrow(): Arrow chunks were appended to the backend without flushing pending cells first, reordering unflushed rows after the new ones.
• Shape inference: stack() with a negative axis inserted the new dimension one position too early, so a lazyexpr's reported .shape disagreed with its computed result; vecdot() also normalized positive axes as if they were negative.
• Chunked matmul(): broadcast (size-1) operand batch dims were sliced with the result-chunk coordinates, producing empty slices when the broadcast dim spans several result chunks.
• DictStore.__setitem__(): overwrite semantics depended on value size — embedded keys refused overwrite ("already exists"), while an embedded-to-external overwrite double-stored the key and resurrected the stale embedded value after a delete. Assignments now behave uniformly dict-like, dropping any previous value.
• Explicitly passing cparams=None or dparams=None to NDArray constructors crashed; both now mean "defaults".
• Fixed a builtin-shadowing bug that made store detection in blosc2.open() recurse ~250 times (silently swallowed) on every .b2z/.b2d open; opening a .b2z is now ~10x faster under allocation tracing.

Documentation

• Restructured docs (#674), with a new Optimization tips section, including new tips on grouping related data into a single memory-mapped .b2z file and on using mmap_mode="r" with many concurrent readers.

Sharing containers across processes

• New locking storage parameter (and the BLOSC_LOCKING environment variable to enable it fleet-wide) serializes accesses to an on-disk SChunk/NDArray/EmbedStore/DictStore against other handles and other processes, via a small sidecar lock file (.b2lock). Advisory: every handle touching the container must opt in.
• SChunk.holding_lock() / NDArray.holding_lock(): a context manager to hold the exclusive lock across several operations, making a multi-step mutation atomic to other locked handles.
• New SChunk.refresh(), mirroring the existing NDArray.refresh().
• Fixed a data-loss bug in NDArray.append(): it read the cached, unrefreshed shape before computing the resize target, so under concurrent growth/shrink — even inside holding_lock() — another writer's just-appended data could be silently deleted.
• EmbedStore and DictStore (.b2d) now support cross-process writers under locking: transactional writes plus key-map re-sync, so readers follow keys added or removed by another process.
• DictStore.to_b2z() (and TreeStore, which inherits from it) now replaces the target file atomically, so concurrent readers always see either the old or the new archive, never a torn one.
• Growth-SWMR (single writer, multiple readers): a reader NDArray handle opened before a resize() made through another handle follows the new shape on its next data access, or via the new explicit NDArray.refresh().
• New user guide page, Sharing containers across processes, covering all of the above plus the caveats (NFS, mmap_mode, Windows in-use-file rename).

Bug fixes

• Fixed detect_aligned_chunks() (used internally to fast-path aligned slice reads/writes): a floor-division undercounted the chunk grid for arrays whose shape isn't a multiple of the chunk shape, which could silently return the wrong chunk's data for an otherwise-aligned slice with a nonzero start in an earlier dimension.

Others

• Raised the manylinux wheel baseline from manylinux2014 (CentOS 7, glibc 2.17, GCC 10.2) to manylinux_2_28 (AlmaLinux 8, glibc 2.28, GCC 12), fixing a build failure with NumPy >=2.5 which requires GCC >=10.3.

DSL → JavaScript backend for WebAssembly (jit_backend="js")

• Under WebAssembly/Pyodide, @blosc2.dsl_kernel kernels can now be transpiled to JavaScript and run via the browser's JIT. It is the default there for transpilable floating-point kernels (silently falling back to miniexpr for anything it can't handle), and beats the WASM TinyCC JIT on compute-heavy kernels (e.g. ~2.8x on a Newton-fractal kernel). Request it explicitly with compute(jit_backend="js"); outside WebAssembly that raises.
• Supports index/shape symbols (_i0/_n0/_ndim/_flat_idx) and integer inputs with a floating-point output. Integer/complex output, reductions, and unsupported constructs stay on miniexpr. Native builds are unaffected.

New blosc2.validate_dsl_jit()

• A new introspection helper reports whether a DSL kernel actually JIT-compiles (vs. silently falling back to the interpreter) for a given set of operand and output dtypes, without running it on real data::

  status = blosc2.validate_dsl_jit(kernel, [np.float64, np.float64], np.float64)
  status["jit"]  # True if a runtime JIT kernel was produced
  

miniexpr fixes

• DSLValidator now rejects ;-joined sibling statements with a clear "one statement per line" error, and assigning to an input parameter raises a targeted error naming the param.
• Fixed (in miniexpr) a name collision where DSL variables named out, idx, nitems, inputs or output clashed with codegen-internal identifiers, causing the generated C to fail to compile and silently fall back to the interpreter. Codegen identifiers are now namespaced under __me.

CTable.sort_by(view=True): zero-copy sorted views

• CTable.sort_by() now accepts view=True, returning a lightweight sorted view that shares the parent's column data and gathers rows on demand in sorted order — no whole-table copy. This is ideal for reading a sorted slice of a large (possibly on-disk) table::

  t.sort_by("col", view=True)[:10]      # top-10 without materialising
  

  Sorting on a fully indexed column streams directly from the index, so the table is never materialised. Multi-column sorts and dotted (nested) leaf names are supported (e.g. t.sort_by(["trip.begin.lon", "payment.fare"], ascending=[True, False])).

where on dictionary (string) columns

• where expressions now work over dictionary-encoded (string) columns, including membership tests such as '"Acme" in company', so categorical text columns can be filtered without decoding the whole column.

b2view is now an opt-in extra

• The b2view terminal browser and its TUI stack (textual, textual-plotext) are no longer core dependencies: a plain pip install blosc2 no longer pulls them, keeping the compression library lean (and dropping deps that are unusable under wasm32, which has no TTY). Install the viewer with pip install "blosc2[tui]", or pip install "blosc2[hires]" to also get the high-res h view. The b2view command prints this hint if the dependencies are missing.

group_by: flexible aggregation naming

• CTable.group_by(...).agg() now accepts a list of (column, ops) pairs and explicit output names (pandas-style keyword arguments), alongside the existing auto-suffixed mapping; the forms can be combined::

  g.agg({"sales": ["sum", "mean"]})              # auto: sales_sum, sales_mean
  g.agg([(t.sales, ["sum", "mean"])])            # auto, but accepts Column objects
  g.agg(revenue=("sales", "sum"))                # explicit: revenue
  g.agg({"sales": "sum"}, n=("*", "size"))       # combined, with a named row count
  

  The list-of-pairs and named forms accept Column objects (t.sales), which the mapping form cannot because Column is unhashable and so cannot be a dict key.

• Aggregation ops may also be given as the matching blosc2 reduction functions (blosc2.sum, mean, min, max, argmin, argmax), matched by identity -- e.g. g.agg([(t.sales, [blosc2.sum, "mean"])]). This is a naming shorthand only; arbitrary/UDF callables (and look-alikes such as np.sum or a user function named sum) are rejected rather than silently misinterpreted.

group_by / group_reduce: tri-state sort=

• Vectorized dictionary group ordering: group_by() result building now batch-decodes dictionary (string) keys in one pass (decode_batch) instead of one decode() per group, making high-cardinality string group-bys dramatically faster (end-to-end group_by().size() dropped from seconds to milliseconds on ~100k-group workloads).
• sort= is now a tri-state (None / True / False) on both CTable.group_by() and blosc2.group_reduce():
  • True — always return groups sorted by key.
  • False — never sort; deterministic but unspecified order.
  • None (the new default) — auto: sort only when cheap. Integer and dictionary keys are sorted (free / vectorized); float and multi-key results, whose only ordering is an O(G log G) Python sort over every distinct group, are left unsorted to avoid a cost that can rival the grouping itself on high-cardinality data.
• Behavior changes (the two APIs had different prior defaults, so they move in opposite directions):
  • CTable.group_by() previously returned results always sorted. Under the new None default, float-key and multi-key group-bys are no longer key-sorted by default — pass sort=True to restore sorted output. This is a deliberate divergence from pandas (which defaults to sort=True), suited to blosc2's large / on-disk datasets.
  • blosc2.group_reduce() previously defaulted to sort=False (unsorted). Under the new None default its cheap kernels now sort by default — most visibly float keys, which previously came out in hash order. Integer keys were already ascending; the generic Python fallback stays unsorted. Pass sort=False to opt out.

Accelerated reductions from index summaries

• min/max on indexed Columns, and argmin/argmax inside group_by, are now accelerated using the index's per-block min/max summaries: when an index is available these reductions run from the precomputed summaries instead of decompressing the underlying data, which is dramatically faster on large columns. A fast path also builds min/max envelope plots from any index.
• The last group_by operation is memoized and reused when the same grouping is requested again, avoiding recomputation in interactive / repeated workflows (e.g. b2view).

b2view: group-by, sort, and richer plots

• Interactive group-by (G): group a CTable by a column (integer, string, or now float keys) directly in the viewer, with a three-list / two-column menu; while grouped, S/R operate on the grouped result and the data panel's subtitle shows a G(roup) chip. The last grouping is memoized for instant reuse.
• Sort by column (S): sort a CTable by a fully indexed column via a dropdown (R toggles reverse) as a zero-copy sort_by(view=True) that streams from the index — the table is never materialised, Esc restores the original order, and a SORTED chip shows in the status bar. Non-indexed columns can now be sorted too. Sort and filter are mutually exclusive; a row window composes over a sort, and an filter is preserved across Sort / Group.
• Better plots of grouped/sorted views: a grouped view plots bars for a categorical key and lines for a numeric key; numeric-key group plots render as stem/impulse charts rather than misleading connected lines. Bar plots gain an hi-res counterpart mirroring the line/scatter plots, and +/- zoom about the view's left edge.
• --max maximizes the current panel, and escape is now the single, consistent way to back out of every modal.

Other / bug fixes

• C-Blosc2 upgraded to 3.1.5.
• Open-file cache correctness: cached open handles are now validated against the file's fingerprint (st_mtime_ns, st_size) and cached index handles are released when a table closes, so a file changed underneath an open handle is no longer served stale.
• NumPy 2.5 compatibility: adjusted for deprecations in NumPy 2.5.
• Substantially reduced test-suite runtime, and emscripten builds no longer attempt to spawn subprocesses (unsupported there).

This follow-up release builds the b2view terminal viewer into a richer data-exploration tool — a scatter plot, a searchable column picker, a one-shot demo download, refreshed chrome, and several interaction fixes — and upgrades the bundled C-Blosc2 to 3.1.4. WASM/Pyodide is now a fully supported platform, and CTable.info reports per-column compressed sizes.

b2view: richer exploration

• Scatter plots: from a column plot, press s to scatter the current column (X) against another column (Y) chosen from a list, over the current (zoomed) row range; h then opens a high-resolution matplotlib scatter.
• High-res for 1-D series is now an envelope plot (matching the in-terminal view), and a new r key toggles between the min/max envelope and the raw values (strided-sampled when the range is wide).
• Searchable column picker: the c go-to-column key now opens a searchable, selectable list (type to filter, ↑/↓, Enter) for CTables, instead of a text field; N-D arrays still go by numeric index.
• Show/hide columns: / opens a searchable multi-select to pick which CTable columns are displayed.
• Demo download: b2view --download fetches a demo bundle (chicago-taxi-flat.b2z by default) into the current directory if it is not already there, then opens it.
• Refreshed chrome: a branded header, a left-docked filename label in the title, and clearer status chips.

b2view: interaction fixes

• Go-to-row/column pre-fill is now pre-selected, so the first keystroke replaces the current index instead of appending to it (typing a column name no longer produced e.g. 0payment.fare).
• Escape keeps its layered exit while a panel is maximized: with the data panel maximized, escape now unlocks a plot's locked row window (and clears filters) as documented, instead of being hijacked into restoring the panel — use r to restore (ESCAPE_TO_MINIMIZE = False).
• Test-suite robustness fixes (a timing flake and a Windows rendering glitch).

Other

• C-Blosc2 upgraded to 3.1.4.
• WASM/Pyodide is now a fully supported platform, with more frequent CI runs.
• CTable.info shows per-column compressed sizes (cbytes and cratio), and print_versions() uses clearer Python-Blosc2 / C-Blosc2 labels.

This release teaches the b2view terminal viewer to plot — peak-preserving envelope line plots of any series, with zoom, a row-window lock, and an optional high-resolution matplotlib view — and gives CTable a pandas-like display and CSV experience. It also publishes WASM/Pyodide wheels to PyPI and adds faster strided reads for NDArray and Column.

b2view: plotting and data inspection

• In-terminal plots: press p on a numeric series (a CTable column or an array row) to draw a braille line plot. Plots are peak-preserving min/max envelopes by default, so no spike or trough is hidden however large the series is; large local series stream their envelope exactly in bounded spans (only remote c2arrays fall back to a labeled strided sample).
• Zoom and row-window lock: zoom the plot into a row range and pan it; press v to lock the data grid to the plotted range so paging stays inside it (escape unlocks). The plot and high-res views honor the locked window.
• High-resolution view: h opens a high-res matplotlib image of the plotted range (new optional hires extra: matplotlib + textual-image).
• On-demand cell decode: enter decodes a single skipped/expensive CTable cell, and SChunk nodes now preview as a paged hex dump.
• Fixes and polish: row paging re-aligns to the page grid after dim-mode single-row scrolls; the data panel now focuses correctly with --path ... --panel data; status chips are branded yellow.

CTable display

• CTable.to_string() now renders the whole table by default (every row and every column), like pandas' DataFrame.to_string(). New max_rows and max_width parameters truncate on demand. Behaviour change: previously to_string() returned the truncated view; code that relied on that should pass max_rows=/max_width= (or use str()).
• The [N rows x M columns] dimensions footer now follows pandas: omitted by to_string() (pass show_dimensions=True to force it), and shown by str/repr/print only when the view is actually truncated. Previously it was always appended.
• repr(ctable) now shows the same truncated table as str(ctable) (pandas/polars convention), instead of the one-line CTable<…> summary. The compact summary remains available via ctable.info.
• New display options in set_printoptions: display_width controls the column-fitting width budget (None = auto-detect terminal, -1 = show all columns, positive int = fixed budget), and display_rows now accepts -1 to show all rows (0 still shows none).
• New blosc2.printoptions(...) context manager temporarily sets the display options and restores them on exit, e.g. with blosc2.printoptions(display_rows=-1, display_width=-1): print(t).

CTable I/O

• CTable.to_csv() now accepts no path, returning the CSV as a string like pandas' DataFrame.to_csv(). Passing a path still writes the file (and returns None); the returned string is byte-for-byte the same as the file.

Performance

• Faster strided reads: NDArray.__getitem__ gains a sparse-gather fast path for large strides, and Column.__getitem__ short-circuits when the logical positions equal the physical ones.
• Fix: a negative step in Column getitem could return []; it now returns the reversed selection.

Indexing

• Fix: a sidecar-handle cache collision could return the wrong SUMMARY index for a compact-store column.
• Cross-column index pruning is now enabled for compact CTable queries, so more predicates prune blocks before any data is materialized. The docs also note when summary indexes are not created automatically.

Packaging

• WASM/Pyodide wheels on PyPI: the main wheel build now also produces pyemscripten wheels for CPython 3.13 (2025 ABI) and 3.14 (2026 ABI) and uploads them to PyPI, so blosc2 is micropip-installable in Pyodide, and b2view prints a clear message instead of crashing when run under WASM. Known limitation: slicing an in-memory SChunk loaded from a frame fails on the Pyodide 0.29.x Emscripten toolchain (cp313); it works on Pyodide 314 (cp314) and natively. See issue #664.
• cibuildwheel updated to 4.1.

Note: 4.4.4 was skipped due to a failure during the release process.

This release promotes the b2view terminal viewer to a core feature — installed by default, with new interactive row and column filtering — and makes BatchArray block layouts (and hence compression ratios) reproducible across CPUs.

b2view, the terminal data viewer

• Installed by default: textual and rich are now regular dependencies, so the b2view CLI works out of the box (the [tui] extra is gone). A getting-started walkthrough was added to the docs, and the README now lists the CLI tools.
• Row filtering: pressing f on a CTable node opens a modal that takes the same string expressions as CTable.where() (dotted nested names, and/or) and pages through the matching view. Filters are remembered per node for the session, the data header shows the active filter plus the unfiltered total, and escape (or an empty expression) clears it.
• Column filtering: / narrows the visible columns by case-insensitive substring; column paging and the c goto-column modal then operate on that subset. Combines freely with the row filter; escape clears one layer per press (rows first, then columns).
• Mouse handling: the terminal owns the mouse by default, so native text selection/copy works like in any CLI program; --mouse lets b2view capture it instead (click-to-focus, wheel scrolling by half a page, paging at the edges).
• Navigation: ? opens a help screen listing all keys; c jumps to a column by index, exact name or unique name prefix; s/e jump to the first/last column window; row paging and jumps keep the cursor on its column; dim-mode index/viewport movements clamp at the boundaries instead of wrapping around.
• Rendering: column windows are fitted from measured rendered widths (and re-fitted on terminal resize and panel maximize/restore), and float columns use a uniform number of decimals so decimal points align down the column.
• Test suite: first automated tests for the TUI — Pilot-driven keyboard journeys against a deterministic generated store (marker tui), plus render unit tests. Skipped on wasm, where Textual apps cannot start (no termios).

BatchArray

• Reproducible block layouts: automatic variable-length block sizing now uses fixed byte budgets (1 MiB for clevel 1-3, 8 MiB for 4-6, 16 MiB for 7-8) instead of the CPU cache sizes, so the layout — and hence the compression ratio — no longer depends on the machine that created the array.

Build and docs

• Installing test dependencies: the docs now use pip install . --group test (a PEP 735 dependency group); the stale [test] extra syntax was removed.
• cibuildwheel updated to 4.0.

This is a maintenance release focused on faster CTable cold-start, printing and groupby performance, a lighter import blosc2, new raw-storage access for columns, and support for the new J2K/HTJ2K codec plugins.

CTable performance

• Lazy column opening in views: select() (and other view-producing operations) no longer open every projected column up front. A column is only opened from storage when the view actually reads it, so selecting and then touching a subset of columns — or aggregating a single one — skips the cold-start cost of the rest.
• Lazy index opening in queries: query planning no longer opens every SUMMARY-indexed column on a wide persistent table; only indexes for columns actually referenced by the predicate are loaded.
• Faster table printing: repr()/to_string() now memoise per-column sparse gathers for the duration of a render and combine the head and tail rows into a single sparse read per column. Each column is read from storage once instead of ~6 times (precision detection, width sizing and row rendering all hit the cache).
• Groupby with integral float keys: float key columns whose values are integral and fit a compact non-negative range (e.g. float32 id/second columns) now take the dense single-key fast path instead of the markedly slower generic float-hash path. Fractional or non-finite keys fall back automatically.
• No tempdir in read mode: opening a .b2z/.b2d store in 'r' mode no longer creates a temporary working directory, since nothing is ever written.

Lighter imports and prefetcher rework

• asyncio dependency dropped: the on-disk chunk prefetcher used by the UDF and numexpr fallback engines now uses plain concurrent.futures instead of an asyncio event loop. import blosc2 no longer pulls in ~30 asyncio modules, saving ~3 MB of memory footprint at import time.
• Prefetcher deadlock fixed: an exception during evaluation could leave the generator finalizer blocked forever in thread.join() while the reader thread was stuck on a full prefetch queue. A stop event now makes the producer bail out when its consumer goes away.

New features

• Column.raw accessor: returns the underlying storage container of a column (NDArray, ListArray, DictionaryColumn, …) directly. Unlike Column.__getitem__, which always materializes NumPy arrays, this is the column as a blosc2-native compressed object — usable as a lazy-expression operand without decompressing, and exposing storage details like schunk, chunks or cparams. Note that this is a physical view: fixed-width containers are over-allocated to chunk capacity, so slice to len(table) to get just the live rows, and no validity-mask or null-sentinel processing is applied. Raises AttributeError for computed columns, which have no backing storage.
• J2K and HTJ2K codec IDs: blosc2.Codec.J2K and blosc2.Codec.HTJ2K expose the IDs for the new JPEG 2000 codec plugins (installable with pip install blosc2-j2k and pip install blosc2-htj2k).

Fixes

• --float-trunc-prec and nested columns: the precision-truncation filter of the parquet_to_blosc2 CLI now propagates to float fields inside nested (struct/list) columns too.
• Guard for unsupported computed-column expressions: expressions that would serialize to an empty or non-round-trippable string are now rejected with an early, actionable ValueError at add_computed_column() time, instead of silently breaking on reload.

Build

• C-Blosc2 updated to 3.1.3.

This is a feature and maintenance release that promotes DSL kernels to first-class CTable computed columns, adds a new CTable.__setitem__ assignment idiom, optimises bulk NDArray writes, and fixes several correctness issues.

DSL kernels as first-class CTable columns

• add_computed_column() accepts DSL kernels: @blosc2.dsl_kernel-decorated functions can now back virtual computed columns directly, in addition to the existing string-expression form. The column survives save/open round-trips via persisted dsl_source.
• add_generated_column() accepts DSL kernels: stored generated columns (written during append/extend and on refresh_generated_column()) now support DSL kernels as their transformer.
• CTable.where() accepts UDF/DSL kernels: filter predicates are no longer limited to expression strings — any DSL kernel can be passed directly.
• dtype inference for DSL kernels: when dtype is omitted, lazyudf() infers the output dtype via NumPy type promotion of the input column dtypes. Pass dtype explicitly for type-changing kernels (comparisons, casts).
• kernel_from_source() utility: new dsl_kernel.kernel_from_source() reconstructs a DSLKernel from its stored source text, shared by the CTable DSL-column loaders and the persisted LazyUDF decoder.
• Security note: .b2d files from untrusted sources that contain DSL computed columns execute stored Python source on open. A warning is now included in the documentation.

New CTable.__setitem__ column-assignment API

• t["col"] = arr: new shorthand equivalent to t["col"][:] = arr. Accepts any array-like including blosc2.NDArray. Raises KeyError for unknown columns and ValueError for views or read-only tables.

Chunked NDArray writes in extend() and Column.__setitem__

• extend({"col": ndarray}) decompresses chunk-by-chunk: when a blosc2.NDArray is passed as a column value to extend(), it is now written in chunks instead of being fully decompressed upfront. Pass validate=False to avoid a transient full decompression during constraint checking.
• col[:] = blosc2_ndarray fast path: a new no-holes fast path in Column.__setitem__ skips the O(n) validity-mask gather and writes the NDArray one chunk at a time using contiguous slice writes. Works for both scalar and fixed-shape ndarray columns. Falls back to a chunked fancy-index path when deleted rows are present.

BLOSC_ME_JIT environment variable override

• Full CLI override: BLOSC_ME_JIT now takes unconditional priority over both the jit= and jit_backend= keyword arguments, making it easy to switch JIT backends from the command line without modifying code.

Correctness fixes

• View corruption in Column.__setitem__: a None == None guard evaluation on view-backed columns could fire the NDArray fast path, bypassing physical-position remapping and silently corrupting rows. Fixed by explicitly checking base is None before activating the fast path.
• CTable.__setitem__ view guard: the new t["col"] = arr API now raises ValueError on views, matching the contract of all other mutating CTable methods.
• Fast path enabled for disk-opened tables: the fast path previously remained dormant for tables opened from disk because _last_pos starts as None. The guard now calls _resolve_last_pos() to lazily initialise it.
• DSL column jit_backend preserved in _empty_copy: the jit_backend setting was silently dropped during internal table copies; it is now retained.
• lazyexpr Column unwrapping: convert_inputs() now automatically unwraps CTable.Column objects to their backing NDArray so that shape and identity checks work correctly.

Documentation and examples

• Parquet-to-blosc2 walkthrough: new step-by-step tutorial added to the getting-started section. Thanks to @SyedIshmumAhnaf.
• CTable performance tips: new section in the overview covering when to prefer computed vs. generated columns, chunk sizing, and query optimisation.
• Simplified docstring examples: examples throughout ndarray.py and ctable.py now use blosc2.array(), blosc2.arange(), and blosc2.linspace() directly instead of two-step numpy-then-asarray patterns.
• udf-computed-col.py example: new end-to-end example demonstrating DSL kernel computed and generated columns.

This is a feature release focused on a new interactive data viewer, automatic SUMMARY indexes for fast WHERE queries, chunk-aligned Arrow/Parquet imports, expanded where() acceleration via miniexpr, and a range of CTable ergonomics and performance improvements. Python 3.10 support has been dropped; Python 3.11 is now the minimum.

b2view: interactive Text User Interface data viewer

• New b2view command: a terminal-based interactive viewer for all blosc2 containers — NDArray, CTable, SChunk, BatchArray, and more. Launch it with b2view <file> or as blosc2.b2view() from Python.
• Full 1-D and 2-D browsing: arrays with more than two dimensions are sliceable along any axis; 1-D arrays are shown as a single-column table.
• CTable navigation: scroll through rows with keyboard shortcuts; t/b jump to the top/bottom; --panel jumps straight to a named panel on launch.
• CTable.vlmeta panel: variable-length metadata is exposed in a dedicated panel.
• New dim mode: navigate along all dimensions freely for N-D arrays.

SUMMARY indexes for fast WHERE queries

• Automatic SUMMARY index creation: when a CTable is closed after a write session, SUMMARY indexes (per-block min/max) are built by default for all eligible scalar columns with no extra configuration needed.
• Incremental build during writes: indexes are accumulated block-by-block during extend() and Arrow import, so closing the table costs almost nothing beyond the write already done.
• Block-skip prefilter: the miniexpr prefilter uses SUMMARY bitmaps to skip entire blocks whose min/max range cannot satisfy the WHERE predicate, reducing decompression work for selective queries.
• Conjunction support: per-column SUMMARY block masks are combined with bitwise AND so multi-column conjunctions prune blocks efficiently.
• Cost gate: a cost model guards index use; the SUMMARY path is skipped when block skipping is unlikely to help (e.g. very low selectivity).
• --no-summary-index: new CLI flag for parquet-to-blosc2 to disable automatic index creation on import.

CTable column grid alignment

• Shared chunk/block grid for scalar columns: fixed-size columns are now written on a shared chunk/block grid derived from the numeric column widths, so all columns have identical chunk boundaries. This makes multi-column SUMMARY scans and chunk-parallel reads significantly faster.
• Chunk-aligned Arrow import: incoming Arrow/Parquet batches are buffered and flushed in exact chunk-sized blocks, so each chunk is compressed exactly once instead of being split across batch boundaries.
• Vectorized dictionary-column import: dictionary codes are now written in bulk at full chunk capacity rather than element by element.
• Small fixed strings on the grid: fixed-length string columns narrow enough to share the numeric grid are admitted to it, reducing the number of distinct chunk sizes.
• --reduce-mem: new CLI option for parquet-to-blosc2 to cap the Arrow read-batch size on nested list<struct> imports, keeping peak RSS low at a modest speed cost.

CTable.copy() enhancements

• C-level bulk copy for ListArray and BatchArray: a new chunk_copy() method transfers pre-compressed chunks directly at the C level, bypassing Python-level serialization and recompression. CTable.copy() uses this path automatically.
• chunks= / blocks= overrides in CTable.copy(): callers can now specify target chunk and block sizes for the output copy.
• cparams and blocks overrides: CTable.copy() accepts cparams and blocks to recompress the copy with different settings.
• --chunks / --blocks added to the parquet-to-blosc2 CLI.

Take/gather APIs

• Added NDArray.take() following Array API take shape semantics, including axis=None flattening and N-dimensional integer indices. One-dimensional gathers use a new sparse C-level path (b2nd_get_sparse_cbuffer) internally.
• Extended top-level blosc2.take() to dispatch to NDArray.take(), CTable.take(), and Column.take() while preserving the input container type.
• Added CTable.take() and Column.take() for logical row/value gathers that preserve order and duplicate indices, unlike mask-based views.
• For ndim > 1 axis-based take, orthogonal selection is used internally for better performance.

where() and miniexpr acceleration

• where(cond, x) via miniexpr: the single-argument where (fill-with-zero variant) is now handled directly by the miniexpr engine when the condition is a boolean array, avoiding a numexpr round-trip.
• where(cond, x, y) via miniexpr: the two-argument flavor is likewise dispatched to miniexpr for element-wise conditional selection.
• Sparse boolean mask fast path: when a boolean indexing result is very sparse (high selectivity), auto-detection switches to a fast gather path instead of a full-array scan.
• Early boolean key check: NDArray.__getitem__ with a boolean array key now detects it before the general process_key / nonzero path, avoiding wasted work.
• Compressed transient masks: temporary boolean masks created during queries are now stored as LZ4-compressed blosc2 arrays, reducing memory pressure without measurable speed regression.
• BLOSC_ME_JIT / BLOSC_ME_JIT_TRACE: new environment variables to control and trace the miniexpr JIT backend at runtime.

CTable views and lazy sorting

• sort_by() on a view is now lazy: calling sort_by() on a filtered view returns a position-reordered view without materializing data; the sort positions are cached and used directly on column access.
• Lazy column materialization in filtered views: select() on a view no longer materializes unneeded columns eagerly; columns are resolved only when accessed.

NestedColumn and .info improvements

• NestedColumn public class: the previously internal _NestedColumnNamespace has been renamed and promoted to NestedColumn, providing aggregate metadata (col_names, nrows, nbytes, cbytes, cratio) and a structured .info report over a group of dotted columns.
• Uniform .info across containers: Column.info, CTable.info, NestedColumn.info, and related classes now follow a consistent field order (identity → shape/grid → sizes → content → compression params).

Context manager support for blosc2.open()

• All objects returned by blosc2.open() — NDArray, SChunk, CTable, BatchArray, ListArray, and stores — now support the with statement. The __exit__ method flushes and closes the underlying storage.

Performance improvements and fixes

• CTable.nrows stored persistently: row counts are written to metadata on close and read back on open, avoiding a full column scan at startup.
• Index sidecar loading from .b2z: SUMMARY/BUCKET sidecars inside .b2z archives are now read in-place rather than extracted to a temporary directory, cutting open latency for indexed tables.
• Compressed query cache: the hot query-result cache is now stored LZ4-compressed, reducing its memory footprint with negligible overhead.
• Query cache consistency fixes: on-disk query cache side effects and a miniexpr chunk-cache race condition on Apple Silicon have been resolved.
• macOS L2 floor for chunk sizing: on macOS the full L2 cache is used as a floor for automatic chunk sizing, giving better compression/speed trade-offs.
• Better Apple Silicon L3 handling: missing L3 cache on Apple Silicon is handled more gracefully in the cache-size heuristic.
• Table capacity management: large CTables grow more conservatively, and capacity is trimmed on close and after Arrow import to reclaim over-allocated space.
• Faster iteration with iterchunks_info(): several hot loops switched to iterchunks_info() for lower overhead per chunk.
• Cost-model index refinement threshold: the previously hardcoded threshold for switching between index and scan has been replaced with a data-driven cost model.
• Index prefetch reuse: data already prefetched during an index lookup is reused in the refinement phase, avoiding redundant I/O.
• Simplify index sidecar filenames in _indexes/{col}/ directories.
• DictStore embed disabled by default: embedding a store inside a dict store is now opt-in (it was error-prone as the default).
• Fixed wasm32 issue: a 32-bit platform arithmetic fix for reduce operations.
• Chunks never exceed array dimension: compute_chunks_blocks now guarantees chunk dimensions are capped at the array shape dimension.
• max_rows robust to older PyArrow: truncation logic no longer depends on PyArrow APIs that are absent in older releases.
• cratio display: compression ratio is now shown with an explicit x suffix (e.g. 2.47x) throughout .info output.
• Updated bundled C-Blosc2 to the latest release.

Dropped Python 3.10 support

• Python 3.11 is now the minimum supported version.

note: 4.3.2 was an internal pre-release that was not published to PyPI.

This is a maintenance release focused on CTable display ergonomics, indexed-query correctness, and query-planner performance.

CTable display and print options

• Pandas-like CTable display by default: str(table) / print(table) now use a compact, pandas/DuckDB-style table representation, including a displayed logical row index, numeric alignment, compact spacing, and a trailing footer such as [726017 rows x 5 columns].
• Configurable display options: added blosc2.set_printoptions() and blosc2.get_printoptions() for CTable rendering. The supported options are display_index, display_rows, display_precision, and fancy.
• CTable.to_string(): added a one-off formatting API for producing CTable string representations without changing global print options.
• Compact truncation for large tables: when a table exceeds the configured display_rows threshold, only the first five and last five rows are shown, with an ellipsis row in between.
• Float display refinements: compact mode uses pandas-like fixed precision for floating-point columns, and integer-valued float columns are displayed with a single decimal place.
• Fancy display preserved: set_printoptions(fancy=True) restores the more decorated display with dtype rows, separator rules, and hidden row/column counts.

Indexed queries and sorting

• Cross-column exact index refinement: multi-column conjunctions can now use exact positions from a selective indexed column (FULL, PARTIAL, or OPSI) as a compact pre-filter, then refine the remaining predicates on those positions instead of scanning the full table.
• NaN-safe index boundary navigation: fixed sorted-boundary navigation for floating-point indexes containing NaN values, so indexed results match scan results for bucket/full index lookups.
• Better index-planner heuristics: the planner now avoids low-value indexed paths when segment pruning is unlikely to help, and avoids expensive scalar specialization for non-scalar arrays.
• Faster filtered sorting: small filtered views can be materialized and sorted directly, avoiding an extra gather of sort keys.

Performance and fixes

• Avoid full materialization of valid_rows in several CTable code paths.
• Keep row counts lazy for views and avoid unnecessary nrows calls in the query planner.
• Reduced overhead in root-column iteration and small query-planner operations.
• Fixed dictionary-column capacity handling during Arrow import and a regression affecting dictionary columns.
• Marked additional long-running tests as heavy to reduce default test-suite runtime.

Documentation

• Updated the containers tutorial with dedicated ListArray and CTable sections, including CTable's columnar storage model and support for columns backed by NDArray, BatchArray, ObjectArray, ListArray, and related containers.

This is a maintenance release focused on CTable nested-column ergonomics, grouped reductions, and API/documentation polish.

CTable nested columns and grouped reductions

• Nested column names in group_by() results: grouped output columns can now preserve dotted/nested names such as trip.sec instead of requiring valid Python identifiers.
• Column-object selectors: CTable.group_by() and CTable.sort_by() now accept Column objects as well as string names, enabling idioms such as t.group_by(t.trip.sec) and t.sort_by(t.trip.sec).
• Grouped arg reductions: CTableGroupBy now supports argmin() and argmax(), plus agg({"col": "argmin"}) / agg({"col": "argmax"}). Results are logical row positions in the grouped table or view; groups with no non-null values return -1.

NDArray constructor ergonomics

• blosc2.array(): added a NumPy-like constructor for NDArrays. It mirrors blosc2.asarray() but defaults to copy=True, so passing an existing NDArray creates a copy unless copy=False or copy=None is requested.

Documentation

• Expanded the CTable reference with RowTransformer, Column.row_transformer, and CTableGroupBy.argmin / argmax documentation.
• Added blosc2.ndarray(), blosc2.dictionary(), and related public schema factory functions to the Schema Specs reference.
• Moved blosc2.group_reduce() into the Reduction Functions reference and updated its example to use Blosc2 NDArrays.

### CTable: N-dimensional (ndarray) columns

• Multidimensional columns: CTable columns can now hold NDArray-backed cells, allowing each row of a column to contain a full n-dimensional compressed array. This enables use cases such as embedding vectors, image patches, time-series windows, or any other multidimensional per-row payload.

• CSV and DataFrame import/export: Multidimensional column data can be imported and exported via CSV and pandas DataFrames, with automatic detection of array-valued cells.

• Nullable ndarray columns: Multidimensional columns fully support the nullable semantics (null_count, sentinel handling, null_policy) already available for scalar columns.

• `from_pandas()` improvements: CTable.from_pandas() now creates the correct specialized backing storage for DictionarySpec, ListSpec, VLStringSpec, VLBytesSpec, and other variable-length scalar specifications.

• Improved schema coverage: New CTable timestamp schema type and extended Column.info output with shape, chunks, and blocks descriptors.

• Arg reductions: Added argmin() and argmax() for scalar and ndarray CTable columns, plus row-transformer support for generated columns such as per-row peak-hour or dominant-embedding-dimension features.

### CTable: Group-by and filtered aggregation

• `CTable.group_by()`: The primary group-by interface. Call t.group_by("city", sort=True).agg({"qty": "mean"}) to produce a new CTable with aggregated results. Single-key and multi-key groupings are supported, along with convenience methods such as .size(), .count(), .sum(), .mean(), .min() and .max():

  by_city = t.group_by("city", sort=True)
  by_city.size()  # COUNT(*)
  by_city.sum("sales")  # SUM(sales) per city
  by_city.agg({"sales": ["sum", "mean"]})  # SUM(sales), AVG(sales) per city

• Performance accelerators: Dedicated Cython fast paths deliver significant speedups: ~25× for float32/64 group-by keys, ~8× for integer and dictionary-code keys, and a general-purpose hash table for arbitrary float keys.

• Filtered aggregate pushdown: The where= parameter is now accepted in aggregation methods, pushing the filter into the compute engine so that only matching rows are read and reduced.

• Persistent grouped output: Group-by results can be saved directly to persistent storage via the urlpath= parameter.

• `blosc2.group_reduce()`: New public function that performs group-by reduction over NDArray instances and CTable columns, with Cython-accelerated backends for common key/reduction combinations.

### CTable: Dictionary / categorical columns

• `DictionarySpec` column type: Introduced a new dictionary-encoded (categorical) column type that stores string or integer codes mapped to a shared dictionary, providing compact storage and accelerated equality and membership queries.

• Dictionary types in `where` clauses: Dictionary columns can be queried with the same where= expression syntax as other column types, including nested dotted-name access.

• Improved display: CTable printing now adapts to the terminal width, and dictionary values are shown in their decoded form. Column.info has been extended with type details, shape, chunks, and blocks.

### CTable: Nested columns and field-name escaping

• Dotted nested column access: Columns whose names contain literal . (e.g., "root.nested") are now fully addressable via the dotted accessor syntax in where expressions, __getitem__, and the public API.

• Hierarchical `_cols` storage paths: The internal column storage layout now preserves a hierarchical structure that mirrors the logical nesting, improving introspection and interop.

• Nested-field pipeline: A new flattened-storage pipeline with logical mapping preserves nested schema structure (field names, types, and hierarchy) through Arrow and Parquet import/export. For unnamed top-level list<struct<...>> Parquet files, the logical schema round-trips faithfully, though the original physical row grouping is intentionally not preserved.

• Field-name escaping: Special characters (. and /) in column names are automatically escaped during schema construction and metadata round-trips.

### Parquet import/export improvements

• Arrow serializer by default: CTable.from_parquet() now defaults to the Arrow serializer, providing better schema fidelity and nested-type support.

• Progress reporting: A --progress flag and an ETA estimator have been added to the parquet-to-blosc2 CLI for long-running imports.

• `--max-rows` parameter: CTable.from_parquet() and the CLI now accept max_rows to limit the number of imported rows.

• `--timestamp-unit`: New CLI option to control timestamp unit conversion on import.

• `--float-trunc-prec`: New CLI option to truncate floating-point precision on import.

• Separated nested columns enabled by default: The separate_nested_cols flag is now True by default for both the Python API and the CLI, ensuring nested Arrow structs are always expanded into flat columns.

• `list_serializer` parameter: New option to control how list-type columns are serialized, with sensible defaults for different list layouts.

• Validation optimizations: Arrow datetime values are validated only during import, reducing runtime overhead on subsequent operations.

### TreeStore: Inline CTable support

• CTables inside TreeStore: CTable objects can now be stored inline as items inside a TreeStore, enabling hierarchical storage that mixes arrays and tables in a single persistent container.

• Cache hardening: TreeStore cache assignments now use defensive copies and cache effective object roots to avoid aliasing and stale-cache errors.

• Examples and tutorials: New tutorials and docstring examples demonstrate how to store, retrieve, and query CTables within a TreeStore.

### Performance and usability enhancements

• Faster open and import: blosc2.open() and store constructors now assume valid file extensions and defer column metainfo loading, making CTable.open() and package import noticeably faster.

• `CTable.nrows` is now lazy: The row count is computed on demand rather than eagerly, speeding up open and schema-inspection workflows.

• Accelerated scalar and small-slice access: The batch/list path for reading scalar values or small column slices has been overhauled, eliminating internal placeholder materialization and yielding lower latency.

• Late-import optimizations: Heavy optional dependencies are imported lazily at the blosc2 package level, reducing the baseline import blosc2 overhead.

• `iter_arrow_batches()` optimization: Avoids full Python object materialization of batches during iteration, reducing memory pressure.

• `NDArray`-to-list conversion: Small optimization when converting NDArray objects to Python lists.

• `_last_pos` invalidation skipped: Mid-table deletes no longer eagerly invalidate cached positional state, improving delete latency.

### Documentation, examples and benchmarks

• API reference expanded: blosc2.group_reduce() has been added to the Sphinx reference, along with updated CTable, Column, and TreeStore pages.

• New tutorials and examples: Added sections on CTable–TreeStore integration, nested fields, dictionary columns, aggregates, grouping and querying with where=.

• New benchmarks: Graph benchmarks for CTable insert time, column count, memory usage, and where= queries, plus dedicated group-by, nested-filter, and Parquet round-trip benchmarks.

### Fixes and compatibility

• Null and NaN handling: NumPy scalar null sentinels are now normalized to plain Python scalars, and floating-point NaN sentinels are treated consistently with Python float('nan').

• Empty aggregate results: Filtered aggregations that produce no rows now handle the empty result gracefully.

• Generated column safety: Accessing a stalled (unfillable) generated column now raises a clear exception instead of producing undefined results.

• Miniexpr bundling: Miniexpr’s bundled libtcc and related runtime files are now kept inside the blosc2 package, avoiding conflicts with other TCC installations.

• Test improvements: Torch-dependent tests are marked as heavy, PyArrow-optional tests are skipped when the library is absent, and parametrization matrices have been trimmed to reduce CI time.

• Missing Cython validation: Added validation guards for several Cython extension functions that previously lacked explicit error checking.

• C-Blosc2 update: Bundled C-Blosc2 has been updated to the latest version (3.0.3).

• ``blosc2.open()`` default mode changed from 'a' to 'r': Removed the FutureWarning that was added to prepare for this transition.

CTable: columnar compressed tables

• Introduced blosc2.CTable, a new columnar table container for compressed, typed columns. CTables support dataclass- and schema-based construction, row iteration, column access, table views, head() / tail() / sample(), sorting, selection and compact where expressions.
• Added persistent CTables backed by TreeStore, with support for blosc2.open(), CTable.open(), CTable.load(), CTable.save(), CTable.to_b2d() and CTable.to_b2z(). CTable views can be saved too, and .b2z/.b2d path handling has been tightened.
• Added mutation operations for CTables, including append(), extend(), delete(), compact(), add_column(), drop_column(), rename_column() and related schema validation.
• Added computed columns, including virtual computed columns backed by lazy expressions, materialized computed columns and automatic filling of materialized computed columns during inserts.
• Added CTable indexing support, including persistent indexes, direct expression indexes, ordered index reuse, boolean LazyExpr/NDArray masks in CTable.__getitem__, iter_sorted() and indexing support for .b2z tables.
• Added nullable schema support and null policies for CTable scalar columns, preserving nullable scalar Parquet round-trips.
• Added variable-length CTable column support via ListArray / ObjectArray, including vlstring and vlbytes schema specs, fixed-length string/bytes import support and list/struct Arrow/Parquet round-trips.
• Added Arrow, Parquet and CSV interoperability for CTables, including batch-wise Arrow/Parquet import/export, Arrow schema metadata preservation, CTable.from_arrow_batches() improvements and a new parquet-to-blosc2 CLI utility.
• Added CTable documentation, tutorials, examples and benchmarks covering schema definition, persistence, querying, indexing, mutations, nullable columns, computed columns and variable-length columns.

Indexing and ordering

• Added a new indexing subsystem for NDArrays and CTables, including full, partial/bucket, light/medium and OPSI-style index kinds, out-of-core index builders and sidecar storage.
• Added blosc2.Index as the unified public index handle, plus APIs such as create_index(), compact_index(), iter_sorted(), will_use_index() and related query explanation support.
• Added materialized expression indexes for NDArrays and direct expression indexes for CTables.
• Added persistent query-result caching for indexed lookups, with FIFO pruning and cache accounting.
• Added blosc2.argsort() and refactored indexing APIs around explicit index enums and sorting helpers.
• Improved indexed query performance with Cython accelerators, threaded chunk batching, zero-copy/cached mmap reads, chunk-aware and reduced-order layouts and faster scattered row gathering.
• Reduced memory usage during index creation and lookup by avoiding full sidecar materialization, replacing memmap staging with Blosc2 scratch arrays and adding tmpdir support for full out-of-core indexes.

Persistence, stores and serialization

• Added structured Blosc2 serialization based on b2object carriers, including persisted C2Array, LazyExpr and DSL LazyUDF objects.
• Added blosc2.Ref for serializing external references, plus examples for b2object bundles and persisted expressions/UDFs.
• Added blosc2.load() as a convenience loader.
• Added vlmeta support to LazyArray objects.
• Improved store handling by preserving lazy b2object carriers in DictStore, allowing reopened proxies to refill caches after read-only opens, relaxing DictStore/TreeStore suffix requirements and adding DictStore.to_b2d().
• Accelerated blosc2.open() by trying standard opens first and warning on implicit append mode.

Arrays, computation and containers

• Added ObjectArray for fully general object data and renamed the earlier VLArray work accordingly; added ListArray docstrings and Arrow integration improvements.
• Added schema helpers including numeric specs, blosc2.struct() and blosc2.object() for nested/fully general column declarations.
• Improved fromiter() with direct chunked construction and substantially lower peak memory use.
• Improved asarray() behavior for NDArray inputs when copy-inducing keyword arguments are supplied.
• Added SChunk.reorder_offsets().
• Improved BatchArray defaults and documentation; the default compression level is now tuned for faster lookup/scan behavior.
• Continued matmul/linalg optimization work and shared-thread-pool integration.

CLI, docs and examples

• Added the parquet-to-blosc2 command with options such as --max-rows, --parquet-batch-size, --blosc2-items-per-block and --use-dict.
• Added new CTable, ObjectArray, BatchArray, containers, indexing and serialization tutorials and examples.
• Reorganized and expanded the API reference for CTable, Column, schema specs, Index, save/load helpers and miscellaneous APIs.
• Updated benchmark suites for CTables, indexing, Parquet import/export, BatchArray and NDArray construction/indexing.

Fixes and compatibility

• Updated bundled C-Blosc2 to v3.0.2 and require C-Blosc2 >= 3.0.0 when building against a system library.
• Updated bundled C-Blosc2 and miniexpr sources multiple times.
• Restored compatibility with NumPy < 2.
• Fixed Windows and mmap/file-locking issues in index creation, rebuilds and temporary file cleanup.
• Fixed full-index query failures for large CTable columns and full out-of-core merge failures on systems with small /tmp.
• Fixed stale sidecar/cache reuse and targeted cache invalidation when persistent sidecars are replaced.
• Fixed .b2z double-open corruption caused by GC-triggered repacking and made temporary .b2z unpacking default to the source file directory.
• Fixed a regression when reopening persisted proxies in read-only mode.
• Fixed GC-induced thread hangs on macOS with Python 3.14 and hardened async chunk reading/cache cleanup paths.
• Fixed lazy-chunk source-size handling in decode/getitem callers.
• Fixed nullable validation, dictionary extend validation, CTable close propagation, print alignment and NumPy mask support.
• Fixed arange() regressions and several pre-existing set_slice error-handling issues.
• Clamped indexing/thread defaults for wasm32.

• A new fast path for src/blosc2/linalg.py that uses the matmul prefilter machinery in src/blosc2/blosc2_ext.pyx.
  • The fast path is only used for supported cases:
    • blosc2.NDArray inputs
    • 2-D only
    • floating-point only
    • matching dtypes
    • aligned chunk/block layouts that satisfy the current kernel assumptions
  • All other valid cases fall back to the existing chunk-by-chunk implementation in src/blosc2/linalg.py.
  • Some benchmarks for the supported cases show significant speedups over the chunked implementation:
    • aligned 400x400 float32: about 3.7x faster over chunked
    • aligned 400x400 float64: about 3.0x
    • aligned 800x800 float32: about 1.5x
    • misaligned case: auto correctly stays on chunked

• Update miniexpr to fix bug on ubuntu with ARM64.

• Add DSL kernel functionality for faster, compiled, user-defined functions which broadly respect python syntax and implement the LazyArray interface. See the introductory tutorial at: https://blosc.org/python-blosc2/getting_started/tutorials/03.lazyarray-udf-kernels.html
• Add read-only mmap support for store containers: DictStore, TreeStore, and EmbedStore now accept mmap_mode="r" when opened with mode="r" (including via blosc2.open for .b2d, .b2z, and .b2e).
• New .meta entry for store containers, allowing better store recognition at blosc2.open() time. Fixes #546.
• Add cumulative_sum and cumulative_prod functions for Array API compliance.
• Add Unicode string arrays, support comparison operations with them, and optimised compression path.
• Add endswith and startswith and extend contains to support strings and offer miniexpr multithreaded computation when possible.
• Use DSL kernels to accelerate arange/linspace constructors by 6-10x.
• Improve documentation for filters and filters_meta.
• Fix edge case issues with resize and constructors so that chunks may be set independently of shape, and arrays may be extended from empty consistently.
• Continued work on miniexpr integration, interface, and support.
• Ruff fixes and implementation of PEP recommendations.

• On Windows, miniexpr is temporarily disabled for integral outputs and mixed-dtype expressions. Set BLOSC2_ENABLE_MINIEXPR_WINDOWS=1 to override this for testing.
• Handle thread workers for computation to ensure never exceeds NUMEXPR_MAX_THREADS. Thanks @skmendez!

• PEP 427 compatibility changes to ensure C-blosc2 files and binaries are stored under blosc2/ subdirectories in shipped Python wheels
• Introduce miniexpr for hyper-fast multithreaded element-wise computations and reductions (on macOS and Linux). This justifies the major version number bump.
• Indexing with None for LazyExpr now matches Numpy behaviour (i.e. newaxis)
• Improvements to open and generally handle Treestore objects and b2z, .b2d, .b2e files. Thanks @bossbeagle1509!
• Minor changes to support new blosc2-openzl plugin

• Hotfix to change WASM wheel hosting to separate repo

• Hotfix for security - disallow import in (saved) LazyUDF objects
• Automate WASM wheel upload via YAML file

• LazyUDF objects can now be saved to disk
• Calls to __matmul__ NumPy ufunc now passed to blosc2.matmul
• Streamlined LazyUDF.compute is now much more robust and functional
• The get_chunk method for LazyExpr is more efficient and enabled for general LazyArray objects
• LazyExpr calculation can now be done even with expressions with pure scalar operands, e.g 10 * 3 +1..

• Change the NDArray.size to return the number of elements in array, instead of the size of the array in bytes. This follows the array API, so it is considered a fix, and takes precedence over a possible backward incompatibility.
• Tweak automatic chunk sizing of results for certain (e.g. linalg) operations to enhance performance
• Bug fixes for lazy expressions to allow a wider range of functionality
• Small bug fix for slice indexing with step larger than chunksize
• Various cosmetic fixes and streamlining (thanks to the indefatigable @DimitriPapadopoulos)

• Small optimisation for chunking in lazy expressions
• Extend Blosc2 computation machinery to accept general array inputs (PR #510)
• Refactoring and streamlining of get/setitem for non-unit steps (PR #513)
• Remote array testing now performed with cat2cloud (PR #511)
• Added argmax/argmin functions (PR #514)
• Change squeeze to return view (rather than modify array in-place) (PR #518)
• Modify setitem to load general array inputs into NDArrays (PR #517)

• LazyExpr.compute() now honors the out parameter for regular expressions (and not only for reductions). See PR #506.

• Bumped to numexpr 2.14.1 to improve overflow behaviour for complex arguments for tanh and tanh
• Bug fixes for lazy expression calculation
• Optimised computation for non-blosc2 chunked array arguments (e.g. Zarr, HDF5)
• Various cleanups and most importantly shipping of python 3.14 wheels due to @DimitriPapadopoulos!
• Now able to use blosc2 in AWS Lambda

• Improved documentation on thread management (thanks to @orena1 in PR #495)
• Enabled direct ingestion of Zarr arrays, and added examples for xarray ingestion
• Extended string-based lazy expression computation using a shape parser and modified lazy expression machinery so that expressions like "matmul(a, b) + c" can now be handled (PR #496).
• Streamlined inheritance from Operand to ensure access to basic methods like __add__ for all computable objects (NDArray, LazyExpr, LazyArray etc.) (PR ##500).

• Bumped to numexpr 2.13.1 to incorporate new maximum/minimum NaN handling and +/* for booleans which matches NumPy behaviour.
• Refactoring in order to ensure Blosc2 functions with NumPy 1.26.
• Streamlined documentation by introducing Array Protocol

Most changes come from PR #467 relating to array-api compliance.

• C-Blosc2 internal library updated to latest 2.21.3, increasing MAX_DIMS from 8 to 16

• numexpr version requirement pushed to 2.13.0 to incorporate round, sign, signbit, copysign, nextafter, hypot, maximum, minimum, trunc, log2 functions, as well as allow integer outputs for certain functions when integr arguments are passed. We also add floor division (//) and full dual bitwise (logical) AND, OR, XOR, NOT support for integer (bool) arrays.

• Extended linear algebra functionality, offering generalised matrix multiplication for arrays of arbitrary dimension via tensordot and an improved matmul. In addition, introduced vecdot, diagonal and outer, as well as useful indexing and associated functions such as take, take_along_axis, meshgrid and broadcast_to.

• Added many ufuncs and methods (around 60) to NDArray to bring the library into further alignment with the array-api. Introduced a chunkwise lazyudf paradigm which is very powerful in order to implement clip and logaddexp.

• Fixed a subtle but important bug for expand_dims (PR #479, PR #483) relating to reference counting for views.

• C-Blosc2 internal library updated to latest 2.21.2.

• numexpr version requirement pushed to 2.12.1 to incorporate isnan, isfinite, isinf functions.

• Indexing is now supported extensively and reasonably optimally for slices with negative steps and general boolean arrays, with both get/setitem having equal functionality. In PR #459 we extended the 1D fast path to general N-D, with consequent speedups. In PR # we allowed fancy indexing and general slicing with negative steps for set and getitem, with a memory-optimised path for setitem.

• Various attributes and methods for the NDArray class, as well as functions, have been added to increase compliance with the array-api standard. In addition, linspace and arange functions have been made more numerically stable and now strictly comply even with difficult floating-point edge cases.

• C-Blosc2 internal library updated to latest 2.21.1.

• Revert signature of TreeStore.__init__ for making benchmarks to get back to normal performance.