bitnet_flutter_ai

Releases

1. ▸▾0.2.0-beta.111 Aug 2026pre-release

   Bumps to 0.2.0-beta.1 and finalizes the CHANGELOG Unreleased section covering Web WASM inference, the PrismML Q2_0 backend, the 10-model catalog, and

   Bumps to 0.2.0-beta.1 and finalizes the CHANGELOG Unreleased section
   covering Web WASM inference, the PrismML Q2_0 backend, the 10-model
   catalog, and iOS support (dual-backend xcframeworks + SPM layout).
   Also refreshes README.md, which still described iOS as Planned and
   had stale build instructions.

   Open source →

   Additional notes

   Added

   • iOS support (device + simulator), both backends. Ships prebuilt bitnet_bridge.xcframework (static archive merged with llama/ggml) and prism_bridge.xcframework (self-contained dynamic framework) in ios/Frameworks/, covering arm64 device and arm64/x86_64 simulator slices. The two frameworks export the same bn_* names over incompatible ggml copies, so they resolve differently at runtime: bitnet through DynamicLibrary.process(), prism through .open(path). The plugin moved to the SwiftPM layout (ios/bitnet_flutter_ai/Sources/) so CocoaPods and SPM build the same sources, with Package.swift added. bitnet_force_link.m keeps the vendored static archive's bn_* symbols — reached only via FFI dlsym, so nothing references them at link time — from being dead-stripped, without resorting to -force_load (which breaks clean Xcode builds because the linker path is validated before the Copy-XCFrameworks phase stages it). Verified end-to-end in the simulator with both the bitnet (I2_S) and prism (Q2_0) backends.
   • ModelCache model-picker helpers. Added isCached, hasPartialDownload, and cachedStatus so a model-picker UI can show download state without triggering a load. A .verified stamp is written after a successful SHA-256 check so later loads skip re-hashing a multi-GB file, hashing now runs on a background isolate instead of the UI thread, and onProgress is throttled to 0.5% steps.
   • Expanded model catalog (10 models). Added six TII models that ship pre-built I2_S GGUFs and run on the existing bitnet.cpp native stack with no rebuild — Falcon3-{1B,3B,7B,10B}-Instruct-1.58bit and Falcon-E-{1B,3B}-Instruct — plus three PrismML Ternary-Bonsai-{1.7B,4B,8B} models on a new second backend (below). All are SHA-256 pinned in BitNetCatalog.all; compatibleWith(ram) now surfaces a real size ladder. contextLength is capped at 4096 per entry (the native layer allocates the full KV cache up-front, and history is trimmed separately).
   • Per-model chat templates. New lib/src/session/chat_templates.dart maps a model's ChatTemplateFamily (bitnetLabeled / falconTagged / chatml / chatmlQwen3) to its prompt formatter and end-of-turn stop markers. The wrong template produces degenerate output (the "runaway !" failure), so BitNetSession and both engines now dispatch per model instead of hardcoding BitNet's format. Verified: Falcon3-1B coherent via falconTagged.
   • Second native backend: PrismML Q2_0 (Ternary Bonsai). PrismML's models use a qwen3/Q2_0 ggml format incompatible with bitnet.cpp, so they route to a new self-contained libprism_bridge.so (native/prism_bridge.cc over the PrismML llama.cpp fork, built by tool/build_{linux,android}_prism.sh). It exposes the same bn_* C ABI as the bitnet bridge (zero Dart FFI changes) and a version script exports only bn_* so its modern ggml symbols can't clash with the bitnet stack loaded in the same process. BitNetModelInfo.engineBackend selects the library at load time. Verified: Bonsai-1.7B coherent via chatmlQwen3. Not available on web yet (no WASM build of the fork).
   • Working Web (WASM) inference. The web engine is now backed by a real WebAssembly build of bitnet.cpp (ggml + llama.cpp) + the embind bridge (native/bitnet_bridge_wasm.cc), produced by tool/build_web.sh. The module runs inside a Web Worker (web/bitnet_worker.js) — the analogue of the native inference isolate — so model loading and token generation never block the UI thread; web/bitnet_glue.js is a main-thread async proxy that exposes window.BitNetWasm. The worker downloads the GGUF (provisionModel, Cache-API persisted, streamed into the in-memory FS) and frees it after bn_init. Verified end-to-end against the real 1.1 GB I2_S model (coherent output, clean EOS stop). Single-threaded scalar (~1–2 tok/s for the 2B model); threading and a wasm_simd128 kernel are documented speed follow-ups. Desktop browsers only — the 1.1 GB model needs multi-GB per-tab memory.
   • Quiet WASM logging. The bridge installs a llama_log_set filter on web that drops everything below ERROR. llama.cpp's INFO/WARN chatter (the full GGUF metadata dump, a per-special-token "not marked as EOG" line, …) otherwise crosses the WASM→JS boundary as thousands of stack-traced console.error calls during load, which froze the tab/devtools on its own.
   • DeviceProfile compiles on web. Platform detection moved behind a conditional import (platform_info_impl.dart / platform_info_web.dart) so the file no longer imports dart:ffi / dart:io on web (which failed to compile). Web reads navigator.hardwareConcurrency for the core count.
   • Mobile-first SDK surface. Added the public types BitNetCatalog, BitNetModelInfo, BitNetCapability, BitNetQuantization, DeviceProfile, BitNetSession, ChatRole, ChatTurn, BitNetRag, RagAnswer, KnowledgeSource, KnowledgeChunk, RetrievedChunk, AssetKnowledgeSource, InMemoryKnowledgeSource, Bm25Retriever, and TextChunker.
   • Catalog & compatibility. BitNetCatalog.all, BitNetCatalog.byId, BitNetCatalog.withCapability, and BitNetCatalog.compatibleWith(ramBytes) for picking a model that fits the device.
   • Device profile & speed estimate. DeviceProfile.current() returns OS / ABI / cores / RAM, and DeviceProfile.estimateTokensPerSecond(model) produces a heuristic tok/s anchored at 12.5 tok/s on an 8-core x86_64 reference.
   • Chat session. BitNetSession provides streaming chat(), setSystemPrompt(), clearHistory(), plus convenience helpers summarize(text, {style}) (paragraph/bullets/tldr) and assist(question, {context}).
   • On-device RAG. BitNetRag indexes AssetKnowledgeSource(assetPaths: [...]) (or InMemoryKnowledgeSource) with Bm25Retriever + TextChunker and exposes ask() returning RagAnswer{answer, citations}.
   • Android prebuilts. android/src/main/jniLibs/{arm64-v8a,x86_64}/ ships libbitnet_bridge.so, libllama.so, libggml.so. Reproducible via tool/build_android.sh (NDK 28).
   • Example app. example/lib/main.dart now demos every public API across four tabs: Device, Chat, Summarise, Ask (RAG) — backed by example/assets/knowledge/bitnet_faq.md.
   • End-to-end Linux verification. The full pipeline (bitnet.cpp → libllama.so → libbitnet_bridge.so → Dart FFI → token stream) is now validated on x86_64 Linux. A Ryzen 7 5700U produces ~12 tok/s for the BitNet b1.58 2B-4T I2_S model with 749 ms cold model load.
   • tool/dart_smoke.dart — Dart-only CLI integration test that drives NativeLibrary directly against a local GGUF, useful for iterating on the FFI bridge without going through Flutter.
   • native/smoke_test.c — standalone C harness for verifying the bridge before Dart is in the loop.
   • @visibleForTesting BitNetModel.forTest — lets unit tests build models with arbitrary (or computed) SHA-256 fixtures now that the canonical hash is pinned.

   Changed

   • C bridge updated for bitnet.cpp's pinned llama.cpp. Renamed llama_model_load_from_file → llama_load_model_from_file, llama_model_free → llama_free_model, llama_model_n_ctx_train → llama_n_ctx_train. Switched llama_batch_get_one to its 4-argument form and adapted llama_model_meta_val_str to the buffer-output variant (the bridge now caches the result in a per-context scratch buffer).
   • BitNetModel.bitnet2B4T.ggufSha256 is now pinned to 4221b252fdd5fd25e15847adfeb5ee88886506ba50b8a34548374492884c2162, enabling real download integrity checks. ggufSizeBytes updated to the exact 1,187,801,280-byte upload.
   • native/CMakeLists.txt: the Linux/macOS shared library now keeps the canonical lib prefix (libbitnet_bridge.so/.dylib) so it matches what DynamicLibrary.open expects. The PREFIX "" strip is now Windows-only.
   • pubspec no longer declares the web/ JS files as Flutter assets. A package asset resolves under assets/packages/..., which can never claim the service-worker scope index.html needs, and it would otherwise be bundled into every Android/iOS build for nothing. See "Web Setup" in the README for the copy-into-your-own-web/-directory workflow.

   Fixed

   • Degenerate logits on arm64 (NaN/garbage output, iOS device + Android arm64-v8a). The classic tree's scalar ggml_vec_dot_i2_i8_s block layout was wrong, and its buggy ggml-cpu-i2s.c GEMM/GEMV fast path was reachable on arm64, which has no AVX2. Patch 0002-i2s-arm-scalar-and-fastpath-gate fixes the scalar layout and gates that fast path to AVX2/AVX512 only; the x86_64 ABI already used the verified AVX2 kernels either way. Applied by tool/build_android.sh.
   • Prefill SIGBUS/SIGSEGV past ~130 tokens (classic-tree I2_S gemm). The prefill path sized a stack VLA as rows-per-thread * n_ubatch floats, but ggml worker threads only get the pthread default stack (1 MB on Android, 512 KB on Darwin) — a prompt long enough to fill the micro-batch overran it and crashed inside ggml_vec_dot_i2_i8_s. Patch 0003-i2s-gemm-no-stack-vla accumulates straight into dst instead, so prefill stack use no longer scales with the prompt.
   • Runaway ! output on Android (and any non-AVX2 target). The BitNet I2_S matmul kernels in bitnet.cpp shipped only AVX2 and NEON implementations with no fallback, so on the Android x86_64 emulator (NDK baseline has no AVX2) they compiled to empty functions: every matmul wrote nothing, logits came out flat, and greedy sampling emitted an endless stream of token 0 ("!"). Added a portable scalar fallback (validated bit-exact against the AVX2 kernel) plus a canonical-layout NEON path, distributed as native/patches/0001-i2s-portable-kernels.patch and applied by tool/build_android.sh. Also fixes an arm64 PARALLEL_SIZE mis-dispatch that corrupted 4-row prefill tail blocks, and a wrong-layout quantize_i2_s NEON path that produced corrupt GGUFs when quantizing on ARM.
   • Multi-piece end-of-turn markers not detected. <|eot_id|> / <|end_of_text|> arrive as literal text spread over several BPE pieces; the old per-piece equality check never matched, so the marker leaked into replies and generation ran on into hallucinated turns. Added TurnStopDetector (streaming, rolling-tail match) and wired it into both the native and Web engines.
   • Degenerate-logits guard. bn_prompt now detects flat/non-finite logits after prefill and fails with a clear error instead of streaming garbage — surfaces a broken-kernel build immediately rather than as "!" spam.
   • Prefill stack overflow on long prompts. bn_init caps n_ubatch at 128 to bound the stack VLA in the I2_S gemm path, which could otherwise exceed a mobile thread's ~1 MB stack for wide FFN tensors.
   • Stop button / cancelGeneration() had no effect. The native inference isolate ran generation as a synchronous loop over the blocking FFI nextToken(), which starved its own message port — the _CmdCancel sent by BitNetSession.stop() was queued but never delivered until generation had already finished. The worker now yields to the event loop between tokens, so a cancel is honoured at the next token boundary.
   • RAM gate was model-agnostic. Both engines gated load() on a hardcoded 3072 MB constant (DeviceInspector.meetsMinimumRam) instead of the selected model's minimumRamBytes — wrong in both directions now that models range 0.6–4 GB on disk: a small model was needlessly blocked on a <3 GB device, and a large one passed the gate then OOM'd inside bn_init. The gate now checks the specific model, and BitNetUnsupportedDeviceException reports the actual required RAM and model name.
   • i2_s gemm tail-block out-of-bounds (crash on WASM, silent corruption elsewhere). Under ACT_PARALLEL, ggml_gemm_i2_i8_s calls the kernel with nrc = cur_r (activation rows) meaning "one weight row · nrc columns" — _Nx1 semantics. The dispatcher routed any nrc not a multiple of PARALLEL_SIZE (e.g. a 23-token prefill → tail of 3) to _1x1, which treats nrc as weight rows and reads the weight pointer out of bounds. On bounds-checked WebAssembly this traps (memory access out of bounds); on native it reads adjacent heap and corrupts the tail rows' logits. The scalar and NEON-dotprod _Nx1 handle any nrc, so the tail now routes to _Nx1; the AVX2 path (which blocks by PARALLEL_SIZE) is left byte-identical. Folded into native/patches/0001-i2s-portable-kernels.patch.

   Documentation

   • BUILD_GUIDE.md rewritten with the verified Linux recipe (gcc-only requirement, kernel codegen step, staging symlinks, and bridge smoke tests).

   ---