llama_cpp_dart

release: 0.9.0-dev.10

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

fix(build): disable MTMD_VIDEO across all native builds

The d6d0ce82 bump pulled in mtmd video decoding, which #includes
vendor/sheredom/subprocess.h and calls posix_spawn to launch ffmpeg. That
broke the Android CPU AAR build (posix_spawn isn't in the NDK at API 26),
and the feature is useless in every target we ship (it needs an ffmpeg
binary in PATH at runtime). MTMD_VIDEO defaults ON upstream; force it OFF
in all four build scripts. Only the mtmd_helper_video_* path drops out;
the bitmap-init symbols the binding uses are unaffected.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Pure-Dart release on the same llama.cpp pin (tag b9360, sha 6b4e4bd58) — no native rebuild required.

MTP (NextN) speculative decoding

• MtpSpeculativeDecoder — self-speculative decoding driven by a model's own Multi-Token Prediction / NextN heads (no separate draft model). Pair a normal target LlamaContext with a draft context created ctxType: ContextType.mtp off the same model. Mirrors upstream llama.cpp's MTP loop (PR #22673): the target emits pre-norm hidden states, the NextN head proposes tokens conditioned on them, and the target verifies a round in one pass. Output is byte-identical to plain greedy decoding.
• Works on M-RoPE / multimodal models (Qwen3.6, etc.): rejected drafts roll back via PARTIAL_ONLY | ON_DEVICE state checkpoints rather than partial seq_rm, which those models forbid.
• Reads pre-norm hidden states through the llama_set_embeddings_pre_norm / llama_get_embeddings_pre_norm_ith staging symbols (resolved by hand since they are absent from the public header). This supersedes the dev.7 note that MTP-as-draft was blocked — it is now implemented.
• Performance is hardware-dependent: high acceptance (85–92%, output identical to greedy) but on Apple Metal it is ~break-even on MoE and modestly slower on dense vs plain decode (per-submission overhead); expected to win on cheaper-dispatch backends. Probe: tool/probe_mtp.dart.

KV-cache quantization

• KvCacheType.iq4_nl — exposes GGML_TYPE_IQ4_NL, the last upstream-supported KV cache type the binding was missing. ~4× compression like q4_0 but better quality from a non-linear codebook. See the README's KV-cache quantization section (incl. the symmetric _0 integer-dot-product property and a note on fork-only TurboQuant).

llama.cpp submodule bumped from gguf-v0.18.0-791-g5d56effde to tag b9360 (sha 6b4e4bd58). 328 commits of upstream history, purely-additive C API delta (llama_context_type, llama_n_rs_seq, new llama_state_seq_flags, mtmd_get_cap_from_file, plus the context-params fields ctx_type / n_rs_seq). Bindings regenerated; existing wrapper code unchanged.

Embeddings

• LlamaEngine.embed(text) — pooled and per-token embeddings via the worker isolate, with optional L2 normalization. Returns EmbeddingResult covering both pooled (mean/cls/last/rank) and unpooled outputs.
• BatchEmbedder (sync) + LlamaEngine.embedBatch(texts) (off-thread) — embed N texts in a single decode pass by assigning each its own sequence id; amortizes per-token compute across the batch for RAG-style ingest. Requires embeddings: true, a pooled pooling type, and nSeqMax >= texts.length.

Speculative decoding

• SpeculativeDecoder — synchronous greedy and exact stochastic speculative decoding over a target + draft LlamaContext sharing a vocab. Greedy output is byte-identical to plain greedy on the target; temperature > 0 runs the min(1, p/q) accept rule with residual resampling (distributionally identical to sampling the target), seed for reproducibility. See example/probes/speculative_generate.dart. (MTP-as-draft is blocked upstream — it needs the non-public llama_set_embeddings_pre_norm; the draft-model variant works today.)
• ContextType + nRsSeq on ContextParams — build an MTP draft context against an MTP-capable target model for raw-FFI use.

New high-level surfaces

• LlamaLora + LoraBinding, with LlamaContext.setLoraAdapters / clearLoraAdapters / setControlVector. LoRA stack swaps, metadata accessors, aLoRA invocation-token reads, and ReFT-style control vectors.
• MtmdBitmap / MtmdChunk / MtmdChunks / MtmdCapabilities — bitmap construction (raw RGB, raw audio, file decode, buffer decode), mtmd_input_chunks introspection (kind, nTokens, nPos, id, text-token reads), plus a cheap mtmd_get_cap_from_file probe.

Context introspection and ops

• nCtxSeq, nRsSeq, effective poolingType.
• Runtime toggles: setThreads, setEmbeddings, setCausalAttn, setWarmup, synchronize.
• Memory ops: memoryClear, memorySeqRm, memorySeqCp, memorySeqKeep, memorySeqAdd, memorySeqDiv, memorySeqPosMin/Max — covers forking, rollback, position shifting.
• Logits & probs: lastLogits, logitsAt, sampledTokenAt, sampledProbsAt, sampledCandidatesAt, sampledLogitsAt.

Diagnostics

• ContextPerf / SamplerPerf snapshots with perf() / resetPerf() / printPerf() on LlamaContext and Sampler. Includes prompt/decoded tokens-per-second convenience getters.

Model / library accessors

• LlamaModel: isDiffusion, isHybrid, nSwa, nEmbdInp, nEmbdOut, decoderStartToken, nClassifierOut, classifierLabel(i), ropeType (new RopeType enum), ropeFreqScaleTrain, metaCount + metaKeyAt / metaValueAt / metaValue(key) / metaEntries.
• LlamaLibrary: supportsMmap / Mlock / Rpc, maxParallelSequences, maxTensorBuftOverrides, timeUs, systemInfo(), initNuma(NumaStrategy.*).
• SplitPath.compose / decomposePrefix for split-gguf filenames.

Sampler chain introspection

• name, seed, chainCount, chainGet(i) (borrowed), chainRemove(i) (owned), clone(), apply(arr).

Session state

• captureRawStateExt / restoreRawStateExt accepting the new StateSeqFlags (mirrors LLAMA_STATE_SEQ_FLAGS_* including the b9360 on-device snapshot bit).

Build

• tool/build_native.sh disables LLAMA_BUILD_SERVER and LLAMA_BUILD_APP: upstream b9360's tools/server/ references mtmd symbols missing from its own public header. We ship neither target, so turning them off keeps --with-mtmd building cleanly.

params: expose missing llama.cpp options across sampler/context/model

- SamplerParams: Mirostat (v1/v2), grammar (incl. lazy patterns), DRY,
XTC, dynamic temperature, adaptive-P, top-n-sigma, infill, logit
bias, shared min_keep. SamplerFactory.build now accepts model: for
vocab-dependent stages.
- ContextParams: RoPE scaling/freq, full YaRN knobs, pooling and
attention type, defrag threshold, no_perf, op_offload, swa_full,
kv_unified.
- ModelParams: split mode, main GPU, tensor split, device list, GGUF
kv overrides (int/float/bool/str), use_direct_io, use_extra_bufts,
no_host, no_alloc.
- README: link to aichat sample app.
- Bump to 0.9.0-dev.6 + CHANGELOG.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Consolidates 0.9.0-dev.0 through 0.9.0-dev.5 (none of dev.0–dev.4 were published to pub.dev). The 0.2.x line is a separate package shape — see MIGRATION.md.

Highlights since 0.2.x

• LlamaEngine worker isolate is the primary public API. Streaming token output via Stream<GenerationEvent> (sealed: TokenEvent | ShiftEvent | DoneEvent). Cancellation via stream subscription cancel.
• EngineSession (raw prompt) and EngineChat (message-history with chat template) on top of the engine isolate.
• Multimodal (vision + audio) via llama.cpp's mtmd.
• Persistence: EngineSession.saveState/loadState and EngineChat.saveState/loadState with metadata-validated reload.
• llama-server-style context shift (ContextShiftPolicy.auto) gated on engine.canShift.
• Three platform artifacts shipped from GitHub Releases:
  • macOS dylib (for dart test)
  • Apple xcframework (ios-arm64, ios-arm64-simulator, macos-arm64)
  • Android AAR for arm64-v8a, two flavors: CPU+mtmd (~2 MB) and Hexagon NPU + OpenCL + mtmd (~3.7 MB)

Validated end-to-end on real devices

• Galaxy S23 Ultra (Snapdragon 8 Gen 2, Android 14) — Hexagon NPU reachable from a third-party Flutter app on commercial firmware.
• Galaxy Fold7 (Snapdragon 8 Elite, Android 16) — same APK runs unchanged.
• MacBook Pro M1 Max (macOS 26) — Metal via dylib path.
• iPad M1 (iOS 26) — Metal + Accelerate BLAS via the bundled CocoaPods llama_cpp.podspec.

Bindings

• Backend inspection. engine.devices (List<BackendDevice>), engine.hasAccelerator, engine.primaryAcceleratorName, and the pre-engine LlamaBackends.list(). Tells you which backends loaded on the current device.
• primaryAcceleratorName priority orders by registry name (HTP → Hexagon → Metal → CUDA → Vulkan) before type, so Snapdragon HTP wins over OpenCL even when ggml reports both as type=gpu.
• KV-cache quantization. ContextParams.typeK / typeV accept any of KvCacheType.{f32, f16, bf16, q8_0, q4_0, q4_1, q5_0, q5_1}. q8_0 halves KV memory at small quality cost; useful on 8 GB Android devices with longer contexts.
• Stderr capture. LlamaLog.captureToFile(path) / LlamaLog.restoreStderr(). Toggleable redirect of llama.cpp/ggml log lines for Android, where stderr is not connected to logcat.
• Auto ADSP_LIBRARY_PATH. LlamaLibrary.load() reads /proc/self/maps on Android and exports ADSP_LIBRARY_PATH so FastRPC finds libggml-htp-v*.so skeleton libs without app-side MethodChannel plumbing.
• LlamaBindings is now exported. Lets callers using the raw FFI surface type variables / pass them around without reaching into src/.
• LlamaVersion is generated at build time. Exposes the package version, the llama.cpp submodule SHA + author date, and a runtime systemInfo() wrapper around llama_print_system_info() (e.g. MTL : EMBED_LIBRARY = 1 | CPU : NEON = 1 | ACCELERATE = 1 | ...).

Removed since 0.2.x

• The Llama god-class.
• LlamaParent / LlamaChild / IsolateScope (replaced by LlamaEngine).
• LlamaService multi-session scheduler. Mobile apps do one conversation at a time; multi-session can be added back as a higher layer if needed.
• The MCP client / server / agent surface.
• TextChunker (RAG helper).
• Hand-written chat-format classes (ChatML, Alpaca, Gemma, Harmony). Modern llama.cpp embeds Jinja templates in the GGUF; we use llama_chat_apply_template instead.
• All non-mobile platform code: Linux, Windows, CUDA, Vulkan desktop. macOS is kept as a dev/test target.
• Bundled binary distribution. Native artifacts ship from GitHub Releases instead.

Known limitations

• Custom Jinja chat templates (some Unsloth quants) require manual prompt rendering. Real Jinja support is post-1.0.
• HTP only engages Q4_0 / Q8_0 quants in upstream ggml-hexagon. K-quants (Q4_K_*, Q5_K_*) and I-quants (IQ*) run on OpenCL+CPU.
• HTP REPACK budget is ~2 GB per session; ≥7B-class models need a multi-session pattern not yet exposed by the binding.
• Multimodal generation does not auto-shift on context overflow (matches llama-server's behaviour).
• Cosmetic: ggml_metal_device_free asserts at process exit because the worker doesn't dispose model/context. Harmless.

• allow freeing the active slot by switching/detaching and reselecting a fallback
• ensure isolate child always replies on dispose/free, even when already torn down
• keep parent subscription alive through shutdown so free-slot confirmations are received
• cancel scope work before freeing slots to avoid in-flight races
• add opt-in KV auto-trim (sliding window) with example example/auto_trim.dart

• Android: Added OpenCL support for GPU acceleration (#91).
• Vision:
  • Fixed crash in mtmd context disposal.
  • Stable Qwen3-VL support.
• Chat: Added experimental support for Qwen3-VL chat format (_exportQwen3Jinja).
• Fixes:
  • Improved logging initialization (#88).
  • Fixed stream processing crash in chat.
• Core: Updated llama.cpp submodule.

• forgot to update version

• State load / save
• llama.cpp 25ff6f7659f6a5c47d6a73eada5813f0495331f0
• harmony prompting syntax
• isolate has vision and verbose support
• mcp server / agent
• scope generation stopping

• Multimodal support - vision

• Major internal refactoring to improve code organization and maintainability
• Fixed critical bug where subsequent prompts would fail due to batch seq_id memory management
• Improved position tracking for continuous conversation support
• Enhanced error handling and debugging capabilities
• Added foundation for future chat optimization features
• Breaking change: Internal API restructuring (public API remains stable)

• disabled llava
• compatible with llama.cpp 42ae10bb
• add typed_isolate
• removed llama processor

• updated binding
• performance imporvement and bugs fix

• added initial support to load lora
• dart cli example
• fixed #3 by @danemadsen

• removed assets defination
• added static property Llama.libraryPath to set library path, in order to support linux and other platforms

• ModelParams disabled options splitsMode, tensorSplit and metadataOverride

• LlamaProcessor now take context and model parameters

• refactored code to follow dart package structure

• TODO: Describe initial release.