v0.5.0 Latest

Latest

Compare

Choose a tag to compare

Overview

llama.cpp 0.4.0 adds initial Qwen3.8-Flash-Next and Nemotron-3-Puzzle support, on-demand tensor reading, per-slot server context limits, video input options, and a ggml update to 0.23.0 with major sparse flash attention and RDMA work.

API changes

• Added llama_lazy_mode and lazy_mode (#27794).
• Added max_buf_size to quantize params (#27795).
• Bumped session/state versions for KV-cell token tracking (#27762).
• Added mtmd_input_part and mtmd_tokenize_from_parts() (#28250).
• Made mtmd_tokenize const-correct (#28310).
• Made mtmd_bitmap_init_lazy const-correct (#28307).
• Added mtmd helper video/init options (#24318).
• Server multimodal helpers accept mtmd_helper_init_opt (#24318).

New models

• Added initial Qwen3.8-Flash-Next (qwen4exp) architecture support; optimization improvements are still pending (#27742).
• Added NVIDIA Nemotron-3-Puzzle-75B-A9B support (#25444).
• Added DSpark support for Nemotron 3.5 (#27804).
• Added support for nanbeige4.2-3B (#27730).

Core changes

• Added lazy tensor reading (#27794).
• Renamed option to --lazy-mode (#27969).
• Added --n-cpu-ffn (#26622).
• Added per-layer expert routing/FFN (#28323).
• Added KV-cell token tracking (#27762).
• Added n-gram history lookup (#28040).
• Optimized KV cache restore (#27991).
• Stopped early sequence scans (#28011).
• Added quantizer RAM cap (#27795).
• Added quantizer row-slab streaming (#27830).
• Prevented RAM peaking during load (#27483).
• Autoscaled YaRN training context (#28030).
• Disabled non-fused GDN/LID (#27877).
• Simplified MiniMax-01 graph (#27790).
• Added DFlash2 support (#27816).
• Fused DFlash encoder into KV injection (#27310).
• Added sparse flash attention for DeepSeek-V4/GLM and Qwen4exp (#27970).
• Added Apple RDMA as an RPC transport (#26421).
• Added RPC event/async backend APIs (#18626).
• Fixed Qwen4exp graph issues (#27941).
• Added Qwen4exp recurrent rollback (#28123).
• Summed Qwen4exp indexer heads (#28023).
• Reduced Qwen4exp graph splits (#27880).
• Fixed DeepSeek-V4 input vision (#28154).
• Fixed Gemma-4 vision handling (#28335).
• Fixed Gemma-4 assistant (#28183).
• Added synthetic spec acceptance options (#27711).
• Made build info stream configurable (#28322).
• Fixed JSON-schema empty object grammar (#28279).

Multi-modality changes

• Added mtmd_tokenize_from_parts() (#28250).
• Added DeepSeek-V4-Flash-Vision-Exp (#28133).
• Added DeepSeek-V4 input vision handling (#28154).
• Fixed Gemma-4 vision handling (#28335).
• Fixed Idefics3 preprocessing (#28273).
• Fixed Qwen3-TTS (#28231).
• Added video parameters/--video-* (#24318).
• Made image preprocessors const-correct (#28310).
• Made audio preprocessors const-correct (#28307).

Server changes

• Added per-slot context limit (#24124).
• Accept data: URLs for media (#27735).
• Enabled preserve_reasoning by default (#28174).
• Reject prefilled assistant tool calls (#27626).
• Added synthetic spec acceptance options (#27711).
• Added pytest-xdist server tests (#28298).

UI changes

• Improved Chat Form Actions UI/UX (#27746).
• Replaced MCP overrides with tool policy (#27745).
• Moved Settings/MCP routes to dialogs (#27744).
• Updated dialog styling (#27743).
• Copy grouped agentic response text (#27832).
• Updated ESLint config (#27700).
• Updated package scripts (#28270).

ggml changes

• Bumped ggml from v0.22.0 to v0.23.0 (release).

Version 0.23.0 adds new operators and backend scheduling APIs for sparse attention, asynchronous execution, and allocation-dependency tracking. The most notable additions are sparse flash attention (ggml_flash_attn_ext_set_n_kv_max), RPC event/async APIs, and Apple RDMA transport support. It also includes broad correctness and performance work across CPU, GPU, and accelerator backends.

Assets

Nightly build: b10809

More info

• Releases and versioning of ggml-org projects

Changelog since v0.3.0

5266f24 llama.cpp : bump version to 0.4.0 (#28386)
64a155d sync : ggml (#28379)
163a407 model, mtmd: fix gemma4 vision handling (#28335)
8f83678 metal : add remaining fa-vec tunings for M3 Max (#28373)
86b351f ggml : replace compile definitions with version.h.in (#28364)
d509cb1 Don't use npx inside a package.json script (#28270)
4cbe8b0 ggml : don't crash when backend search path can't be read (#28271)
24f5bf8 ggml : remove GGML_CUDA_PEER_MAX_BATCH_SIZE (#28177)
a529af9 docs : update maintainer PRs link and regenerate AUTHORS (#28365)
38521ec vendor: update BoringSSL to 0.20260903.0 (#28354)
0ef4d56 ci : disable failing openvino tests (#28347)
c390d0a common : make build info output stream configurable (#28322)
832fd6f ggml-cpu(s390x) : fix q5_1 uninitialized v_acc (#28332)
9a4843c src : add n_expert_used_max function (#28323)
6703d78 sycl: fuse rms_norm+mul+add and add+add residual chains (#27610)
f9f09f0 SYCL: Refactor GGML_SYCL_ENABLE_MKL_FA to global var (#26863)
d230ddd llama: fix whole source code rebuilt on each new commit (#28278)
c5a5535 common/json-schema : fix GBNF grammar generation for empty object schemas (#28279)
95ef7fc opencl: quant lm_head / decode GEMV and medium-batch GEMM optimizations (speculative decoding/MTP) (#26477)
8c1a251 tune MMVQ to MMQ crossover for SM87 (#28285)
d30500b snapdragon: ci updates to use new run script (#28293)
e107984 ops: add Hexagon to ops.md and update main README.md (#28263)
42f0225 server : use pytest-xdist for server tests (#28298)
de8656b mtmd: propagate const to preproc class (#28310)
7bb0fc1 metal : add sparse FA (#28098)
0df017d metal : fix glu dispatch with ne00 = 1 (#28306)
f45576a mtmd : add const in various places (#28307)
0ba6499 CUDA: Allow concurrent streams per split for multi-GPU (#28198)
c7bda03 vulkan: fix FA dequant path engagement (#28190)
0df974d sycl : enhance the api to support peer-to-peer copy (#27550)
d646c9d convert : skip bias_vl tensor in DeepSeek-V4 DSpark conversion (#28294)
5ec4eab misc : prevent RAM peaking at model loading stage (#27483)
4aa6ffb sycl: reduce redundant work in Q4_K multi-column MMVQ (#27062)
c61b98b model: add NVIDIA Nemotron-3-Puzzle-75B-A9B (NemotronHPuzzle) support (#25444)
67a17c1 mtmd: fix idefics3 preproc (#28273)
159b741 finetune: fix no KV cache (#27199)
9cffdcc server : accept data: URLs for input_video and input_audio (#27735)
f027c4f ggml-hexagon: add F16 support for unary ops (#28228)
7339054 mtmd: add mtmd_tokenize_from_parts() (#28250)
9cc3394 metal : add fa-vec tunings for M3 (#28236)
8c0b9cd metal : fix memory query under low-memory conditions (#27701)
03dbcc5 ci : check for missing autoreleasepools (#27884)
cff1844 Update ROCm to 10.0.0 release (#27803)
9400c89 model: correctly support input vision for deepseek4 (#28154)
d5fec32 ci : enable hf-jobs on server-cuda (#28258)
3d3d7c8 ggml-cuda : remove unused vars (#28235)
e750b88 common, server : enable preserve_reasoning kwarg by default, log its effective state (#28174)
7798007 mtmd: support DeepSeek-V4-Flash-Vision-Exp (#28133)
8e93a97 CUDA + ggml: add sparse-fa for DSV4/GLM (#27970)
0f3a71b mtmd: Fix Qwen3-tts-0.6b (#28231)
b81c99b ggml: avoid KleidiAI buffer type init on dispatch (#27891)
960dffa hexagon: MUL_MAT and MUL_MAT_ID fusion and fixes (#28202)
ba8818c vulkan: handle larger batch sizes (>4) efficiently for IQ3_S mat-vec (#27449)
56dd815 vulkan : only request VK_KHR_shader_bfloat16 extension if supported (#28155)
2637dfe ggml-cpu : conditionally add SpacemiT IME kernel sources (#27961)
43d87ff opencl: fix out‐of‐bound reads in the Adreno image kernels (#27632)
69320fe hexagon: add missing FARF logs for cpy/get_rows/set_rows/gdn ops (#28217)
b96806d metal : add metallib build support for xcframework (#28163)
3466812 cuda: fuse MoE weighted expert reduction (#25952)
b356fa2 kv-cells: look up the n-gram history in the sequence position index (#28040)
dfc29b6 context : autoscale n_ctx_train when yarn scaling specified (#28030)
f28493c models : appropriately flag noscan ssm_a tensors (#28121)
73159c3 model : fix gemma4-assistant (#28183)
d11b3cc model : load relevant arrays with n_layer_all (#28173)
c845263 Revert "sycl : add Kronecker product FWHT support for sizes 384, 640, 768, 12…" (#28184)
1f3d318 sycl : add Kronecker product FWHT support for sizes 384, 640, 768, 1280 (#28016)
8887a48 metal : add fa-vec tuning for M2 Pro (#28122)
be789c3 metal : add fa-vec tunings for A18 Pro (MacBook Neo) (#28152)
9d81721 model : load hparams.n_layer_nextn before n_layer() calls (#28159)
fe2120b metal : fix more leaks due to missing autoreleasepools (#27883)
d08c787 metal : add fa-vec tuning for M2 Max (#28015)
5eec3ad sycl : support limit max alloc memory within 2GB for host-pinned memory (#27559)
36b1015 qwen4exp: fix seq_cp, block position keying, mtmd input, cuda abort, add tests (#27941)
d086dbb tests : fix log verbosity for test-llama-archs (#28147)
1b89a43 quantize: row-slab stream to avoid thread starvation (#27830)
d5d993a metal: enable Metal 4.0 tensor API on M5+/A19+ (#27461)
234a6eb ci: Bump ggml-org/ccache-action to v1.2.24 (#28083)
518b762 kleidiai : Update KleidiAI Documentation (#26078)
0eadefe qwen4exp: support recurrent state rollback (#28123)
09412af qwen4exp: sum the indexer heads by slices (#28023)
458681e metal : add fa-vec tunings for M1 Ultra (#28088)
e4b9af0 CUDA: XOR swizzle flash attn K,V smem fp16 tiles (#25635)
ab0b3bd metal : add concat support for quantized types (#28116)
85c5522 AVX2: Speed up large batch size prompt processing of IQ models (#27402)
2a74817 metal : add top-k radix implementation (#28073)
2d8d612 kv-cache : optimize restoring non-contiguous cells (#27991)
010be96 opencl: tune the quant paths for Intel Xe-LP GPUs to improve its TG and PP performance (#26438)
774ee0e ui: copy the displayed text of grouped agentic responses (#27832)
8e53fce webgpu : avoid crash when offset is not multiple of 4 in WebGPU ggml_backend_tensor_get() implementation (#28045)
f8dbcd6 ROCm: add radix TOP_K for long rows (#27466)
5d4a3be metal : add fa-vec tunings for M1 (#28078)
41ef91f CUDA: extend MOE fusion to specdec, earlier MOE glu fusion and topk-router fusion were restricted to 1 token (#27621)
a32af33 sycl : Enhance to get the free memory of Intel GPU (

Note truncated.

Overview

llama.cpp 0.3.0 introduces the dots3-note multimodal model (with a new DSA-ISWA KV cache), MTP support for GLM-4.5-Air, and tensor-split (-sm tensor) plus multi-sequence rollback fixes for DeepSeek 4. ggml is bumped to v0.22.0 (meta-backend tensor split, per-op Metal kernels with parallel compilation, non-in-place ggml_clamp), while mtmd gains dots3-note vision/audio, WebP decoding and a Pillow-accurate resize. The server adds a LLAMA_SERVER_SLOTS_N_DIFF debug knob, and the web UI gets tabbed chat navigation.

New models

• Add dots3-note model with a new DSA-ISWA KV cache type (#27060)

Core changes

• DeepSeek 4: add tensor-split mode via -sm tensor (#26490)
• DeepSeek 4: fix rollback with multiple sequences (#26756)
• Fix meta tensor split state propagation for tensor parallel (#27574)
• GLM-4.5-Air: add MTP (multi-token prediction) support (#26534)
• bailingmoe3: support DSpark (#27508)
• mamba2: flatten in/out projections to dispatch GEMM instead of GEMV (#27513)
• Models: use ggml_rope_set_offset in deepseek2/4, dflash, minicpm3 and plm (#27382)
• Grammar: parse \- in char classes as a literal hyphen (#27591)
• Common: add json.h abstraction (#27511) with a clang LTO fix (#27575)
• Common: fit moved out of the server and now takes n_streams into account (#27496)
• Common: fix draft-mtp with embeddings (#27400)
• Arg: remove the -no-cnv CLI option (#27542)

Multi-modality changes

• Support dots3-note vision and audio (#27524)
• Support WebP images via ffmpeg (#27520)
• Fix loading videos with the moov atom at the end of the file (#27596)
• Use a Pillow-accurate resize algorithm and correct resize_algo for all models (#27594)
• Use ggml_rope_set_offset in the CLIP graph (#27521)

Server changes

• Add LLAMA_SERVER_SLOTS_N_DIFF env var to widen the slot debug diff window (#27600)
• Slot fitting logic moved to the common fit, now accounting for n_streams (#27496)
• Adopt the common json.h abstraction (#27511)

UI changes

• Tabbed navigation for chat conversations (#27263)
• Fix keyboard shortcuts for the chat tabs navigation (#27609)

ggml changes

• ggml bumped to v0.22.0 (ggml/1607):
  • This release adds tensor-split support to the multi-backend (meta) backend with improved split-state propagation, reworks the Metal kernels into per-op sources with parallel compilation, and fixes ggml_clamp to be a proper non-in-place op. It also brings new ops
    (POOL_1D, PAD_REFLECT_1D), Q2_K SYCL kernels, MoE bias fusion on OpenCL, and assorted fixes across the CUDA, Metal, SYCL, Vulkan, OpenCL and WebGPU backends.

Assets

Nightly build: b10621

More info

• Releases and versioning of ggml-org projects
• Help tune Metal Flash-attention kernels

Changelog since v0.2.0

c1d0e7a llama.cpp : bump version to 0.3.0 (#27696)
81191af sync : ggml
9388236 ggml : bump version to 0.22.0 (ggml/1607)
eb25b72 grammar : parse - in char classes as literal hyphen (#27591)
814d84b sycl : mark tq2_0 as not supported (#27660)
5ea87dd webgpu : fix handling of infinity values during ARGSORT and TOP_K (#27538)
f280b26 metal : per-device tuned (Q, NE) for flash-attn vec (#26570)
b615f5b metal: per-op source split + parallel compile (#26561)
b3c3b96 misc : read repetition_penalty from generation_config.json (#27659)
7584430 tests : disable DOTS3NOTE arch test for WebGPU (#27654)
71cc86f convert: fix GLM regression in index_tensors (#27655)
a14dba6 ggml : shorten virtual device naming in CUDA and Metal (#27608)
c1c766d webgpu : reorder includes since V that appears in common_decls.tmpl may be defined as K in flash_attn_decls.tmpl if KV_OVERLAP (#27545)
160c6b0 mtmd: video: fix moov atom at the end of file (#27596)
985b149 ci : apply ccache-clear with older/min/dry-run to all ccache jobs (#27602)
6036c63 ggml : fix ggml_clamp (#27644)
a130532 mamba2 : Flatten in/out projections to dispatch GEMM instead of GEMV (#27513)
bf0a29c Deepseek 4: -sm tensor (#26490)
c060ca9 model : support MTP in GLM-4.5-Air (#26534)
ccc8fd2 readme : update links (#27617)
d05f895 fix: Change chat tabs nav shortcuts (#27609)
8d9af25 test : fix multi-GPU server tests (#27614)
4a08fa2 test: move tools/parser to tests (#27548)
56db501 mtmd: use pillow-accurate algo, correct resize_algo for all models (#27594)
95b8e33 ci : add test-llama-archs tensor split for Metal (#27598)
a278dce contrib : recommend waiting for CI before merging (#27603)
e8eed45 server : add LLAMA_SERVER_SLOTS_N_DIFF (#27600)
ba8e0ed common : skip device_info loop if it's not going to be printed (#26692)
b0539c4 DeepseekV4: fix rollback with multi-seq (#26756)
d337192 [Tensor parallel] Fix meta tensor split state propagation (#27574)
8144f31 ui: Chat Conversation Tabbed navigation (#27263)
6657ded vendor : update subprocess.h (#27409)
29ea941 cuda : add POOL_1D support (#27573)
70adb1b common: json.h: fix clang lto (#27575)
3f545be vulkan : added the PAD_REFLECT_1D operation (#26586)
b21e4de mtmd: use ggml_rope_set_offset (#27521)
d9f918d common: add json.h abstraction (#27511)
2fb989b fit: also take into account n_streams (#27496)
9fee29e arg: remove -no-cnv from cli [no ci] (#27542)
e85caa8 ci : Restore ROCm job for Ubuntu (#27399)
2115b73 model : support DSpark for bailingmoe3 (#27508)
54ee5ee mtmd: support dots3-note vision+audio (#27524)
3a653fe ci : add older, min and dry-run options to ccache-clear (#27504)
369e1cd ggml: optimize concat op by replacing per-element memcpy with row-level memcpy (#24575)
2c6b141 common : fix draft-mtp with embeddings (#26352, #27299) (#27400)
8672290 sycl : add Q2_K reordered MMVQ and ESIMD kernels (again) (#27490)
3aeb924 readme : fix server badge alt (#27533)
2100e59 readme : update badges (#27531)
d775b89 mtmd: support webp via ffmpeg (#27520)
3af988f opencl: fold the gpt-oss MoE per-expert bias adds into the epilogue (op/kernel fusion) (#26431)
9a286ac docs: improve Windows build instructions (#27381)
a3b9c23 ci : fix empty release_id in make-release upload step (#27516)
5a32f7b model: add dots3-note (#27060)
873e5d8 model: use ggml_rope_set_offset() (#27382)
d7fa69b ci : run ccache-clear as the last step of release jobs (#27503)

v0.2.0

Compare

Choose a tag to compare