NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
PyPI, #3266 most downloaded on PyPI
SGLang is a fast serving framework for large language models and vision language models.
Latest stable
0.5.23
Last release 15 days ago
30 Oct 2026
Ships on a steady schedule
a new release about every 2 weeks
Some releases are documented
notes for 28 of the last 60 stable releases
Nothing withdrawn
no release was ever pulled
3 years old
160 releases · first in 2024
Drag across the ruler to compare, or tap a release to open it.
Unified radix tree by default. The unified tree is now the cache for every model, not just hybrid ones ( #35081 , see Breaking Changes). It also picke…
786 PRs from 214 contributors.
New models in this release (see the cookbook for all supported models):
| Model | Type | PRs | Cookbook |
|---|---|---|---|
| Qwen3.8 (2.4T-A95B) | Autoregressive | #35758 ⭐ | link |
| Qwen3.8-27B | Autoregressive | #34859 | link |
| dots3.note | Autoregressive | #33829 ⭐ | link |
| Ling-3.0-flash | Autoregressive | #33561 | link |
| Ling-3.0-tiny | Autoregressive | #33561 | link |
| Spark2.5 | Autoregressive | #35963 ⭐ | |
| MiniCPM-SALA | Autoregressive | #30360 | |
| Granite 4.2 | Autoregressive | #36286 | link |
| LongCat-Image-Edit & Edit-Turbo | Diffusion | #35829 |
Cookbook updates:
Beam search. SGLang can now do beam search. Pass beam_width in your request and you get back the n best sequences instead of a single sample. It works out of the box next to regular requests, though it does not yet mix with speculative decoding, disaggregation, DP attention, or HiCache (#31626).
DeepEP v2. DeepEP's new ElasticBuffer engine is available as --moe-a2a-backend deepep_v2 for DeepSeek-V3/V4 and Qwen3-MoE in FP8. Its buffers have a fixed size, so decode can run under CUDA graphs even across nodes. Performance is on par with the classic backend (#35634, #34923).
LayerNorm sequence parallelism. With --enable-layernorm-sp, each tensor-parallel rank normalizes only its own share of the prefill tokens instead of all of them. That takes 3.5% off Qwen3-8B prefill on H100 and 5.6% on B200, and the saving grows with the TP degree. Dense Qwen3 models only for now (#30915).
W4A8 MoE on Hopper. If you serve MXFP4 experts on Hopper, you can now quantize the activations to FP8 as well with --flashinfer-mxfp4-moe-precision fp8. DeepSeek-V4-Flash gains about 12% output throughput with no change in GSM8K accuracy. Needs FlashInfer 0.6.18 (#34967).
DCP on the default Blackwell MLA backend. Decode context parallelism now runs on trtllm_mla, not just CuTe DSL and Tokenspeed. It pays off at long context: at 128K input, plain TP stops scaling around 680 tokens per second on eight B200s, while DCP keeps going as concurrency grows (#33926).
Faster speculative kernels. DSA prefill top-k moves to the v2 kernel, 1.3 to 1.8 times faster on B200 (#35175). KDA models get an opt-in fused accept path, SGLANG_OPT_KDA_FUSED_ACCEPT_STATE=1, that cuts MTP verify-and-commit time by 45% to 63% on Kimi-Linear shapes with bit-identical output (#33722).
Unified radix tree by default. The unified tree is now the cache for every model, not just hybrid ones (#35081, see Breaking Changes). It also picked up three things this cycle: PD decode workers can reuse cached prefixes for SWA hybrid models like gpt-oss (#27770), you can attach or detach L3 storage on a running server (#35269), and pipeline parallelism with HiCache L3 stays consistent across ranks (#27010).
Lean attention on AMD. Long or uneven decode batches used to leave many compute units idle on MI300X and MI355X. The new persistent Lean kernel spreads the work across all of them, for up to 1.52x more throughput and up to 3.62x lower inter-token latency on MI355X. It turns on by itself where it helps, and SGLANG_DISABLE_LEAN_ATTENTION=1 turns it off (#33576).
DSA models on ROCm. Disaggregated GLM-5.2 serving now uses the fused top-k seed remap, which brings decode TPOT from 23 ms down to 8 ms on eight MI355Xs (#36714). DeepSeek-V4 gets the v2 top-k kernel, up to three times faster (#36684), and a shared-experts gate fix gives GLM-5.2 up to 16% better TPOT (#36124).
Dependencies. FlashInfer moves to 0.6.18 (#36954), sgl-deep-ep to 0.1.2 (#35450), sgl-deep-gemm to 0.1.7 (#37279), and mooncake to 0.3.13 (#36493). There is a new CUDA 13.4 preview image for Rubin (#36233) and new ROCm 10 images for gfx942, gfx950, and gfx1250 (#36434, #36871).
Full release notes by category below; breaking changes and known issues are at the end.
KimiK3ForConditionalGeneration: #36211<optional> include in AITER topk kernel: #36216Note truncated.
Your coding agent can read these notes before it upgrades. Set up the MCP server →
The first launch after upgrading recompiles once; see Breaking Changes ( #32434 ).
710 PRs from 212 contributors.
New models in this release (see the cookbook for all supported models):
| Model | Type | PRs | Cookbook |
|---|---|---|---|
| Muse Glimmer | Autoregressive (Multimodal) | #34262 | link |
| Intern-S2-Mobius | Autoregressive | #33691 | link |
| SANA-Video | Diffusion | #32921 | link |
| LingBot-Video-MoE | Diffusion | #32341 | link |
| LTX-2.5 | Diffusion | #34471 | link |
| Cosmos3 Edge & Distilled | Diffusion | #31590 | link |
| LongCat-Image | Diffusion | #23274 |
Plus cookbook recipes for the Qwen3.8 family, Ling-3.0, Nemotron 3.5 Lightning, Dots3-Note, and DeepSeek-V4-Pro-0813 (#34809).
Overlapped checkpoint staging at startup: Checkpoint pages now stage from storage while CUDA graphs capture. Qwen3-32B on H100 starts 8.6-11.7% faster than serial with prefetch, and 2.38x faster (35.6s vs 84.8s) than the plain default. Opt in with --startup-weight-load-mode overlap (#32017).
TP LMHead with All-to-All: The TP LMHead's allgather + scatter becomes a single all-to-all for pure-DP dp-attention. On DeepSeek-V4-Pro B200 decode, LMHead time drops 320us to 169us and TPOT improves 36.97ms to 35.67ms (#32313).
FlashInfer MNNVL for pure allreduce: Non-fused allreduce sites now reuse the FlashInfer MNNVL workspace instead of falling back to NCCL. DeepSeek-V4-Flash TP4 decode on Blackwell gains up to +6.9% at small batches. Auto-enabled for DeepSeek-V3/V3.2/V4; elsewhere --enable-flashinfer-pure-allreduce (#30700).
NVFP4 checkpoints run on AMD: --quantization quark_mxfp4 dequantizes ModelOpt and Quark NVFP4 weights and requantizes to MXFP4 at load, never holding a full-precision copy. 97.5-100.2% GSM8K recovery vs the NVFP4 reference across MiniMax-M2.7, GLM-5.1, Kimi-K2.6, Qwen3.5-397B, and DeepSeek-R1 (#29328).
Kimi K3 tuned for MI355X: a grouped-head MLA verify kernel replaces the MHA-shaped split-KV path that re-read the shared latent once per head, for 1.37-1.77x throughput and 1.45-2.42x ITL at concurrency 2-32; the AITER MLA prefill kernel now accepts K3's 12-head shape (TTFT up to -14.9%); a gfx950-tuned decode stage-1 geometry adds 46-73% ITL at 68k input, opt-in via SGLANG_MLA_DECODE_TUNE=1. GSM8K holds at 0.951-0.957 (#33981, #34261, #34837, #34580).
One compiled-kernel cache directory: Triton, FlashInfer, Inductor, DeepGEMM, and CUDA driver caches all move under SGLANG_CACHE_DIR. The first launch after upgrading recompiles once; see Breaking Changes (#32434).
Dependencies: torch 2.13.0 with triton 3.7.1 (#28836), flashinfer 0.6.17 (#33997), CuTeDSL 4.6.2, fixing an FA4 startup regression on Blackwell (#34372), DeepEP now installed from released sgl-deep-ep wheels (#33932), and sgl-kernel 0.4.6.post1 (#33842).
Full release notes by category below; breaking changes and known issues are at the end.
req.prefix_indices when the prefix cache is disabled: #34644is_image_understandable_model: #34217cuda-tile to 1.6.0rc5 to unblock Python 3.10 x86_64 installs: #34321Note truncated.
Full release notes by category below; breaking changes and known issues are at the end.
582 PRs from 194 contributors.
Kimi K3 day-0 support: A 2.8T-parameter multimodal LatentMoE (896 experts, top-16, routed in a 3584-dim latent space) with a 1M-token context, 69 KDA linear-attention layers interleaved with 24 MLA layers, and a MoonViT3d vision tower, shipping as a native MXFP4 checkpoint. SGLang serves it from day 0 with DCP, DSpark speculative decoding, chunked-prefill PP with TP decode, KDA-aware prefix caching, HiCache L2 over DCP, LoRA on the quantized weights, and reasoning, tool-call and OpenAI-compatible serving, verified on NVIDIA GB300 and AMD MI35x (#32541, #32828, #32890, #33025, #33112, blog, cookbook, roadmap).
MiniMax-H3 day-0 support: MiniMax's video generation model that produces a video and a synchronized stereo audio track in one request, served natively on SGLang-Diffusion across all three public task profiles: text-to-video-and-audio (t2va), first/last-frame conditioning (fl2va), and image/video/audio reference conditioning (ref2va, which also covers video-to-video). Verified on B200 (TP2 + Ulysses4), H100 (TP2 + Ulysses2), AMD MI300X and MI355X (Ulysses1/2/4/8), and 2x RTX 5090 with layerwise offload (#33275, cookbook).
Other new models added: EmbeddingGemma and LFM2.5 embedding models, nvidia/MiniMax-M3-NVFP4, plus cookbook recipes for Poolside's Laguna-S-2.1 family and Inkling-Small.
Initial support for the Rust frontend: Migrates the front half of the server, everything from network ingress up to the point a tokenized request is handed to the GPU scheduler, from Python to a multi-threaded Rust implementation (#29799).
DCP communication backends and q-replicate (Helix): The DeepSeek-MLA decode context-parallel path gains pluggable comm backends. a2a exchanges packed attention output plus fp32 LSE in a single NCCL collective per layer, with fp8 KV carried as uint8 byte transport; fi_a2a delegates the cross-rank exchange to the FlashInfer MNNVL kernel on GB200. --dcp-replicate-q-proj projects full-head Q locally and skips the per-layer Q head-dim all-gather. Select with --dcp-comm-backend {ag_rs, a2a, fi_a2a} (#21637).
DWDP for MoE prefill: A new prefill parallelism strategy that prefetches peer expert weights over NVLink P2P and computes all experts locally, removing EP all-to-all token dispatch. On 4x B200 with gpt-oss-120b, prefill-only, DWDP4 reaches 1.92x over DEP4 at MNT 32K / ISL 32K, and 506K vs 329K tok/s (1.54x) at saturation (CONC=128, ISL=8K). Enable with --dwdp-size; the authors mark it early-development (#29778).
Session-reference-aware Unified Radix Cache: For agentic and RL-rollout workloads, requests can carry a stable session_id so eviction knows which prefixes an active session still references, instead of evicting purely by cache policy. Release the references with /close_session. Opt in with --enable-session-radix-cache (#29173).
SM90 FP8 MegaMoE for DeepSeek-V4: Adds the DeepGEMM MegaMoE A2A path on SM90 for DeepSeek-V4-Flash/Pro FP8, including the pre-dispatch JIT kernel and FP8 expert weight preparation. Guarded behind SGLANG_OPT_USE_DEEPGEMM_MEGA_MOE=1 (#29016).
Faster large-MoE model loading: Oversized or non-contiguous CPU weight views were driving pathological H2D transfers, with DeepSeek-V4-Pro TP8 spending 27 to 32 minutes in H2D on some ranks. Copying those views into contiguous storage before H2D cuts full model loading from about 35 minutes to 6m20s (5.6x), and GPT-OSS-20B BF16 from 545s to 70s (7.8x), with Qwen3.5-397B measured at 1.93x to 2.3x. Opt in with SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D, off by default (#32315).
Lower DeepSeek-V4 memory on AMD: Removing unnecessary expert padding drops MI355X FP4 MoE model weights from 159.07 GB to 112.36 GB, and bringing the HIP compress-state pool into the memory_saver KV_CACHE region lets colocated RL reclaim it, cutting the measured training-phase footprint from about 143 GiB to 87 GiB per GPU (#31450, #31747).
Faster engine recovery: Large-model restarts cost 3 to 6+ minutes today, about 6.5 minutes for Qwen3-235B FP8 on 4 GPUs, because weights reload from storage and CUDA graphs recapture. A weight-cache daemon holds weights per GPU so a restarting engine can recover from cache instead (#27139).
Lower host overhead in hybrid-linear MTP decode: Under spec-v2 overlap scheduling each decode step runs draft, verify and extend CUDA graphs, and the eager seams between them become GPU idle time at low concurrency. This trims that host work so the host stays off the critical path (#32219).
Dependencies: flashinfer 0.6.15.post1 (#31927), sgl-deep-gemm 0.1.5.post1 (#32345, #33143), helion 1.4 (#32562), mooncake 0.3.12.post1 (#32302), dynamo-tokenizers 1.7.0 (#32981). PyTorch stays at 2.11.0 and the CUDA base image at 13.0.1.
Full release notes by category below; breaking changes and known issues are at the end.
| Model | Type | PRs | Cookbook |
|---|---|---|---|
| Kimi K3 | Autoregressive (Multimodal) | #32541, #32828, #32890, #33025 | link |
| MiniMax-H3 | Diffusion | #33275 | link |
| MiniMax-M3-NVFP4 | Autoregressive | #31989 | |
| EmbeddingGemma | Autoregressive (Embedding) | #32375, #32383 | link |
| LFM2.5 | Autoregressive (Embedding) | #28691 | link |
A native Rust serving layer: tokenizer manager, ingress validation and egress, an OpenAI-compatible API server, and PD disaggregation support, shipped as prebuilt release artifacts.
trtllm_mha support for Gemma 4 MTP draft attention backend: #25545_online_combine_kernel: #32277load_weights v2 dispatch (#28671): #31988GrammarMask type across all decode paths: #32409retrive_next_token: #32380Note truncated.
Breaking Changes & Upgrade Notes
574 PRs from 169 contributors.
DSpark: confidence-driven speculative decoding: A new speculative algorithm. It drafts semi-autoregressively in blocks, then sizes each verify window from the draft's own confidence instead of a fixed draft length. Reaches 383.7 tok/s at accept length ~5 on DeepSeek-V4-Pro, TP8 on B300 (bs=1). Enable with --speculative-algorithm DSPARK and SGLANG_RAGGED_VERIFY_MODE=compact; tune the block with --speculative-dspark-block-size (#30261, #31434, blog).
Inkling support: A 975B-parameter multimodal MoE with a 1M-token context. It mixes sliding-window, full and Mamba2 linear attention, and adds an NVFP4 MoE, optional vision/audio towers and native MTP. On Blackwell it reaches up to 71.7k tok/s input and 171.0 tok/s per-user decode. Verified on Blackwell TP4/TP8, H200 and AMD MI350X / MI355X (#31681, blog, cookbook).
Other new models added: LongCat 2.0 FP8, JetBrains Mellum v2, Pi0.5, plus diffusion support for LongLive 2.0.
UnifiedRadixTree is now the default for SWA, Mamba and DSA models. Replay SSM and Mamba int8 checkpoints are synced onto it, and a cache hit now resets only the state it used (#30468, #30636, #30626, #31643).
GLM-5.2 DSA cache layer split under prefill CP: KV and indexer cache layers are sharded across CP ranks. Each rank owns a disjoint layer range instead of all layers. That cuts per-rank KV memory by ~74% (0.77 to 0.20 GB/rank) at 8192 tokens on GLM-5.2-FP8, 78 layers, cp_size=4. Enable with --enable-dsa-cache-layer-split, which needs --enable-prefill-cp --cp-strategy interleave (#29421).
ReplaySSM Ring Spec-Verify (GDN): Drops the per-draft SSM snapshot. Speculative scratch goes from 11.5 GB to 1.8 GB per GPU (6.4x smaller) on Qwen3.5-35B-A3B at TP1, at accuracy and throughput parity. Opt in with --enable-gdn-replayssm-spec (default off; GDN with a linear draft chain only, --speculative-eagle-topk in {None, 1}), and tune the ring via --linear-replayssm-cache-len (#28695).
Linear attention on Blackwell (SM100): The first correct KDA MTP path. Its recurrent_kda decode kernel runs at 29.6 us vs 36.8 us for Triton (ncu, B=64). The full decode path reaches parity by B=128 and 1.35x at B=256, and is slower below that (#30113). Separately, GDN/KDA CuteDSL prefill fuses state I/O into the chunk-h kernel (#30169).
QServe and FBGEMM FP8 quantization are removed: the experimental QServe (QoQ) W4A8 and FBGEMM FP8 paths are gone. --fp4-gemm-backend cutlass goes too, along with the in-tree NVFP4 JIT kernels, so NVFP4 GEMM now requires FlashInfer (#31109, #30448).
Dependencies: flashinfer 0.6.14 (#29910), CuTe DSL 4.6.0 (#31714), sgl-kernel 0.4.5 (#31496), llguidance 1.7.6 (#31484).
--fp4-gemm-backend cutlass is removed along with the in-tree NVFP4 JIT kernels, so NVFP4 GEMM now requires FlashInfer. Use auto, which picks flashinfer_cutedsl on SM100 and flashinfer_cutlass on SM120: #30448UnifiedRadixTree is now the default for SWA, Mamba and DSA models. A behavior change on those architectures: #30468docs/ removed; the Mintlify cutover is complete: #28964sglang.kernels namespace: kernels are relocated verbatim and only import paths change; public wrappers keep defaulting to the AOT sgl_kernel backend, so code reaching past them to internal paths must update (RFC #29630): #30044, #31582num_tokens_per_bs renamed to num_tokens_per_req across spec-decoding runners: #30977--enable-deepep-waterfill is renamed to --enable-waterfill with no deprecated alias, so existing launch commands fail with unrecognized arguments: #27350--optimistic-prefill-retries is renamed to --optimistic-prefill-attempts with no deprecated alias: #30951application/msgpack instead of JSON, with tensors as raw msgpack bytes rather than base64 (tensor_to_base64 / base64_to_tensor become tensor_to_bytes / bytes_to_tensor), so RL rollout consumers must be upgraded in lockstep with the server: #31565Full release notes by category below.
| Model | Type | PRs | Cookbook |
|---|---|---|---|
| Inkling | autoregressive | #31681 | link |
| LongCat 2.0 | autoregressive | #30275, #30320 | link |
| JetBrains Mellum v2 | autoregressive | #27375 | wip |
| Pi0.5 | vla | #30633 | link |
| LongLive 2.0 | diffusion | #27639 | link |
Landed this cycle but not yet usable end-to-end: MiniMax-M3 completes its four-part landing (#28715, begun in v0.5.14) but its cookbook still points at a dev image (#31819).
SGLANG_MAX_NEW_TOKENS_LIMIT to cap per-request max_new_tokens: #22591run_batch launch: #31687LogprobResult and rename chunk env vars: #31733--enable-flexkv routes the KV cache through FlexKV's KVManager for host-tier offload, configured via --flexkv-config-file: #29701SGLANG_HICACHE_FILE_BACKEND_ENABLE_METADATA_CACHE, off by default): #29716sglang.kernels, RFC #29630)Full Changelog: v0.5.15...v0.5.16
v0.5.15.post1 includes a few patches, mostly for GLM 5.2
v0.5.15.post1 includes a few patches, mostly for GLM 5.2
GLM-5.2 NVFP4, tuned for production : We took time this cycle to tune GLM-5.2 NVFP4 on Blackwell for optimized production serving. It now runs at 500+
GLM-5.2 NVFP4, tuned for production: We took time this cycle to tune GLM-5.2 NVFP4 on Blackwell for optimized production serving. It now runs at 500+ tok/s/user on 8x B300, 450 on 4x GB300 (bs=1). Run GLM-5.2 with our cookbook.
New Model Support: Hunyuan 3 (Hy3), Hierarchical Reasoning Model (HRM-Text), NVIDIA LocateAnything-3B, Baidu Unlimited-OCR, JoyEcho multi-shot A/V, plus Qwen3.6 NVFP4 support.
Native web search (Exa): Built-in web_search support backed by Exa (#29342).
Breakable CUDA Graph on by default: Breakable CUDA Graph is now the default capture path, reducing per-step kernel-launch overhead (#29458); full CUDA Graph support for the prefill phase lands as experimental (#27988).
Linear-attention kernels (KDA / GDN): New FlashKDA prefill backend for safe-gate KDA linear attention (#29472), plus ReplaySSM buffered output-only decode for linear attention (#28451).
FlashInfer A2A for routed MoE: Adds FlashInfer all-to-all with the flashinfer_trtllm_routed MoE runner (#22394).
DeepSeek-V4 Optimization:
Decode Context Parallelism: decode context parallelism lands for MLA models, including DeepSeek V3 and Kimi K2 series (#14194).
Dependency upgrades: transformers bumped to 5.12.1 (#29393); tvm-ffi / sgl-deep-gemm / tilelang upgraded (#29554).
and . See the DeepSeek-V4 cookbook.
Full release notes by category below.
has_separate_cfg: #29688All PRs included in this release: v0.5.14...v0.5.15
Approximate list from git history; GitHub's "Generate release notes" produces the canonical, deduped list.
Full Changelog: v0.5.14...v0.5.15
New Model Support: GLM-5.2 , LiquidAI LFM2.5 , Kimi-K2.7-Code , Poolside Laguna-M.1 , DiffusionGemma , Zyphra ZAYA1, MiMo-V2-ASR
New Model Support: GLM-5.2, LiquidAI LFM2.5, Kimi-K2.7-Code, Poolside Laguna-M.1, DiffusionGemma, Zyphra ZAYA1, MiMo-V2-ASR
DeepSeek-V4 on GB300 since Day 0: 5x higher throughput at the same interactivity, serving DeepSeek-V4 on NVIDIA GB300 with SGLang (blog).
Waterfill & LPLB MoE load balancing: Two dispatch-time load-balancing methods for DeepEP expert parallelism: Waterfill for shared-expert dispatch and LPLB for redundant expert replicas, improving throughput for DeepSeek-V3/R1 and DeepSeek-V4 (blog).
KDA CuteDSL prefill kernel on Blackwell (SM100): New CuteDSL prefill kernel for Kimi-Linear (KDA), 1.08-1.52x faster than the Triton path via a reusable scratch workspace, plus a cuda-graph padding fix (#27488); see the Kimi-Linear cookbook.
Linear-attention prefix-cache memory savings: An int8 checkpoint pool stores recurrent states compactly in the Mamba radix cache, substantially increasing prefix-cache capacity for KDA / GDN models (#28185); the speculative conv-window intermediate cache is deduplicated with a sliding-window layout, halving its footprint with no numerical change (#28302).
LPLB: linear-programming load balancer for MoE expert parallelism: Balances token routing across redundant expert replicas by solving a per-layer LP; opt-in via --ep-dispatch-algorithm=lp, default behavior unchanged (#24515).
MSCCL++ integration & MNNVL allreduce fusion: MSCCL++ migrates to the upstream mscclpp Python package (Executor + DSL compiler) with auto-tuned collectives for TP=8 single-node and TP=16 two-node (#22734); FlashInfer fused allreduce + residual + RMSNorm re-enables an MNNVL backend behind --flashinfer-allreduce-fusion-backend (auto / trtllm / mnnvl), fixing the piecewise-CUDA-graph interaction (#23402).
Nemotron DP attention + MTP: Data-parallel attention for the hybrid Nemotron-H (Mamba2 + full attention + MoE), plus MTP support (#24955); see the Nemotron 3 Ultra cookbook.
AMD: breakable CUDA graph on ROCm/HIP: The breakable CUDA graph execution path now runs on AMD GPUs (#28173).
NVFP4 MoE for DeepSeek-V4: Adds an NVFP4 MoE quantization path for DeepSeek-V4 on Blackwell for higher MoE throughput; enable with --moe-runner-backend flashinfer_trtllm_routed (#25820); see the DeepSeek-V4 cookbook.
DeepSeek-V4 decode & quantization optimizations: FP8 group quantization now emits power-of-two (UE8M0) scales directly from the per-token group-quant kernel, dropping a separate rounding pass (#26766); MLA decode q-heads are padded to 64 under attention-TP so FlashMLA dispatches the ~2x cheaper head64 kernel instead of head128 (#27954); the MHC prenorm kernel is prewarmed at startup to remove the first-run JIT slowdown on a fresh server (#27986); and BF16 mixed-dtype compression states are supported on the C4 / C128 paths (#27277); see the DeepSeek-V4 cookbook.
Full release notes by category below.
fast_prefill_plan for EAGLE draft-extend CUDA graph: #28854No security-tagged PRs in this release.
All PRs included in this release: v0.5.13...v0.5.14
Full Changelog: v0.5.13...v0.5.14
Nothing published for this version
One column per month.