NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
PyPI
PyPI, #3266 most downloaded on PyPI
SGLang is a fast serving framework for large language models and vision language models.
Latest stable
0.5.23
Last release 23 days ago
30 Oct 2026
Ships on a steady schedule
a new release about every 2 weeks
Some releases are documented
notes for 29 of the last 60 stable releases
Nothing withdrawn
no release was ever pulled
3 years old
160 releases · first in 2024
Drag across the ruler to compare, or tap a release to open it.
The `tokenspeed_mla` attention backend is removed. tokenspeed_mla still starts as a deprecated alias for cutedsl_mla with a warning. To switch, pass -…
1007 PRs from 257 contributors.
| Model | Type | Cookbook |
|---|---|---|
| Muse Spark Open | LLM / VLM | link |
| Beam-1.0 | LLM / VLM | link |
| MiniMax-M3.1-Flash | LLM / VLM | link |
| MiniCPM-V 4.7 | LLM / VLM | link |
| Kev | LLM / VLM | link |
| Cosmos3-Nano-Sim (Bimanual, Transfer, Auto) | Diffusion | link |
| FastH3 Omni | Diffusion | link |
| FLUX.3 Image Klein | Diffusion | link |
| HY-Nomas | Diffusion | link |
| MOSS-Image | Diffusion | link |
uv pip install --prerelease=allow sglang==0.5.23
| Platform | Docker |
|---|---|
| NVIDIA (CUDA 13) | docker pull lmsysorg/sglang:v0.5.23 |
| AMD MI35x | docker pull lmsysorg/sglang:v0.5.23-rocm10-mi35x |
| AMD MI30x | docker pull lmsysorg/sglang:v0.5.23-rocm10-mi30x |
| Intel GPU | docker pull lmsysorg/sglang:v0.5.23-xpu |
| Intel CPU | docker pull lmsysorg/sglang:v0.5.23-xeon |
| Google TPU7x (TorchTPU) | docker pull lmsysorg/sglang:v0.1.0-tpu |
--speculative-skip-dp-mlp-sync: #45043deep_gemm for 32x32 block-FP8 linears: #45585tool_choice="none": #41233--cpu-offload-gb on a 16 GB card): #42557SGLANG_USE_ROCM_AITER_TGEMM=1; Wan2.2 denoising 3.55x faster on MI450): #41988--quantization fp8 on gfx1250 (torch.compile lowering + per-channel weight scales): #45304SGLANG_GFX1250_W4A8_KERNEL_GATHER=1; gpt-oss-120b W4A8 output throughput +30%, TTFT 40% lower): #45758Note truncated.
Your coding agent can read these notes before it upgrades. Set up the MCP server →
Add --attn-dp-size and deprecate --enable-dp-attention: #41818
950 PRs from 245 contributors.
| Model | Type | Cookbook |
|---|---|---|
| Clef / Clef Flash | LLM / VLM | link |
| PPLX-Decider v1 / v1.1 | LLM / VLM | link |
| EmbeddingGemma 2 | LLM / VLM | link |
| LLaDA2.2 | LLM / VLM | |
| HunyuanImage-3.0-Instruct | Diffusion | link |
| Kandinsky 6 / Kandinsky 6 SR | Diffusion | link |
| Wan-Animate-2 | Diffusion | link |
| SANA-Video 2.0 | Diffusion | link |
| LLaDA-Image / LLaDA-Image-Turbo | Diffusion |
SGLang now requires Python 3.11 or newer.
uv pip install --prerelease=allow sglang==0.5.22
| Platform | Docker |
|---|---|
| NVIDIA (CUDA 13) | docker pull lmsysorg/sglang:v0.5.22 |
| AMD MI35x | docker pull lmsysorg/sglang:v0.5.22-rocm10-mi35x |
| AMD MI30x | docker pull lmsysorg/sglang:v0.5.22-rocm10-mi30x |
| Intel GPU | docker pull lmsysorg/sglang:v0.5.22-xpu |
| Intel CPU | docker pull lmsysorg/sglang:v0.5.22-xeon |
MarkovSampleStep, one op for a markov draft step's add and token pick: #43621MarkovSampleStep: #43622SGLANG_DSPARK_FOLDED_SAMPLING: #43668dspark-draft: #43669SGLANG_DSPARK_USE_GREEDY_DRAFT, an argmax draft accepted as one-hot: #43670prefix_lens: #42797cache_unfinished_req to checkpoint_req and count cache hits only when a request finishes: #41520refresh_fill_ids public: #42194UnifiedRadixCache; reject unverified tree caches: #42295UnifiedRadixCache when the radix cache is disabled: #42354SGLANG_OPT_SWA_RELEASE_LEAF_LOCK_AFTER_WINDOW to SGLANG_OPT_RELEASE_PREFILL_SWA: #42657match_prefix; read KV indices off the node path: #42923init_load_back; drop empty_device_indices: #43023PureSWARadixCache: #43198UnifiedRadixCache: #43199UnifiedRadixCache: #43200ChunkCache: #43201The experimental Rust router and frontend (experimental/sgl-router, sglang-processor) keep moving toward parity with the Python server.
sglang-processor lib: #41896model name on the OpenAI chat and completions routes: #43923SGLANG_SIMULATE_ACC_LEN inside the accept instead of switching paths: #43623convrot_int8_customkernel): #38040gate_up LoRAs and diffusers metadata alpha: #42174--offload-during-compile false): #43602Note truncated.
[Misc] Remove deprecated endpoints, env vars and aliases past two releases: #40795
779 PRs from 227 contributors.
| Model | Type | Cookbook |
|---|---|---|
| DeepSeek-V4.1 Flash | LLM / VLM | link |
| GigaChat 3.5 | LLM / VLM | link |
| IQuest-Q1 | LLM / VLM | link |
| MiMo-V2.6 / MiMo-V2.6-Pro | LLM / VLM | link |
| Ling-3.0-flash-VL | LLM / VLM | link |
| DiffusionGemma | Diffusion | link |
| Qwen-Image 2.1 | Diffusion | link |
| Anima Base v1.0 | Diffusion | link |
| Ming-Image 0.1 Design / Design-Layer | Diffusion | link |
| FLUX 3 Action | Diffusion | link |
/v1/decisions) turns an LLM / VLM into a low-latency classifier and scorer (docs, #41208)./v1/score) scores all candidates in one request (#38965, #41188).uv pip install --prerelease=allow sglang==0.5.21
| Platform | Docker |
|---|---|
| NVIDIA (CUDA 13) | docker pull lmsysorg/sglang:v0.5.21 |
| AMD MI35x | docker pull lmsysorg/sglang:v0.5.21-rocm10-mi35x |
| AMD MI30x | docker pull lmsysorg/sglang:v0.5.21-rocm10-mi30x |
| Intel GPU | docker pull lmsysorg/sglang:v0.5.21-xpu |
| Intel CPU | docker pull lmsysorg/sglang:v0.5.21-xeon |
TreeSpeculativeSamplingTargetOnly: #35798none decode retraction backup and subclass seams in the PD queues: #41103/server_info: #39993cpp_extension loader: #40989RadixCache no-insert cleanup with kv_len_to_handle: #35204owned_kv_len on radix cache insert: #40075send_with_url, fix broken tests: #40502/v1/decisions route it builds on): #41208The experimental Rust router and frontend (experimental/sgl-router, the Rust renderer) keep moving toward parity with the Python server.
exclude: #39317page_first layout: #40770, #39572nvidia/cuda:13.4.1-cudnn-devel-ubuntu24.04 base, sgl-kernel 0.4.7, sgl-deep-gemm 0.2.0, sgl-deep-ep 0.1.2: #40987; Triton 3.8 compatibility for DeepSeek-V4.1-Flash on Rubin: #40805acf8fdf9: #39965SGLANG_UNIFIED_RADIX_TREE_CORE_BACKEND=python to force the Python core everywhere: #39627SGLANG_FLASHINFER_MOE_FUSED_FINALIZE=1 to restore the fused atomic finalize: #40105SGLANG_DISAGGREGATION_DEFERRED_DECODE_KV_RELEASE_TIMEOUT). Set SGLANG_DISAGGREGATION_DEFERRED_DECODE_KV_RELEASE=False to restore immediate release: #41023chat_template key inside chat_template_kwargs on /v1/chat/completions or /v1/tokenize now returns an error unless the server starts with --trust-request-chat-template; other chat_template_kwargs are unchanged: #28135/model_info instead of /get_weight_version and /weight_version, /v1/loads instead of /get_load, and POST /hicache/storage-backend/clear instead of /clear_hicache_storage_backend. SGLANG_NSA_* aliases give way to SGLANG_DSA_*, SGLANG_ENABLE_SPEC_V2 is gone, --elastic-ep-rejoin becomes --elastic-ep-join-mode recover, --kv-cache-dtype fp4_e2m1 becomes fp4_mx_block16, and python -m sglang.bench_offline_throughput / sglang.bench_one_batch move to sglang.benchmark.offline_throughput / sglang.benchmark.one_batch: #40795SGLANG_EXPERIMENTAL_CPP_RADIX_TREE) is gone (#40775), and SGLANG_ENABLE_UNIFIED_RADIX_TREE no longer does anything, though it still prints a deprecation warning (#40780). Code importing SWARadixCache, MambaRadixCache, or HiRadixCache must move to UnifiedRadixCache: #40313, #40787CustomLogitProcessor that returned a reduced shape such as (1, vocab) and relied on broadcasting now raises at decode time. The built-in processors are unaffected: #39234update_weights_from_tensor and update_weights_from_distributed must be called between begin_weight_update() and end_weight_update(). UpdateWeightsFromTensorReqInput.disable_draft_model is replaced by selector="target", and ChecksumInfo.parallelism_info is now a list with one entry per role: #40777sglang.srt.distributed; read groups through get_parallel() (for example get_parallel().tp_group) or import from parallel_state directly: #40342, #40344, #40707Note truncated.
*Full release notes by category below; breaking changes are at the end.*
713 PRs from 237 contributors.
New models in this release (see the cookbook for all supported models):
| Model | Type | PRs | Cookbook |
|---|---|---|---|
| GLM-5.3-Flash | Autoregressive | #36507, #38621 | link |
| Hy4-Preview | Autoregressive | #36805 | link |
| Qwen3.8-Flash-Next | Autoregressive | #37500 | link |
| K2 Horizon | Autoregressive | #37654, #38033 | link |
| Nanbeige4.2 | Autoregressive | #32151 | |
| SenseNova-U1.5-8B-MoT | Diffusion | #36606 | link |
| FastH3 (4-step MiniMax-H3 distill) | Diffusion | #37480 | link |
| VDN-H3 (hybrid-attention MiniMax-H3 distill) | Diffusion | #37903 | link |
Sampling masks for RL rollouts. With return_sampling_mask, each decode step returns the exact token support the sampler drew from and the log-probability of the sampled token under it, so a trainer can replay the rollout without reconstructing top-k or top-p (#36630). Masks now run under overlap scheduling: on Qwen3-8B, decode throughput is 17% higher at batch 1 and 52% higher at batch 64 than the previous implementation. Capacity is set by --sampling-mask-max-tokens (default 4096) (#36631). DisallowedTokensLogitsProcessor is supported alongside masks (#38279).
Unified radix tree. Branching-point caching for the SWA component keeps the sliding-window state at the point where requests fork from a shared prefix, so branches reuse it instead of recomputing. On DeepSeek-V4-Flash with a shared system prompt, token hit rate rises from 43.8% to 60.8% and mean TTFT falls from 1.57 s to 1.07 s (#34565). An opt-in external linker lets the tree address a shared global memory pool through Mooncake or UMBP (#37381).
DSpark under PD with decode context parallelism. A DCP1 prefill can now transfer its DSpark draft KV to a DCP-N decode, so hybrid models such as Kimi-Linear run DSpark in disaggregated, context-parallel serving. Verified on 8x B300 over NIXL and Mooncake up to 256K input (#37709).
Responses API storage is opt-in. /v1/responses no longer retains results in memory unless the server starts with --enable-response-store. Without it, retrieval, previous_response_id chaining, and background requests return 400; PD deployments cannot enable it (#39122).
SGLang Simulator. A CPU-only simulator runs the real scheduler, radix cache, and hierarchical cache with a latency predictor in place of the model forward. Against measured serving traces it predicts TTFT within about 6% on most traces (up to 10% on the longest 32K to 128K ones) and prefix reuse within 0.05 percentage points, for cache and scheduling studies without GPUs (#33824).
Prefill context parallelism v1 removed. The strategy-based implementation is now the only prefill CP path; the v1 runtime and its CLI options are gone. Prefill CP on HIP, NPU, and MUSA is rejected until those platforms are ported (#36228).
Faster model loading on ROCm. Large pageable host-to-device copies are staged rather than pinned in place, which stops the driver from suspending GPU queues on every eviction. GLM-5.2 at TP4 on 4x MI355X loads in 40.4 s instead of 505.7 s (#37720).
Intel XPU joins the release images. Every tagged release now builds and publishes lmsysorg/sglang:vX.Y.Z-xpu from the XPU Dockerfile, so Intel GPU users get a versioned image instead of relying on nightly builds (#37340).
DeepSeek-V4 on Blackwell. TRT-LLM attention kernels now cover DeepSeek-V4's CSA and HCA layers on SM100 and SM103: about 1.2x faster prefill and 1.45x faster decode than FlashMLA at the kernel level on B200 (#30805). FlashInfer MegaMoE is available as --moe-runner-backend flashinfer_megamoe; on DeepSeek-V4-Flash NVFP4 at TP4/DP4 it adds up to 11.9% prefill throughput at 8192 tokens per rank, with decode within 2% of the trtllm runner at saturation (#31470).
DeepSeek-V4 on RTX PRO 6000. On SM120 the sparse-MLA indexer now runs on DeepGEMM's paged-MQA kernel and the DeepGEMM FP4 MoE backend is enabled, replacing the torch fallback that was the only working path. On 4x RTX PRO 6000, DeepSeek-V4-Flash decode TPOT drops from 36.1 to 10.5 ms at batch 1 and TTFT falls 20% from 8K to 128K input. Opt-in through the environment flags in the PR (#29927).
Dependencies and images. The CUDA 12 lane is retired; v0.5.19 was the last release with -cu12x wheels and images (#38404). sglang-kernel moves to 0.4.7 (#39346) and sgl-deep-gemm to 0.2.0 (#39371). New images: ROCm 10 for MI30x and MI35x with a matching kernel wheel (#38763), gfx1151 for Strix Halo / Ryzen AI MAX+ (#33939), and Moore Threads MUSA (#36709). ROCm 7.0 CI, images, and kernel wheel are retired (#38632, #38767).
Full release notes by category below; breaking changes are at the end.
SGLANG_DISAGGREGATION_ENGINE_INIT_TIMEOUT: #37874DynamicChunkSizer scheduler component: #37674--schedule-policy hrrn; mean TTFT -69% and p99 -8% vs FCFS on a GLM-5.2 production trace): #32911all_reduce in check_hicache_events for PP (TTFT -7% on DeepSeek-V4-Flash with HiCache L3, PP4 TP2): #37562free_swa sync-free on page_size == 1: #36723torch.unique sync from the SWA page expansion: #37463free_segment and drop the boundary trim: #37729free_segment: #37876allocator/ and split the composites out: #38072page_size > 1: #38159<image> sentinel on artifact fast path: #39278A CPU-only inference simulator under tools/sglang-simulator that reuses the real scheduler and cache stack with a latency predictor in place of the model forward.
First installation path for Moore Threads GPUs: a MUSA Dockerfile, install guide, and runtime extras.
-cu129 / -cu12 wheels or images are published from this release on; already-published tags are unaffected, and v0.5.19 is the last release with a CUDA 12 lane: #38404SGLANG_ENABLE_CP_V2 gate, the v1 runtime, and the generic v1 CLI options are gone, and CP APIs are renamed to versionless names: #36223, #36228, #36229. --enable-prefill-cp is rejected on HIP, NPU, and MUSA until their CP refactor lands; decode CP and non-CP inference are unaffected: #38293/v1/responses storage is off by default. Start with --enable-response-store to keep retrieval, previous_response_id chaining, and background requests; PD deployments cannot enable it and reject those features at admission. background=true with store=false is rejected: #39122--sampling-mask-max-tokens (default 4096). SGLANG_DISAGGREGATION_SAMPLING_MASK_MAX_TOKENS is rejected at startup, and masks under PD require SGLANG_ENABLE_DISAGG_SAMPLING_MASK=1 on both prefill and decode servers: #36631cutlass_mla attention backend, the non-Marlin gptq quantization choice, the AWQ AOT dequantize kernel, and Dual Chunk Flash Attention are deleted. GPTQ checkpoints continue through gptq_marlin; auto-round and moe_wna16 lose their non-Marlin GPU fallback: #32114get_global_server_args now raises; read resolved values from the namespace bags or the raw input from get_server_args(): #38375. The config tier, including ServerArgs, is now msgspec.Struct, so dataclasses.asdict and dataclass introspection on it no longer work: #38753, #38958--dcp-comm-backend now resolves per platform for every model: fi_a2a on Blackwell when the DCP group sits inside one node or MNNVL domain, otherwise a2a on CUDA and ROCm; pass --dcp-comm-backend ag_rs to restore the old default: #37767, #39165ValueError instead of hanging at warmup: #37610GetIsReady is replaced by the WatchEngineState stream, which publishes health, pause state, instance ID, and discovery snapshots: #39915--dit-cpu-offload is passed explicitly (#38001); the SDPA attention backend is chosen by measurement, retiring the no-op allow_cudnn_sdp (#38689); component loader capability switches are replaced by resolver contracts that fail closed (#36824)rocm700 kernel wheel are retired; ROCm 10 release images ship for MI30x and MI35x with a +rocm1000 kernel wheel, and ROCm 7.2.x images remain: #38659, #38632, #38767, #38763.git-blame-ignore-revs entry for the reformat commit: #37210, #37695Note truncated.
Unified radix tree by default. The unified tree is now the cache for every model, not just hybrid ones (#35081, see Breaking Changes). It also picked…
786 PRs from 214 contributors.
New models in this release (see the cookbook for all supported models):
| Model | Type | PRs | Cookbook |
|---|---|---|---|
| Qwen3.8 (2.4T-A95B) | Autoregressive | #35758 ⭐ | link |
| Qwen3.8-27B | Autoregressive | #34859 | link |
| dots3.note | Autoregressive | #33829 ⭐ | link |
| Ling-3.0-flash | Autoregressive | #33561 | link |
| Ling-3.0-tiny | Autoregressive | #33561 | link |
| Spark2.5 | Autoregressive | #35963 ⭐ | |
| MiniCPM-SALA | Autoregressive | #30360 | |
| Granite 4.2 | Autoregressive | #36286 | link |
| LongCat-Image-Edit & Edit-Turbo | Diffusion | #35829 |
Cookbook updates:
Beam search. SGLang can now do beam search. Pass beam_width in your request and you get back the n best sequences instead of a single sample. It works out of the box next to regular requests, though it does not yet mix with speculative decoding, disaggregation, DP attention, or HiCache (#31626).
DeepEP v2. DeepEP's new ElasticBuffer engine is available as --moe-a2a-backend deepep_v2 for DeepSeek-V3/V4 and Qwen3-MoE in FP8. Its buffers have a fixed size, so decode can run under CUDA graphs even across nodes. Performance is on par with the classic backend (#35634, #34923).
LayerNorm sequence parallelism. With --enable-layernorm-sp, each tensor-parallel rank normalizes only its own share of the prefill tokens instead of all of them. That takes 3.5% off Qwen3-8B prefill on H100 and 5.6% on B200, and the saving grows with the TP degree. Dense Qwen3 models only for now (#30915).
W4A8 MoE on Hopper. If you serve MXFP4 experts on Hopper, you can now quantize the activations to FP8 as well with --flashinfer-mxfp4-moe-precision fp8. DeepSeek-V4-Flash gains about 12% output throughput with no change in GSM8K accuracy. Needs FlashInfer 0.6.18 (#34967).
DCP on the default Blackwell MLA backend. Decode context parallelism now runs on trtllm_mla, not just CuTe DSL and Tokenspeed. It pays off at long context: at 128K input, plain TP stops scaling around 680 tokens per second on eight B200s, while DCP keeps going as concurrency grows (#33926).
Faster speculative kernels. DSA prefill top-k moves to the v2 kernel, 1.3 to 1.8 times faster on B200 (#35175). KDA models get an opt-in fused accept path, SGLANG_OPT_KDA_FUSED_ACCEPT_STATE=1, that cuts MTP verify-and-commit time by 45% to 63% on Kimi-Linear shapes with bit-identical output (#33722).
Unified radix tree by default. The unified tree is now the cache for every model, not just hybrid ones (#35081, see Breaking Changes). It also picked up three things this cycle: PD decode workers can reuse cached prefixes for SWA hybrid models like gpt-oss (#27770), you can attach or detach L3 storage on a running server (#35269), and pipeline parallelism with HiCache L3 stays consistent across ranks (#27010).
Lean attention on AMD. Long or uneven decode batches used to leave many compute units idle on MI300X and MI355X. The new persistent Lean kernel spreads the work across all of them, for up to 1.52x more throughput and up to 3.62x lower inter-token latency on MI355X. It turns on by itself where it helps, and SGLANG_DISABLE_LEAN_ATTENTION=1 turns it off (#33576).
DSA models on ROCm. Disaggregated GLM-5.2 serving now uses the fused top-k seed remap, which brings decode TPOT from 23 ms down to 8 ms on eight MI355Xs (#36714). DeepSeek-V4 gets the v2 top-k kernel, up to three times faster (#36684), and a shared-experts gate fix gives GLM-5.2 up to 16% better TPOT (#36124).
Dependencies. FlashInfer moves to 0.6.18 (#36954), sgl-deep-ep to 0.1.2 (#35450), sgl-deep-gemm to 0.1.7 (#37279), and mooncake to 0.3.13 (#36493). There is a new CUDA 13.4 preview image for Rubin (#36233) and new ROCm 10 images for gfx942, gfx950, and gfx1250 (#36434, #36871).
Full release notes by category below; breaking changes and known issues are at the end.
KimiK3ForConditionalGeneration: #36211<optional> include in AITER topk kernel: #36216/rerun-test on commenter trust and remove /rerun-stage: #35750SGLANG_ENABLE_UNIFIED_RADIX_TREE is deprecated and can be unset: #35081Spark2_5Config, model_type spark2_5), model classes, and the function-call parser; the tool-call parser name changes from spark to spark25: #36416--moe-runner-backend is left at auto; an explicit backend is preserved: #35919--enable-w4a4-megamoe; SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_FP4_ACTS and SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_MXF4_KIND now only warn: #35918--dsa-topk-backend flashinfer and the CuTe DSL NVFP4 W4A16 mode have no fallback for earlier FlashInfer APIs: #36954, #33237, #35120ServerArgs no longer resolves it. Programs that build their own record must call resolve_once() before reading resolved values; the launcher does this for the engine: #35907--enable-symm-mem under CUDA graphs again, since torch 2.13.0 fixes the allocator bug that forced it off: #34842--enable-lean-attention=False or SGLANG_DISABLE_LEAN_ATTENTION=1: #33576SGLANG_USE_SGL_XPU flag is removed; XPU now uses the sgl-kernel MoE kernels by default, and --moe-runner-backend triton restores the Triton path: #34492SGLANG_CACHE_DIT_* env vars remain as server defaults and are slated for deprecation: #35339/rerun-stage is removed and /rerun-test is gated on commenter trust (zero cooldown or repo write): #35750kill_process_tree waits for the reap by default in CLI, benchmark, and test teardowns: #36589The first launch after upgrading recompiles once; see Breaking Changes (#32434).
710 PRs from 212 contributors.
New models in this release (see the cookbook for all supported models):
| Model | Type | PRs | Cookbook |
|---|---|---|---|
| Muse Glimmer | Autoregressive (Multimodal) | #34262 | link |
| Intern-S2-Mobius | Autoregressive | #33691 | link |
| SANA-Video | Diffusion | #32921 | link |
| LingBot-Video-MoE | Diffusion | #32341 | link |
| LTX-2.5 | Diffusion | #34471 | link |
| Cosmos3 Edge & Distilled | Diffusion | #31590 | link |
| LongCat-Image | Diffusion | #23274 |
Plus cookbook recipes for the Qwen3.8 family, Ling-3.0, Nemotron 3.5 Lightning, Dots3-Note, and DeepSeek-V4-Pro-0813 (#34809).
Overlapped checkpoint staging at startup: Checkpoint pages now stage from storage while CUDA graphs capture. Qwen3-32B on H100 starts 8.6-11.7% faster than serial with prefetch, and 2.38x faster (35.6s vs 84.8s) than the plain default. Opt in with --startup-weight-load-mode overlap (#32017).
TP LMHead with All-to-All: The TP LMHead's allgather + scatter becomes a single all-to-all for pure-DP dp-attention. On DeepSeek-V4-Pro B200 decode, LMHead time drops 320us to 169us and TPOT improves 36.97ms to 35.67ms (#32313).
FlashInfer MNNVL for pure allreduce: Non-fused allreduce sites now reuse the FlashInfer MNNVL workspace instead of falling back to NCCL. DeepSeek-V4-Flash TP4 decode on Blackwell gains up to +6.9% at small batches. Auto-enabled for DeepSeek-V3/V3.2/V4; elsewhere --enable-flashinfer-pure-allreduce (#30700).
NVFP4 checkpoints run on AMD: --quantization quark_mxfp4 dequantizes ModelOpt and Quark NVFP4 weights and requantizes to MXFP4 at load, never holding a full-precision copy. 97.5-100.2% GSM8K recovery vs the NVFP4 reference across MiniMax-M2.7, GLM-5.1, Kimi-K2.6, Qwen3.5-397B, and DeepSeek-R1 (#29328).
Kimi K3 tuned for MI355X: a grouped-head MLA verify kernel replaces the MHA-shaped split-KV path that re-read the shared latent once per head, for 1.37-1.77x throughput and 1.45-2.42x ITL at concurrency 2-32; the AITER MLA prefill kernel now accepts K3's 12-head shape (TTFT up to -14.9%); a gfx950-tuned decode stage-1 geometry adds 46-73% ITL at 68k input, opt-in via SGLANG_MLA_DECODE_TUNE=1. GSM8K holds at 0.951-0.957 (#33981, #34261, #34837, #34580).
One compiled-kernel cache directory: Triton, FlashInfer, Inductor, DeepGEMM, and CUDA driver caches all move under SGLANG_CACHE_DIR. The first launch after upgrading recompiles once; see Breaking Changes (#32434).
Dependencies: torch 2.13.0 with triton 3.7.1 (#28836), flashinfer 0.6.17 (#33997), CuTeDSL 4.6.2, fixing an FA4 startup regression on Blackwell (#34372), DeepEP now installed from released sgl-deep-ep wheels (#33932), and sgl-kernel 0.4.6.post1 (#33842).
Full release notes by category below; breaking changes and known issues are at the end.
req.prefix_indices when the prefix cache is disabled: #34644is_image_understandable_model: #34217cuda-tile to 1.6.0rc5 to unblock Python 3.10 x86_64 installs: #34321SGLANG_CACHE_DIR, so the first launch after upgrading recompiles once. Copy or symlink the old directories if you pre-warm or volume-mount them (deep_gemm and flashinfer are the expensive ones): ~/.triton to {SGLANG_CACHE_DIR}/triton, ~/.cache/flashinfer to {SGLANG_CACHE_DIR}/.cache/flashinfer, ~/.cache/deep_gemm to {SGLANG_CACHE_DIR}/deep_gemm, /tmp/torchinductor_$USER to {SGLANG_CACHE_DIR}/inductor, ~/.nv/ComputeCache to {SGLANG_CACHE_DIR}/nv. Setting any of TRITON_CACHE_DIR, FLASHINFER_WORKSPACE_BASE, SGLANG_DG_CACHE_DIR, TORCHINDUCTOR_CACHE_DIR, or CUDA_CACHE_PATH keeps that one where it is: #32434--torchao-config had raised ImportError for every accepted value since the torchao pin moved to 0.17.0): #34304sgl-deep-ep wheels instead of building from source: #33932flashinfer_trtllm path with an unfused, non-TP1 shared expert (the DeepSeek-V3 family); set SGLANG_ENABLE_MOE_DEFERRED_FINALIZE=False to fall back: #33618SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS=False to restore the old behavior: #34653Full Changelog: v0.5.17...v0.5.18
*Full release notes by category below; breaking changes and known issues are at the end.*
582 PRs from 194 contributors.
Kimi K3 day-0 support: A 2.8T-parameter multimodal LatentMoE (896 experts, top-16, routed in a 3584-dim latent space) with a 1M-token context, 69 KDA linear-attention layers interleaved with 24 MLA layers, and a MoonViT3d vision tower, shipping as a native MXFP4 checkpoint. SGLang serves it from day 0 with DCP, DSpark speculative decoding, chunked-prefill PP with TP decode, KDA-aware prefix caching, HiCache L2 over DCP, LoRA on the quantized weights, and reasoning, tool-call and OpenAI-compatible serving, verified on NVIDIA GB300 and AMD MI35x (#32541, #32828, #32890, #33025, #33112, blog, cookbook, roadmap).
MiniMax-H3 day-0 support: MiniMax's video generation model that produces a video and a synchronized stereo audio track in one request, served natively on SGLang-Diffusion across all three public task profiles: text-to-video-and-audio (t2va), first/last-frame conditioning (fl2va), and image/video/audio reference conditioning (ref2va, which also covers video-to-video). Verified on B200 (TP2 + Ulysses4), H100 (TP2 + Ulysses2), AMD MI300X and MI355X (Ulysses1/2/4/8), and 2x RTX 5090 with layerwise offload (#33275, cookbook).
Other new models added: EmbeddingGemma and LFM2.5 embedding models, nvidia/MiniMax-M3-NVFP4, plus cookbook recipes for Poolside's Laguna-S-2.1 family and Inkling-Small.
Initial support for the Rust frontend: Migrates the front half of the server, everything from network ingress up to the point a tokenized request is handed to the GPU scheduler, from Python to a multi-threaded Rust implementation (#29799).
DCP communication backends and q-replicate (Helix): The DeepSeek-MLA decode context-parallel path gains pluggable comm backends. a2a exchanges packed attention output plus fp32 LSE in a single NCCL collective per layer, with fp8 KV carried as uint8 byte transport; fi_a2a delegates the cross-rank exchange to the FlashInfer MNNVL kernel on GB200. --dcp-replicate-q-proj projects full-head Q locally and skips the per-layer Q head-dim all-gather. Select with --dcp-comm-backend {ag_rs, a2a, fi_a2a} (#21637).
DWDP for MoE prefill: A new prefill parallelism strategy that prefetches peer expert weights over NVLink P2P and computes all experts locally, removing EP all-to-all token dispatch. On 4x B200 with gpt-oss-120b, prefill-only, DWDP4 reaches 1.92x over DEP4 at MNT 32K / ISL 32K, and 506K vs 329K tok/s (1.54x) at saturation (CONC=128, ISL=8K). Enable with --dwdp-size; the authors mark it early-development (#29778).
Session-reference-aware Unified Radix Cache: For agentic and RL-rollout workloads, requests can carry a stable session_id so eviction knows which prefixes an active session still references, instead of evicting purely by cache policy. Release the references with /close_session. Opt in with --enable-session-radix-cache (#29173).
SM90 FP8 MegaMoE for DeepSeek-V4: Adds the DeepGEMM MegaMoE A2A path on SM90 for DeepSeek-V4-Flash/Pro FP8, including the pre-dispatch JIT kernel and FP8 expert weight preparation. Guarded behind SGLANG_OPT_USE_DEEPGEMM_MEGA_MOE=1 (#29016).
Faster large-MoE model loading: Oversized or non-contiguous CPU weight views were driving pathological H2D transfers, with DeepSeek-V4-Pro TP8 spending 27 to 32 minutes in H2D on some ranks. Copying those views into contiguous storage before H2D cuts full model loading from about 35 minutes to 6m20s (5.6x), and GPT-OSS-20B BF16 from 545s to 70s (7.8x), with Qwen3.5-397B measured at 1.93x to 2.3x. Opt in with SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D, off by default (#32315).
Lower DeepSeek-V4 memory on AMD: Removing unnecessary expert padding drops MI355X FP4 MoE model weights from 159.07 GB to 112.36 GB, and bringing the HIP compress-state pool into the memory_saver KV_CACHE region lets colocated RL reclaim it, cutting the measured training-phase footprint from about 143 GiB to 87 GiB per GPU (#31450, #31747).
Faster engine recovery: Large-model restarts cost 3 to 6+ minutes today, about 6.5 minutes for Qwen3-235B FP8 on 4 GPUs, because weights reload from storage and CUDA graphs recapture. A weight-cache daemon holds weights per GPU so a restarting engine can recover from cache instead (#27139).
Lower host overhead in hybrid-linear MTP decode: Under spec-v2 overlap scheduling each decode step runs draft, verify and extend CUDA graphs, and the eager seams between them become GPU idle time at low concurrency. This trims that host work so the host stays off the critical path (#32219).
Dependencies: flashinfer 0.6.15.post1 (#31927), sgl-deep-gemm 0.1.5.post1 (#32345, #33143), helion 1.4 (#32562), mooncake 0.3.12.post1 (#32302), dynamo-tokenizers 1.7.0 (#32981). PyTorch stays at 2.11.0 and the CUDA base image at 13.0.1.
Full release notes by category below; breaking changes and known issues are at the end.
| Model | Type | PRs | Cookbook |
|---|---|---|---|
| Kimi K3 | Autoregressive (Multimodal) | #32541, #32828, #32890, #33025 | link |
| MiniMax-H3 | Diffusion | #33275 | link |
| MiniMax-M3-NVFP4 | Autoregressive | #31989 | |
| EmbeddingGemma | Autoregressive (Embedding) | #32375, #32383 | link |
| LFM2.5 | Autoregressive (Embedding) | #28691 | link |
A native Rust serving layer: tokenizer manager, ingress validation and egress, an OpenAI-compatible API server, and PD disaggregation support, shipped as prebuilt release artifacts.
trtllm_mha support for Gemma 4 MTP draft attention backend: #25545_online_combine_kernel: #32277load_weights v2 dispatch (#28671): #31988GrammarMask type across all decode paths: #32409retrive_next_token: #32380sglang.kernels, RFC #29630)bmm_fp8, use flashinfer.bmm_fp8: #31202New recipes: RTX 5090 DeepSeek-V4, DGX Spark for Inkling-Small, MiniMax-H3 on 2x RTX 5090 (TP2 + layerwise offload).
helion jumps 0.2.6 to 1.4, a major-version move for anyone depending on helion-backed kernels: #32562sglang.jit_kernel is retired into sglang.kernels, completing RFC #29630. Imports from the old module path must move: #32072, #31666, #32015, #32045bmm_fp8 is deleted in favor of flashinfer.bmm_fp8; the AOT router GEMM and fused A GEMM are also removed: #31202, #30280ServerArgs is being made read-only behind namespace config bags. Code that mutated ServerArgs at runtime must route through the new accessors: #31809, #31811, #33334, #33338test_update_weights_from_disk is skipped on ROCm pending a reload fix (#31924): #31925Full Changelog: v0.5.16...v0.5.17
Full Changelog: https://github.com/sgl-project/sglang/compare/v0.5.16...v0.5.17
`--enable-deepep-waterfill` is renamed to `--enable-waterfill` with no deprecated alias, so existing launch commands fail with unrecognized arguments:…
574 PRs from 169 contributors.
DSpark: confidence-driven speculative decoding: A new speculative algorithm. It drafts semi-autoregressively in blocks, then sizes each verify window from the draft's own confidence instead of a fixed draft length. Reaches 383.7 tok/s at accept length ~5 on DeepSeek-V4-Pro, TP8 on B300 (bs=1). Enable with --speculative-algorithm DSPARK and SGLANG_RAGGED_VERIFY_MODE=compact; tune the block with --speculative-dspark-block-size (#30261, #31434, blog).
Inkling support: A 975B-parameter multimodal MoE with a 1M-token context. It mixes sliding-window, full and Mamba2 linear attention, and adds an NVFP4 MoE, optional vision/audio towers and native MTP. On Blackwell it reaches up to 71.7k tok/s input and 171.0 tok/s per-user decode. Verified on Blackwell TP4/TP8, H200 and AMD MI350X / MI355X (#31681, blog, cookbook).
Other new models added: LongCat 2.0 FP8, JetBrains Mellum v2, Pi0.5, plus diffusion support for LongLive 2.0.
UnifiedRadixTree is now the default for SWA, Mamba and DSA models. Replay SSM and Mamba int8 checkpoints are synced onto it, and a cache hit now resets only the state it used (#30468, #30636, #30626, #31643).
GLM-5.2 DSA cache layer split under prefill CP: KV and indexer cache layers are sharded across CP ranks. Each rank owns a disjoint layer range instead of all layers. That cuts per-rank KV memory by ~74% (0.77 to 0.20 GB/rank) at 8192 tokens on GLM-5.2-FP8, 78 layers, cp_size=4. Enable with --enable-dsa-cache-layer-split, which needs --enable-prefill-cp --cp-strategy interleave (#29421).
ReplaySSM Ring Spec-Verify (GDN): Drops the per-draft SSM snapshot. Speculative scratch goes from 11.5 GB to 1.8 GB per GPU (6.4x smaller) on Qwen3.5-35B-A3B at TP1, at accuracy and throughput parity. Opt in with --enable-gdn-replayssm-spec (default off; GDN with a linear draft chain only, --speculative-eagle-topk in {None, 1}), and tune the ring via --linear-replayssm-cache-len (#28695).
Linear attention on Blackwell (SM100): The first correct KDA MTP path. Its recurrent_kda decode kernel runs at 29.6 us vs 36.8 us for Triton (ncu, B=64). The full decode path reaches parity by B=128 and 1.35x at B=256, and is slower below that (#30113). Separately, GDN/KDA CuteDSL prefill fuses state I/O into the chunk-h kernel (#30169).
QServe and FBGEMM FP8 quantization are removed: the experimental QServe (QoQ) W4A8 and FBGEMM FP8 paths are gone. --fp4-gemm-backend cutlass goes too, along with the in-tree NVFP4 JIT kernels, so NVFP4 GEMM now requires FlashInfer (#31109, #30448).
Dependencies: flashinfer 0.6.14 (#29910), CuTe DSL 4.6.0 (#31714), sgl-kernel 0.4.5 (#31496), llguidance 1.7.6 (#31484).
--fp4-gemm-backend cutlass is removed along with the in-tree NVFP4 JIT kernels, so NVFP4 GEMM now requires FlashInfer. Use auto, which picks flashinfer_cutedsl on SM100 and flashinfer_cutlass on SM120: #30448UnifiedRadixTree is now the default for SWA, Mamba and DSA models. A behavior change on those architectures: #30468docs/ removed; the Mintlify cutover is complete: #28964sglang.kernels namespace: kernels are relocated verbatim and only import paths change; public wrappers keep defaulting to the AOT sgl_kernel backend, so code reaching past them to internal paths must update (RFC #29630): #30044, #31582num_tokens_per_bs renamed to num_tokens_per_req across spec-decoding runners: #30977--enable-deepep-waterfill is renamed to --enable-waterfill with no deprecated alias, so existing launch commands fail with unrecognized arguments: #27350--optimistic-prefill-retries is renamed to --optimistic-prefill-attempts with no deprecated alias: #30951application/msgpack instead of JSON, with tensors as raw msgpack bytes rather than base64 (tensor_to_base64 / base64_to_tensor become tensor_to_bytes / bytes_to_tensor), so RL rollout consumers must be upgraded in lockstep with the server: #31565Full release notes by category below.
| Model | Type | PRs | Cookbook |
|---|---|---|---|
| Inkling | autoregressive | #31681 | link |
| LongCat 2.0 | autoregressive | #30275, #30320 | link |
| JetBrains Mellum v2 | autoregressive | #27375 | wip |
| Pi0.5 | vla | #30633 | link |
| LongLive 2.0 | diffusion | #27639 | link |
Landed this cycle but not yet usable end-to-end: MiniMax-M3 completes its four-part landing (#28715, begun in v0.5.14) but its cookbook still points at a dev image (#31819).
SGLANG_MAX_NEW_TOKENS_LIMIT to cap per-request max_new_tokens: #22591run_batch launch: #31687LogprobResult and rename chunk env vars: #31733--enable-flexkv routes the KV cache through FlexKV's KVManager for host-tier offload, configured via --flexkv-config-file: #29701SGLANG_HICACHE_FILE_BACKEND_ENABLE_METADATA_CACHE, off by default): #29716sglang.kernels, RFC #29630)Full Changelog: v0.5.15...v0.5.16
v0.5.15.post1 includes a few patches, mostly for GLM 5.2
v0.5.15.post1 includes a few patches, mostly for GLM 5.2
One column per month.