NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
PyPI
PyPI, #2774 most downloaded on PyPI
A high-throughput and memory-efficient inference and serving engine for LLMs
Latest stable
0.33.0
Last release 20 days ago
02 Nov 2026
Ships on a steady schedule
a new release about every 2 weeks
Nearly every release is documented
notes for 60 of the last 60 stable releases
1 version withdrawn
withdrawn after publishing
3 years old
101 releases · first in 2023
Drag across the ruler to compare, or tap a release to open it.
Breaking changes : Medusa and the MLP speculator removed ( #61701 ); the legacy Ray compiled-graph executor removed ( #61712 , #62228 ); --mamba-cache…
This release features 927 commits from 389 contributors (122 new)!
-cu129 / -cu129-ubuntu2404 images are no longer published (#62575); use the default CUDA 13.0 artifacts instead.vllm/vllm-openai:v0.33.0-efa image (x86_64 and aarch64) bundles the AWS EFA libraries and the EFA build of the Mooncake transfer engine, for KV transfer and multi-node serving on EFA-enabled EC2 instances (#58768).vllm/vllm image alias: every release image is now also published under the shorter vllm/vllm name, e.g. vllm/vllm:v0.33.0, vllm/vllm:v0.33.0-efa and vllm/vllm:v0.33.0-rocm.lm_head (draft_lm_head_quantization, +18% decode at concurrency 1) (#59973); lookahead block hashes so EAGLE/MTP prefix-cache hits keep their last block, on by default (#61589); and MTP drafts in the Transformers modeling backend (#61270).-rocm72 (#58761); MonoKernel decode via --monokernel for DeepSeek-V4.1-Flash on MI300X/MI325X (2.32x P50 interactivity at concurrency 1, #60731) and on every MI355X layer (#61082), and for MiniMax-M3-MXFP4 on MI355X (16.6% lower TPOT, #62712, #62994); the ROCM_OCTAVE 3-4 bit KV cache (4.6x the KV tokens of fp16 on RX 7900 XTX, #59774); and gfx1250 enablement for gpt-oss-120b, DeepSeek-R1, DeepSeek-V4-Flash and MiniMax-M3 (#58800)./v1/decisions endpoint, on by default for supported generation models (#60465, #61550); token embeddings with optional late chunking on /v1/embeddings (#61772, #63051, #60164); output_mode="derender" on /inference/v1/generate (#60082); a large Rust frontend parity push, including no more HTTP 500 on long conversations (#62374) and unsupported flags failing at startup (#62431); concurrent worker startup with TP > 1 (#60028); and a watchdog that dumps stack traces from stuck engine and worker processes (#55700).--mamba-cache-mode deprecated (#60043); requests sent while generation is paused now get HTTP 503 (#51488); the NIXL connector needs NIXL 1.5 or newer (#61032); LM Format Enforcer and --ir-op-priority deprecated (#63423, #63371).| Platform | Install |
|---|---|
| PyPI (CUDA 13.0) | pip install vllm |
| PyPI (CUDA 13.0, uv) | uv pip install vllm --torch-backend=auto |
| ROCm 10.0 | pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.33.0/rocm100 |
| ROCm 7.2 | pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.33.0/rocm72 |
| XPU | uv pip install vllm --extra-index-url https://wheels.vllm.ai/0.33.0/xpu --extra-index-url https://download.pytorch.org/whl/xpu --index-strategy unsafe-best-match |
| Platform | Docker Image |
|---|---|
| CUDA 13.0 (Default) | docker pull vllm/vllm-openai:v0.33.0 |
| CUDA 13.0 + AWS EFA | docker pull vllm/vllm-openai:v0.33.0-efa |
| ROCm 10.0 | docker pull vllm/vllm-openai-rocm:v0.33.0 |
| ROCm 7.2 | docker pull vllm/vllm-openai-rocm:v0.33.0-rocm72 |
| CPU | docker pull vllm/vllm-openai-cpu:v0.33.0 |
| XPU | docker pull vllm/vllm-openai-xpu:v0.33.0 |
Every image is also available under the shorter vllm/vllm name: vllm/vllm:v0.33.0, vllm/vllm:v0.33.0-efa, vllm/vllm:v0.33.0-rocm (-rocm100, -rocm72), vllm/vllm:v0.33.0-cpu and vllm/vllm:v0.33.0-xpu. latest works the same way, e.g. vllm/vllm:latest-rocm.
Pre-built release artifacts are available in the Assets section at the bottom of this page, including:
BeamForCausalLM) day-0 support with its DFlash2 drafter and fused Blackwell kernels (#62074, #62042), plus beam reasoning and tool parsers (#62052); Yandex AliceAI-Foundation-80B-A3B (Kimi Delta Attention hybrid, 512-expert MoE, 262K context) (#57963); Recursive Flint BF16 and NVFP4 GGUF checkpoints with small-batch Blackwell decode kernels (#62938); Muse Spark Open (#61879) with reasoning and tool parsers (#62207); Step-5 Preview (#61727, #62126, #62132), running on SM100 (#63147) with LoRA (#62498); NVIDIA Alpamayo 2 Super (#62957); Nemotron-Labs Dory1.0-4B (#60149); Nemotron Parse 3.5 with an MTP proposer on MRV2 (#61651); MiniCPM-OCR 1.0 (#63095); Domyn Edge (#62544); ColUltraVBERT (#62237); LFM2.5-Embedding-350M (Lfm2BidirectionalModel, #62201); Kolibri 1 through the Transformers backend (#60827); and Decision 3.0 models (#61565, #61778) and Cloudflare Clef-Flash (#61308) on /v1/systemone.reload_weights works (#60096); fused_wqa_wkv split by token across TP on SM90 (#58872); MegaAttn wo_a writes MXFP8 input for wo_b (#60118); native small-head FlashMLA sparse prefill on SM90 (#61615); FlashMLA interleaved KV layout (#61371); redundant EPLB expert slots loaded (#59611); GPU Engram tables read from peer HBM, fixing DEP decode hangs on GB200 (#62091); and fixes for Engram under PP (#62208), Engram all-to-all hangs (#62269), prefill index-workspace NaNs (#60613), C128 compressed-KV writes under PIECEWISE graphs (#59709), routed-expert rows under replay (#61643, #62652) and the updated 1017 reference encoding (#62199, #62200).concat_and_cache_mla scales (1.2-3.0% TPOT on B300, #62102); FP8 indexer WK pairs kept across load_weights calls, fixing thinking loops with runai_streamer (#59223).compile_mm_encoder (#61838); DeepSeek-VL2 ViT full CUDA graphs (#46005); Gemma4Unified batched embedding (#60643); Pixtral real-valued RoPE under compile (#63145); and fixes for EXAONE-4.5 images (broken since v0.29.0, #61104), Gemma 4 encoder graphs (#61365, #60687, #63538), DeepSeek-VL2 encoder cache and PP (#63120, #63192) and Gemma 4 placeholders (#61196).lm_head (#59973), lookahead block hashes (#61589), MRV2 skipping drafts no request can use (up to 7.3% prefill throughput with DFlash, #62784), Triton split-K verify (#56148), varlen Mamba2 decode for adaptive verification (#53036), and fixes for stop-trimmed token counts (#58668), block verification randomness (#61161), MTP block tables (#61174), grammar masks (#52477), EAGLE SWA prefix hits (#62206), DFlash/DSpark Mamba checkpoint reuse (#54163) and PP warmup (#62129).input_ids on later PP stages (#61016), and memory-profiling fixes (#57926, #58243, #61509, #63431).flashinfer_trtllm (#61979), text-only LoRAs on multimodal wrappers (#62224), unsupported models rejected at validation (#61884), and fixes for load_inplace reloads (#41482), max_loras accounting (#60306), SM90 lora_expand NaNs (#61906) and wake-up from level-2 sleep (#62358).return_weight_versions (#61347), NCCL M2N transfer with per-parameter meshes (#60934), parameter metadata sent once (#35911), the ParamMeta / iter_full() / iter_local() weight-transfer API (#63256), and a silent MoE weight corruption fix with packed NCCL transfer plus layerwise reload (#62555).VLLM_BATCH_INVARIANT (#61118).Note truncated.
Your coding agent can read these notes before it upgrades. Set up the MCP server →
Breaking changes: Python 3.10 dropped (#60402); default CORS origins restricted (#61009); transformers>=5.16.1,<5.20.0 and tokenizers>=0.23.2 (#57387,…
This release features 814 commits from 363 contributors (121 new)!
POST /v1/systemone endpoint answers choice, noul and score questions over a structured state (#59299), is served by default (#60677, #60696), and also runs on Cloudflare/clef-omni (#61064), LiquidAI/d1-3B (#60676) and DiffusionGemma (#60466). The new hf tool and reasoning parser (--tool-call-parser hf, --reasoning-parser hf) reads the checkpoint's response_template, so models that ship it need no model-specific parser (#58604, #58602), in both the Python and Rust frontends (#59005).fp8_ds_mla on SM90 gives GLM-5.3-Flash up to 31% more output throughput (#59246); ROCm gains the 4-bit ultraquant_4bit KV cache with about 2x the density of FP8 (#57057).vllm download-kernels installs FlashInfer's precompiled kernels (#58765); vllm preload --preload-autotune keeps the FlashInfer autotune table warm across restarts (#60085); a shared-memory tensor arena broadcasts large multimodal tensors to workers with one copy (#51207); tokenizers and processors load from the local snapshot (#60825); and with --api-server-count above 1 each connection goes to the least-loaded API worker, for 43-85% more req/s on short requests (Qwen3-0.6B, 32 input / 1 output tokens) in the PR's benchmark (#60723).max_token_id, #59491) and GHSA-frvg-qw88-hww9 (P2P ZMQ socket exhaustion, #51504); CORS no longer allows every origin by default (#61009); untrusted images are limited to raster formats (#60022); shared-memory multimodal cache handles are signed (#59357); and JSON schemas nested deeper than 64 levels are rejected (#60036).transformers>=5.16.1,<5.20.0 and tokenizers>=0.23.2 (#57387, #59796); torch.compile sequence parallelism and AsyncTP passes removed (#60517); cascade attention removed (#61126); mamba_cache_mode="all" removed (#58997); max_tokens no longer reserved from the context window (#57035); Responses API reasoning stream events renamed (#59652).| Platform | Install |
|---|---|
| PyPI (CUDA 13.0) | pip install vllm |
| PyPI (CUDA 13.0, uv) | uv pip install vllm --torch-backend=auto |
| ROCm | pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.32.0/rocm723 |
| XPU | uv pip install vllm --extra-index-url https://wheels.vllm.ai/0.32.0/xpu --extra-index-url https://download.pytorch.org/whl/xpu --index-strategy unsafe-best-match |
| Platform | Docker Image |
|---|---|
| CUDA 13.0 (Default) | docker pull vllm/vllm-openai:v0.32.0 |
| CUDA 12.9 | docker pull vllm/vllm-openai:v0.32.0-cu129 |
| CUDA 13.0 + Ubuntu 24.04 | docker pull vllm/vllm-openai:v0.32.0-ubuntu2404 |
| CUDA 12.9 + Ubuntu 24.04 | docker pull vllm/vllm-openai:v0.32.0-cu129-ubuntu2404 |
| ROCm | docker pull vllm/vllm-openai-rocm:v0.32.0 |
| CPU | docker pull vllm/vllm-openai-cpu:v0.32.0 |
| XPU | docker pull vllm/vllm-openai-xpu:v0.32.0 |
Pre-built release artifacts are available in the Assets section at the bottom of this page, including:
google/embeddinggemma-2), an omni-modal text/image/audio/video embedding model, with unused modality encoders skipped through --limit-mm-per-prompt (#60254, #60289); MiniCPM-V 4.7 (#58674); Nemotron 3.5 ASR (nvidia/nemotron-3.5-asr-streaming-0.6b) complete-audio transcription; native realtime streaming is not yet supported (#59827); Cloudflare/clef-omni (#61064) and LiquidAI/d1-3B (#60676) on /v1/systemone; PLaMo 3 reasoning and tool parsers (#56575); and the zerank-2 reranker score template (#60415).fp8_ds_mla on SM90 (#59246), qlnorm skipped for MHA (4.4-7.7% TTFT, #58845), long-context indexer prefill sharded across TP (up to 25% lower TTFT at 1M context, #54951), DCP for the kpool indexer (#59211, #60032), KDA prefill checkpoints with FlashKDA (#56960), FP8 KV on SM90 sparse MLA (#55222), --mamba-ssm-cache-dtype honored for KDA state (#59147), a generic packed KV layout (#57169), MTP index sharing restored (#61079), and fixes for vision-tower crashes (#59126), non-square image encoder budgets (#59565), DP startup (#56974), video timestamps (#55647), DFlash2/EAGLE3 aux hidden states (#60763) and null-block writes (#59528).compilation_config.decoder_replay_trim_threshold, #59532) plus last KV-source layer trimming (+3.6%, #59894), packed FP8 Engram exchange across DP (#59923), faster CPU-offloaded Engram lookups (#59327), fused DeepSeek V4 + MTP drafting (#58463), encoder CUDA graph packing for V4.1 and Kimi-K3 (11% lower mean TTFT, #58925), EPLB for DSpark drafts (#57549), and fixes for NVFP4 checkpoints with 0% draft acceptance (#59655), MegaMoE shared-expert finalize ordering (#59927), worst-case vision dummy inputs (#57898, #59271), compressor ring null-block writes (#58560), Triton 3.9 indexer quantization (#60716) and decoder-replay CUDA graphs with FlashInfer EP MoE backends (#61041).--enable-hc-sp (#56322), SM100 low-latency decode GEMM plans (up to 2.7x vs cuBLAS on small-M, #59214), --kv-cache-dtype-skip-layers honored (#60023), and PLE loading for INC (#59990), Quark (#59443) and pre-SM89 GPUs (#60142); Qwen3-VL/Qwen3.5/Qwen3.8 M-RoPE cache 4x smaller and Qwen3.5 YaRN matching Transformers (#59853); Qwen3-Omni M-RoPE fixes (#58890, #59842); Qwen3-ASR on the upstream processor (#57146); Qwen3-VL unknown video fps (#60501) and per-video options with the multimodal cache (#55203).compile_mm_encoder (#59195); encoder inputs sorted by modality (6.5-8.2% more req/s, #60196); Mistral 3 preprocessing 8% faster (#57531); Pixtral compile_mm_encoder (#57168); and fixes for Llama 4 / Mistral Large 3 EAGLE drafters (regression since v0.28.0, #57633), Phi-4 MM audio OOM (#60614), DeepSeek-OCR zero-sum pixels (#59417), MiMo-V2.5/V2.6 with fastsafetensors (#58142), MiMo-V2.6 FP8 KV cache (#58128), Voxtral realtime (#45022), Nemotron Super vision LayerNorm (#61401), K-EXAONE config and MTP RoPE (#60187), Nemotron Parse tied lm_head (#60272), LongCat-Flash MTP loading (#60080) and Ling-3.0-flash-VL (#59026).STOCK_TORCH_COMPILE mode (#59079), dynamic K in autoregressive speculators (#57053), MTP sampling-mask replay (#59359) and per-module LM heads (#58921), non-blocking H2D copies on WSL2 (#60215), and fixes for int64 index offsets at long context (#60736), warmup (#61013), GDN prefill checkpoints (#59536) and MRV2+PP+MTP align state (#60210).sleep_mode_offload_cudagraph (#59160, #59523); token-level weight-version spans on outputs (#59582, #61070); weight-transfer communicators released without stopping inference (#61060); --sleep-preserve-parameter-names 'draft.*' (#61046); modelexpress weight-transfer backend (#58399); POST /weight_checker (#51350); weight-transfer metrics (#55781); --enable-nccl-comm-suspend releasing the MNNVL workspace (#59360); and sleep(level=2) keeping FP8 KV scales (#57163).--enable-shm-tensor-arena, on by default, #51207), Fast Start with PP (#55477), daemon-held weights counted against gpu_memory_utilization (#57298), autotune preload (#60085), local tokenizer snapshots (#60825), sparse_mla_prefill_workspace_factor saving several GiB at 1M context (#61023), cached memory reclaimed during weight loading (#57516), fewer Triton sampler warmup compiles (#58605), sharded state loader with PP (#52110), and reload_weights with Run:ai streamer (#58124).tl.make_block_ptr (#60502), fusion passes restored under PyTorch 2.15 (#57299) and static FP8 quant fusion on recent torch (#61108).VLLM_BATCH_INVARIANT=1 (#58623), deterministic split-K LoRA shrink (#59377), every attention backend checked (#60271), SM120 TP configs (#58495), and ROCm support (#52231).vllm:prompt_tokens_cached_by_source_total (#56318), vllm:num_requests_kv_fetch_by_stage (#58874), vllm:admission_rejections_total (#55812), vllm:watermark_requests (#58715), --custom-histogram-buckets (#48867), a JSON log formatter via LoggingConfig.formatter (#58739), and per-session profiling controls (#57875).--moe-backend flashinfer_cutedsl (#54049), FP8 combine for the FlashInfer one-sided all2all (+4% at DEP8, #57995), NCCL symmetric-memory MoE reduce-scatter (9.6-16% lower latency, #49194), DeepEPv2 hybrid NVLink+RDMA mode on by default (#57991), Kimi-K3 shared experts with DeepEP v2 (#56875), and opt-in EPLB migration batching (#52641).ray[default] (#57648).--moe-backend aiter_mega_moe (#59685) with opt-in shared-expert fusion (3-8%, #61004), opt-in mono decode for DeepSeek-V4.1-Flash on MI355X (VLLM_ROCM_MONO_DECODE=1, #60397), opt-in Gluon sparse MLA (up to +16% on GLM-5.2, #53492), paged MXFP4 indexer (#58671), opt-in a4w4 MoE for DSv4 (#58819), ultraquant_4bit KV cache (#57057), aiter_flydsl GDN prefill (#57560), Kimi-K3 prefill checkpoints (about 20% better TTFT, #58344), fused MLA decode (#57640) and per-rank latent MoE shards (#59591), GLM-5.3-Flash split-K sparse MLA decode (5-15% lower ITL, #58584), native merge_attn_states (2.4x, #60159), ROCM_SEGMENTED_ATTN for RDNA (#59132), gfx11 W4A16 tuning (up to 23% TTFT, #56301), MRV2 default with model-specific MRV1 exceptions removed (#56360), and fixes for sleep-mode segfaults on ROCm 7.2+ (#60367) and MLA weight reloads (#60389).FLASH_ATTN_MLA (#58297), Arc Pro B70 FP8 GEMM (2.0-2.73x, #56063) and fused MoE configs (#53065), fused SYCL LayerNorm (#57172), and fixes for Gemma-2 softcap (#58936) and DP+EP MoE (#57787).dtype=auto choosing bfloat16 (#58528).--quantization-config.linear fp8_per_channel, #55684), per-token NVFP4 CuTe-DSL MoE (#50030) with ReLU2 (#56740), MXFP4 MoE oracle with CUTLASS W4A4 and BF16 fallback (#60079), compressed-tensors MXFP4 MoE with FP8 activations (#59815), ModelOpt IQ2_XS/IQ2_XXS/Q8_0 on SM120/SM121 (#59604), int4_per_token_head KV with non-power-of-two heads (#56198), and Humming preferred over Marlin on SM90 (#56997).moe_wna16 8-bit asymmetric zero points (#58950), and ModelOpt mixed-precision NVFP4 detection (#56050)./v1/systemone (#59299, #60677, #60696), hf parser (#58604), vllm download-kernels (#58765), output_mode: "text" and stop_reason in /inference/v1/generate (#58588, #60251), mm_payload for render requests (#61427, #60728), Responses min_p and stop_token_ids (#48084, #56592), LLM.chat(priority=...) (#57243), and /generative_scoring cache_salt (#60841).max_tokens is an upper bound instead of a reservation, so long prompts are no longer rejected (#57035); least-loaded API worker selection (#60723); tool parser compatibility checked at startup (#59749).parallel_tool_calls=false enforced (#50502, #59298), tool_choice="required" streaming (#60354) and native XML parsers (#47512), Step-3.5 and minimax_m3 on the parser engine (#59321, #59743), xgrammar propertyNames (#57594), and LoRA PEFT rank_pattern/alpha_pattern applied with unsupported features rejected (#59801, #60225).--grpc-port (#59659), --stream-interval (#60589), logprob_token_ids (#59549), Control.Shutdown with vllm-proto 0.5.0 (#59316), qwen3_json parser (#60136), and Kimi K3 grammar fixes (#59190, #59395).VLLM_CHAT_TEMPLATE_RENDER_TIMEOUT), user-supplied chat_template in Responses chat_template_kwargs is rejected, and the Rust frontend caps template execution (#50300).max_token_id off-by-one fixed (#59491).VLLM_P2P_MAX_PEERS (default 128), VLLM_P2P_HANDSHAKE_TIMEOUT_S (default 30) and VLLM_P2P_IDLE_TIMEOUT_S (default 300), so unique peer addresses can no longer exhaust ZMQ sockets (#51504).--allowed-origins to permit browser origins (#61009).ALLOWED_IMAGE_FORMATS), denying EPS/PostScript and WMF (#60022); shared-memory multimodal cache handles are HMAC-signed and forged handles rejected (#59357)./v1/embeddings requests keep cache_salt (#47696); Phi-4 MM audio no longer holds multi-GiB attention matrices (#60614); DeepSeek-OCR zero-sum pixel tensors no longer kill EngineCore (#59417).aiohttp>=3.14.3, pillow>=12.3.0, cryptography, urllib3, pyjwt and others (#59249, #59315, #59427).transformers>=5.16.1,<5.20.0 (minimum raised from 5.10.4, #57387, #59762) and tokenizers>=0.23.2 (#59796).humming-kernels==0.1.17 with initial Rubin support (#61556, #61858), InstantTensor 0.2.2 (#61167), and flashinfer-jit-cache excluded from wheel install_requires (#60052).VLLM_USE_PRECOMPILED finds a compatible published wheel within the last 20 commits (#58871); the vendored DeepGEMM uses TORCH_LIBRARY (abi3, #48962).* to none; set --allowed-origins for browser clients (#61009).transformers>=5.16.1,<5.20.0 and tokenizers>=0.23.2 are required (#57387, #59796).PassConfig.enable_sp, fuse_gemm_comms and sp_min_token_num; model-level sequence parallelism is unaffected (#60517).--disable-cascade-attn still parses but only warns and will be removed in v0.33.0 (#61126).mamba_cache_mode="all" was removed; --mamba-cache-mode all now fails validation (#58997).wait_for_inflight_requests (use mode="wait"), AsyncLLM.pause_generation(), IOProcessor.output_to_response, vllm.scripts.main() and vllm bench throughput --dataset (#60923). The sonnet benchmark dataset was removed as scheduled (#60129). Tensorizer support was removed from LoRA (#60024), and the PyO3 vllm._rust_tool_parser bridge was removed (#59744).max_tokens is treated as an upper bound on output rather than reserved from the context window (#57035); Responses API streaming uses response.content_part.added/done with part.type="reasoning_text" instead of response.reasoning_part.added/done (#59652); SamplingParams.prompt_logprob_token_ids is per row (#56984); /inference/v1/generate logprobs use integer token IDs (#58181); sleep API responses are standardized and invalid levels return 400 (#52864); return_mm_kwargs is deprecated in favor of mm_payload (#61427).false opts out; unsupported combinations warn and fall back (#56801).qwen3_xml now parses Qwen3-Coder XML (alias of qwen3_coder); use the new qwen3_json for the JSON format (#60136).--kv-cache-dtype auto (#60140, #60281); VLLM_DEEPEP_V2_ALLOW_HYBRID_MODE=1 (#57991); Humming preferred over Marlin on SM90 (#56997); sparse_mla_prefill_workspace_factor 2 above 256K max_model_len (#61023); MRV2 on ROCm with model-specific MRV1 exceptions removed (#56360); VLLM_ROCM_AITER_MLA_DCP_VERIFY=auto (#59965); dtype=auto picks bfloat16 on PowerPC (#58528); MXFP4 W4A4 checkpoints fall back to W4A16 with a warning when no W4A4 kernel exists (#60079).--prefix-match-unit (#58021), max_num_scheduled_tokens=0 (#60714), HiSparse with cudagraph_mode=FULL (#59688), and negative --device-ids (#49053) are rejected at startup.@BugenZhao, @aoshen02, @stefankoncarevic, @mgoin, @khluu, @hmellor, @yewentao256, @lesj0610, @MatthewBonanni, @jperezdealgaba, @ZJY0516, @AndreasKaratzas, @aarushjain29, @Sunt-ing, @NickLucche, @vadiklyutiy, @hickeyma, @divakar-amd, @yma11, @jzakrzew, @djramic, @simondanielsson, @microslaw, @Thangnguyenvn98, @mjkvaak-amd, @njhill, @chaojun-zhang, @sfeng33, @zhenwei-intel, @andyxning, @matteso1, @vMaroon, @alec-flowers, @simon-veitner-redhat, @TheEpicDolphin, @cagrikymk, @atalman, @shijie-lyu, @cheese-cakee, @Juntian777, @ivanium, @chaunceyjiang, @louie-tsai, @Dovis01, @apex-mochen, @wzhao18, @venkywonka, @pmanczak, @stecasta, @jinzhen-lin, @gty111, @noooop, @Rohan138, @rasmith, @laithsakka, @Fangzhou-Ai, @tlrmchlsmth, @Isotr0py, @cjackal, @vllmellm, @liusy58, @reidliu41, @peperunas, @shimib, @ojeda-e, @yonigozlan, @ShuoleiWang, @lucamotz, @ai-jz, @he-yufeng, @DimensionSTP, @taneem-ibrahim, @LucasWilkinson, @wangshangsam, @whx-sjtu, @micah-wil, @bzsuni, @robertgshaw2-redhat, @arpera, @sricursion, @mawong-amd, @mmastrac, @rmhaskarnvidia, @nvbfalk, @debanshd, @abmfy, @shaohuaxi, @jin-amd, @xijiaat, @ashraf-bhuiyan, @beenpow, @LioEinaudi, @Ronald1995, @R3hankhan123, @ZhengGong-amd, @Akashcodes732, @S1ro1, @yuvalluria, @mganczarenko, @DarkLight1337, @pisceskkk, @YidaWeng, @TQCB, @gau-nernst, @askliar, @benchislett, @jackLei0901, @vx120, @amd-mghanimi, @YueshenZ, @aniskumar-nv, @fululi12, @liuzijing2014, @zyongye, @Prudhvivuda, @kliuae, @wjabbour, @AbhayShuklaIIT, @WarrierRajeev, @Jeff-Tseng-dev, @yifanFengg, @elvircrn, @zupengwang, @tangzzycc, @jeejeelee, @roikoren755, @JohnQinAMD, @LeizhenZhang, @Chinmay-Kulkarni-AMD, @sungbin1015, @zhangchengzhucufe-dev, @fxmarty-amd, @lucianommartins, @nishantrevur, @hyukjlee, @andylolu2, @wangxiyuan, @stu-cao, @vineethsaivs, @zzaebok, @WoosukKwon, @franciscojavierarceo, @hpdkhoa, @gcanlin, @zigzagcai, @ultramancode, @warrendeng, @surajsharan, @subhashpolisetti, @ilmarkov, @awesome-pro, @faaany, @spandantiwari, @almogtavor, @ahmed-bsod, @danigarciaoca, @Alnusjaponica, @NNNtrance, @vaclavcadek, @YashasviAsthana, @QwertyJack, @Ubospica, @AlanFokCo, @jungjiyu, @lcskrishna, @YukioZzz, @hubunt, @sashko-zakharchuk, @staryxchen, @yzong-rh, @Ostring24, @taking-lying-flat, @drakosha, @CorgiBoyG, @QHarshil, @zhou9402, @shiyuan680, @fjosw, @a-sidorova, @GOavi101, @mrodden, @fuscof-ibm, @Taimys, @Rukhaiya2004, @KernelClint, @almayne, @JulienDarve, @sshlyapn, @eldarkurtic, @rbrugaro-amd, @GuyStone, @ZhenchengLin, @ArcheyChen, @djmmoss, @hjjq, @namgyu-youn, @jiangyunfan1, @ganeshr10, @markmc, @yantinglai, @Id545, @roipony, @JadenMathias, @hclsys, @Anurag-M1, @ukannika, @blueberry808, @nv-hwoo, @guanxingithub, @stmatengss, @mohit-sarvam, @Legend-2727, @xiaoyu-xyz, @hdq66666, @Lcy0801, @gaby, @erictang000, @jdebache, @Ayu190505, @DOCCA0, @C1ves, @maeehart, @wangyicong52, @AIwork4me, @Woolgathererer, @yongqiw-i, @divyvasal, @1fanwang, @TyroneNel, @ShengleiFu, @bluemelov1, @kamb-code, @LiRunGuo, @Thiago4532, @harshaljanjani, @ieaves, @NolenLiang, @sudhanshu112233shukla, @abhayjoshi201, @charlesli640, @Sohaib-Ahmed21, @akii96, @notimesea, @JRJRJPRO, @usberkeley, @Mazukiri, @brandonfzhang, @amd-xavierwang, @sawsa307, @czhu-cohere, @wtdcode, @tarukumar, @LostFox11, @big-yellow-duck, @SherifWaly, @roG0d, @esmeetu, @mgehre-amd, @samnordmann, @SunnyR, @ovidiusm, @cleonard530, @eky-amd, @yinli-systems, @MadhavJivrajani, @cquil11, @ywang96, @PatrykSaffer, @PatchouliTIS, @aditi-amd, @git-jxj, @vovani, @MohammadHijjawi97, @TommyTranX, @viktorkou, @xaguilar-amd, @MohitAMD, @danichan-mkm, @pondzikk, @ebfio, @mkhazraee, @ai-yang, @xuantengh, @dhairyashilRG, @www6v, @v-shobhit, @ejrtks1020, @Kylinny, @bnkaiteam, @huishi-hs, @jiahanc, @shen-shanshan, @zxd1997066, @Aravind-11, @usmanmateen, @devshah-cohere, @praveingk, @JartX, @TylerMa-debugg, @lobo235, @dougbtv, @Tialo, @kyleliang-nv, @YeonwooSung, @Kevinli24, @aijanai, @spa5k, @DGlan, @Btwestyo, @xianbaoqian, @rdlh, @hao-aaron, @errmakov, @amd-sriram, @ntheanh201, @LCAIZJ, @damienlaine, @shreyanshj10, @ischencheng, @0xViking, @yisustc, @bigPYJ1151, @Roy214, @Gregory-Pereira, @YuvrajSinghBhadoria2, @surawut-jirasaktavee, @Oxygen56, @andakai, @BolinSNLHM, @Dao007forever, @siliangchen-amd, @xiaofei-zheng, @gc-fu, @puneetsharma21, @linitra24, @devpramod, @yankay, @hlin99, @Godmook, @bvolpato, @UNIDY2002, @tc-mb, @JaredforReal, @hubert-marek, @Zhenzhong1, @AMD-yanfeiwang, @thuongvu, @JimmyWang0417, @MatthewCYM, @i-m-aditya, @lianakoleva, @peakcrosser7, @hoaivoke, @njriasan, @tjtanaa, @haotiaa, @superdiaodiao, @josh-schell-amd, @ztysth, @sungsooha, @Etelis, @natsala13, @ZhouXiaoya25
Breaking changes: per-request multimodal kwargs gated (#58830); tokenizer_mode="slow" removed (#58545); --enable-mamba-fine-grained-prefix-cache renam…
This release features 717 commits from 307 contributors (96 new)!
wo_b GEMM fused with the sequence-parallel reduce-scatter (#57428); Engram wkv sharded across TP ranks (#58678) and Engram host tables shared across co-located DP replicas by default (#57651); encoder CUDA graphs for the vision tower (#56625); and SWA bounded replay that keeps the sliding-window KV out of prefix caching (#56227).vllm preload CLI launches the weight-cache daemon that keeps post-quantized weights resident in GPU memory across engine restarts (#56680), now with data parallelism (#57386), MTP draft models (#57312), a /health endpoint (#58552) and a readiness wait (#58370). Experimental initialized-engine snapshots (vllm snapshot create/restore) use CRIU to restore a fully initialized TP1 engine (#51360).--all2all-backend moonep (#52101), prefill context parallelism with data parallelism (#57075), a low-SM multimem reduce-scatter for SM100/SM103 (#55072), DeepEPv2 with sequence parallelism (#57210) and EPLB with shared-expert overlap (#57236), a sharding-aware NCCL M2N weight-transfer backend for RL (#51520), and KV offloading back-pressure detection (#50045).--max-num-active-seqs caps RUNNING admission independently of max_num_seqs (#56758), --long-prefill-token-threshold now adapts to the number of waiting prefills instead of chunking a lone request (#57951, #58459), the waiting queue was reworked so requests already holding KV blocks are scheduled first (#58947), and the KV connector + MTP deadlock under KV pressure was fixed (#57104).mm_processor_kwargs and media_io_kwargs are rejected unless --trust-request-mm-kwargs is set (#58830); prefix-cache extra keys are tagged by source so a LoRA name and a cache_salt can no longer collide (#51899), and the LoRA path is part of the block hash (#59335); stale multimodal receiver-cache entries can no longer replace fresh payloads (#57833).tokenizer_mode="slow" removed (#58545); --enable-mamba-fine-grained-prefix-cache renamed to --enable-mamba-shared-prefix-checkpoint (#57382); online quantization through quantization="fp8" replaced by the fp8_per_tensor shorthand (#53585) and Quark silent online quantization removed (#51800); the AllSpark INT8 W8A16 backend removed (#58001); --enforce-eager now also disables JIT kernel warmup (#58197); XPU graphs enabled by default with VLLM_XPU_ENABLE_XPU_GRAPH removed (#51600).| Platform | Install |
|---|---|
| PyPI (CUDA 13.0) | pip install vllm |
| PyPI (CUDA 13.0, uv) | uv pip install vllm --torch-backend=auto |
| ROCm | pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.31.0/rocm723 |
| XPU | uv pip install vllm --extra-index-url https://wheels.vllm.ai/0.31.0/xpu --extra-index-url https://download.pytorch.org/whl/xpu --index-strategy unsafe-best-match |
| Platform | Docker Image |
|---|---|
| CUDA 13.0 (Default) | docker pull vllm/vllm-openai:v0.31.0 |
| CUDA 12.9 | docker pull vllm/vllm-openai:v0.31.0-cu129 |
| ROCm | docker pull vllm/vllm-openai-rocm:v0.31.0 |
| CPU | docker pull vllm/vllm-openai-cpu:v0.31.0 |
| XPU | docker pull vllm/vllm-openai-xpu:v0.31.0 |
Pre-built release artifacts are available in the Assets section at the bottom of this page, including:
granite_thinking_parser for Granite 4.2 (#55957).wo_b GEMM + reduce-scatter (#57428), overlapped mHC coefficients for small TP batches (#57603) limited to FULL CUDA graphs (#57874), Engram wkv sharded across TP (#58678), serialized Engram lookups with huge-page host tables (#56926), Engram tables shared across DP replicas by default (#57651, #57914, #59068), native shared-expert MegaMoE fusion without padding (#56568, #57204), faster MegaMoE staging and NVFP4 cache gathers (#57604), the fused query RMSNorm + MXFP8 path restored (#57679), KV-only DSpark context insertion (#56441), vision encoder CUDA graphs (#56625, #58499), SWA bounded replay (#56227), causal image SWA restored to match the reference (#57152), updated reasoning effort mappings (#58316), and startup fixes for NaN-scored candidate blocks (#57454), DeepSelect sentinels (#58215), DSpark non-causal attention on FlashInfer (#57432), runtime JIT of offset candidate buffers (#57667) and imports without Triton (#57654).suffix (#44229), missing string= in tool calls parsed (#56271), request tools attached to an existing system message (#51856), image block spacing preserved (#56882), and DSpark width separated from MTP stage validation (#54631).index_kpool mismatch (#58704), kpool tail strides (#57477), 500k-token prompts (#57317), dense MLP layers under sequence parallelism (#58061), indexer top-k backend selection (#58594) and varlen paged MQA launches (#55270).Conv3dLayer patch embedding (about 62x, #58512), triton_mrope in the vision tower (#58526), fused MiniMax2 routing with non-unit scaling (#58880), and processor fixes (#58460, #59613).diffusion_constrained reads over logprob_token_ids (about 25% faster, #58216), fewer logit rows for prefill-only batches (#57416), logprob_token_ids support (#57417), and fixes for concurrent logprobs (#57414), eager fallback dtype (#57462), multimodal inputs (#57589), quantized LM heads (#48521) and CPU execution (#58964).mm_input_norm kernel (#56798, #56711), raw pixels kept through the DP-sharded ViT path (#56872), Qwen2.5-VL video fps honored for temporal M-RoPE (#47736), Whisper and Qwen2-Audio clips longer than 30s (#57769, #56912), Mistral3 placeholder grid (#53758), Idefics3 unsplit patches (#48760), Ovis2.5 tokens (#52623), Aria expert weights (#57487), MiMo-V2.5 fused FP8 qkv_proj sharding (#57508), Gemma4 AutoWeightsLoader (#55911) and buffer scalars (#54213), Gemma4 FP8 KV with FA4 at head dim 512 (#53175), Laguna RoPE (#57189), Mistral-Large-3 accuracy regression (#57563), malformed EXIF (#56527, #57234), JinaVL labels (#57347), Sarvam MLA routing with FP32 router logits (#56034), fused CohereASR attention scores (#55190), and fused DFlash2 grouped convolution (#55960).modules_to_save sequence-classification heads (#53555) with per-adapter num_labels (#57766).max_model_len (#58149) and intermediate_tensors during capture (#57745).--max-num-active-seqs (#56758), adaptive --long-prefill-token-threshold (#57951, #58459), skipped_waiting replaced by a KV-holding waiting queue (#58947), atomic admission of n > 1 requests (#53936), KV connector + MTP deadlock (#57104), a throughput cliff when max_num_seqs is not a multiple of 8 (#57355), streaming continuations keeping logprobs and refreshing max tokens (#57447, #57676), a resumable request + async scheduling race (#58259), and a non-blocking structured-output grammar poll (#55931).--enable-mamba-shared-prefix-checkpoint (#57382), prompt-tail hits with MTP restored (#58368), prompt-end checkpoints kept under sparse retention (#59146), align-mode checkpoint reservation (#59175), stateless GDN first chunks (#51565), MTP draft KV cache groups annotated on the hybrid grouping path (#55390), a generalized prefill checkpoint builder (#57783), batched Mamba2 prefill state saves without GPU-CPU syncs (#49371), skip_reading_prefix_cache honored for connector hits (#57269), incremental multimodal block hashing (#51694), a KV block size every attention backend supports (#49845) with clearer errors (#58557), and CacheConfig.effective_attention_block_size for DCP (#56538).--enforce-eager (#58197) unless fault tolerance is on (#58593), DeepGEMM warmup reusing the MoE workspace (#57268), allocator fragmentation no longer shrinking the KV cache during profiling (#58430), sparse prefill buffers reserved before KV sizing (#57575), stale FlashMLA workspace views released (#56902), DeepGEMM FP8 workspace halved (#53914), shared Marlin/Humming workspaces (#57421), FlashInfer BF16 MoE weights converted in place (#54699), UniProc startup threads bounded by the CPU quota (#58946), and runtime threads set before profiling (#55891).GateLinear for all MoE models (#58234), non-local expert slots skipped in TritonExperts (#58051), deferred TRT-LLM-Gen top-k finalize on the modular path (#58635), SM120 batch-invariant matmul configs (#57456), FlashInfer prefill dequant scratch bounded (#57918), and fixes for Triton softcap NaNs (#56579), QuIP Hadamard transforms (#43462), CUTLASS FP8 linear on A100 (#55884), FlashInfer sampling on unsupported GPUs (#48956), SM100 FP8 blockwise scale padding (#57377), mixed FULL graph prefill capture (#58275) Inductor custom-op pattern matching (#58189), CuteDSL BF16 GDN prefill diverging from FLA on Qwen3.5 (#53864), SM100 fp8_ds_mla cache scales (#49435), ragged decode batches in the sparse indexer (#52500), stale allowed_token_ids masks after batch reordering (#48419, #43931), and prompt_embeds tensors held after requests finish (#57988).VLLM_BATCH_INVARIANT (#57586), sequence parallelism and async TP disabled (#56377), and NCCL 2.31 collectives kept enabled (#58179).release_kv_cache_memory() frees only KV cache memory (#44890), --sleep-preserve-parameter-names retains frozen weights across level-2 sleep (#57891), NCCL M2N weight transfer (#51520), dense DP weight updates by DP index (#56950), allocator config kept when toggling expandable segments (#57982), and cache reset failures propagated during sleep (#54581).vllm preload (#56680), DP (#57386), MTP drafts (#57312), readiness wait (#58370), /health (#58552), ModelOpt MXFP8 pre-processed weights (#57316), and initialized-engine snapshots (#51360).LoggingConfig via --logging-config and --log-level (#57205), JSON logging fixes (#57957, #58747), startup log suppression fixed (#51366), unified platform-aware torch profiling (#57460), no 0.0% prefix cache hit rate before any query (#54990), KV transfer metrics formatting (#57068), MFU activation sizing from the model dtype (#57070), platform-overridable env var checks (#48599), VLLM_TARGET_DEVICE=empty pip install vllm for out-of-tree backends (#41074), and the correct vLLM version reported when installing from source (#57295, #57744).CUSTOM_MEM_POOL (#49300), request-level load failures under HMA (#56855, #57174) and bootstrap retries (#58919); MoRIIO hybrid Mamba/KDA state in READ mode (#51052) and a multi-decode routing race (#51681); prefill cache hits in prompt_tokens_details (#54222); KV-event publishers bound at port 0 (#55844); KV cache metadata GET restored for external consumers (#56925); partial-block KV events keep every multimodal feature (#58288); saves finalized on steps without a forward (#57775); abort-safe ExampleHiddenStatesConnector (#56841); DecodeBench FP8 fills (#58472); a KvHints request envelope (#53423); and the AuxOutput connector for routed-expert outputs (#45635, #58150, #58205).max_load_tokens (#55885), back-pressure detection (#50045), SimpleCPUOffloadConnector Prometheus metrics (#57251), non-prefix-cacheable (#56810) and scratch (#57145) groups skipped, replicated layouts for multi-group MLA (#57652), regions of 512 GiB or more registered in chunks (#51081), cgroup memory checked before SHM allocation (#54014), and fixes for cache recency (#51787), MTP-retained sliding windows (#56709), canonical MLA rows (#56799), event metadata (#57453) and ROCm pinned memory (#57160).--mm-encoder-only (#58086), EC connector metrics (#54960), and encoder-only fixes (#58490, #58287, #57696).triton_kernels 3.8 MXFP4 MoE for gpt-oss and DeepSeek-V4 (#55934), a ROCR host segfault fixed (#57328); DeepSeek-V4/V4.1 HCA dual-stream (#56853) and layer-aware CSA2 overlap (#57407), FP8 wo_a (#54894), inverse RoPE fused into the sparse decode reduce (#57435, #57451) which now emits MXFP8 for a grouped FP8 wo_a (#58456), MXFP8 GEMM on native 32x32 scales on gfx950 (#58510), reused top-k ragged metadata (#57434), faster candidate block selection (#58208), Engram tables in host memory (#57491) and Qwen3.8-Flash-Next PLE tables offloaded to host memory (#57497), an opt-in AITER ASM decode route for DCP + speculative decoding (#56861), get_top_tokens() on the DeepSeek V4 MTP drafter (#57568), DSpark adaptive verification (#52362), opt-in VLLM_DSV4_LOGITS_FIX for sparse-indexer logits on gfx950/gfx942 (#50455), an accuracy revert of #56433 and #51692 (#57132), and clear errors for FSE with DPA+ETP (#57919); Kimi-K3 a4w4 FlyDSL kernels (#53940) with VLLM_ROCM_USE_AITER_MOE_SITUV2=a16w4|a8w4|a4w4 (#58201), low-concurrency speculative KDA (#58045), sharded latent MoE under EP (#54956) and fewer projection copies (#50592); a ROCm Hy4 path with torch.compile (13x lower decode latency, #57526); MiniMax-M3 AITER QK-norm fusion (#54535), copy-free K/V insert (#56849) and MXFP8 fixes (#53674, #58089); GLM-5.3-Flash boot fixes (#57192, #57252, #57425); AITER QuickReduce + RMSNorm (#48249), BF16 AsyncTP (#58098), static FP8 attention output fusion (#58099), QK-norm/RoPE/KV-cache fusion for MRoPE (#50212), AITER GDN decode for flat layouts (#53623), wvSplitK for single-output GEMMs (#53283), 69 fewer copies per decode step on the skinny GEMM path (#58566), tuned GEMM lookup through AITER (#55001), a narrower Triton prefill KV tile on RDNA3/RDNA4 (#58225), staged large pageable H2D copies (#56343), and fixes for MXFP4 MoE padding starving the KV cache (#56359), AITER MLA FP8 prefill OOM (#57923), unquantized cache descales (#56726) and AITER MoE fallbacks (#56590, #57866, #57426). VLLM_ROCM_USE_AITER_FP4_ASM_GEMM is restored and off by default (#57055); SWA bounded replay is disabled on ROCm (#57906).--device-ids honored (#56015), and device pointer overflow fixed (#54514).--device-memory-utilization alias (#56547), vLLM Recipes in the CPU image (#58796, #57306), s390x protobuf pin (#54978) and torchcodec video (#58693), and fixes for Ministral FP8 (#56985), FP32 router weights (#56168), zentorch import failures (#54923), macOS multimodal SHM (#57142), CPU affinity per local rank (#53636) and NIXL GDN state layout (#53300).mm_mxfp8 shape limits (#54223), online NVFP4 scales on reload (#57954), LM head linear metadata (#58444), and the fused SiLU-mul block-quant path skipped under a SwiGLU clamp (#57984).--tool-strict-level (#56268), response_format with tool_choice=auto (#56086), DeepSeek-V4 FIM completions (#44229), release_kv_cache_memory() and POST /release_kv_cache_memory (#44890), fixed-token prefill scoring via prompt_logprob_token_ids (#54335), per-request speculative decoding metrics in /inference/v1/generate (#43310), per-request metrics (#55084) and cache_write_tokens (#57222) in the Responses API, Anthropic thinking in /v1/messages (#58613), streaming reasoning and tool calls from the derender endpoint (#50550) with offloaded detokenization (#57528), vllm chat streaming thinking output (#57045), request body debug logging with --enable-log-requests (#58163), and only the summary line of config docstrings in --help (#57357).structural_tag (#47450), outlines EOS handling (#58612, #57743), parser-suppressed streaming logprobs (#58583), Inkling tool names after reasoning (#58792), and length finish_reason for truncated streaming tool calls (#46303).max_output_tokens in the tool loop (#58551), batched chat completions using the adjusted requests (#58929, #58958), a fresh parser per choice (#58939), deferred reasoning recounts (#56067), Responses MCP cleanup (#56988) and image detail default (#57241), Anthropic inline system detection (#58754) and disabled thinking with P/D (#58786), stop strings rejected on --tokens-only servers (#57058), invalid prompt_embeds returning 400 (#55451, #57006), prompts bounded after multimodal expansion (#57076), --override-generation-config penalties (#50769), Hub revisions (#56092, #57461), full logprobs in token-in/token-out responses (#58488), generative scoring cancellation (#57729, #58788), gRPC keepalive pings (#55102), and run-batch diarized transcriptions (#57948).--hf-overrides (#56931), --sse-keep-alive-interval (#58306), custom chat roles (#58311), MiMo V2.5 parsers (#57933), normalized reasoning controls (#56998), parser-owned output grammars (#55269, #57340), sampling masks over gRPC (#56777), local DP size in gRPC metadata with vllm-proto 0.3.0 (#57116, #57233), a per-request preemption histogram (#57033), lock-free histograms (#58574), Nemotron-H vision context (#57634), model-owned vision processors (#58109), an mm-processor benchmark (#51922, #58084, #58378), unsupported serve args recognized (#58330), NaN logprobs no longer killing the engine client (#51026), appended EngineCoreOutput fields accepted (#56533), and vllm-rs on PATH in the CUDA image (#57606).openai-responses backend for vllm bench serve (#54628) and model_id in latency/throughput JSON (#58112).mm_processor_kwargs and media_io_kwargs are rejected by default; trusted deployments opt in with --trust-request-mm-kwargs (#58830).min_tokens above the filled max_tokens default is rejected instead of wedging the engine (#57731); message sanitization filters upper-case memory addresses (#58832); LoRA adapters named after a served model are rejected (#59286).oss-harmony replacing openai-harmony (#55128), DeepGEMM fork pin bump (#57218), FlashKDA bump (#58846), and humming-kernels 0.1.16 (#58054).-x86_64-zstd) is published (#55608); Rubin CUDA 13.4 nightly images (#55953).mm_processor_kwargs and media_io_kwargs now return an error unless the server is started with --trust-request-mm-kwargs; server-level --mm-processor-kwargs / --media-io-kwargs and offline LLM are unchanged (#58830).tokenizer_mode="slow" was removed; it already behaved like "hf" under Transformers v5 (#58545).--enable-mamba-fine-grained-prefix-cache was renamed to --enable-mamba-shared-prefix-checkpoint (#57382).quantization="fp8" now redirects to the fp8_per_tensor online shorthand (#53585); Quark-specific silent online MXFP4 quantization was removed in favor of the online quantization API (#51800).return_assistant_tokens_mask option of /render and the assistant_tokens_mask response field were removed (#57520).VLLM_PLE_CPU_OFFLOAD was removed; use --engram-config (#57937). VLLM_XPU_ENABLE_XPU_GRAPH was removed and XPU graphs are on by default (#51600).--collect-detailed-traces was removed; use the list syntax (#55702).--enforce-eager also disables JIT kernel warmup unless fault tolerance is enabled (#58197, #58593); VLLM_BATCH_INVARIANT=1 uses breakable CUDA graphs without torch.compile (#57586) and disables sequence parallelism and async TP (#56377); DeepSeek-V4.1 SWA bounded replay is on (#56227) except on ROCm (#57906); Engram host tables are shared across co-located DP replicas when possible (#57651); FlashMLA mega attention is the DeepSeek-V4.1 default on SM100 (#56935); the AITER w4a4 ASM GEMM is off by default on ROCm (#57055).json_object is rejected at validation with the outlines backend (#57743).transformers now has an upper bound in requirements (#59614).@AndreasKaratzas, @khluu, @mgoin, @njhill, @BugenZhao, @stefankoncarevic, @robertgshaw2-redhat, @taneem-ibrahim, @Thangnguyenvn98, @hmellor, @Juntian777, @WoosukKwon, @yewentao256, @gau-nernst, @mmastrac, @LucasWilkinson, @Fangzhou-Ai, @NickLucche, @aoshen02, @JaredforReal, @mawong-amd, @DarkLight1337, @zyongye, @sfeng33, @vllm-agent, @okorzh-amd, @ZJY0516, @shen-shanshan, @gty111, @Isotr0py, @djramic, @zixi-qi, @wzhao18, @alec-flowers, @aarushjain29, @Rohan138, @yma11, @mjkvaak-amd, @ivanium, @yisustc, @MatthewBonanni, @wangxiyuan, @reidliu41, @chaojun-zhang, @shaohuaxi, @gcanlin, @ganeshr10, @linitra24, @yzong-rh, @divakar-amd, @atalman, @ShuoleiWang, @simondanielsson, @rasmith, @chaunceyjiang, @micah-wil, @liusy58, @LioEinaudi, @louie-tsai, @LiuYinfeng01, @jeejeelee, @zhenwei-intel, @jperezdealgaba, @hlin99, @zxd1997066, @lucamotz, @lucifer1004, @eopXD, @ashraf-bhuiyan, @TheEpicDolphin, @JulienDarve, @mayuyuace, @danisereb, @bigPYJ1151, @Etelis, @JohnQinAMD, @jiangkuaixue123, @tianmu-li, @akii96, @vllmellm, @RyanMa29, @afriedri, @elvircrn, @mustafayildirim, @yuzhouo7, @wtdcode, @biswapanda, @adtygan, @hickeyma, @itayalroy, @jiangLLM, @Hotragn, @faaany, @xhx1022, @jinzhen-lin, @sheralskumar, @Wauplin, @matteso1, @wjabbour, @Sunt-ing, @arpera, @hclsys, @S1ro1, @HDCharles, @fxmarty-amd, @liuzijing2014, @UNIDY2002, @samuelkim7, @markmc, @errmakov, @zhejiangxiaomai, @KernelClint, @i-m-aditya, @ColinZ22, @ppalanga, @franciscojavierarceo, @maithilijoshi20, @mfylcek, @almersawi, @albertoperdomo2, @0z5a, @jacklin78911-collab, @lzhan011, @Alex-ai-future, @freyfwt, @ZhengGong-amd, @mindungil, @amasen02, @devtyagi3909, @wenjinhust, @jbyczkow, @AdaAibaby, @harshit-sarvam, @vineethsaivs, @tripathiarpan20, @sashko-zakharchuk, @semerandre, @rebklee, @git-jxj, @cjackal, @xinnywinne, @andrewor14, @kyleliang-nv, @thillai-c, @dongluw, @ChuanLi1101, @omerpaz95, @jiaran-king, @liuyao0322, @coderfornow, @drakosha, @laulopezreal, @melcheikh, @shantipriya-amd, @Rukhaiya2004, @yuchenwang3, @lxy-alexander, @tlrmchlsmth, @HieDean, @Zoe923, @YCH188, @amd-sriram, @andylolu2, @MicheleCampi, @sdougbrown, @vMaroon, @shimib, @andakai, @thegoldenflow, @Levius-Fubuki, @olka-amd, @limitmhw, @yuwenzho, @adenzhou1350, @Josephasafg, @kylesayrs, @jdebache, @kaijunli-infr, @afierka-intel, @frida-andersson, @bnellnm, @ys2025-AI, @waizuichougou, @touch869, @rjrock, @sawsa307, @pavelzak, @ScarWar, @fadara01, @Mi-Jiazhi, @dilberx, @mahird3, @Zyann7, @roikoren755, @YukioZzz, @Ronnie-Rui, @acsoto, @czhu-cohere, @twu3202, @oliverholworthy, @weitliao, @sergiofigueras, @tuukkjs, @Sip4818, @karen-sy, @garrett361, @GirasoleY, @Navjot10, @taking-lying-flat, @wangyicong52, @majunze2001, @gangula-karthik, @ubwzwd, @linnea-lin-00638949, @kushaldabbe, @Yatimai, @guanxingithub, @jiakangkangfuzhe, @MichaelLapshin, @jhu960213, @tpopp, @mganczarenko, @qiching, @tangzzycc, @lijipeng787, @nikhilkulkarni1755, @QwertyJack, @Ankit-Jaiswal-AMD, @vorapolsiloai, @Dao007forever, @BPbruce, @divyvasal, @simpleqt, @blipbyte, @jhaotingc, @karya0, @farzad-elastix, @grYe99, @talorabr, @jackLei0901, @Yejing-Lai, @Priyjain-amd, @KEYS-A15, @LinzeShi, @bohnstingl, @nightcityblade, @vcave, @AARONKANG04, @CZT0, @snadampal, @microslaw, @haosenwang1018, @netanel-haber, @khushali9, @sammaji, @SIDDARTHAREDDY8, @100milliongold, @gongwei-130, @fululi12, @mkunredd, @mrodden, @noooop, @valarLip, @shallow10, @200lz, @askliar, @chuan932, @Monishver11, @YashasviChaurasia, @gokay-ai, @jiacao-amd, @yousafshah, @positive666, @TQCB, @lk-chen, @Ricardo-M-L, @huthvincent, @baljinderhothi-cohere, @vschandramourya, @kwen2501, @xiao-llm, @zhecfy, @Willian-Zhang, @simon-veitner-redhat, @mevince, @voidxb, @hongxiayang, @YannikHinteregger, @V-3604, @Rakul-Chauhan, @frankwang28, @LCAIZJ, @PeganovAnton, @jz-yolo, @QHarshil, @R3hankhan123, @jayzuccarelli, @yannicks1, @BaoYunkai, @xaguilar-amd, @xiaohuguo2023, @wxsIcey, @harshaladhav-amd, @varun-sundar-rabindranath, @kliuae, @LostFox11
Breaking changes: scale-out endpoints are opt-in on plain vllm serve via --enable-scale-out, replacing VLLM_ENABLE_SCALE_OUT_ENDPOINTS (#54579, #55176…
This release features 762 commits from 315 contributors (104 new)!
--load-format ipc_cache instead of reloading from disk (#54921), now covering FP4 checkpoints (#55465) and multi-node TP (#55468).HiSparseConnector (#53781), with Prometheus counters (#56061), a host cache shared across TP ranks (#56629), and the attention config inferred from the connector (#57041).--return-sampling-mask compacted on GPU, fixing an about 2x RL step-time regression (#54901).--engram-config (#54371), and torch.compile removed from the NVIDIA implementation so FP8 fits on a single GB300 (#55272).quantization_config.targets (#51285) and on partially pre-quantized checkpoints from any quant method (#51392), W4A16 DSA with the nvfp4_fp8_ds_mla KV cache (#51724), FlashInfer CuTeDSL NVFP4 W4A16 default over Marlin on SM100/103 (#53014), NVFP4 in the torch linear backend (#53319), per-quantization linear backend overrides (#51204), AutoRound 2/3/5/6/7-bit on CUDA (#52890), and DeepSelect top-k for the DSA sparse indexer (#56464).vllm serve via --enable-scale-out, replacing VLLM_ENABLE_SCALE_OUT_ENDPOINTS (#54579, #55176); GPTQ activation ordering (g_idx) removed (#54809); items deprecated for 0.29 removed, including the VLLM_PREFIX_CACHE_RETENTION_INTERVAL and VLLM_MM_HASHER_ALGORITHM env vars (#55353); the all Mamba cache mode deprecated (#55041); python -m vllm.entrypoints.grpc_server deprecated in favor of vllm serve --grpc (#56746); YaRN aligned with Transformers so vendor YaRN aliases no longer re-scale max_model_len (#56446).| Platform | Install |
|---|---|
| PyPI (CUDA 13.0) | pip install vllm |
| PyPI (CUDA 13.0, uv) | uv pip install vllm --torch-backend=auto |
| ROCm | pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.30.0/rocm723 |
| XPU | uv pip install vllm --extra-index-url https://wheels.vllm.ai/0.30.0/xpu --extra-index-url https://download.pytorch.org/whl/xpu --index-strategy unsafe-best-match |
| Platform | Docker Image |
|---|---|
| CUDA 13.0 (Default) | docker pull vllm/vllm-openai:v0.30.0 |
| CUDA 12.9 | docker pull vllm/vllm-openai:v0.30.0-cu129 |
| CUDA 13.0 + Ubuntu 24.04 | docker pull vllm/vllm-openai:v0.30.0-ubuntu2404 |
| CUDA 12.9 + Ubuntu 24.04 | docker pull vllm/vllm-openai:v0.30.0-cu129-ubuntu2404 |
| ROCm | docker pull vllm/vllm-openai-rocm:v0.30.0 |
| CPU | docker pull vllm/vllm-openai-cpu:v0.30.0 |
| XPU | docker pull vllm/vllm-openai-xpu:v0.30.0 |
Pre-built release artifacts are available in the Assets section at the bottom of this page, including:
--kv-cache-dtype auto resolving to fp8_ds_mla with FlashMLA (#45091), prefill sparse index workspace sentinel seeded (#55299), dequant gather grid sized by rows (#55061), optional Q-norm in the fused MLA epilogue and group_size=32 packed FP8 quant (#56215), sparse settings read from the text config for composite models (#56160), MegaMoE startup without EP fixed (#55914), and a Triton iHC pre/post fallback for HY V4 (#55059).glm5next NVIDIA subtree now packaged in wheels (#55214).VLLM_SPARSE_INDEXER_MAX_LOGITS_MB (#54513), fused PLE kernels and merged K/V projections (#54517), sparse GQA skipping padded indices (#54873), FP8 indexer cache (#54890), fused PLE residual and QSA output gate (#55309), UVA PLE offload and Engram tensor parallelism via --engram-config (#54371), torch.compile removed from the NVIDIA path (#55272), compact indexer logits workspace (#54915), HC combine-norm reused for MTP input (#54687), PLE MTP metadata transfers (#55054), Hopper LL-GEMM tuning table (#54560), B200 FP8 MoE tuning (#55890), and fixes for FP8 PLE weight scales (#54722, #54882) and fused PLE conv strides (#55375).n_predict from the text config (#55369), Qwen3 DSpark padded-vocab drafts (#55133), and GLM-OCR MTP weight loading (#49869) and position masking under CUDA graphs (#56447).audio_backend in --media-io-kwargs (#51826) with automatic decoding kept soundfile-first (#55642), torchaudio as the default resampler (#52598), media_io_kwargs in multimodal hashes (#54241), cache hash kwargs scoped by modality (#54918), empty video URLs with multimodal UUIDs (#54220), cached audio inputs with UUIDs (#56310), encoder CUDA graphs for MiniCPM-V 2.5/2.6/4.0 (#42785), Voxtral Realtime with FULL_DECODE_ONLY graphs (#51167), Triton/FlashInfer composite attention for multimodal prefixes (#56305), pruned sliding-window tiles for Gemma 4 multimodal prompts (up to 3-4x E2E, #53147), SDPA for BLIP-2 Q-Former (#55285), baddbmm Conformer scores (#55062), fused DeepEncoder relative bias (#55629), Gemma3n sparse GELU Triton kernel (#48498), and no duplicate text embedding in Qwen2.5-Omni (#55415).LLM.chat() for multimodal models (#55288), decoded frame lists parsed as one video (#55326), swapped H/W in dummy video profiling (#56385), base64 video validation (#54323), Sentence Transformers 5.4+ configs (#54533), ColQwen3.5 pooler projector (#54847), torch.compile enabled for StableLM (#54969) and Sarvam MLA (#55817), and Nemotron latent-MoE skipping a redundant all-reduce at TP>1 (about 13% decode win, #52301).--lora-modules name=path with = in the path (#53353).apply_write (#55819), fast-prefill support (#56145), DP sync skipped for all speculator uniform decodes (#54856), prompt embeddings in decoder pooling (#56639), pooling post-processing on non-final PP ranks (#56666), kernels warmed before graph capture (#55341), CUDAGraphStat (#52358), Proton CUDA graph attribution (#51084), platforms without pinned memory such as WSL (#56908), draft moe_backend (#54788) and attention_backend (#54826) overrides honored, DCP metadata initialized after batch partitioning (#55212), and a record_stream fix in the PP draft broadcast (#55745). The MRV2-specific Triton JIT warmup migration was reverted (#56654).disable_eagle_block_drop (#53388), adaptive verification on FLASHINFER_MLA_SPARSE_DSV4 (#52724) and deferred until after kernel warmup (#55455), full-vocab EAGLE3 drafters skipping vocab remapping (#53458), DSpark preserving the target DCP config (#55472), DFlash drafters dropping FlashAttention's AOT schedule (#54374), DP token padding excluded from draft metadata (#55458, #56181), fastsafetensors deadlock avoided for PP drafts (#54416), Mamba state cached at the EAGLE resume position (#53945) and preserved for padded prompt tails (#55178), resumed requests padded in P/D (#55126), and warmup device selection (#55245).all Mamba cache mode deprecated with MRV1 fallback (#55041), and vllm:request_num_preemptions histogram (#49984).--attention-backend B12X_ATTN (#52017), FlashInfer XQA decode on SM90 (#50439), DeepSelect top-k with --sparse-indexer-topk-backend (#56464), BF16x3 router GEMM default on SM100 for FP32-router models (#55899), split-row Triton top-p for small batches (#54651), manual ActivationQuantFusionPass producer for Llama (#51415), fused add-RMSNorm + NVFP4 (#51925), prefix-prefill tiling independent of KV page size (#54194), FlashInfer GDN prefill on SM12x (#55715), fused GDN MTP decode for SM110 (#53835), masked MoE activation for padded layouts (#51217), cooperative batched_moe_align_block_size (#53280), PDL in fusedQKNormRopeKernel (#55755) and weight prefetch in fused q/kv RMSNorm (#55020), XD-RoPE folded into M-RoPE (#56078), and the FlashAttention pin synced with upstream (#54819).--load-format ipc_cache (#54921), FP4 checkpoints (#55465), multi-node TP (#55468), GPU UUID socket folders (#56669), per-client IPC tensor export so a copy-mode client cannot release zero-copy weights (#56472), and a clear error on non-CUDA/ROCm platforms (#56010).allow_target_only_watermarking and context deduplication (#56122).ncclUniqueId rendezvous for torch-free trainers (#53784), stale HPC QK-norm weights after refit fixed (#56706), and session_id on GPU BlockStored events (#51381).FULL_DECODE_ONLY graphs (#55095), MRV1 breakable graphs scoped to the piecewise path (#56312), sync checks cover paged async copies (#53491) and torch.compile (#56904), more H2D copies pinned (#54660, #55202, #56138), GDN capture metadata built without a device sync (#55404), queued work carried over when materializing the dedicated stream (#56382), FlexAttention recompiles avoided (#55888), persistent top-k falling back on low-shared-memory GPUs (#54110), KV page size scaled for hidden-state extraction with TP (#50894), out-of-bounds access in SM90 sparse MLA mixed batches (#56969), sparse-MLA piecewise capture crash (#56825), TRT-LLM ragged prefill regression (#55499), MoE fused sum int32 overflow (#50220), odd-row per-token-group quant cliff (#56478), dummy-load re-initialization (#56682, #56688), warning_once retaining the LLM instance (#54098), DSpark cache-group capability under python -O (#55234), sparse-MLA DCP stabilized for GLM PCP configurations (#55879), DeepGEMM FP8 warmup coverage (#56452), Triton autotune inspection skipped without Triton (#56676), mm_device_do_normalize under encoder CUDA graphs (#55370), VidCom2 frame budgets read once (#55331), the CUDA profiler start fix (#55237), FlashInfer all-reduce workspace failures logged as errors (#55127), and backend-neutral KV cache and MFU log lines (#55650).ROCM_AITER_UNIFIED_ATTN (#53695), MoRI-IO SWA+HMA (#53721), attention sinks in AITER sparse MLA (#54404), non-causal DSpark drafts in AITER MLA decode (#55966), shared KV prefill (#55887), V-cache padding masked (#53856), elastic EP deadlock (#56610), profiler fixes (#54171, #56190), breakable CUDA graphs auto-enabled for DeepSeek-V4.1 (#56349), the AITER paged-MQA outputs guard removed for DeepSeek-V4 (#55808), DSV4 indexer gather cold-compile churn removed (#56153), PDL JIT warmup skipped where unsupported (#56035), Kimi-K3 fused prefill state_indices crash (#56526), Ray NIXL agents for sharded RDT (#56687), Qwen3-Omni audio encoder TP with non-divisible heads (#45900), Quark MXFP4 Qwen3.8-Flash-Next shared experts on gfx950 (#55213), AITER preshuffled FP8 block-scale kernel (#56433), linear biases with requires_grad=False (#56161), a clearer unsupported KV dtype error (#55236), Mooncake in the ROCm image (#55002), and AINIC pins (#54112, #56459).fused_input_norm (#52945) and fused grouped_topk (#53580), LoRA for DeepSeek V4 (#53689), DP external LB device assignment (#53037), triton-xpu 3.8.0 shim (#56014), auto-round-lib 0.15.0 (#54515), and fixes for FalconH1 PP (#55913), Ernie4.5-VL RoPE (#55942), gated RMSNorm ops (#54968), Qwen DFlash context keys (#56431), INC quant selection (#47237), and moe_wna16 loading (#52651).lmcache_mp_connector state transitions guarded (#47505); HiSparse per-layer offsets (#57027), P/D region-mapped pulls (#57077) and NIXL full-block imports (#57049); NixlPush remote-agent reads guarded (#56317).unbound_store_timeout_s and late-fetch rejection (#53453), OffloadingConnector retention interval (#51886) and offload hits under MTP/EAGLE (#52771), SimpleCPUOffload fine-grained hybrid prefix hits (#54736), eager registration and final flush (#53532), BlockStored metadata (#54325), mixed page sizes for DSA models (#54756), prefix-cache bypass (#54998), no-forward-step stores (#56621); UniformTypeKVCacheSpecs portability (#51690) and DCP scaling (#50883); and fixes for async lookups (#54872, #55075, #55823), SWA reachability and coverage (#54362, #55712, #56486), final-token slot (#54288), oversized offers (#54759), ARC fallback (#55461), disk alignment (#55424), DiskBackend buffer race (#51667), offload key ordering (#52923), prefetch slot ownership (#54975), load boundaries (#52807), and tiering shutdown (#52290).linear_backend_per_quant (#51204), AutoRound 2/3/5/6/7-bit (#52890), TRT-LLM FP8 block-scale MoE with SwiGLU clamp (#55069), and Quark per-block FP8 MoE on ROCm (#52263).QuantKey (#52958) and ModelOpt linear methods unified into one QuantKey-driven method (#49381).weight_scale (#54770), GPT-OSS FP8 MoE with Quark (#54824), FSE detection for Quark models (#54573), DSv4 MXFP4 backend alias selection (#53586), unset group_size on compressed-tensors WNA16 MoE (#53163), block FP8 MTP in ModelOpt mixed checkpoints (#55513), and unquantized MoE weight storage preserved on ROCm for RL updates (#46009)./v1/responses/render (#50195), scale-out endpoints behind --enable-scale-out (#54579, #55176), count_reasoning_tokens so usage reports reasoning_tokens (#54982), prompt metadata from /inference/v1/generate (#53187), vllm:request_num_preemptions (#49984), MFU/MBU for SWA and hybrid layers (#55624), consolidated Prometheus bucket defaults (#48866), site-packages reasoning and tool parser plugins (#45241), and max_num_queued_reqs shared across API server processes (#54746).patternProperties/propertyNames/unevaluatedProperties (#42904), full-history reasoning scans eliminated (#55223) with is_reasoning_end derived from the grammar (#56200), invalid structured-output requests no longer stopping the engine (#51450), a unified Cohere Command parser (#56392), MCP SDK 2.x schemas (#53870), incomplete tool-call markup dropped consistently (#47562), DSML tool calls without the wrapper (#55954), misspelled wrappers (#56141) and implicit parameter closes (#54838), Gemma4 malformed openers (#53444), Seed-OSS turn boundaries (#54264), Step-3.5 with structured outputs (#53174), reasoning parser plugins loaded before headless config (#53124), and a warning for removed guided_* fields (#54285).browser.find action type (#55305), Responses validation boundary restored (#55761), validation errors migrated to VLLMValidationError (#50254, #50257, #55701, #55735), unsupported reasoning_effort as a client error (#54022), content format detected through macro parameters (#53824), Anthropic stop_sequence in message_delta (#55325), Cohere priorities bounded (#56146), translation rejecting string files (#55618), tokenless chat and audio streams rejected (#54708), beam search respecting skip_special_tokens (#56211) and aborts (#56249), nested cancellation handlers removed (#55602, #57024), extension integers validated (#55606), routed-expert prompt offsets validated (#56844), MistralCommonBackend avoided for HF tokenizers (#54192), chat template fallbacks packaged (#53762), and the content format warning restored (#54622).assistant_tokens_mask (#54539) and token offsets (#54692) kept aligned with truncated prompts, skip_decoder_start_token honored (#54799), renderer warmup bounded (#55448), LoRA adapter names in pooling responses (#56004), max_embed_len for chunked embeddings (#55551), body request_id honored (#55665), actual input usage for scoring (#56573), STEP token pooling (#55307), offline Jina token limits (#56415), and a warning for Qwen3 rerankers without a chat template (#56017).--lora-modules (#54837), HTTP RL weight synchronization (#56567), HF revisions, offline mode and cache directory (#56386), {% generation %} chat template blocks (#56378), the vllm-proto crate on crates.io (#56365), reasoning_tokens in usage (#54883) built on token-attributed text (#52910, #54884), Mooncake/NIXL connector metrics (#52755) and vllm:iteration_tokens_total (#56990), gRPC preprocessed multimodal features (#55047), remote-prefill multimodal metadata (#54814) and engine errors surfaced as INTERNAL (#56405), Qwen4-exp multimodal (#54813), optional --max-model-len for render (#55328), per-request vocabulary counting removed (#54449), decoded chunks coalesced per update (#55012), bounded recursive parsers (#54303), tekken 0.2.0 (#53056), strict whitespace framing (#55417), consistent parser resolution (#56018), empty structured-output values rejected (#54821), selected-token logprobs (#56406), undefined token ids skipped (#55240), DeepSeek renderer fixes (#53281, #54854, #56593, #56260, #56366), and HF ShareGPT multi-turn datasets in vllm bench serve (#51104, #56915).vllm bench serve accounts for client queue time (#54136) with consistent streaming TTFT/E2E accounting (#55508), non-positive --max-concurrency rejected (#54887), mm-processor crash fixed (#56300), complete pooling responses measured (#56760), explicit empty and zero arguments honored in vllm chat/complete (#55710), vllm launch render shutdown hang fixed (#54913), double /stop_profile crash fixed (#51678), and remaining non-OpenAI content moved out of the openai folder (#56369).cache_salt validated before reaching LMCache so one request cannot take down the engine (#51444); scale-out multimodal features validated before engine handoff (#51898); late-interaction query caches keyed by server-generated ids instead of X-Request-Id (#51445).openai >= 2.25.0 (#49104), huggingface_hub >= 1.31.0 with httpx imported through it (#56460)._USE_MATH_DEFINES defined for FlashMLA targets (#54007)./render, /derender, /inference/v1/generate) are no longer registered on plain vllm serve unless --enable-scale-out is passed; VLLM_ENABLE_SCALE_OUT_ENDPOINTS was removed. vllm launch render and vllm serve --tokens-only always register them (#54579, #55176).g_idx is ignored and the related Marlin, GPTQ, CPU and RDNA3 kernels are gone (#54809).VLLM_PREFIX_CACHE_RETENTION_INTERVAL and VLLM_MM_HASHER_ALGORITHM env vars (use the config fields), the use_fp4_indexer_cache alias (use indexer_kv_dtype), the CUDA_VISIBLE_DEVICES fallback on ROCm (use HIP_VISIBLE_DEVICES), and the seq_lens_cpu / num_computed_tokens_cpu attention metadata properties (#55353).all Mamba cache mode is deprecated and falls back to Model Runner V1 (#55041).python -m vllm.entrypoints.grpc_server is deprecated; use vllm serve --grpc (#56746). sse_keep_alive moved to vllm.entrypoints.serve.utils (#56369).max_position_embeddings by the factor a second time, so derived max_model_len drops for some models (TeleChat3-36B-Thinking 131072 to 32768, sarvam-105b 5242880 to 131072); attn_factor and extrapolation_factor are ignored in favor of mscale/attention_factor (#56446).--disable-hybrid-kv-cache-manager; use READ mode otherwise (#53721).auto (#51826, #55642).VLLM_DEEPEP_V2_COMBINE_OVERLAP=0 to disable (#52781), AITER custom AG/RS on for DP attention + TP experts with VLLM_ROCM_USE_AITER_CUSTOM_AR=0 to disable (#48247), native CUDA AttnRes for Kimi-K3 on SM100 (#54261), and VLLM_USE_BREAKABLE_CUDAGRAPH=1 no longer overriding FULL graphs on MRV1 (#56312).@khluu, @AndreasKaratzas, @taneem-ibrahim, @BugenZhao, @yewentao256, @njhill, @NickLucche, @aoshen02, @LopezCastroRoberto, @WoosukKwon, @LucasWilkinson, @zyongye, @hmellor, @Fangzhou-Ai, @sfeng33, @Isotr0py, @gau-nernst, @JaredforReal, @wzhao18, @fxmarty-amd, @stefankoncarevic, @Alex-ai-future, @mgoin, @micah-wil, @mayuyuace, @djramic, @ZJY0516, @mawong-amd, @Rohan138, @sychen52, @waizuichougou, @chaunceyjiang, @simondanielsson, @KernelClint, @AdaAibaby, @gty111, @zhenwei-intel, @zxd1997066, @majunze2001, @divakar-amd, @zhewenl, @giuseppegrossi, @Levius-Fubuki, @gcanlin, @jperezdealgaba, @bigPYJ1151, @stecasta, @linitra24, @chaojun-zhang, @shaohuaxi, @reidliu41, @atalman, @mfylcek, @lucifer1004, @ZeldaHuang, @askliar, @peakcrosser7, @yzeyu71, @hungnnvidia, @zupengwang, @TheEpicDolphin, @hlin99, @liusy58, @jyizheng, @jimmy-adams, @lucamotz, @jeejeelee, @shen-shanshan, @MatthewBonanni, @cjackal, @aarushjain29, @noooop, @Juntian777, @FeathBow, @Hotragn, @Jankwi, @lzhan011, @subhashpolisetti, @maithilijoshi20, @ilmarkov, @lukealonso, @mingg26, @franciscojavierarceo, @mevince, @zdtsw, @bnellnm, @omerpaz95, @jikunshang, @rbrugaro-amd, @bohnstingl, @specture724, @afierka-intel, @mohit-sarvam, @andylolu2, @pisceskkk, @starkwj, @wangyicong52, @eastwood-c, @wuhangxian, @yzong-rh, @yma11, @TQCB, @Sy0307, @jbyczkow, @itayalroy, @xiaguan, @alec-flowers, @ayush1399, @orestis-z, @mobicham, @ArcheyChen, @Zhenzhong1, @ZhengGong-amd, @jorge-menjivar, @IwakuraRein, @linamy85, @wentian-byte, @afriedri, @dharak-cohere, @DarkLight1337, @lvliang-intel, @zigzagcai, @harjothkhara, @bojiang3, @maireneu, @z-zanez, @JaynouOliver, @pra2107tham, @jdebache, @aijanai, @Dao007forever, @juhi10071998, @odashi, @Etelis, @fjosw, @jackLei0901, @tarukumar, @positive666, @tzielinski-habana, @drakosha, @iyastreb, @Harry-Chen, @sheralskumar, @ccrhx4, @aaab8b, @982945902, @tanyuqian, @DebugSy, @jasonozuzu-cohere, @byshiue, @yifjiang, @BolinSNLHM, @wseaton, @Yuzu23, @andyxning, @andy-neuma, @foraxe, @xuhuan51, @AnshulDesai, @luyixiao95, @null-Exception1, @yiz-liu, @zhang-keliang, @erdholion, @happyyzy, @GirasoleY, @SubSir, @natsala13, @yongqinwang-cmd, @StevenWang-CY, @LauraGPT, @wei-core, @zhejiangxiaomai, @bakiburakogun, @Whamp, @almogtavor, @zhouyou9505, @jschmied, @wenhuach21, @Big2Wheel, @vhagor, @frankie-ys, @zufangzhu, @stmatengss, @seanwestfall, @kamb-code, @somuai, @Josephasafg, @Aj2280, @am-cohere, @Sip4818, @liuyao0322, @heliubj18, @krsish, @Oskii, @xijiaat, @benenzhu, @tdoublep, @akshaver, @SherifWaly, @TensorRaya, @shikamd123, @Roderick-Wu, @lcskrishna, @BANANASJIM, @soodoshll, @mlazos, @Akine-Ko, @miyakido, @meena-at-work, @nikhilkulkarni1755, @jcayab, @wincent8, @abrahamzewoudie, @ronensc, @haic0, @gc-fu, @ykamiset, @Xarbirus, @zvier, @avininjamay8, @ColinZ22, @YunzhuLu, @jiacao-amd, @kliuae, @tobymao, @zexplorerhj, @ywang96, @YoavMiron, @xonder, @bjf-frz, @cenab, @josiahdavis, @juliendenize, @JiataiWang, @colinmcnamara, @tlrmchlsmth, @LearningMachine621, @jamesETsmith, @YukioZzz, @kushaldabbe, @lzzzzzc, @SalikovAlex, @pmanczak, @fadara01, @JulienDarve, @nicole-lihui, @zRzRzRzRzRzRzR, @100milliongold, @labAxiaoming, @GuyStone, @Wauplin, @SorenDreano, @chengchengpei, @circlepen, @amd-xavierwang, @tahsintunan, @biswapanda, @junuxyz, @walterbm, @arcusbuilds, @JartX, @arpera, @jcotant-inferact, @PatrykSaffer, @maeehart, @surajm20061998, @Woolgathererer, @kebe7jun, @ShengleiFu, @pavelzak, @stu-cao, @CedricHwong, @lxy-alexander, @BWAAEEEK, @Chinmay-Kulkarni-AMD, @mgehre-amd, @BruceLoveDecimal, @hickeyma, @jiangkuaixue123, @xiaozhenbi, @ganeshr10, @Ronald1995, @arkohut, @russellb, @zqlcode, @tomasruizt, @thillai-c, @samnordmann, @Ianniu123, @Luosuu, @netanel-haber, @MicheleCampi, @zhaoguochun1995, @JohnQinAMD, @liranschour, @jacklin78911-collab, @migarci2, @404mario, @ItsRoy69, @mkhazraee, @Ubospica, @Thangnguyenvn98, @YeonwooSung, @alexliluz, @ECMGit, @luoyuctl, @averma12, @drslark, @ppalanga, @zwang86, @wangshangsam, @jiahanc, @vllm-agent, @connorcarpenter15
Breaking changes: ten deprecated model architectures removed (#53608); FlexOlmo, Olmo3 and Hunyuan V1/VL migrated to the Transformers modeling backend…
This release features 594 commits from 277 contributors (91 new)!
extract_hidden_states speculation (#49811), padded FULL cudagraph dispatch for uniform decode under spec decode (#53407), and DP-sync skipping before EAGLE/MTP draft prefill (#53694). MRV1 remains in use for a few ROCm models and features MRV2 does not yet support.eh_proj (12.9-25.2% kernel speedup, #53942), GEMM-RS extended to GEMM-AR (#53053), MLA gate merged into the QKV-A projection (#54015), K3 DCP with DSpark (#52188) and DCP partial prefix cache hits (#50493); DeepSeek V4 shared experts fused into MegaMoE (#53040), adaptive top-k width re-landed (#52823), a native SwiGLU clamp kernel for Humming MoE (#53685), and an opt-in FlashInfer moe_ep expert backend (#49636).--per-request-spec-decode-metrics (#48915), adaptive verification extended to logprobs (#52242), SM100 sparse MLA for GLM-5.2 (#52783) and DeepSeek V4 on SM90 (#52795), Qwen3-Omni DSpark drafts (#52560), PLaMo3 EAGLE-3/DFlash (#54239), and DFlash2 loading from speculators format (#53797).sharded_rdt P2P backend where each worker pulls only its TP/EP slice over NIXL or Ray Direct Transport (#43375), rank-local IPC weight updates (#52497), sparse checkpoint-coordinate updates through native weight loaders (#50723, #53751), and routed expert loading for gpt-oss (#52209).prefix_cache_retention_interval is now a CLI argument defaulting to 0 (#52216), with dense retention automatically restored for hybrid models using EAGLE/MTP (#55760, #55861).VLLM_ALLREDUCE_USE_FLASHINFER=0 (#52998); prefix-cache NONE_HASH is deterministic by default so distributed KV cache users no longer need to pin PYTHONHASHSEED (#51875); new --max-num-queued-reqs / --max-num-queued-tokens admission-control flags (#49445).python -m vllm.entrypoints.openai.api_server deprecated in favor of vllm serve (#52131); VLLM_TEST_FORCE_FP8_MARLIN (#52182) and VLLM_ROCM_USE_AITER_FP4_ASM_GEMM (#53141) removed.Now that Model Runner V2 is used by default, we are considering Model Runner V1 deprecated and are targeting v0.32 for its removal. We do not intend to accept any more MRV1-specific improvements or optimizations.
Some features are not yet supported in MRV2 but we are planning for these gaps to be closed within the next 2-3 weeks. These include sequence parallelism, dual-batch overlap, elastic expert parallellism, custom logits processors and certain speculative decoding methods. For now, vLLM will still fall back to use MRV1 if any of these features are configured.
| Platform | Install |
|---|---|
| PyPI (CUDA 13.0) | pip install vllm |
| PyPI (CUDA 13.0, uv) | uv pip install vllm --torch-backend=auto |
| ROCm | pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.29.0/rocm723 |
| XPU | uv pip install vllm --extra-index-url https://wheels.vllm.ai/0.29.0/xpu --extra-index-url https://download.pytorch.org/whl/xpu --index-strategy unsafe-best-match |
| Platform | Docker Image |
|---|---|
| CUDA 13.0 (Default) | docker pull vllm/vllm-openai:v0.29.0 (v0.29.0-cu130 also works) |
| CUDA 12.9 | docker pull vllm/vllm-openai:v0.29.0-cu129 |
| CUDA 13.0 + Ubuntu 24.04 | docker pull vllm/vllm-openai:v0.29.0-ubuntu2404 |
| CUDA 12.9 + Ubuntu 24.04 | docker pull vllm/vllm-openai:v0.29.0-cu129-ubuntu2404 |
| ROCm | docker pull vllm/vllm-openai-rocm:v0.29.0 |
| CPU | docker pull vllm/vllm-openai-cpu:v0.29.0 |
| XPU | docker pull vllm/vllm-openai-xpu:v0.29.0 |
Pre-built release artifacts are available in the Assets section at the bottom of this page, including:
RMSNormFuser.fuse performance fixed (#52766).lm_head is loaded even when the config claims tied embeddings (#51665, #53170).mm_processor_kwargs honored (#53808), Qwen3-VL profiling honors cap_pixels_per_frame (#54380), video frame sampling respects MP4 edit-list trims (#48608), oversized items skip the MM processor cache instead of crashing (#53016), encoder cache entries survive until their last use (#54284), and repeated multimodal requests with the SHM cache no longer terminate the engine (#54994).extract_hidden_states (#49811), padded FULL cudagraph dispatch (#53407), DP-sync skipping for drafts (#53694), decoupled draft/target gumbel noise streams (#54282), encoder-only path split out (#53176), and memory released correctly on shutdown and sleep (#53508, #54246, #54162, #53955, #53682).DSparkDraftModel configs for Qwen3 (#52197), and Gemma4 MTP under CUDA graphs (#53884).prefix_cache_retention_interval argument (#52216, #55760, #55861), deterministic NONE_HASH (#51875), queue admission control (#49445), KV null block reserved when validating max_model_len (#47272), spec decode no longer padded up to max_model_len (#53962), and negative external block allocation prevented (#52707).sharded_rdt P2P weight sync (#43375), rank-local IPC updates (#52497), sparse checkpoint updates (#50723, #53751), gpt-oss routed expert loading (#52209), stable DeepSeek V4 mHC broadcast buffers across weight sync (#52626), and packed weight transfer stream reuse capping reserved-memory waste (#52951).trace_decode_token_ids for deterministic decode replay (#46701), per-arch tuned batch-invariant matmul configs (about 3x decode kernels on RTX 4090D/H20, #53247), Blackwell autotuning with 33.6% E2E latency reduction (#53649), deterministic MoE combine under DP+EP (#45683), and fuse_allreduce_rms disabled under batch invariance (#51292).--linear-backend flashinfer_cutedsl (#50572), replicated embedding and norm fusion for DSV3 flat models (#48484), standardized fused shared-expert selection (#51695), GPT-OSS MoE topk metadata reuse (#45457), tuned cooperative topk (#53382), and tuned FP8 fused_moe for Qwen3.5 on L40S (+7%, #53819).KVCacheLayout enum (#51718), JIT warmup provider registry (#50174), --cpu-offload-params now reaches vision/audio towers (#53120), attention backend probe failures no longer crash init (#51703), FlashInfer XQA falls back on unsupported head_dim (#53111), FlashInfer prefill LSE normalized before merging to fix prefix-cache logits divergence (#52796), seed preserved when a batch mixes seeded and unseeded requests (#51866), startup thread allocation accounts for local DP workers (#52385), int32 overflow fixes in fused SiLU block quant (#53409) and LoRA kernels (#53034), a shared-memory race in fused groupwise RMSNorm quantization (#54111), BLHNC addressing for FlashInfer sparse MLA (#54465), Mamba state copy race (#50729), and a start_profile no-op after auto-stop (#51839).--dcp-q-replicate with query replication default-on for GLM sparse attention (#50382), DCP fused attention fix for DeepSeek-V3.2 / GLM-5.2 (#50005), sparse MLA metadata and kernel block sizes under DCP (#52377, #51031), PCP PIECEWISE cudagraph fixes (#53869, #53515), and PCP compatibility checks delegated to the PCP manager so plugins can enable GQA+PCP (#53853).--enforce-eager startup (#51824) and the decode/cudagraph path (#52632), NCCL>=2.31 heap overflow fix (#53008), cross-node MNNVL all-reduce gated by capability (#53253), and MNNVL all-reduce buffers sized for DSpark (#50932).save_decode_cache (#52466) and hybrid DCP prefix caching (#53324), externally transferable KV cache group identification (#53779), async KV loads deferred past forward launch (#53333), kv_transfer_params for /inference/v1/generate (#42644), MoRIIO shared KV region registration after the layout refactor (#53698), and Mamba fixes for Mooncake (#51362, #51358, #53663) and NIXL (#53523)./dev/shm leak on crash fixed (#52596), CPU->GPU loads ordered against compute stream (#50696), store_threshold counting fixed (#52227), in-flight primary keys cascaded (#53329), and padded GPU cache storage handled (#54021)./v1/messages/render for the Anthropic Messages API (#45803), /cohere/v2/chat/render (#53219), SSE keep-alive comments for idle streams (#51034), per-request spec decode metrics (#48915), video embeds input (#54242), --max-num-queued-reqs / --max-num-queued-tokens (#49445), and pooling requests can set padding (#51157).vllm_xargs forwarded to sampling params (#53308), stop_sequence stop reason reported (#45807), Cohere citation/tool 500s fixed (#52175), and stop string limit applied (#53750).logprobs=-1 in Completions (#46175), streamed logprob offsets with echo (#47815), batched chat echo (#52529), all choices returned from /inference/v1/generate with n>1 (#52399), cache_salt forwarded for content parts (#54315), stop_token_ids validated against vocab (#54196), and 4xx for client errors in /detokenize (#52622), malformed namespace tools (#53763), malformed base64 audio (#53744), and unknown chat roles in DeepSeek encoders (#53071); tool-call arguments in replayed history are parsed defensively (#48922), empty FlatLogprobs slices are handled (#53704), and empty bad-word tokenizations are rejected (#53433).min_tokens (#54218), unsupported pattern+length schemas rejected (#49996), MistralCommonBackend tokenizers (#52720), XGrammar termination in batches (#52805), spurious FSM errors after speculative reasoning end (#53046), shared parser engine adapters (#52830), Gemma4 parenthesized tool calls (#53657) and enable_thinking default (#52430), HY-V3 parallel calls in one delta (#53965), GPT-OSS Harmony strict grammar (#52222), Kimi K3 reserved markers excluded from response text (#52889), and unused request-local reasoners skipped (#52573).--generation-config vllm (#53044), truncate_prompt_tokens (#48584), OpenAI edge-case alignment (#53218), optimized SSE hot path (#51321), pure-Rust protox replacing protoc (#52892), RL world-size reporting (#53204) and routed expert prompt offsets (#52703), and fixes for GLM-5.2 template parity (#51426), Qwen parser auto-detection (#51169), Kimi K3 reasoning_effort="none" (#53043), n > 1 rejection on /inference/v1/generate (#52844), and the LogprobsTensors wire schema (#53939).vllm bench serve (#53213).vllm launch runs the serve argument checks (#52825), run_batch.py moved out of the openai folder (#53500) with its upload retry fixed (#50588), vllm bench warns on a warm prefix cache for random runs (#53920) and restores multimodal datasets on the vllm throughput backend (#52168), and the vLLM recipes tool gained sweep recommendations and alias parsing (#53325, #53946).cache_salt length bounded to 1024 to prevent scheduler CPU exhaustion (#54353).VLLM_MAX_AUDIO_CLIP_FILESIZE_MB enforced on all audio paths (#53561).api_key and hf_token redacted from startup logs, compile cache factors, and the Rust frontend launch log (#52523, #53625, #53738).RWForCausalLM and StableLMEpochForCausalLM aliases, and PrithviGeoSpatialMAE (superseded by Terratorch) (#53608).python -m vllm.entrypoints.openai.api_server is deprecated; use vllm serve (#52131).prefix_cache_retention_interval default changed from dense to 0 for SWA/SSM models; the env var is deprecated in favor of the argument (#52216).VLLM_TEST_FORCE_FP8_MARLIN removed in favor of --linear-backend / --moe-backend (#52182); VLLM_ROCM_USE_AITER_FP4_ASM_GEMM removed (#53141); dead --attention-config.use_prefill_decode_attention removed (#52557); other long-deprecated parameters cleaned up (#53559).@AndreasKaratzas, @khluu, @mgoin, @taneem-ibrahim, @yewentao256, @njhill, @BugenZhao, @hmellor, @GirasoleY, @DarkLight1337, @mayuyuace, @WoosukKwon, @jeejeelee, @noooop, @jperezdealgaba, @LucasWilkinson, @wzhao18, @stefankoncarevic, @gau-nernst, @okorzh-amd, @ZJY0516, @HollowMan6, @connorcarpenter15, @gcanlin, @aoshen02, @linitra24, @mhuzaifa3, @Isotr0py, @Etelis, @MatthewBonanni, @TheEpicDolphin, @zupengwang, @atalman, @NickLucche, @rasmith, @elvircrn, @JasonKeyiL, @Hotragn, @waizuichougou, @pmanczak, @divakar-amd, @he-yufeng, @xuebwang-amd, @qgallouedec, @esmeetu, @ZeldaHuang, @Prudhvivuda, @chaunceyjiang, @zxd1997066, @louie-tsai, @andyxning, @pisceskkk, @tjtanaa, @qli88, @tlrmchlsmth, @LopezCastroRoberto, @micah-wil, @hongxiayang, @BabyDrangoner, @yimdev, @mganczarenko, @SageMoore, @biswapanda, @sfeng33, @russellb, @itayalroy, @bigPYJ1151, @drakosha, @qwerqwerqwe8688-jpg, @akii96, @shen-shanshan, @meiyeh123, @reidliu41, @yma11, @frgossen, @lukealonso, @tanchao, @Fangzhou-Ai, @vllm-agent, @djramic, @andrewbcohere, @zyongye, @gty111, @ShuoleiWang, @ZHIHANCHEN03, @KurodaKanbei, @alexeldeib, @khushali9, @xiaohuguo2023, @hungnnvidia, @thisjiang, @arpera, @ShengleiFu, @zhenwei-intel, @jungjiyu, @yzong-rh, @charlifu, @hclsys, @Ronald1995, @mkhazraee, @YukioZzz, @theamalsebastian, @030611, @cr-zhao, @jbyczkow, @tommy-asai-sonarsource, @jikunshang, @bobboli, @LH-and-FPGA, @SayHelloToWorld, @kkt-cohere, @floatlibai, @rajathpi, @lxy-alexander, @gangula-karthik, @ActiveSky, @fxmarty-amd, @sstamenk, @mpashkovskii, @LiuYinfeng01, @chaojun-zhang, @Oxygen56, @daviswer, @vineethsaivs, @Agoni-02, @Andy365-365, @wangxiyuan, @haoyangqian, @Naveassaf, @eilamc14, @sseanliu, @y0hnn, @dineshchitlangia, @yiliu30, @Lossfull, @lxyxinyi, @vanshbhatia-amd, @jcotant-inferact, @chengy-sysu, @92hyungjun, @xyang16, @AmitMY, @zllion, @AnkitNakhawa, @dmvevents, @lengrongfu, @JC-ut0, @Eoin-Houstoun, @sagearc, @wjabbour, @sandeep-maddipatla, @seonjinn, @Yiqin-17, @kyleliang-nv, @thanhpt1110, @MKQuantum, @matthewkotila, @vhagor, @tthakkal, @ShuaiShao93, @hagaikwa-redhat, @shijuzhao, @elwhyjay, @Gregory-Pereira, @ErenAta16, @morrison-turnansky, @DCoEngine, @SoluMilken, @xianbaoqian, @kliuae, @nascheme, @waynehacking8, @stecasta, @vMaroon, @zwischenraum, @Rohan138, @Zhou248, @brianosaurus, @anmolgupt, @wyettzeng, @hao-aaron, @frank-suwen, @danisereb, @fuzzifikation, @KernelClint, @thunguo, @studioego, @shipiyouniao, @cjackal, @LioEinaudi, @guan404ming, @libinta, @mhoqueanik, @minjang, @Edge-Explorer, @shepark, @JiataiWang, @jiahaoliang, @therealnaveenkamal, @ColinZ22, @tolleybot, @almogtavor, @simon-veitner-redhat, @hyeongyun0916, @nicholaskh-ai, @mawong-amd, @adisivaprasad, @ray24777, @robertgshaw2-redhat, @new-TonyWang, @oliverholworthy, @fanxingran, @lucianommartins, @hallerite, @avininjamay8, @VBS2004, @omerpaz95, @Sunt-ing, @Hert4, @wangshangsam, @cogniera, @jeffreywang88, @Xarbirus, @fynnsu, @jiahanc, @zixi-qi, @prakharPant, @haic0, @maobaolong, @xinyu-intel, @CHIPMUNK-T0T, @hangy-amd, @afriedri, @canlahlah, @Alnusjaponica, @ukannika, @pranavthakur0-0, @zzaebok, @Xuan-1998, @JulianZJN, @roachsinai, @ivanium, @CalvinXKY, @yu-xin-c, @rchalamala, @QwertyJack, @Kaif10, @yudigege86, @yiz-liu, @dkrisman, @simondanielsson, @machero, @CherryLemon, @ima-helikoptaaa, @RookieCoder-Camera, @LironKesem, @peakcrosser7, @eligotts, @foraxe, @zufangzhu, @tianmu-li, @jl9876, @lucifer1004, @liranschour, @maithilijoshi20, @li-ukumar, @andyluo7, @Zhenzhong1, @faaany, @joerowell, @juhi10071998, @ganeshr10, @Priyjain-amd, @SubSir, @luyixiao95, @djw8605, @codex
Breaking changes: bitsandbytes support migrated to an out-of-tree plugin (#43529); Transformers bumped to 5.15.0 (#51668); the deprecated calculate_kv…
This release features 584 commits from 270 contributors (76 new)!
thinking_token_budget support (#46727).module_path (#51007), partial secondary-tier load results (#50321), tiering metrics (#48798), and a canonical CPU layout for parallelism-agnostic offload (#48414).max_num_batched_tokens raised from 8192 to 16384 (#51726), prefix caching enabled by default for Mamba models (#50991), and the Blackwell CUDA graph capture default raised to 1024 (#49390).calculate_kv_scales runtime KV scale calculation was removed (#49389); override_attention_dtype was removed (#48684).| Platform | Install |
|---|---|
| PyPI (CUDA 13.0) | pip install vllm |
| PyPI (CUDA 13.0, uv) | uv pip install vllm --torch-backend=auto |
| ROCm | pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.28.0/rocm722 |
| Platform | Docker Image |
|---|---|
| CUDA 13.0 (Default) | docker pull vllm/vllm-openai:v0.28.0 (v0.28.0-cu130 also works) |
| CUDA 12.9 | docker pull vllm/vllm-openai:v0.28.0-cu129 |
| CUDA 13.0 + Ubuntu 24.04 | docker pull vllm/vllm-openai:v0.28.0-ubuntu2404 |
| CUDA 12.9 + Ubuntu 24.04 | docker pull vllm/vllm-openai:v0.28.0-cu129-ubuntu2404 |
| ROCm | docker pull vllm/vllm-openai-rocm:v0.28.0 |
| CPU | docker pull vllm/vllm-openai-cpu:v0.28.0 |
| XPU | docker pull vllm/vllm-openai-xpu:v0.28.0 |
Pre-built release artifacts are available in the Assets section at the bottom of this page, including:
CuMemAllocator.discard() for tag-selective GPU memory release (#52514), level-2 sleep/wake/reload fixed with LoRA enabled (#39935), and rewritten weight-transfer docs with standardized examples (#51729).get_open_port() livelock on DP-reserved ports was fixed (#50965), and NVML is no longer re-initialized on every device-capability check (#50393).module_path (#51007), partial secondary-tier load results (#50321), tiering offloading metrics (#48798), data-parallel topology exposed to offloading backends (#51879), a canonical CPU layout for parallelism-agnostic offload (#48414), and quadratic ARC batch eviction avoided (#50992).count_reasoning_tokens in the streaming parser engine (#45802), content_parts on /inference/v1/generate (#51478), model optional on all /derender request classes (#51463), output token IDs logged at DEBUG level (#52098), and vLLM Recipes connected to native config-based deployment and benchmarking (#51308, #51878)./v1/messages (#52246), disable_parallel_tool_use preserved (#52021), and stop sequences bounded (#51997).structured_outputs.regex (#51796), negative token IDs rejected as out-of-vocabulary (#51795), and VLLMValidationError raised from validators (#52394).cache_salt now required to be non-empty (#50816)._load_ov2_processor guarded with resolve_trust_remote_code (#52952).--api-key does not gate all endpoints (#51999).calculate_kv_scales runtime KV scale calculation was removed (#49389).override_attention_dtype was removed (#48684).reasoning_content output removal is documented as a breaking client change (#50624).kv_offload_tiering_block_{queries,hits} to ..._chunk_... (#52812).@yewentao256, @AndreasKaratzas, @njhill, @mgoin, @aoshen02, @khluu, @hmellor, @stefankoncarevic, @taneem-ibrahim, @LucasWilkinson, @chaunceyjiang, @jikunshang, @zufangzhu, @bigPYJ1151, @NickLucche, @askliar, @fxmarty-amd, @Rohan138, @gty111, @zhenwei-intel, @Isotr0py, @jeejeelee, @ZJY0516, @wangxiyuan, @BugenZhao, @yma11, @zyongye, @DarkLight1337, @jperezdealgaba, @zhou9402, @connorcarpenter15, @kliuae, @lucifer1004, @Alex-ai-future, @aarushjain29, @TheEpicDolphin, @chaojun-zhang, @pmanczak, @WoosukKwon, @BabyDrangoner, @noooop, @almogtavor, @hongxiayang, @fuscof-ibm, @gau-nernst, @zxd1997066, @tlrmchlsmth, @music-dino, @zexplorerhj, @S1ro1, @jdebache, @sfeng33, @mayuyuace, @ganeshr10, @benchislett, @tzulingk, @gcanlin, @ivanium, @divakar-amd, @R3hankhan123, @sagearc, @vhagor, @ronensc, @micah-wil, @qyYue1389, @vanshbhatia-amd, @hao-aaron, @chengy-sysu, @elvircrn, @taking-lying-flat, @omerpaz95, @Etelis, @vllmellm, @frank-suwen, @KernelClint, @Fangzhou-Ai, @louie-tsai, @simondanielsson, @maxyanghu, @dmai-afk, @KurodaKanbei, @ziqifan617, @bastefaniak, @ECMGit, @haregali, @cmiyai, @fede-kamel, @drakosha, @vineethsaivs, @zcxGGmu, @TQCB, @skysnow2001, @shenoyvvarun, @karen-sy, @fattchris, @RyanJHamby, @shikamd123, @namgyu-youn, @zzt93, @abmfy, @reidliu41, @Rapisurazurite, @tandixit95, @mganczarenko, @yimdev, @anujbolewar, @LiuYinfeng01, @lk-chen, @NVShreyas, @huangzhilin-hzl, @varoudis, @Yejing-Lai, @mkhazraee, @jzakrzew, @TrainToGPB, @waynehacking8, @zixi-qi, @Sundaresan-G, @mindungil, @bitborne, @Wauplin, @jacobzhang22, @zhewenl, @bnellnm, @pisceskkk, @Lin-z-w, @gabriel-peracio, @SilenNaihin, @baodii, @YunzhuLu, @xwu-intel, @BWAAEEEK, @thisjiang, @maobaolong, @anhtra3889, @JaredforReal, @lvhan028, @xiaolong-intel, @andyxning, @cleonard530, @gnovack, @MatthewBonanni, @wangxian001, @lengrongfu, @Tejas-Raj01, @simon-mo, @vitamin-chaos, @arpera, @jairitAge, @jimmy-adams, @ILikeIneine, @woosebastian, @haic0, @edwinlim0919, @fcui-amd, @jhu960213, @jinzhen-lin, @coltonottley, @walterbm, @meenchen, @matteso1, @djramic, @gchinora, @davidjpyu, @tianmu-li, @xiaopusun, @majunze2001, @Vegetog, @puririshi98, @janeyx99, @RobbieJ, @oonyshch, @thegoldenflow, @Srinivasoo7, @fatday, @acheamponge, @efschu, @rajfirke, @fanxingran, @xudonlyu, @lcskrishna, @xijiaat, @GirasoleY, @d4l3k, @samuelkim7, @tarukumar, @acmore, @theminghuang, @khushali9, @wzhao18, @Priyjain-amd, @yiz-liu, @lkm2835, @dmholtz, @Dao007forever, @liushujia122, @LopezCastroRoberto, @UgaTheDev, @tuukkjs, @aditi-amd, @guan404ming, @yiliu30, @zou3519, @Luosuu, @JoursBleu, @varun-sundar-rabindranath, @mpashkovskii, @yu-xin-c, @WillZZZy, @vrdn-23, @xyang16, @ccrhx4, @tanpinsiang, @russellb, @fxfxfxfxfxfxfxfx, @afriedri, @yifjiang, @Akashcodes732, @HF-001, @ovidiusm, @arthurgao2003, @TomerBN-Nvidia, @hotTea123, @vx120, @bohnstingl, @qwerqwerqwe8688-jpg, @jasonozuzu-cohere, @vineetatiwari27, @ruirui6946, @linitra24, @syedalijaseem, @nickus, @yzong-rh, @s3woz, @jhaotingc, @lukealonso, @Jie-Fang, @kzwrime, @xianbaoqian, @velonica0, @ccaadaro, @yisustc, @fangchenli, @iwannagotobed, @zobinHuang, @rchalamala, @shanjiaz, @jamesETsmith, @stacyroberts, @guanxingithub, @biswapanda, @shanewidanagama, @UranusSeven, @hsusul, @tobymao, @mispa-ms, @jeffreywang88, @SayHelloToWorld, @jyan-R, @oops-oom, @shantipriya-amd, @andakai, @akii96, @shen-shanshan, @Kaif10, @yitingdc, @positive666, @pavelzak, @SubSir, @ywang96
This is a patch release on top of v0.27.0.
This is a patch release on top of v0.27.0.
This release features 561 commits from 242 contributors (64 new)!
This release features 561 commits from 242 contributors (64 new)!
vllm-bench integrated into the vllm CLI (#48930).sm_107 target for NVIDIA Rubin (#49387) with NVLink all-reduce paths on SM107 (#49647), and ROCm gfx1250 architecture enabled (#46516).fx tracer (#49957), fused residual-add + RMSNorm compilation pass (#48757), and fixes for MLA padding + grouped topk routing (#49982), MQA with TP (#49987), and Qwen3-VL M-RoPE (#49292)./wake_up crash on hybrid models (#41602).sample_from_anchor loaded from speculators config (#48639), earliest-completing stop string selected (#49391).CustomOp.forward_native compiled for ReLU^2 (#50244), HF config used for HF tokenizers (#49907), batch-invariant RMSNorm via pinned block size (#48391).reduce_scatter regression fix restoring 5% E2E throughput (#48763), non-grouped bias-less topk routing dispatched to the fused path (#49618), tuned LL BF16 router GEMM (#48774) with warmup skipped for non-MoE models (#49659), Triton tensor-descriptor path for fused MoE via VLLM_TRITON_USE_TD (#42436), cudagraph/DP padding skipped in topk (#48979), coalesced HBM access in the Marlin INT4-FP8 AWQ preprocess kernel (#47268).sm_107 for Rubin (#49387), NVLink all-reduce paths on SM107 (#49647), fixed CUDA arch detection producing kernel-less builds on SM121 (#49904).prepare_value on the KV load path (#48531), full external hits re-derived on stored boundaries (#49481).has_pending_push_work (#49582).--linear-backend honored for ModelOpt W4A16 (#50273).find_matched_target prioritizes fused-name matches (#49483).aiter/emulation backends (#49348, #48949) moved to kernel abstractions, CT WNA16 Marlin/MoE methods merged (#44570), Quark W4A8 (INT4-FP8) MoE CI coverage (#48050).vllm-bench integrated into vllm-rs and the vllm CLI (#48930) with opt-in Rust delegation for vllm bench serve (#50081); zero-copy multimodal tensor slicing (#48781), multimodal tensors in auxiliary frames (#49341), --limit-mm-per-prompt (#49604), ordinary-text tokenizer encoding (#49992).cache_salt in the Anthropic Messages API (#49498), strict tool calling and constrained decoding for GPT-OSS Harmony (#45560), unified engine-based Mistral parser for reasoning and tool calls (#48947), stream_interval exposed as a per-request sampling param (#49754), additional sampling parameters for the translation API (#45839), diarized_json for MOSS-Transcribe-Diarize (#48543), cumulative speech-to-text chunk timestamps (#41131).vllm bench serve (#49611).max_num_partial_prefills and max_long_partial_prefills arguments (#49244).@AndreasKaratzas, @njhill, @hmellor, @mgoin, @BugenZhao, @yewentao256, @khluu, @taneem-ibrahim, @guan404ming, @NickLucche, @stefankoncarevic, @Change72, @zhenwei-intel, @reidliu41, @oonyshch, @Isotr0py, @MatthewBonanni, @andyxning, @WoosukKwon, @tlrmchlsmth, @ZJY0516, @fxmarty-amd, @ivanium, @aoshen02, @xwu-intel, @connorcarpenter15, @lengrongfu, @bnellnm, @mikekg, @kylesayrs, @aarushjain29, @fadara01, @netanel-haber, @Etelis, @LopezCastroRoberto, @jikunshang, @chaunceyjiang, @umut-polat, @divakar-amd, @gau-nernst, @ColinZ22, @jcotant-inferact, @R3hankhan123, @chaojun-zhang, @jongukc, @zxd1997066, @bigPYJ1151, @yzong-rh, @hickeyma, @yushangdi, @majunze2001, @Akashcodes732, @zufangzhu, @sagearc, @FeathBow, @xiaolong-intel, @coltonottley, @okorzh-amd, @sungsooha, @cleonard530, @microslaw, @vllm-agent, @LucasWilkinson, @eicherseiji, @Fangzhou-Ai, @peizhang56, @ZeldaHuang, @noooop, @atalman, @wzhao18, @matteso1, @GirasoleY, @Dao007forever, @chaeminlim-mb, @music-dino, @amd-sourjya, @BadrBasowid, @Johnny-Liou, @Rohan138, @brandonpelfrey, @harjothkhara, @tianmu-li, @fxmarty, @oops-oom, @TheEpicDolphin, @yma11, @itayalroy, @shen-shanshan, @JaredforReal, @wskr00, @janeyx99, @mayuyuace, @omkar-droid, @mganczarenko, @Spycsh, @hclsys, @EdalatiAli, @wangqia0309, @tc-mb, @wkutak, @charlifu, @fynnsu, @xuanyu-mistral, @ormandj, @flutist, @simon-mo, @yuan-alex, @esmeetu, @stefan-kaestle, @bastefaniak, @markyangcc, @ilmarkov, @jjmiao1, @stecasta, @evantakahashi, @ariG23498, @sfeng33, @rasmith, @gnovack, @boe20211, @bbrowning, @S1ro1, @davidjpyu, @nikhilkulkarni1755, @vllmellm, @yuyue0225sc, @euisuh, @zhou9402, @li-jinpeng, @hotTea123, @mgazz, @thomas-fahrner-parasail, @elvircrn, @djramic, @adobrzyn, @qtris123, @harshaljanjani, @mosya415, @gcanlin, @fangyuchu, @anthonsu, @Palaiologos1453, @Achyuthan-S, @athrael-soju, @LiuLi1998, @liranschour, @Alex-ai-future, @dsocek, @galletas1712, @nickus, @edwinlim0919, @walterbm, @thegoldenflow, @Zhenzhong1, @haoyangli0109, @tjtanaa, @ronensc, @neweyes, @liminfei-amd, @garrygale, @jesse996, @TobyB1702, @puririshi98, @andreatassi, @avininjamay8, @frida-andersson, @nvbfalk, @varun-sundar-rabindranath, @mindungil, @ayush1399, @afriedri, @ShuoleiWang, @Liangliang-Ma, @xin3he, @liangel-02, @Ibrahim2595, @qianlihuang, @jiacao-amd, @brian-dellabetta, @labAxiaoming, @johnnyychiu, @siddhant-bharti, @jdebache, @mrn3088, @kevglynn, @krishnateja95, @danielafrimi, @zqzten, @philippesic, @afierka-intel, @PerkzZheng, @omerpaz95, @cinnamonica02, @fallintoplace, @yintong-lu, @chanh, @roikoren755, @zaristei, @bugkeep, @vecheruk-amd, @molly-ting, @lkk12014402, @LiuYinfeng01, @jperezdealgaba, @stacyroberts, @juliendenize, @hao-aaron, @cagrikymk, @zou3519, @RyanClark2k, @samlaf, @jpvillam-amd, @mawong-amd, @vanshbhatia-amd, @rjrock, @majian4work, @woosebastian, @Mi-Jiazhi, @jeejeelee, @latent-9, @oguzhankir, @LG-0927, @ylangtsou, @meiyeh123, @skavulya, @Lafunamor, @Amir-19, @andrewbcohere, @yudigege86, @aeon-x, @wentian-byte, @almogtavor, @hongxiayang, @loulanyue, @jasonlizhengjian, @chun-wan, @wjabbour, @yamt, @amitz-nv, @JianDan0212, @lkm2835, @lk-chen
This release features 411 commits from 212 contributors (61 new)!
This release features 411 commits from 212 contributors (61 new)!
fused_topk_bias (1.5–2x kernel, #47463), and redundant repeat/copy removal (1.8% E2E TPOT, #48137), plus ROCm two-stage compressor for HCA prefill (#47718), sparse decode/prefill optimizations (#48519, #48788, #46275), and DSpark speculative decoding on AMD (#47419) and XPU (#47677).lm_head for generation models via head_dtype (#48390), extended to the LoRA path (#48525) and given a ROCm torch.mm fast path (#48688), improving accuracy for generation heads.vllm-bench port (#48107).lm_head on the LoRA path (#48525), optimized TrtLlmLoRAExperts (#48759).torch.compile (#48901).lm_head for generation models via head_dtype (#48390); lower memory for capturing large CUDA graph sizes (#48483); opt-in persistence and reuse of the memory-profiling result across boots (#47388); improved InstantTensor loading (#46868).kv_cache_dtype for speculative_config (#48787).blocks_per_chunk config for heterogeneous KV groups (#48878), P2P default host/port env vars (#47636).new_block_ids (#44490), DSv3.2 + MTP + sequence-parallel accuracy (#48036).fused_topk_bias 1.5–2x (#47463), redundant repeat/copy removal (1.8% TPOT, #48137)._copy_mamba_state_block to uint64 (#48110), stop upcasting logits to fp32 in the sampler (#48641).head_dtype torch.mm fast path (#48688), DSv4 two-stage compressor kernel (#47718), sparse decode/prefill optimizations (#48519, #48788, #46275), DSv3.2 sparse MLA KV-split heuristic (#46832) and MTP CUDA-graph mode (#45149), MXFP8 GEMM for MiniMax-M3 (#46117), AITER sparse paged attention + spec decode for MiniMax-M3 (#47287, #47984), MiniMax-M2 fused QK-norm + all-reduce via AITER (#44849), HybridW4A16 linear kernel (#40977), Qwen3-30B-A3B QK-Norm+RoPE+KV runtime fusion (#42749).nvfp4_per_token online MoE quantization (#48538), CuTe-DSL FlashInfer MXFP4 quantization (#48417); bounded peak memory when repacking FP4 MoE weights for Marlin (#47851) and for NVFP4 MoE weight loading (#46276).kv_cache_dtype_skip_layers support (#47309).vllm-bench port (#48107), continue_final_message handling with renderer sentinel (#47844).bad_words in /v1/completions (#46793), expose logprob_token_ids on Python OpenAI endpoints (#43463), include_reasoning param for non-Harmony models (#44301), populate num_cache_creation_tokens on Messages responses (#48535)./abort_requests on the RLHF dev API router (#47173); Deepstream video decoding backend (#42424); overlap preprocessing and computation for pooling models in offline inference (#47699).@mgoin, @yewentao256, @NickLucche, @njhill, @LucasWilkinson, @micah-wil, @khluu, @AndreasKaratzas, @WoosukKwon, @aoshen02, @BugenZhao, @vanshbhatia-amd, @jperezdealgaba, @jeejeelee, @hmellor, @tlrmchlsmth, @MatthewBonanni, @Change72, @chaojun-zhang, @gau-nernst, @gnovack, @ZJY0516, @reidliu41, @Yejing-Lai, @taneem-ibrahim, @Srinivasoo7, @Sunt-ing, @AmeenP, @zhenwei-intel, @LopezCastroRoberto, @matteso1, @yzong-rh, @stefankoncarevic, @Isotr0py, @djramic, @charlifu, @peizhang56, @giuseppegrossi, @drakosha, @NickCao, @benchislett, @Rohan138, @hickeyma, @muhammadfawaz1, @rasmith, @zixi-qi, @music-dino, @BWAAEEEK, @xianbaoqian, @wendyliu235, @atalman, @joerowell, @ErenAta16, @AlejandroParedesLT, @gcanlin, @liranschour, @omerpaz95, @Alex-ai-future, @tanpinsiang, @Fangzhou-Ai, @KKothuri, @zxd1997066, @akii96, @nemanjaudovic, @Etelis, @afierka-intel, @DaoyuanLi2816, @HDCharles, @tahsintunan, @xiaohongchen1991, @sagearc, @mikekg, @kliuae, @qli88, @arpera, @yushangdi, @edwinlim0919, @tjtanaa, @ariG23498, @lucifer1004, @netanel-haber, @kl527, @Rukhaiya2004, @ronensc, @guan404ming, @shaunkotek, @liulanze, @pierDipi, @eldarkurtic, @simon-mo, @robinguo23, @CienetStingLin, @RishabhSaini, @Sahil170595, @jasonlizhengjian, @jacklin78911-collab, @walterbm, @amd-ethany, @zqzten, @nicklasfrahm, @hongxiayang, @alexeldeib, @ManaEstras, @sungbin1015, @aoright, @Saddss, @vivek8123, @voipmonitor, @shawntsai, @almayne, @ilmarkov, @cleonard530, @kjiang249, @chaunceyjiang, @bigPYJ1151, @tsvikas, @deng451e, @tvirolai-amd, @zhewenl, @zihaomu, @ap9272, @staugust, @Yancey0623, @GongLei-HW, @albertoperdomo2, @guoriyue, @ViranjanPagar, @Functionhx, @XuZhou26, @MynameFelix, @larryli2-amd, @ashwing, @thisisjimmyfb, @robertgshaw2-redhat, @mayuyuace, @ibondarenko1, @zhejiangxiaomai, @vx120, @hugo-cen, @tanish-malekar, @zzt93, @guybd, @R3hankhan123, @mmangkad, @omera-nv, @yma11, @Gavin-Morris-04, @pavanimajety, @shanjiaz, @wenpengw-nv, @atalhens, @langzhao-netizen, @emricksini-h, @Zhenzhong1, @DanBlanaru, @mgehre-amd, @mwoodson, @wangxiyuan, @adsridhar, @hnt2601, @gangula-karthik, @tomerg-nvidia, @adhi29, @rishitdholakia13, @divakar-amd, @chaeminlim-mb, @joanvelja, @russellb, @janeyx99, @aarushjain29, @wjabbour, @mahadrehmann, @krishy91, @tzielinski-habana, @avalliappan-nvidia, @ruikangliu, @majian4work, @maxyanghu, @brijrajk, @lengrongfu, @Josephasafg, @elvircrn, @xiaguan, @ricky-chaoju, @iyastreb, @ovidiusm, @tuukkjs, @noooop, @samnordmann, @AndyDai-nv, @xiao-llm, @DiegoCao, @yuvalluria, @jhu960213, @woosebastian, @Debasish-87, @esmeetu, @hao-aaron, @zhangj1an, @wendadawen, @juliendenize, @passtoor-agi, @mosya415, @labAxiaoming, @devalshahamd, @wangxingda, @xuebwang-amd, @fuscof-ibm, @alexxu-roblox, @frida-andersson, @lishunyang12, @izhuhaoran
This release features 2 commits from 2 contributors (1 new)!
This release features 2 commits from 2 contributors (1 new)!
v0.25.1 is a patch release containing two targeted bug fixes on top of v0.25.0.
import torchcodec raised a RuntimeError at import time when system FFmpeg was missing, which blocked startup (e.g. vllm serve Qwen/Qwen3-VL-2B-Instruct) even when TorchCodec was not in use. The error is now deferred to runtime so it only surfaces if TorchCodec is actually needed.!!!!! tokens. A dtype-match guard now routes incompatible mixed-dtype graphs to the safe path, while same-dtype models retain the full allreduce + RMSNorm + quant fusion.@Isotr0py, @hugo-cen
FP8: weights padding for per-block online quantization (#44763); deprecated the old FP8 online MoE quantization class (#44514).
This release features 558 commits from 232 contributors (64 new)!
tok_sparse_select from MSA replacing Triton kernels (#47502).mm_token_type_ids fix (#46552), tied-embedding lm_head.bias fix (#46835).architectures crash fix (#46037), DeepSeek-V2 hidden-size and aux-hidden-state fixes (#46986, #46973).FLASH_ATTN_MLA_SPARSE Hopper sparse-MLA backend (#46189), DCP + FP8 KV cache in MLA decode (#44044), XQA decode kernels (#43232).LookupResult enum (#46363).VLLM_GPU_SYNC_CHECK env var (#44800), VRAM semaphore infrastructure (#44465), skip detokenization in online beam search (#46422), several int32-overflow fixes in sampler/attention kernels (#46560, #47383, #47671).fused_indexer_q_rope_quant Triton kernel (1.9–3.3% E2E throughput) (#46862), reduce-scatter MoE all-reduce (3.1–3.2% E2E) (#46635), op fusion for GLM5/DSV3.2 (#46876), token_to_req_indices cache for DSv4 (5–6x kernel speedup) (#47474), better DSv4 MXFP8 kernel (#47229), redundant-op removal (#47198, #46651).fused_qk_norm_rope (#44010) and silu_and_mul_per_block_quant (#43994), Triton MLA logits workspace (#46819), swap-AB optimization for fused MoE (#36559), vectorized fp32 moe_sum supporting any top-k (#46643), blocking CUDA events to avoid busy-polling the driver lock (#47081).ROCM_AITER_FA (#45033); fused shared-expert for GLM-4.5/6/7 (#44313) and MiniMax-M3 (#46474, #46545); AITER MoE optimization for DeepSeek-V4 (#46122); AITER custom all-reduce in CudaCommunicator (#46065); INT3 quantization for quickreduce (#45666).get_memory_info (#47134).get_memory_info (#44825).kv_transfer_params merging (#46777), usage field exposed for disaggregated serving (#42748).metrics field on Chat/Completions responses (#46768), token offsets on render endpoints (#44226), return_loss_mask for training-data generation (#46846), HTTP 422 for unprocessable image URLs (#47165).process_eos() flush (#46437), raw-output recovery on non-terminal parse (#47062, #47379).repetition_detection sampling param (#46684), unified/combined parser interface (#46583), reduced multimodal tensor copies (#47581), plus many parser and validation fixes.vllm chat (#46775), model_class_overrides for development/debugging (#47148).thinking_token_budget re-entry #43757); rejection of invalid config values (#44070, #44002, #46612) and degenerate structured_outputs that crash EngineCore (#45346).split_audio with NaN audio samples (#46463).truncation_side is set (#47007).api_server.py moved to the examples directory (#46783); gptq_marlin removed from supported ROCm quant schemes (#46655).Thank you to all the contributors who made this release possible!
@AndreasKaratzas, @njhill, @BugenZhao, @hmellor, @yewentao256, @WoosukKwon, @Sunt-ing, @micah-wil, @mgoin, @reidliu41, @peizhang56, @mawong-amd, @TheEpicDolphin, @jeejeelee, @taneem-ibrahim, @chaunceyjiang, @chaojun-zhang, @divakar-amd, @fxmarty-amd, @LopezCastroRoberto, @wzhao18, @mayuyuace, @jperezdealgaba, @noooop, @yzong-rh, @jikunshang, @zxd1997066, @bigPYJ1151, @yma11, @hickeyma, @benchislett, @xianbaoqian, @andakai, @NickLucche, @ivanium, @joerowell, @EazyReal, @mganczarenko, @majunze2001, @hongxiayang, @WindChimeRan, @Rohan138, @tjtanaa, @bbrowning, @thisjiang, @Fangzhou-Ai, @blasrodri, @Isotr0py, @zhenwei-intel, @zyongye, @frida-andersson, @muhammadfawaz1, @lcheng321, @spandantiwari, @Palaiologos1453, @soaringk, @Lynn-hh, @fadara01, @djramic, @Liangliang-Ma, @ronensc, @aarushjain29, @HDCharles, @qianlihuang, @AgenticSpark, @charlifu, @cleonard530, @shen-shanshan, @xaguilar-amd, @xiaohongchen1991, @varun-sundar-rabindranath, @gau-nernst, @tahsintunan, @GirasoleY, @hclsys, @Yejing-Lai, @LucasWilkinson, @matteso1, @akii96, @atalman, @lucianommartins, @I3eg1nner, @rahulssv-ibm, @ZichenYuan, @tanpinsiang, @hillelda, @Srinivasoo7, @Etelis, @Rukhaiya2004, @Oxygen56, @Priyjain-amd, @GuyStone, @nholmber, @CienetStingLin, @xinyu-intel, @JartX, @esmeetu, @hhhhhhhhhhhhhhhhho, @harsha20032020, @walterbm, @Acaciasama, @jessiewei7, @ashwin-phadke, @shivampr, @cyq1017, @kjiang249, @orestis-z, @xyang16, @tianmu-li, @mgehre-amd, @aaarkai, @guybd, @wcynb1023, @Josephasafg, @qyYue1389, @russellb, @haoyangli0109, @sfeng33, @mikekg, @EanWang211123, @ovidiusm, @ItsMatti4, @hyeongyun0916, @qli88, @juliendenize, @calvarado2004, @tdoublep, @brandonpelfrey, @davispuh, @weizhoublue, @jasonozuzu-cohere, @wentian-byte, @skajre, @gty111, @omirosh, @decarpentierg, @fjosw, @ilmarkov, @yuwenzho, @JisoLya, @JohnLangford, @aldenlobo, @bnellnm, @jasonlizhengjian, @zufangzhu, @izhuhaoran, @MatthewBonanni, @deng451e, @ashwing, @sriganesh123, @linitra24, @liranschour, @umarkovi-amd, @aman0603, @adobrzyn, @jwzheng96, @eicherseiji, @ArsalanShakil, @tc-mb, @imargulis, @fangyuchu, @puririshi98, @JeanPaulShapo, @VectorPeak, @tarjan1, @qiching, @Achyuthan-S, @ZJY0516, @lucifer1004, @cinnamonica02, @jmamou, @almayne, @hao-aaron, @Jyothirmaikottu, @andylolu2, @AIvashov, @stevenkuang-tencent, @lcskrishna, @Aneureka, @wan-danfeng, @chengzheng345, @pranavthakur0-0, @zRzRzRzRzRzRzR, @DanBlanaru, @adamkbaranowski, @wendyliu235, @eparshut, @yangyang-cs95, @kalyanamdewri, @maxdebayser, @fenghourun, @tpopp, @okorzh-amd, @labAxiaoming, @sychen52, @ekagra-ranjan, @gausah01, @yuyue0225sc, @cpersson-amd, @lslusarczyk, @alex101-ops, @Zhenzhong1, @velonica0, @zhongjing123, @zhou9402, @llsj14, @majian4work, @akinsella, @BadrBasowid, @afierka-intel, @ayush1399, @LiJzd, @jesco-absolut, @Laurent-Zhang, @Kevin-XiongC, @NathanielMcVicar, @askliar, @ACEEE-1222, @jinzhen-lin, @SherryC41, @simondanielsson, @nv-nedelman-1, @yisustc, @kylesayrs, @jialoop-git, @NicolasHug, @guan404ming, @HumphreySun98, @danielafrimi, @gcanlin, @robertgshaw2-redhat
Dependencies: Upgrade Starlette to ≥ 1.0.1 to fix CVE-2026-48710 (#45675).
This release features 571 commits from 256 contributors (77 new)!
next_n > 2 on SM100 (#45322). It is now enabled on SM120 alongside GLM-5.1 (#43477), with XPU (#44144, #44517, #45240) and ROCm (#44899, #45103, #45681) attention/MoE paths added./tokenize + /detokenize (#44222), /pause /resume /is_paused (#44499), /abort_requests (#44382), /get_world_size (#44801), thinking_token_budget (#46137), a Python bridge for Rust tool parsers (#44624), and many new parsers and validation paths.CUDA_VISIBLE_DEVICES internally; a new device_ids argument is provided instead (#45026). On ROCm, a deprecation window for CUDA_VISIBLE_DEVICES has begun (#46636).mm_prefix support (#42175); many parser/serving fixes — forced-JSON skip for required/named tool choice (#45795), parsing with thinking disabled (#45832), streaming reasoning-state init (#45852), reasoning rendering on assistant turns (#45867), offline-parser truncation/token-leak fix (#45553); legacy Gemma4 parsers replaced with an engine-based implementation (#45588).num_heads_q (#45564), LoRA warmup fix (#35536), more accurate FP32 Gumbel sampling (#45996), min_tokens off-by-one fix in the V2 GPU sampler (#46243), plus assorted model/config compatibility fixes (#45868).ParallelLoader for weight loading (#40183), release of cached device memory under pressure on UMA GPUs (#45179), structured outputs for beam search (#35022), device_ids arg / no internal CUDA_VISIBLE_DEVICES (#45026), graceful fallback when numactl --membind is blocked (#45438), config-class registration before tokenizer init (#40299), async scheduling with prompt embeds for multimodal models (#45673).P2pNcclConnector (#44854).fused_moe FP8 for Qwen3-Next-80B on H100 (+25%) (#44830), native DSA indexer decode on SM100 (#45322), cluster-cooperative topK for DeepSeek low-latency (#43008), PDL support for DeepGEMM (#46006), FlashInfer cutedsl NVFP4 GEMM (#42235) and cute-dsl MXFP8 linear kernel (#46393), new Helion kernels for FP8/RMSNorm quant (#36902, #33790, #36895, #34432)._C library migration [12/n] (#45415).CUDA_VISIBLE_DEVICES on ROCm (#46636).VLLM_TRITON_FORCE_FIRST_CONFIG to skip Triton autotuning (#42425), Triton recompile detection (#45631), fused multi-group block-table staged writes (#44944).modelopt_mixed support extended to Ampere/SM80-86 (#45306) and Turing/SM75 (#45375).flashinfer_cutlass allowed as a clamped NVFP4 MoE backend (#46492), NVFP4/OCP MX MoE emulation fix (#46254), FP8 MoE re-enabled on NVIDIA Thor (#46339).fp8_e5m2 KV cache allowed for non-fp8 checkpoints (#45040)./v1/embeddings support for messages + chat_template_kwargs (#45173), multimodal token counts in usage.prompt_tokens_details (#45458), omit empty tool_calls from chat responses (#44105), Responses API streaming function_call id fix (#44608), Harmony refactor of streaming/non-streaming paths (#45171, #45104)./v1/messages (#40912), mid-conversation system-message handling (#46025), inline system-message position preserved for prefix caching (#44602), tool_use argument-dropping fix (#45287)./tokenize + /detokenize (#44222), /pause /resume /is_paused (#44499), /abort_requests (#44382), /get_world_size (#44801), thinking_token_budget (#46137), parallel_tool_calls=false (#44760), continuous usage stats (#43965), model metadata in /v1/models (#45950), Python bridge for Rust tool parsers (#44624), dedicated runtime for HTTP/ZMQ (#46051), and many validation/correctness fixes.vllm:tool_call_parser_invocations_total (#44448), group-aware KV cache capacity in vllm:cache_config_info (#42206), MLA attention metrics for DeepSeek MFU estimation (#39457)./v2/embed input exclusivity (#45640), non-negative rerank top_n (#46119), matryoshka embedding dimension bounds (#46313).vllm bench serve (#42457), multi-turn benchmark api_key/custom headers (#44516), tokenizer-mismatch auto-correction (#44708).This release ships another coordinated security-hardening batch (much of it from security researcher @jperezdealgaba).
prompt_embeds on M-RoPE models (#45252), regex-compilation timeout guard in structured outputs (#45118), audio upload size limit before full materialization (#45510), audio decode duration limit in the chat-completions path (#45908).temperature/repetition_penalty (#45116), sanitize_message applied to Anthropic and STT error paths (#45119).mistral_common is now optional via deferred import (#45305); CUDA Dockerfiles upgraded from GCC 10 to GCC 12 for C++20 (#44923); spinloop extension skipped on Python < 3.11 (#44783).CUDA_VISIBLE_DEVICES on ROCm (#46636); general deprecations for v0.23/v0.24 (#44992).Thank you to everyone who made this release possible!
@yewentao256, @Sunt-ing, @jperezdealgaba, @AndreasKaratzas, @BugenZhao, @sfeng33, @njhill, @micah-wil, @bbrowning, @mgoin, @jeejeelee, @hmellor, @tlrmchlsmth, @xianbaoqian, @mmangkad, @jikunshang, @Dao007forever, @zhenwei-intel, @noooop, @Isotr0py, @ivanium, @reidliu41, @varun-sundar-rabindranath, @chaunceyjiang, @WoosukKwon, @mawong-amd, @zxd1997066, @chaojun-zhang, @NickLucche, @bigPYJ1151, @ZJY0516, @charlifu, @yzong-rh, @divakar-amd, @khluu, @cleonard530, @wseaton, @xiaohongchen1991, @ywang96, @taneem-ibrahim, @mikekg, @itayalroy, @Alex-ai-future, @sahilsGit, @bnellnm, @littlecircle0730, @majian4work, @ricky-chaoju, @ronensc, @Fangzhou-Ai, @lucianommartins, @Srinivasoo7, @zyongye, @Rohan138, @Etelis, @wentian-byte, @ekagra-ranjan, @LucasWilkinson, @tahsintunan, @waynehacking8, @gau-nernst, @tuukkjs, @stefankoncarevic, @Palaiologos1453, @lucifer1004, @jmamou, @liulanze, @Terrencezzj, @Change72, @LopezCastroRoberto, @he-yufeng, @benchislett, @juliendenize, @s3woz, @panpan0000, @ilmarkov, @zixi-qi, @wcynb1023, @fynnsu, @ZhanqiuHu, @yuwenzho, @tdoublep, @MatthewBonanni, @hickeyma, @majunze2001, @mrn3088, @Yejing-Lai, @vllmellm, @Saddss, @DarkLight1337, @hongxiayang, @m4r1k, @qli88, @jonathanc-n, @felix0080, @djramic, @aoshen02, @fxmarty-amd, @simon-mo, @llsj14, @akii96, @walterbm, @dmaniloff, @zlxi02, @grYe99, @jeffye-dev, @parthash0804, @qyYue1389, @sagearc, @maeehart, @TanNgocDo, @cinnamonica02, @zucchini-nlp, @tykow, @mganczarenko, @yangdian96, @jimmy-evo, @YellowFoxH4XOR, @yzhan1, @shenoyvvarun, @yufufi, @laviier, @xiaohuguo2023, @EanWang211123, @JartX, @shantipriya-amd, @askliar, @hallerite, @appleparan, @effi-ofer, @angelayi, @TheCodeWrangler, @DanBlanaru, @ankrovv, @velonica0, @pjdurden, @cyyever, @wjinxu, @kliukovkin, @x41lakazam, @Jasen2201, @r-barnes, @tc-mb, @nataliepjlin, @KaletoAI, @WineChord, @fangyuchu, @vraiti, @nascheme, @jjppp, @sasindharan, @xiaguan, @snadampal, @chfeng-cs, @thillai-c, @guan404ming, @sridhar-3009, @vincentzed, @j-i-l, @rjrock, @abinggo, @anony-mous-e, @Achyuthan-S, @Harry-Chen, @mfylcek, @amd-asalykov, @noa-neria, @maobaolong, @TheEpicDolphin, @FAUST-BENCHOU, @martin-kukla, @xin3he, @ZiguanWang, @youkaichao, @factnn, @llx-08, @xx-thomas, @gitbisector, @Bortlesboat, @thisisjimmyfb, @JOSH1024, @wendyliu235, @wangxiyuan, @shen-shanshan, @HanHan009527, @amd-lalithnc, @netanel-haber, @fuscof-ibm, @AjAnubolu, @carlyou, @abcd1927, @CienetStingLin, @kouroshHakha, @alexbi29, @jesse996, @sungsooha, @andakai, @cquil11, @nehmathe2, @liangel-02, @hello-args, @j9smith, @nikhilesh-csa, @ruocco, @oguzhankir, @yiliu30, @xaguilar-amd, @amirkl94, @danisereb, @wangjiaxin99, @shanjiaz, @Oseltamivir, @alexeldeib, @wzhao18, @coder3101, @lyd1992, @markmc, @ashishpatel26, @HumphreySun98, @ByteFlowing1337, @nv-nedelman-1, @JaredforReal, @sammshen, @okorzh-amd, @muhammadfawaz1, @vadiklyutiy, @JasonLi314, @SumanthRH, @Sirius29, @tjtanaa, @zhangshuoming990105, @amanchugh89, @umut-polat, @srajabos, @junkang1991, @pst2154, @WindChimeRan, @Zedong-Liu, @gq112, @sunnweiwei, @athrael-soju, @EazyReal, @Liangliang-Ma, @jinzhen-lin, @V-3604, @aarushjain29, @ZewenShen-Cohere, @Bot1822, @BowenBao, @MichaelCao0, @tanpinsiang, @QwertyJack, @nagisa-kunhah, @Meihan-chen, @robertgshaw2-redhat
…failure handling (#43659), scheduled-function deprecations (#43358).
Please note that Minimax M3 is not yet supported in this version. Please follow vLLM recipe for usage guides for M3.
This release features 408 commits from 200 contributors (63 new)!
torch.compile (#43746, #43891), its attention and RoPE paths were refactored (#44569, #44262, #43926), and an XPU attention decode path was added (#42953).generate endpoint (#43779), dynamic LoRA endpoints (#43778), /version (#43854) and /server_info (#43942) endpoints, a server-router extension hook (#43774), request-ID headers (#43883), and many new tool parsers (InternLM2 #43481, hy_v3 #43872, Phi-4-mini #44213, Gemma4 #43850).on_new_request lifecycle hook (#43205).Parser.parse() interface (#44267), with the Responses parser migrated to it (#42977).fetch_audio for transformers≥5.10 (#44559).torch.compile (#43617), EVS for Qwen3-VL (#44205), GLM-5.1 PP loading (#42944), GLM-4.1V processor logits (#43575), GLM-4.6V video loader (#44417), OlmoHybrid init (#43846), HyperCLOVAX remote-code removal (#43860), Bailing-MoE rotary factor (#43770), Step3 PP residual KeyError (#37622), MiniCPM-V-4.6 video (#44509), MiniCPM-O audio unpadding (#38053), MiniCPM-V batched preprocessing (#44609), FunASR-Nano init (#44215), Cohere routing method (#44021), Kimi-K2.5 FlashInfer ViT metadata (#44493).extra_repr() for pooler classes (#44805), LoRA-adapter-name pooling fix (#44410), resettled generative scoring entrypoint (#44153), expanded pooler unit tests (#43818, #44471).max_seq_len for attention metadata (#43991), rejection-sampling acceptance-rate fix (#40651), KVConnector + PP cleanup (#43732), speculator-prefill warmup/capture (#44253).num_heads_q for drafts (#43543), EAGLE/MTP lookahead caching in the SWA prefix-cache mask (#44082).do_not_specialize (#43803), Qwen3.5 mixed prefill+decode split routing (#44700), MiniMax-M2 gate kernel (#38445).KVCacheSpec (#37505), scheduler_block_size threaded into KVCacheManager/Coordinator (#44165), max_concurrent_batches moved to VllmConfig (#44274), config validation rejecting 0/negative knobs (#43794, #44057, #44207), KV-cache scale boilerplate removed from weight loading (#43167).on_new_request) (#43205) and on_schedule_end() hook (#44206), token-offset selective offload (#39983), skip decode-phase blocks in CPU offload (#43797), page-size block alignment (#43689), Triton fast-path for small CPU→GPU swap_blocks_batch (#42212), stale sliding-window block fix (#42959).kv_both role deprecation cycle (#43874), Mooncake fixes (#43742, #44103, #42694), LMCache LMCacheMPConnector (#42865), EC connector shutdown API (#42423) and non-blocking lookup (#41627), KV-transfer tokens excluded from iteration_tokens_total (#43346).Fp8BlockScaledMM new_empty() optimization (#43677), TurboQuant shared dequant buffers (#40941), tuned selective_state_update for H200/RTX PRO (#44251), Inductor fast-path fallback for vLLM/AITER custom ops (#42129), Gemma RMS all-reduce fusion (#42646), NUMA auto-binding on DGX B300 (#43270).permute_cols for ROCm (#44674), blocks-first KV layout for AMD (#43660), N=5 wvSplitK for spec decode (#40687), MoRI connector improvements (#43303, #41751, #40344).block_fp8_moe (#42139), block-scaled W8A8 FP8 path (#39968), WNA16 oracle for GPTQ sym-int4 (#41426), rms_norm/act quant fusions (#43963), GDN-attention MTP (#43565), Triton selective-scan op (#43421), transparent sleep mode (#37149), CPU/tiering offloading on XPU (#36423), DeepSeek-V4 attention decode path (#42953).cpu_awq folded into awq_marlin (#43841), RISC-V RVV WNA16 helpers (#42730), fused GDN gated-delta-rule kernels (#43534), PowerPC SHM communicator (#43754), arm64 CI image (#41303)._has_module trial-import verification (#44035).supports_expert_map (#43108) and the inplace fused-experts mechanism (#43727).system_fingerprint field (#40537), streaming tool/function calling with required (#40700), chat_template_kwargs in Responses (#43761), developer-to-system conversion in the HF renderer (#43590), unstreamed tool-call-args streaming fix (#44348).Parser.parse() (#44267), Responses parser migrated to the unified interface (#42977), unstreamed tool-arg flush moved into the parser (#44017); new/fixed tool parsers — MiniCPM5 XML (#43175), Qwen3 XML JSON-args-first (#43243), DeepSeek DSML incremental streaming (#42879), first-args-chunk serializer fix (#42683), tool_choice="none" honored in streaming (#42752), null-tool-args crash fix (#43862).thinking_token_budget validation (#43402), GPT-OSS instruction rendering (#44330), Harmony stop_token_ids cleanup (#44009), consistent VLLMValidationError in chat/completion validators (#36254), consolidation of dev entrypoints (#44170) and online-serving utils (#44479).generate endpoint (#43779), dynamic LoRA endpoints (#43778), /version (#43854) and /server_info (#43942), server-router extension hook (#43774), --enable-request-id-headers (#43883), recursive tool-parameter conversion (#44299), include_reasoning=false (#44391), --language-model-only skips the multimodal processor (#44500), per-engine batch auto-abort (#44591), UTF-8 char-boundary detokenizer fix (#44620), HF chat-template fixes (#44311), cross-DP aggregation of is_sleeping/reset_prefix_cache (#43429); new tool parsers — InternLM2 (#43481), hy_v3 (#43872), Phi-4-mini JSON (#44213), Gemma4 (#43850).vllm bench serve (#39795), reasoning-model (thinking) benchmarking via --chat-template-kwargs (#44244).thinking_token_budget values (#43402), non-positive ParallelConfig integer knobs (#44057), zero-valued config fields (#43794), and out-of-range max_num_scheduled_tokens (#44207).libcublas-dev (#39855), and CUTLASS DSL cu13 install order (#45204).JAISLMHeadModel (#43784).kv_both role (#43874).Thank you to everyone who made this release possible!
@AndreasKaratzas, @WoosukKwon, @BugenZhao, @yewentao256, @hmellor, @khluu, @njhill, @sfeng33, @bnellnm, @vadiklyutiy, @NickLucche, @JartX, @lucianommartins, @cleonard530, @wzhao18, @yma11, @simondanielsson, @jeejeelee, @zyongye, @chaunceyjiang, @bigPYJ1151, @ronensc, @taneem-ibrahim, @LucasWilkinson, @MatthewBonanni, @mmangkad, @chunyang-wen, @yzong-rh, @JaredforReal, @zixi-qi, @Isotr0py, @noooop, @chaojun-zhang, @Xunzhuo, @ivanium, @zufangzhu, @DaoyuanLi2816, @CienetStingLin, @aoshen02, @akii96, @benchislett, @MengqingCao, @rshavitt, @kliuae, @omerpaz95, @willamhou, @Majid-Taheri, @micah-wil, @ricky-chaoju, @mikekg, @mgoin, @mayuyuace, @Etelis, @ilmarkov, @tlrmchlsmth, @UranusSeven, @bedeks, @izhuhaoran, @ZJY0516, @fadara01, @pschlan-amd, @wangxiyuan, @Oxygen56, @charlifu, @varun-sundar-rabindranath, @shen-shanshan, @TheEpicDolphin, @adobrzyn, @XuZhou26, @tjtanaa, @Terrencezzj, @zhejiangxiaomai, @ILikeIneine, @yubofredwang, @chfeng-cs, @ThibaultCastells, @linzm1007, @javierdejesusda, @meenchen, @zhewenl, @xyang16, @angelayi, @nholmber, @zhangtao2-1, @adityasingh2400, @sts07142, @jatseng-ai, @fallintoplace, @andakai, @he-yufeng, @ignaciosica, @JINO-ROHIT, @tonyliu312, @QwertyJack, @animeshtrivedi, @jzakrzew, @juliendenize, @zexplorerhj, @ruocco, @mgehre-amd, @jasonboukheir, @MaciejBalaNV, @JohnQinAMD, @huanghua1994, @rajkiranjoshi, @rasmith, @harshaljanjani, @ltd0924, @wdhongtw, @yintong-lu, @tianmu-li, @jikunshang, @JMonde, @MHYangAMD, @frida-andersson, @gau-nernst, @Wauplin, @czhu-cohere, @gagandhakrey, @nemanjaudovic, @Liangliang-Ma, @liulanze, @sphinx07, @aadwived, @nightcityblade, @umut-polat, @jeffreywang88, @wcynb1023, @zzt93, @shadeMe, @Dao007forever, @alec-flowers, @Krishnachaitanyakc, @orozery, @BWAAEEEK, @cinnamonica02, @albertoperdomo2, @Rukhaiya2004, @mfylcek, @shreyas269, @Gruner-atero, @TomerBN-Nvidia, @wjinxu, @IdoAtadTD, @xiaozcy, @brian-dellabetta, @zhenwei-intel, @adotdad, @Kartavyasonar, @lesj0610, @ECMGit, @cakeng, @william-rom, @qiching, @NolanHo, @andylolu2, @xwu-intel, @linitra24, @hoobnn, @Dymasik, @wanghenshui, @maobaolong, @oguzhankir, @Jie-Fang, @okorzh-amd, @Kevin-XiongC, @jiahanc, @garrygale, @dsikka, @QiliangCui2023, @wjabbour, @zvik, @tc-mb, @jwzheng96, @divakar-amd, @tushar00jain, @galletas1712, @hanlin12-AMD, @tuukkjs, @viiccwen, @Sunt-ing, @HueCodes, @tianyu-z, @adhithyamulticoreware, @rishitdholakia13, @effi-ofer, @Vikrantpalle, @walterbm, @devin-lai, @Yadan-Wei, @amd-fuweiy, @maeehart, @qyYue1389, @BramVanroy, @SunskyXH, @Holworth, @majian4work, @xaguilar-amd, @Rohan138
This release features 8 commits from 6 contributors (1 new)!
This release features 8 commits from 6 contributors (1 new)!
v0.22.1 is a patch release on top of v0.22.0 with targeted bug fixes plus a couple of additions: new model support for JetBrains' Mellum v2, zentorch-accelerated quantized linear inference on AMD Zen CPUs, and fixes for multi-node Ray data-parallel serving, DeepSeek-V4 initialization, and a few model-loading regressions.
fmin compatibility issue that broke initialization (0decac0d).OlmoHybridForCausalLM failing to initialise after the checkpoint changed rope_parameters from None to {"rope_type": None} (#43846).transformers >= 5.9.0): register the hyperclovax model_type so vLLM uses its vendored config instead of the stale auto_map (#43860).num_api_servers > 1 by excluding the Ray DP backend from the deferred (kernel-assigned) port allocation introduced in #42585 (#43864).flashinfer-jit-cache via --extra-index-url while it is quarantined on PyPI, fixing image builds (#44366).ImportError: libcudart.so.12 when importing nixl_ep on CUDA 13 images (#44266).@khluu, @vadiklyutiy, @aadwived, @shadeMe, @alec-flowers, @hmellor
One column per quarter.