NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
PyPI, #2209 most downloaded on PyPI
FlashInfer: Kernel Library for LLM Serving
Latest stable
0.7.3
Last release 10 days ago
10 Nov 2026
Ships on a steady schedule
a new release about every 2 weeks
Most releases are documented
notes for 45 of the last 60 stable releases
Nothing withdrawn
no release was ever pulled
2 years old
114 releases · first in 2025
Drag across the ruler to compare, or tap a release to open it.
chore: remove deprecated APIs eligible at the 0.7.0 boundary by @aleozlx in #5218
fast-math INF scale when activation contains all-zero row by @xuantengh in #5031lse_base and lse_layout on the prefill wrappers (defaults unchanged; "ln" / "HN" for LSE-merge consumers) by @YangXu1990uiuc in #5257Note truncated.
Your coding agent can read these notes before it upgrades. Set up the MCP server →
Full Changelog : v0.7.0...v0.7.0.post1
Full Changelog: v0.7.0...v0.7.0.post1
Full Changelog: https://github.com/flashinfer-ai/flashinfer/compare/v0.7.0...v0.7.0.post1
These highlights are also published at flashinfer.ai/releases .
These highlights are also published at flashinfer.ai/releases.
FlashInfer 0.7.0 makes the unified mixture-of-experts (MoE) API official, brings TRT-LLM Gen MoE kernels into readable Python source with PrimTS, and introduces Autotuner v2 and a formal experimental-API policy. It also expands sparse and linear attention, expert-parallel serving, and diffusion workloads across Blackwell GPUs.
Read the v0.7 overview and the accompanying deep dives on MegaMoE, Autotuner v2, and the experimental path.
Unified MoE API is official
MoELayer is now an official FlashInfer API, with the lower-level kernel entry points supported alongside it. QuantConfig gives weights, activations, and output explicit QuantFormat fields: QuantConfig(weight=QuantFormat.MXFP4) selects MXFP4 weights with BF16 activations, while adding activation=QuantFormat.MXFP8 selects W4A8. Backend coverage expands with quantization-specific CUTLASS runners, cuTile BF16 and NVFP4 MoE, and CuTe-DSL BF16 MoE on Hopper.
TRT-LLM Gen MoE kernels as Python source with PrimTS
The experimental PrimTS MoE backend exposes kernels built with the CUTLASS DSL Primitives and Task Scheduling APIs, making the expert GEMMs available as readable Python source. It supports BF16, per-tensor and block-scaled FP8, and NVFP4/MXFP4 combinations, while reusing TRT-LLM Gen routing and finalization. Accuracy qualification targets B200; B300 qualification is still pending, and supported configurations have backend-specific restrictions. PrimTS attention also gains paged block-sparse attention, variable-window attention, and a unified plan()/run() contract for reusable wrappers.
Autotuner v2 tunes the way you serve
autotune_v2() lets applications measure candidates in eager or CUDA-graph execution and persist the results in an environment-specific cache managed by FlashInfer. Atomic cache entries support concurrent rank writes, and autotune_v2_reload() lets homogeneous ranks converge on shared results. In the reported vLLM validation, Qwen3-8B-FP8 at TP2 on B200 reduced its tuning window from 128 seconds on a cold start to 1 second on restart with the same cache; total startup was 410.7 seconds and 165.3 seconds, respectively. Applications opt in through the new API; autotune() remains available.
Experimental APIs and backends have an explicit opt-in path
Experimental APIs are marked with @flashinfer_experimental_api, and experimental backends have a dedicated flashinfer.experimental namespace. Calling an experimental API or explicitly selecting a marked backend emits a warning; automatic selection includes marked experimental backends only when FLASHINFER_ALLOW_EXPERIMENTAL_AUTO_BACKENDS=1 is set. The policy defines admission and graduation criteria and keeps experimental implementations JIT-only, outside prebuilt packages.
@flashinfer_experimental_api, and flashinfer.experimental namespaceExpert-parallel MoE expands across Blackwell
moe_ep adds an unquantized BF16 MegaMoE backend and a W4A8 split backend on B200, with MXFP8-packed dispatch for the latter. RTX PRO 6000 and DGX Spark gain an MXFP8 MegaMoE backend with functional correctness validated and performance tuning ongoing. The NCCL-EP split path supports CUDA-graph capture through a reusable handle with per-step update(), validated through vLLM on four B200 GPUs.
DeepSeek-V4 and MiniMax-M3 sparse attention
Blackwell gains paged FP8/MXFP4 indexer logits and more top-K choices, including a CUB backend with variable-length support. DeepSeek-V4 Flash sparse MLA supports an NVFP4 KV cache on SM120/SM121 through kv_cache_format="nvfp4". MiniMax-M3 sparse attention gains source-distributed CAKE-generated kernels on SM100/SM103; the B200 kernel benchmark reports a 2.45× geometric-mean speedup over the MiniMax baseline across 11 comparable prefill, decode, speculative, and boundary cases.
radix_filter)DeviceBatchedTopK top-k backend with variable-length supportLinear attention for Kimi K3, Qwen 3.6, and Nemotron-H
Kimi K3 gains a speculative-verification entry point compatible with vLLM's recurrent verifier, plus CuTe-DSL recurrent prefill on B200/B300 and SM120. The experimental RecurrentKDAPrefillWrapper supports planning and graph-safe prefix checkpoints. Qwen 3.6 gains experimental fused GDN decode steps on SM120 that combine projection, convolution, gating, and recurrent state updates. Nemotron-H gains source-built Mamba SSD-combined and selective-state-update backends on B200/GB300.
Diffusion and video attention on Blackwell
Video Sparse Attention gains generated SM100/SM103 kernels, and the block-64 path adds a native CuTe-DSL implementation with Sage FP8 support. SM120 gains a generated Sage block-sparse backend and an optimized NVFP4 attention path for Cosmos workloads. MiniMax-H3 gains a prepared MXFP8 pre-attention pipeline on B200/B300, while PrimTS block-sparse attention adds proxy compensation for Sol-Attn workloads.
Communication for PCIe and NVLink deployments
PcieIpcAllReduceWorkspace adds an intra-node CUDA-IPC all-reduce for two, four, or eight ranks on PCIe machines without NVLink. Blackwell fused all-gather matmul gains a backend="cake" option and a prepared callable for packed-QKV workloads, with tensor parallelism up to eight GPUs.
Notice: packaging and dependencies
flashinfer-jit-cache is now a small shim that depends on architecture-specific provider wheels. The usual installation command installs the provider set for the selected CUDA and CPU platform; to reduce image size, install only the provider needed for your GPU. If you mirror or vendor wheels, include the provider wheels as well as the shim, and keep their CUDA-specific versions aligned.
| Dependency | 0.6.18.post1 | 0.7.0 |
|---|---|---|
nvidia-cudnn-frontend |
>=1.25.0 |
>=1.29.0 |
apache-tvm-ffi |
>=0.1.6,!=0.1.8,!=0.1.8.post0,<0.2 |
>=0.1.11,<0.2 |
| NCCL-EP Python packages | nccl4py>=0.3.1 |
nccl4py>=0.4.1 and nccl-extensions>=0.1.0 |
nvidia-cutlass-dsl through [cu12] / [cu13] |
>=4.6.2a0 |
>=4.7.0a0 |
The base nvidia-cutlass-dsl requirement remains >=4.6.2a0; the CUDA extras have the higher floor. Environments pinned to CUTLASS DSL 4.6.2 need compatible dependency pins before installing those extras.
Notice: small-batch FP8 groupwise GEMM
⚠️ The CUTLASS
gemm_fp8_nt_groupwisepath on SM100/SM103 can intermittently produce incorrect output forM <= 32withscale_granularity_mnk=(1, 128, 128). This pre-existing issue remains in 0.7.0; see #4396 for the investigation. Validate affected workloads before deployment. A different backend requires its own supported shapes and scale layout; there is no documented switch to disable only this CUTLASS fast path.
Notice: API removals and behavior changes
Update callers of the removed APIs before upgrading:
| Removed | Replacement |
|---|---|
comm.trtllm_custom_all_reduce |
comm.trtllm_allreduce_fusion |
comm.trtllm_create_ipc_workspace_for_all_reduce |
comm.trtllm_create_ipc_workspace_for_all_reduce_fusion |
BatchDecodeMlaWithPagedKVCacheWrapper |
mla.BatchMLAPagedAttentionWrapper |
No-op end_forward() methods on decode, prefill, sparse, cascade, and POD wrappers |
Remove the call |
fused_moe.QuantVariant, QuantConfig(variant=...), QuantConfig.from_variant |
QuantConfig(weight=..., activation=...) using QuantFormat |
mamba.checkpointing_ssu replaces the per-step old_x, old_B, old_dt, old_cumAdt, and cache_buf_idx arguments with ring-buffer caches (x_cache, B_cache, dt_cache, ring_start). Update cache allocation and argument binding; the caller-provided out tensor remains required.
For the TRT-LLM backend of BatchPrefillWithRaggedKVCacheWrapper, skip_all_rows_active_check defaults to True. Calls on the default fast path must have positive query and KV lengths for every row; pass False to request device-derived checking when CPU length mirrors are absent.
gated_delta_rule_mtp still resolves an omitted disable_state_update to True in 0.7.0. Pass disable_state_update=True or False explicitly to select the intended behavior and suppress the stale warning announcing a change in this release.
DeviceBatchedTopK top-k backend with variable-length support by @NaderAlAwar in #4442Note truncated.
Full Changelog: https://github.com/flashinfer-ai/flashinfer/compare/v0.7.0rc3...v0.7.0rc4
Full Changelog: https://github.com/flashinfer-ai/flashinfer/compare/v0.7.0rc3...v0.7.0rc4
fix(release): drop the stale diffusion_ops.minimax_h3 trace-registry entry by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/5216
Full Changelog: https://github.com/flashinfer-ai/flashinfer/compare/v0.7.0rc2...v0.7.0rc3
Full Changelog : v0.6.18...v0.6.18.post1
Full Changelog: v0.6.18...v0.6.18.post1
Full Changelog: https://github.com/flashinfer-ai/flashinfer/compare/v0.6.18...v0.6.18.post1
One column per month.