These highlights are also published at flashinfer.ai/releases.
v0.7.3 Highlights
FlashInfer v0.7.3 brings MTP and NVFP4 MLA decode to Rubin, and speeds up expert-parallel MoE decode steps on Blackwell while Hopper gains MXFP4 and NVFP4 experts. Hybrid Gated DeltaNet models gain paged recurrent state for prefix caching, DeepSeek Sparse Attention models get token-sparse decode, MoELayer adds a low-latency small-batch path from Ampere to Blackwell, and a first set of training kernels covers GLM-5.3-Flash and Qwen hybrid models.
DeepSeek-class MTP decode, Qwen3.5 deterministic MoE and Mamba2 prefill run on Rubin
BatchMLAPagedAttentionWrapper adds a cute-dsl-rubin-mtp backend for FP8 absorbed-MLA multi-token-prediction decode on SM107, and an experimental CAKE backend decodes dense MLA from an NVFP4 paged latent cache in the Kimi K3 and DeepSeek-V3 geometry. The deterministic unfused finalize of the CuTe DSL fused MoE works on Rubin for Qwen3.5 NVFP4 on Vera Rubin NVL72, and knobs="auto" tunes Rubin MegaMoE through autotune_v2(). Rubin also gets an SM107-native blockwise FP8 GEMM behind gemm_fp8_nt_groupwise(backend="auto"), Mamba2 prefill through SSDCombined(backend="cake"), and CAKE programs for DeepSeek routing, BGMV LoRA MoE, GDN and packed KDA decode.
- #6177 feat(mla): add Rubin MTP backend to the planned wrapper
- #6931 feat(cake_rubin_mla_nvfp4_decode): NVFP4 MLA paged decode for Rubin (sm_107a) from cake
- #6950 feat(moe): unfused (deterministic) finalize path for the Rubin CuTe DSL fused MoE
- #6535 feat(moe_ep): integrate Rubin automatic tuning with autotune v2
- #6508 feat(gemm): SM107 CuTe-DSL blockwise FP8 GEMM backend for gemm_fp8_nt_groupwise
- #6746 feat(cake_ssd_combined): Rubin sm_107a programs for the Mamba2 SSD combined prefill
- #6238 feat(cake_kernel_rubin): Cake backends on Rubin (sm_107a / CC 10.7) — routing, BGMV MoE, concat-MLA-K, GDN decode/prefill, packed KDA T=1
Expert-parallel MoE decode steps run faster on Blackwell, and Hopper adds MXFP4 and NVFP4 experts
NVLinkOneSidedAlltoAll adds a Lamport transport for decode-sized steps on B200, B300, GB200 and GB300 (NVLinkOneSidedConfig(lamport=...)), which lowers dispatch and combine time for DeepSeek-V3, Kimi K3 and Qwen3 MoE decode steps, measured end to end on Kimi K3 in vLLM; the backend now shares its kernels with TensorRT-LLM and scales to 256 ranks. Blackwell MegaMoE adds an opt-in GenPhase decode pipeline, and knobs="auto" tunes the Blackwell backends through autotune_v2(). On Hopper, MoEEpLayer runs FP8-activation experts with MXFP4 or NVFP4 weights, and the new flashinfer.megamoe package adds fused NVFP4 and MXFP4 expert-parallel MoE on H100 and H200 for DeepSeek-V4-Flash-class models such as DeepSeek-V4-Flash-NVFP4.
- #7498 feat(moe_ep): Lamport transport for small steps in NVLinkOneSidedAlltoAll
- #6047 feat(moe_ep)!: sync NVLinkOneSidedAlltoAll with TensorRT-LLM's kernels; remove the deprecated MoeAlltoAll/MnnvlMoe
- #6565 add-new-decoding-pipeline-for-blackwell-megamoe
- #7523 feat(moe_ep): integrate Blackwell standard and GenPhase autotuning with v2
- #4843 feat(moe_ep): add fused Hopper MXFP4 MegaMoE with autotuning
- #6999 feat(moe_ep): add SM90 NVFP4 x FP8 CuTe DSL MegaMoE backend
- #6343 [SM90] Add W4A8 MegaMoE support
MoELayer adds a low-latency small-batch path from Ampere to Blackwell and MXFP8 experts for Nemotron 3.5 Super
FrostLowLatencyConfig selects an opt-in cuDNN Frost backend of the unified MoELayer for routed MoE at one to eight tokens, covering BF16, FP16, FP8 (per-tensor and DeepSeek block), MXFP8 and NVFP4 experts and weight-only NVFP4, MXFP4, MXINT4 and E4M3 formats on SM80 (BF16 and FP16 only), SM90, SM100, SM103, SM107 and SM120. cuDNN grouped-GEMM backends (CudnnGroupedGemm*Config) join the same API as a further opt-in, and the cuDNN Frost grouped-GEMM backend covers BF16, MXFP8, NVFP4 and MXFP8-activation/MXFP4-weight experts on SM120, where auto-selection admits it. The CuTe DSL fused MoE adds MXFP8 activations with MXFP8 weights for Nemotron 3.5 Super's non-gated ReLU² experts on B200 and B300, raising serving throughput at high concurrency over BF16 experts.
- #6626 feat(moe): add unified Frost low-latency MoE backend
- #7184 feat(moe): extend Frost low-latency to four-bit weight-only formats
- #7162 feat(moe): add per-tensor FP8 and Hopper FMA to Frost
- #7153 feat(moe): extend Frost low-latency FMA to FP16 and SM120
- #5453 feat(moe): add cuDNN grouped-GEMM backends to the unified MoE API
- #6504 Add SM120 Frost Moe Grouped Gemm Support for Swap-AB, MXFP8, NVFP4 and MXFP8×MXFP4
- #6793 feat(moe): MXFP8 x MXFP8 mode for the CuTe-DSL fused MoE (for Nemotron models)
Hybrid Gated DeltaNet models serve with prefix caching, and Gated DeltaProduct gets decode and prefill kernels
flashinfer.gdn_paged keeps Gated DeltaNet recurrent state in a paged pool with a CSR page table, so a serving engine's prefix cache can share and resume GDN state like paged KV: chunk_gated_delta_rule_paged writes a state at every page boundary and never touches prefix pages, gated_delta_rule_decode_paged updates pages per token, and gdn_prefix_materialize reconstructs a state at a fixed boundary from the decode checkpoint ring. GDN prefill also runs on Ampere. Gated DeltaProduct, the multi-step DeltaNet recurrence, gets cake_gated_delta_product_mtp for decode and MTP decode on H100, B200 and B300 and a backend="cake" chunked varlen prefill on H100. fused_kda_decode covers GLM-5.3-Flash's 64 Kimi Delta Attention heads at TP1 to TP4 and a dimension-first conv-state layout for disaggregated serving.
- #7785 feat(gdn): paged recurrent state for GDN prefill and decode
- #5073 feat(gdn): prefix-cache state materialization kernel
- #5109 feat(gdn): add SM8x prefill for the gated delta rule, fused and chunk-parallel
- #6883 feat(cake_gdp): add Gated DeltaProduct decode backend (sm_100a, sm_103a)
- #6942 feat(cake_gdp): Gated DeltaProduct decode on SM90 (sm_90a programs)
- #6983 feat(cake_gdp): Gated DeltaProduct chunked varlen prefill on SM90 (sm_90a programs)
- #7132 feat(kda): fused KDA decode for 16/64 heads and the DS conv-state layout
Token-sparse decode serves DeepSeek Sparse Attention models, and sparse MLA reads NVFP4 and FP8 caches
trtllm_batch_decode_with_kv_cache(sparse_top_k=K) runs token-sparse MQA decode on the trtllm-gen kernels, where each query token, MTP drafts included, attends only the keys its indexer selected; token_sparse_kv_indices builds the per-token page tables and group_union_masks lowers the same selection onto dense kernels. The experimental Prims-TS sparse-MLA decode wrapper serves DeepSeek Sparse Attention (DSA) decode for GLM-5.3 and DeepSeek-V3.2 from an E4M3 latent cache on B200 and B300, and adds NVFP4 and per-tensor FP8 cache recipes. Query sparse attention (QSA) gets cuDNN-backed qsa_forward and qsa_indexed_forward on B200, B300 and Rubin and an experimental qsa_sparse_paged_attention over NVFP4 paged caches on A100 and H100. MiniMax Sparse Attention ops (msa_*) for MiniMax M3 and MiniMax M3.1 Flash now run on H100.
- #6420 [SDSA 1/6][Attention] Add token-sparse MQA decode to trtllm-gen
- #6421 [SDSA 2/6][Attention] Add key unions and masks for token-sparse attention on dense kernels
- #6536 feat(prims_ts): DSA (GLM-5.3 / DeepSeek-V3.2) sparse MLA decode with an E4M3 KV cache through the sparse MLA wrapper
- #7410 feat(prims-ts): support NVFP4 and per-tensor FP8 sparse MLA caches
- #7532 feat: add cuDNN 1.32 QSA forward APIs
- #6804 feat(qsa_nvfp4): add NVFP4 QSA sparse paged attention
- #7147 feat(qsa_sparse_paged_attention): backend="auto" default runs the Cake kernels on SM80 through SM90
- #6140 SM90 MSA kernels for MiniMax M3
Sliding-window context attention reaches Blackwell and Rubin, and GLM-4.7 runs FP8 prefill on RTX PRO 6000
The TRTLLM-Gen context kernels accept a fixed sliding window and per-token VariableWindow bounds on B200, B300 and Rubin. backend="auto" routes more Blackwell shapes to cuDNN: single-token head-dim-256 paged decode in its measured CTA band, and causal BF16 paged prefill on B200 for both prefix-extension and full-sequence requests (cuDNN frontend 1.32 and cuDNN 9.27 or newer). FA2 takes a per-token, per-head FP8 KV scale for the dynamic fp8_per_token_head KV cache. On RTX PRO 6000, GLM-4.7 and GLM-4.7-Flash get FP8 prefill attention, and SM120 sparse MLA reads DeepSeek-V4.1 inline-scale KV records natively (kv_cache_format="fp8_dsv41_inline").
- #4905 feat: integrate TRTLLM-Gen sliding-window FMHA
- #6152 feat(cudnn): backend="auto" takes cuDNN for single-token d256 paged decode in the measured CTA band; CUDA-graph wrappers declare replay to cudnn-frontend 1.31+
- #6231 [Attention] Reuse bounded cuDNN prefill plans and gate Blackwell paged auto
- #7377 [Attention] Admit proven full paged prefill to Blackwell cuDNN auto
- #5272 feat(attention): FA2 per-(token, head) FP8 KV scale
- #6466 feat(attention): add SM12x FP8 prefill attention support for GLM-4.7 and GLM-4.7-Flash
- #6616 feat(mla): read DeepSeek-V4.1 inline-scale KV records natively on SM120
Training kernels with backward passes arrive for GLM-5.3-Flash, Qwen hybrid models and MoE layers
FlashInfer adds its first CAKE training kernels for Blackwell and Rubin, forward and backward, behind autograd facades: packed chunked Kimi Delta Attention training in the GLM-5.3-Flash geometry with a context-parallel variant (flashinfer.kda_train), context-parallel Gated DeltaNet training for Qwen hybrid GQA/GDN models (flashinfer.gdn_train), a bitwise-deterministic GDN training backend, and BF16 or MXFP8 routed-expert training with recompute (flashinfer.mok). Stable companions cover GLM-5.3-Flash's four-stream mHC block (cake_mhc_train), packed causal conv1d with SiLU, and SiLU-gated RMSNorm. The KDA, GDN and MoE training entry points are experimental and need the source install described in the experimental upgrade notice below.
- #6787 feat(cake_kda): packed KDA chunked training kernels for GLM-5.3-Flash with grad
- #7002 feat(cake_kda): context-parallel packed KDA grad with precise cross-rank state passing and per-document zigzag for GLM-5.3-Flash with grad
- #7169 feat(cake_gdn): context-parallel packed GDN auto grad with precise cross-rank state passing, native grouped Q/K heads and per-document zigzag for Qwen hybrid GQA/GDN with grad
- #7592 feat(cake_gdn_det_train): add deterministic GDN chunked training backend for SM100/SM103
- #6053 feat(cake_moe): BF16 and MXFP8 MoE training with recompute, clamped SwiGLU and variable sequence lengths on Blackwell and Rubin
- #6897 feat(cake_mhc): four-stream mHC kernels (GLM-5.3-Flash, hidden size 4096) on SM100 / SM103 / SM107 with grad (#6486)
- #6782 feat(cake_causal_conv1d): packed causal conv + SiLU kernels for sm_100a, sm_103a and sm_107a with grad
- #7786 feat(cake_rmsnorm_silu_gate): SiLU-gated RMSNorm training forward and backward (SM100 / SM103 / SM107)
⚠️ Upgrade notices
- Install
nvidia-cudnn-frontend>=1.32.0 and nvidia-cutlass-dsl>=4.8.0: pip install -U 'nvidia-cudnn-frontend>=1.32.0' 'nvidia-cutlass-dsl>=4.8.0'. Environments pinned to an older CuTe DSL must upgrade.
- After upgrading
flashinfer-python, run flashinfer download-kernels to replace mismatched flashinfer-cubin and flashinfer-jit-cache wheels; FlashInfer ignores wheels from another version and falls back to JIT compilation and on-demand cubin downloads.
- Expert-parallel MoE:
flashinfer.comm.MoeAlltoAll, MnnvlMoe and the flashinfer.comm.moe_a2a_* functions are removed together with the backend="trtllm" all-to-all kernels; use the flashinfer.moe_ep communication backends (NVLinkOneSidedAlltoAll, CakeAlltoAll, NVLinkTwoSidedAlltoAll, or SplitConfig(comm=...)), and rename dispatch(max_tokens_per_rank=...) to runtime_max_tokens_per_rank=....
⚠️ Upgrade notices (Experimental)
- To use APIs under
flashinfer.experimental, install from source with pip install --no-binary flashinfer-python flashinfer-python==0.7.3 or pip install git+https://github.com/flashinfer-ai/flashinfer.git@v0.7.3. The PyPI wheel carries every stable API and its JIT sources; the experimental kernel sources ship with the source install.
What's Changed
- perf(cake_kimi_k3_latent_moe): plan from the device SM count; qualify 148- and 152-SM parts (SM100a, SM103a) by @yyihuang in #6109
- perf(cake_kimi_k3_attn_res): qualify the native ports on 148- and 152-SM parts (SM100a, SM103a) by @yyihuang in #6108
- perf(cudnn): reduce attention execution and planning overhead by @YangXu1990uiuc in #5526
- feat(cake_ssd_combined): <128-token calls, seq_idx guard, grouped epilogue loads, any chunk_size via cu_seqlens (SM100 / SM103) by @yyihuang in #6100
- perf(prefill): defer CUTLASS work counting until needed by @YangXu1990uiuc in #5747
- fix: initialize paged fallback metadata with an explicit KV maximum by @YangXu1990uiuc in #6079
- fix(ci): handle isolated SM107 nightly tests by @cindyzxq in #5673
- bump version to 0.7.2 by @kahyunnam in #6158
- perf(cake_deepgemm): folded weighted-sum epilogue for the 64-head FP8 MQA logits program; tiered sm_100a paged admission (SM100a, SM103a) by @yyihuang in #6097
- fix: package SM90 BF16 Cake MegaMoE JIT sources by @cindyzxq in #6116
- feat(mla): unify backend execution and add wrapper autotuning by @saltyminty in #6106
- feat(cake_dsa_indexer): round 2 generated programs -- host-dispatched scan pipeline levers and the staged / two-level prefix-popcount rank finalize with pools sized by max_seqlen_k, bulk-copy row I/O, a 16-slot thread form for top_k > 2048 and an SM107 persistent two-row form for top_k <= 2048, bitwise identical (SM100 / SM103 / SM107) by @yyihuang in #6056
- perf(cake_all_gather_matmul): overlap the gather with the GEMM — persistent arrival-ordered schedule, SM push for small shards, PDL chaining (SM100a, SM103a) by @yyihuang in #6153
- perf(cudnn): bound cached MLA prefill geometry by @YangXu1990uiuc in #6134
- fix(cudnn): gate backend="auto" on a caller-owned block table's declared cache length (cudnn-frontend 1.30) by @vedaanta in #6150
- perf(cake_gdn): Qwen3.5 BF16-state T=1 decode bands and five prefill head splits (SM100a, SM103a) by @yyihuang in #6131
- feat(cudnn): backend="auto" takes cuDNN for single-token d256 paged decode in the measured CTA band; CUDA-graph wrappers declare replay to cudnn-frontend 1.31+ by @vedaanta in #6152
- feat(cake_diffusion): MiniMax-H3 SM120 fused quantized full MLP, FC1 + SwiGLU + FC2 + gated residual (RTX 5090 / RTX PRO 6000) by @yyihuang in #6169
- test(cake_backend): cross-backend DeepSeek-V3 routing coverage on real-logit, tie and large-T inputs by @yyihuang in #6171
- fix(cake_all_gather_matmul): re-allocate past torch's stale NVSHMEM rendezvous handle on workspace growth (SM100a, SM103a) by @yyihuang in #6157
- fix(ci): isolate test_sharding child pytest runs from nightly PYTEST_ADDOPTSFix sharding plugin test pytest addopts by @nvamyt in #5722
- fix(rope): gate cuTile FP8 quantization on SM89+ by @cindyzxq in #6188
- feat(moe): add Rubin locality-domain execution by @soodoshll in #5276
- test: bound peak memory in mxfp4 groupwise group GEMM test (32GB OOM) by @waynehacking8 in #3594
- feat(moe): enable TRTLLM-GEN FP8 routed MoE on SM107 (Rubin) by @Vinnie6167 in #5390
- [Patch] Enable SM107 on kda-prefill-cute by @johncalesp in #6201
- feat(cake_deepgemm): SM120 paged FP8 MQA lightning-indexer logits for RTX PRO 6000 / RTX 5090 (experimental generated-program export) by @yyihuang in #6180
- perf(cake_sparse_mla): FP8 H128 prologue rendezvous + BF16 SWA128 two-chunk online softmax (SM100/SM103) by @yyihuang in #6193
- feat(moe): add the remaining Prims-TS quant pairs to the unified API by @feih-nv in #5718
- feat(cake_deepgemm): SM120 paged MQA-logits indexer: one Q atom per request for next_n 4, strided per-layer cache views by @yyihuang in #6199
- feat(cake_warp_decode): add clamped DeepSeek-V4-Flash routed NVFP4 decode for SM100 by @yyihuang in #5558
- refactor(cake_dsv4): fold the NVFP4 decode knob clones into kernel templates and share the binding host code (SM100/SM103) by @yyihuang in #6190
- perf(cake_moe_finalize_allreduce_fusion): hoist the residual row ahead of the Lamport poll and gather the routes four at a time by @yyihuang in #6166
- fix(moe): reject unsupported routing modes and per-token scales in every unified runner by @feih-nv in #5663
- feat(cake_deepgemm): SM120 paged MQA-logits indexer: next_n 3, 5 and 6 programs for 32 and 64 heads by @yyihuang in #6219
- perf(cake_dsv4): dequantize the NVFP4 decode cache with the hardware QMUL4 via PTX ISA 9.4; require CUDA 13.4 (SM100/SM103) by @yyihuang in #6215
- [Perf] Reduce cuDNN linear-attention host overhead by @Anerudhan in #6218
- feat(moe): add cuDNN grouped-GEMM backends to the unified MoE API by @Aneureka in #5453
- perf(cake_diffusion): MiniMax-H3 SM120 quantized MLP, FP8 FC1 raster group 64 and NVFP4 FC1 producer warpgroup + staged y emission (RTX 5090 / RTX PRO 6000) by @yyihuang in #6196
- feat(cake_kimi_k3_attn_res): serving-shape routing (M 1..16384) + snapshot-write variant for SM100 / SM103 by @yyihuang in #6070
- perf(cake_minimax_h3_varlen_attention): round-2/3 BF16 varlen attention schedule and segment plan cache (SM100a, SM103a) by @yyihuang in #6062
- feat(cake_dsa_indexer): round 3 generated programs -- SM100 wide-kind register split (232 math / 40 I/O per lane, host-dispatched), bitwise identical to round 2 on SM100 / SM103 / SM107; lean export (shared headers, lever table in tests, cached device queries), SASS unchanged by @yyihuang in #6230
- perf(cake_sampling): round 11 -- exact early termination of the leader radix passes, per-capability dispatch, hard floors (SM90, SM100a, SM103a, SM107a) by @yyihuang in #6143
- fix(tests): gate SM120-only MiniMax kernels on exact CC by @cindyzxq in #6228
- feat(cake_concat_mla_kv_quant_fp8): generated Cake kernels for the fused MLA K/V pack + fp8 cast by @yyihuang in #6163
- fix(msa): keep earlier blocks when the NVFP4 prefill replay rebases deeply by @yekerr in #6093
- perf(prims-ts): qk-bf16/pv-fp8 and all-fp8 dense fmha context perf bringup by @harrisonzhy in #5900
- fix(tests): count caching-allocator requests in the Cake DSv4 hardening no-allocation checks by @yyihuang in #6244
- perf(cake_ssd_combined): preprocess, exact-scan and chunk-parallel prefill improvements (SM100 / SM103) by @yyihuang in #6206
- ci: move the CUDA 13.4 CI image to cuDNN 9.26.0.51 by @dierksen in #5877
- feat(cli): let install-jit-cache-wheel --sm imply minimal mode by @dierksen in #6095
- fix(cubin_loader): log HTTP error response headers and body at DEBUG by @dierksen in #5980
- fix(jit): ignore stale cubin/jit-cache wheels at import so download-kernels can replace them by @dierksen in #5892
- ci: retry JIT cache image and dependency downloads by @dierksen in #5748
- fix(moe): route the cuDNN grouped-GEMM runners through the shared pack contract by @yyihuang in #6264
- feat(cake_all_gather_matmul): serve torch's default CUDA symmetric-memory backend and capture the prepared launcher (SM100a, SM103a) by @yyihuang in #6104
- feat(moe): add TRTLLM Gen SwiGLU StepFun support by @jiahanc in #6226
- [SM120] Add W4A8 CuTeDSL MegaMoE backend by @Euynaheh in #4632
- test: prune FP8 blockscale GEMM parameter matrix by @righthandabacus in #5423
- test: prune SM-constrained GEMM parameter matrix by @righthandabacus in #5424
- feat(moe): fuse tanh-clamped squared relu in CUTLASS epilogues by @mathemakitten in #5696
- feat: integrate cuDNN linear attention pools and bounded auto dispatch by @Anerudhan in #6259
- cudnn prefill: return a packed LSE, declared as ragged Stats by @egilliam-nv in #5161
- fix(gemm): use the SM107 CuTe-DSL kernel for mm_mxfp8(backend="cute-dsl") on Rubin by @Vinnie6167 in #6136
- test: prune GDN decode parameter matrices by @righthandabacus in
Note truncated.