These highlights are also published at flashinfer.ai/releases.
v0.7.2 Highlights
FlashInfer v0.7.2 brings MoELayer experts to Rubin through the cuDNN Frost backend, widens Rubin coverage across MLA, GEMM, linear attention and expert-parallel MoE, and adds fused-epilogue Prims-TS GEMMs on B200, B300 and Rubin. On Blackwell datacentre parts, DeepSeek-V4 NVFP4 sparse MLA reaches B200 and B300 and speculative-decode verification picks cuDNN automatically; expert-parallel MoE adds Hopper BF16 and SM120 W4A8 backends.
MoELayer experts run on Rubin
MoELayer on Rubin SM107 can run its experts on cuDNN Frost, a CuTe DSL grouped-GEMM backend for BF16, MXFP8, NVFP4 and MXFP8-activation/MXFP4-weight experts; the BF16 kernels also run on Blackwell SM120. Autotuning admits Frost alongside the existing backends, and CudnnFrostBf16Config, CudnnFrostMxfp8Config, CudnnFrostNvfp4Config and CudnnFrostMxfp8Mxfp4Config in flashinfer.fused_moe select it explicitly. Frost needs PyTorch 2.14 (CUDAGraph.retain_object) and, on SM107, a PTXAS or CuTe DSL assembler from CUDA 13.4 or newer.
- #6035 Yanqinz/move frost out from experiment
- #5762 Sm120 cudnn frost moe bf16
DeepSeek-V4 and Kimi K3 serving kernels run on Rubin
BatchMLAPagedAttentionWrapper(backend="auto") plans on B300 and Rubin, and DeepSeek-V4 sparse MLA runs on SM107. Rubin also runs the CuTe DSL GEMM family, mm_mxfp8(backend="cute-dsl") on an SM107-specific kernel, the CuTe packed Kimi Delta Attention decode fast path, the CuTe DSL Mamba2 SSDCombined prefill, hardware stochastic rounding in the Mamba selective state update, and silu_and_mul_mxfp8_quantize. SM107 MegaMoE through MoEEpLayer adds MXFP4 weights with MXFP8 activations, prequantized NVFP4 and MXFP8 checkpoints, and Kimi K3's SiTU activation, functionally validated on Rubin.
- #5984 Enable MLA auto-selection on SM103 and SM107
- #5391 feat(mla): enable TRTLLM-GEN DSv4 sparse MLA/RopeQuant on SM107 (Rubin)
- #5394 feat(gemm): enable TGV, CuTe DSL low-latency block-scaled and contiguous-grouped GEMMs on SM107 (Rubin)
- #6136 fix(gemm): use the SM107 CuTe-DSL kernel for mm_mxfp8(backend="cute-dsl") on Rubin
- #5396 feat(kda,mamba): enable CuTe packed KDA decode, cvt.rs stochastic rounding and Mamba2 SSD on SM107 (Rubin)
- #5267 perf(mamba): enable native SM107 stochastic rounding
- #5392 feat(quant): enable fused gated MXFP8 quantization (SwiGLU + MXFP8) on SM107 (Rubin)
- #5699 feat(moe_ep): support Rubin MXFP4 weights with MXFP8 activations
- #5662 feat(moe_ep): add Rubin SiTU, prequantized weights, and NVFP4 scaling
GEMMs fuse SwiGLU and QKV-norm-RoPE epilogues on B200, B300 and Rubin
flashinfer.prims_ts adds dense FP8 and NVFP4 GEMMs with fused SwiGLU (fp8_linear_swiglu, fp4_linear_swiglu) and fused QKV + QK-norm + RoPE (fp8_qkv_qknorm_rope, fp4_qkv_qknorm_rope) epilogues on B200, B300 and Rubin SM107. The NVFP4 SwiGLU and QKV variants accept an opt-in tmem_overlap=True tactic, which the autotuner also considers. The module needs the CuTe DSL experimental primitives shipped with nvidia-cutlass-dsl.
- #5758 feat(prims-ts): add FP8/NVFP4 GEMMs with SwiGLU and RoPE fusion
- #6255 feat(prims-ts): allow TMEM overlap for NVFP4 SwiGLU epilogues
- #6286 feat(prims-ts): allow TMEM overlap for NVFP4 qkv overlap
DeepSeek-V4 NVFP4 sparse MLA reaches B200 and B300 and adds prefill on RTX PRO 6000
DeepSeek-V4 serving with an NVFP4 KV cache runs sparse-MLA decode on B200 and B300 through trtllm_batch_decode_sparse_mla_dsv4(kv_cache_format="nvfp4", backend="cake"). On RTX PRO 6000 Blackwell and RTX 5090 the same entry point gains a backend="cake" decode and prefill path for the NVFP4 cache and a decode path for the DeepSeek-V4.1 mixed FP8/FP4 cache (SM120 and GB10/SM121), plus fused RoPE, NVFP4 quantize and paged-insert cache writers. These backends are opt-in.
- #6103 feat(cake_dsv4): NVFP4 DeepSeek-V4 sparse-MLA decode on SM100/SM103 (backend="cake")
- #5914 feat(cake_dsv4): SM120 DeepSeek-V4 NVFP4 sparse-MLA decode backend for RTX PRO 6000 / RTX 5090 (#5690)
- #5959 feat(cake_dsv4_sparse_mla): add SM120 DeepSeek-V4 NVFP4 sparse-MLA prefill backend for RTX PRO 6000 / RTX 5090
- #5983 feat(cake_dsv4_sparse_mla): SM120/SM121 DeepSeek-V4.1 mixed-cache sparse-MLA decode (528 B FP8 SWA + 288 B FP4 compressed, BF16 Q)
- #6033 feat(cake_dsv4): SM120 DeepSeek-V4 NVFP4 fused GPT-J RoPE + quantize + paged-insert writers
Speculative-decode verification runs faster on B200 and B300
BatchDecodeWithPagedKVCacheWrapper(backend="auto") on B200 and B300 routes multi-token decode rows (two to four query tokens per request, as in speculative or multi-token-prediction verification) of fp16/bf16 head-dim-128 grouped-query-attention models to cuDNN, which lowers verification decode latency. The route needs nvidia-cudnn-frontend[cutedsl]>=1.31.0. Single-token decode and runs with explicit determinism controls stay on FA2; FLASHINFER_DECODE_AUTO_CUDNN=0 pins FA2.
- #5508 feat(cudnn): backend="auto" takes cuDNN for the multi-token paged-decode rows its default FROST engine serves faster (cudnn-frontend 1.30+)
Expert-parallel MoE comes to H100 in BF16 and to SM120 in W4A8
H100 serves an expert-parallel MoE round entirely in BF16 through MoEEpLayer with Sm90_Bf16_Bf16_Bf16_PushCake_MegaMoeConfig: BF16 dispatch, fused SwiGLU, a deterministic combine over a pre-reduced BF16 wire, CUDA Graph capture and up to 32 ranks on one node. SM120 workstations get a W4A8 MegaMoE backend (Sm120_Mxfp4_Mxfp8_Bf16_Cutedsl_MegaMoeConfig) with CUDA Graph replay and uneven token counts across ranks. SplitConfig(comm=..., kernel=...) picks an NVLink one-sided, CAKE or NVLink two-sided communication backend, or the NCCL-EP or NIXL-EP transports, independently of the expert kernel.
- #5958 feat(cake_mega_moe): SM90 native BF16 MegaMoE expert compute through MoEEpLayer (sm90_bf16_bf16_bf16_push_cake)
- #6058 feat(cake_mega_moe): pre-reduced BF16 combine wire and fixed-cost levers for the SM90 MegaMoE backend
- #4632 [SM120] Add W4A8 CuTeDSL MegaMoE backend
- #5532 refactor(moe_ep): add a MoE-level communication interface for EP dispatch/combine
⚠️ Upgrade notices
- Install
nvidia-cudnn-frontend>=1.31.0. The cuDNN linear-attention backend and the automatic cuDNN decode route need the cutedsl extra: pip install -U 'nvidia-cudnn-frontend[cutedsl]>=1.31.0'.
- After upgrading
flashinfer-python, run flashinfer download-kernels to replace mismatched flashinfer-cubin and flashinfer-jit-cache wheels; --sm sm90a (repeatable) limits it to named architectures.
- Rename callers of
cutlass_fused_moe_prepare_workspace to cake_fused_moe_prepare_workspace, replace prepare_bgmv_moe(..., backend="blackwell") with backend="cake", and import the cuDNN Frost MoE configs from flashinfer.fused_moe (the flashinfer.experimental path is a one-release shim).
- Sampling: batch-length
seed and offset tensors apply one value per output row; callers passing distinct per-row values sample differently from before.
⚠️ Upgrade notices (Experimental)
flashinfer.source_mega_moe (MegaMoEPlan, prepare_mega_moe) is removed; use flashinfer.mega_moe_v3.
prepare_msa_nvfp4_sparse_decode drops workspace_buffer; msa_nvfp4_decode_workspace_size, workspace_layout and bind_decode_payload are removed. DeepGEMM-family prepare_* entry points drop their descriptor arguments.
What's Changed
- perf(cake_sparse_mla): round-6 DSv4 sparse-MLA Cake programs for SM100/SM103 (exact numerics; stacked on #5686) by @yyihuang in #5700
- refactor(cake_fused_moe): rename the Cake SiTU workspace preparation helper to cake_fused_moe_prepare_workspace by @yyihuang in #5710
- chore: bump nvidia-cudnn-frontend floor to >=1.30.0 by @Anerudhan in #5702
- test: guard tests/moe architecture requirements on the T4 lane by @taylor-yb-lee in #5465
- ci: treat a widened Literal annotation as a compatible public API change by @yyihuang in #5713
- feat(cake_mm_fp4): add per-token NVFP4 quantize + mm_fp4 per-token alpha backend for SM100 and SM103 by @yyihuang in #5697
- perf(cake_gemm): two-fragment TMEM drain and routing-aware route selection for the fused grouped FP8 gate_up + SwiGLU + quant program by @yyihuang in #5714
- perf(cake_sampling): round-5 kernels (software-pipelined sampled pass, ballot-gated filter, two-level cluster select, two-warp fused tail, ept-32 streams, per-architecture stage-2/3 forms) and chain-aware dispatch by @yyihuang in #5684
- fix(cute_dsl): wait on grid dependencies before the finalize-fusion kernel reads routing outputs by @yyihuang in #5682
- fix(cake_kda): repack the BF16 inverse fragment before block composition in the native prefill facade bodies by @yyihuang in #5659
- perf(cake_kimi_k3_latent_moe): round 9 -- release-scoped stream-K publication in both prefill GEMMs, symmetric two-way fix-up for the front's trailing-wave K halves (SM100a/SM103a) by @yyihuang in #5721
- fix(moe): initialize TRTLLM routing padding for TMA gathers by @byshiue in #5444
- feat(kda): kda for kda cutedsl impl by @Lyken17 in #5612
- feat(cake_dsa): add native 64-query-head DSA sparse-attention kernels for SM100/SM103 (#5657) by @yyihuang in #5704
- fix(moe): guard Humming MXFP4xFP8 runner behind ENABLE_FP4 by @x41lakazam in #5086
- fix(moe): skip unrouted (-1) expert ids in the SM12x b12x fused MoE by @SamMausberg in #5451
- feat(prims-ts): 8-bit Q/K/V sage attention for the PrimTS block sparse decode kernels by @heyuhhh in #5123
- ci: stage benchmarks for nightly package tests by @dierksen in #5698
- feat(mla): fall back to FA2 for unsupported SM90 auto requests by @saltyminty in #5712
- fix: log expected SM107 GEMM candidate rejections at debug level by @yufeiwu-nv in #5689
- Update Docker CI tags to 20260930-b1420f1 by @flashinfer-bot in #4673
- feat(cake_bgmv_moe): SM90/SM100/SM103 support and cake_bgmv_moe naming for the deterministic multi-LoRA BGMV MoE backend by @yyihuang in #5727
- feat(kda): kda-for-kda CAKE PTX impl by @Lyken17 in #5665
- perf(mla): deepen FP8 and FP16/BF16 decode load pipelines on SM107 by @zcnrex in #5703
- feat(cake_sampling): round-6 radix sampling bundle: coarse-sample stage-1 twins, row-span filter arm and small-k dispatch re-fit by @yyihuang in #5731
- feat(quant): enable fused gated MXFP8 quantization (SwiGLU + MXFP8) on SM107 (Rubin) by @Vinnie6167 in #5392
- perf(cake_mm_fp4): tactic rules and regenerated programs for the per-token NVFP4 route by @yyihuang in #5744
- perf(cake_kimi_k3_latent_moe): round 10 -- final-item staged TMA store for the 256-wide prefill tail GEMM instances (SM100a/SM103a) by @yyihuang in #5734
- feat(kda,mamba): enable CuTe packed KDA decode, cvt.rs stochastic rounding and Mamba2 SSD on SM107 (Rubin) by @Vinnie6167 in #5396
- perf(cake_deepgemm): faster two-CTA prologue, early TMA load start and descriptor prefetch on the mixed FP8xFP4 and native FP4 GEMM routes (SM100a, SM103a) by @yyihuang in #5739
- perf(cake_deepgemm): one bulk TMA store per 16-row slab on the FP32 K-grouped FP4 GEMM swapped route (SM100a, SM103a) by @yyihuang in #5754
- perf(cake_deepgemm): per-group leader pre-ranking on the M = 1 routing-gate cluster split-K route (SM100a, SM103a) by @yyihuang in #5766
- perf(cake_vision_backend): round 6 — split-KV attention with exact fixed-order merge, wide unit table + prefetch, per-row cluster trim, half-N GEMM tail twins (sm_100a/sm_103a) by @yyihuang in #5746
- perf(cake_gqa): serve BF16/page-16 and E4M3/page-64 (head_dim 128/256) DCP speculative decode on SM100/SM103 with the on-device load-balanced programs by @yyihuang in #5750
- feat(kda): kda for kda TIRx impl by @Lyken17 in #5613
- feat(cake_lm_head_loss): add chunked LM-head + fused loss training kernels for SM100/SM103 (#5680) by @yyihuang in #5752
- perf(cake_kda): read packed-row strided q/k/v in place in every prefill body by @yyihuang in #5846
- refactor(cake_comm): fold the MoE all-reduce union to one kernel per route, one launcher and a small route table by @yyihuang in #5848
- refactor(cake_moe_finalize_allreduce_fusion): drop the SM103 copy, fold the PDL/binding clones and replace the import manifest by @yyihuang in #5852
- refactor(cake_comm): drop the unreachable all-reduce-output kernels from the MoE all-reduce fusion bundle and build it through gen_jit_spec by @yyihuang in #5849
- perf(cake_fmha): round-4 balanced DCP E4M3 bodies and deterministic packed-MTP fold on SM100/SM103 by @yyihuang in #5855
- feat(cake_moe_grouped_gemm): ragged BF16 grouped GEMM for MoE experts (forward, dX, dW) on SM100/SM103/SM107 (#5678) by @yyihuang in #5751
- perf(cake_sampling): round 7 -- speculative-sample stage-1 twin, whole-CTA tail for cluster-8 streams, capability-keyed host policy by @yyihuang in #5847
- fix(tests): scale cake/cute SSDCombined parity atol by the output scale by @kahyunnam in #5756
- fix(tests): skip unsupported groupwise GEMM architectures by @saltyminty in #5759
- fix(aot): include CUTLASS fused MoE in SM110 providers by @janbernloehr in #5695
- Add native W4A16 prefetch and grouped-tile tactics on SM121 by @stecasta in #5648
- Add SM12x NVFP4 support to the CuTe DSL GEMM backend by @stecasta in #5100
- feat(cake_kimi_k3_attn_res): add Kimi-K3 AttnRes backend for SM100/SM103 by @yyihuang in #5853
- fix(autotuner): preserve profiling policy in saved configs by @lightseek-bot in #5705
- perf(cake_mm_fp4): half-M pair tile for the narrow-N 2-CTA rows of the per-token NVFP4 route (sm_100a, sm_103a) by @yyihuang in #5869
- refactor(cake_comm): retire the isolated MoE all-reduce fusion bundle; serve calls without the all-reduce output from the union by @yyihuang in #5856
- fix(gemm): keep narrow block-scaled N tiles inside the first 32-token SFB group by @yyihuang in #5890
- fix(tests): skip unsupported cuTile groupwise GEMM architectures by @saltyminty in #5883
- fix(cake_comm): retain replaced MoE all-reduce scratch tensors recorded by captured graphs by @yyihuang in #5894
- revert(cake_fused_moe): remove the Kimi-K3 W4A8 MXFP4 SiTU changes to the CuTe DSL MoE path (#5430) by @yyihuang in #5681
- fix(gemm): fall back from stale MXFP8 split-K tactics by @ZeldaHuang in #5269
- perf(gemm): cute-dsl bf16-nvfp4 dense gemm fix to include no-autotune fallback tactic by @jimmyzho in #4941
- update-frost-kernel-and-name by @yanqinz2 in #5891
- feat(moe_ep): add Rubin SiTU, prequantized weights, and NVFP4 scaling by @akaashrp in #5662
- perf: optimize CuTe DSL W4A4 and W4A16 unfused finalize by @zianglih in #5319
- feat(cake_dsv4): SM120 DeepSeek-V4 NVFP4 sparse-MLA decode backend for RTX PRO 6000 / RTX 5090 (#5690) by @yyihuang in #5914
- fix(gdn): keep CP prefill correct when max_seqlen is omitted by @lucifer1004 in #5661
- refactor(cake kda): one host shim and one argument plan for the SM103a PTX KDA programs by @yyihuang in #5903
- refactor(cake kda): render the generated prefill selectors from Python and compact the module table by @yyihuang in #5919
- refactor(cake kda decode): drop the seven never-selected instances and serve each family from one binding header by @yyihuang in #5901
- refactor(cake mega_moe): drop the smoke-shape programs, the SM-count override and per-call preparation work from both generated MegaMoE families by @yyihuang in #5904
- refactor(cake tinygemm2_sm100): fold the six STAGES x PDL sections into one templated kernel by @yyihuang in #5910
- feat(cake_kimi_k3_attn_res): regenerate the Kimi-K3 AttnRes programs after the schedule-selection round (SM100 / SM103) by @yyihuang in #5916
- perf(cake_mm_fp4): stream-K stripe tail for the partial last wave of the per-token NVFP4 2-CTA GEMM (SM100) by @yyihuang in #5944
- refactor(moe_ep): add a MoE-level communication interface for EP dispatch/combine by @bobboli in #5532
- fix(mla): prefer measured SM12x backends and guard cuTile CTA tiles by @saltyminty in #5896
- Canonical-layout NVFP4 W4A16 MoE in CuTe DSL on SM12x by @stecasta in #5735
- feat(moe): refresh the SM107 CuTe DSL gather+activation MoE kernel from TensorRT-LLM (SiTU, Relu2) by @Vinnie6167 in #5462
- fix(sampling): honor per-row seed and offset tensors by @wdykas in #5745
- fix(gdn): point the gated_delta_rule_mtp deprecation marker at 0.8.0 by @kahyunnam in #5306
- feat(mla): enable TRTLLM-GEN DSv4 sparse MLA/RopeQuant on SM107 (Rubin) by @Vinnie6167 in #5391
- perf(cake_sampling): round 8 -- slab-tail twins of the sample builds, pushed-coarse-sums twin of the cluster-8 default build, Hopper whole-CTA tail policy by @yyihuang in #5950
- feat(cake_dsv4_sparse_mla): add SM120 DeepSeek-V4 NVFP4 sparse-MLA prefill backend for RTX PRO 6000 / RTX 5090 by @yyihuang in #5959
- feat(attention): enable CuTe DSL VSA/BSA and Sage-quant tests on SM107 (Rubin) by @Vinnie6167 in #5397
- fix(cake_dsv4_sparse_mla): SM-count-aware split and head-tile planners for GB10 (48 SMs) by @yyihuang in #5949
- feat(cake_bgmv_moe): fall back to the portable BGMV MoE path for unsupported inputs and launch the portable kernels on the current stream by @yyihuang in #5809
- feat(cake_dsa): SM107 (Rubin) generated programs and strided / packed GLM-5.2 trainer layouts for the native 64-head DSA sparse-attention training kernels (#5675) by
Note truncated.