These highlights are also published at flashinfer.ai/releases.
v0.7.1 Highlights
FlashInfer v0.7.1 enables quantized expert-parallel MoE on Rubin and Blackwell SM12x, adds fused Kimi K3 speculative verification, and adds cuDNN linear-attention prefill on Blackwell. Sparse MLA gains DeepSeek-V4.1 dual-cache support and compact GLM53-NoPE caches, alongside MLA API improvements and per-model NVFP4 quantization recipes.
Run quantized expert-parallel MoE on Rubin and Blackwell SM12x
Rubin SM107 serves NVFP4 and MXFP8 MegaMoE experts through MoEEpLayer, with BF16 combine and output. On Blackwell SM120/SM121, the W4A16 fused MoE shards experts across GPUs through expert_map, so multi-GPU workstations such as RTX PRO 6000 pairs can split a model's experts between cards. New CuTe DSL NVFP4 W4A16 and W4A4 MoE backends on SM12x read one prepared expert layout, so a serving stack can choose W4A16 for decode and W4A4 for prefill from a single weight copy; on DGX Spark the W4A16 backend runs a Qwen3.8-Flash-Next MoE layer 1.36× faster than Marlin at one token. The split MoE layer also supports CUDA Graph capture with both NCCL-EP and NIXL-EP.
- #5384 feat(moe_ep): add SM107 block-scaled MegaMoE backends
- #4302 feat(moe): expert parallelism for the SM12x W4A16 fused MoE
- #5735 Canonical-layout NVFP4 W4A16 MoE in CuTe DSL on SM12x
- #5863 Canonical-layout NVFP4 W4A4 MoE in CuTe DSL on SM12x
- #5287 feat(moe_ep): CUDA-graph support for the split layer (nccl_ep + nixl_ep)
Kimi K3 adds packed speculative decoding and NVFP4 SiTU experts
Kimi K3 integrations can use packed_fused_kda_decode on Blackwell SM10x to fuse short convolution, Kimi Delta Attention, and gated RMS normalization during speculative verification. In vLLM on eight B300s with six-draft-token speculative decoding, the fused path raised total output throughput by 1.8–3.3% at concurrencies 1 to 32. The API supports ragged batches, accepted-token cache updates, CUDA Graph capture, and FP32 or BF16 recurrent state. NVFP4 MegaMoE also supports Kimi K3's SiTU activation on Blackwell SM10x.
- #4809 feat(kda): add fused speculative decode kernel
- #5503 feat(kda): support bf16 state in packed fused KDA decode for T>1
- #5455 feat(moe_ep): Support Kimi K3 SiTU in NVFP4 MegaMoE
Kimi Delta Attention and Gated DeltaNet prefill on Blackwell are supported with cuDNN's open-source kernels
Kimi Delta Attention and Gated DeltaNet prefill run on cuDNN's open-source fused linear-attention engine on Blackwell SM100, selected with recurrent_kda(backend="cudnn") and chunk_gated_delta_rule(backend="cudnn"). The cuDNN path also runs on CUDA 12 builds, where the default Blackwell Gated DeltaNet prefill kernel requires CUDA 13. New chunk_gated_delta_product and chunk_gated_delta_rule2 entry points add Gated Delta Product and Gated Delta Rule 2 prefill on the same engine.
- #4968 Add cuDNN backend for linear attention
Run DeepSeek-V4.1 dual-cache attention on SM12x with 19.5% smaller GLM53-NoPE caches
DeepSeek-V4.1 sparse MLA runs on SM120/SM121 with a primary FP8 cache and an optional FP4 or FP8 extra cache, each with its own page size and top-k selection, in FP8 or BF16 compute. GLM53-NoPE caches can use compact 528-byte rows, cutting per-token KV storage by 19.5% against the 656-byte padded layout; existing 656-byte pools keep working through strided cache views.
- #5197 refactor(sparse-mla): unify SM120 execution, calibration and DSv4.1 support
- #5075 feat(sm120): runtime KV row stride + canonical 528B GLM53_NOPE rows + NaN-safe masked gathers in sparse MLA
Call MLA prefill directly and reuse execution plans
trtllm_prefill_with_kv_cache_mla, available through both flashinfer.mla and flashinfer.prefill, gives integrations a named prefill entry point with the existing MLA decode API's signature and behavior. Multi-token prefill callers must explicitly select a compatible backend; XQA supports one query token. BatchMLAPagedAttentionWrapper prepares TRTLLM-GEN, XQA, and monolithic or modular CuTe DSL backends in plan() and reuses them in run(), giving integrations a shared lifecycle for repeated execution and CUDA Graph capture. On SM100, callers can pass backend="auto" and let plan() choose among all available backends for the planned shapes and features.
- #5158 fix: add discoverable MLA prefill API
- #5070 feat(mla): add planned TRTLLM-GEN, XQA, and CuTe DSL backends
- #5463 feat(mla): MLA SM100 auto mode
NVFP4 recipes can be configured per model
NVFP4 quantization APIs and the unified MoE QuantConfig accept an explicit NVFP44Over6Config, so a serving framework can enable 4over6 from a model's own quantization config and run models with different recipes in one process. The process-wide environment variables remain the default when no recipe is given.
- #5152 feat(quantization): expose NVFP4 4over6 recipe through the public API
Upgrade notices
- Install
nvidia-cudnn-frontend>=1.30.0. The cuDNN linear-attention backend needs the cutedsl extra: pip install -U 'nvidia-cudnn-frontend[cutedsl]>=1.30.0'.
- SM90 CuTe DSL MoE:
cute_dsl_fused_moe_bf16 takes tactic for tuning, and CuteDslBf16MoEWrapper reorders its constructor arguments. Pass output_dtype, enable_pdl, and use_fused_finalize by keyword. moe_a2a_dispatch takes recv_view_cache by keyword only.
Upgrade notices (Experimental)
- Update calls to
fp8_paged_mqa_logits and fp4_paged_mqa_logits: use block_tables, seq_lens, and max_seq_len; pass the page table before sequence lengths in positional calls; remove num_epi_subtiles. FP4 callers must also rename sf_q to q_sf.
- Review PrimTS integrations against the experimental API contract. Migrate callers of the removed
prims_ts_batch_decode_with_kv_cache and prims_ts_batch_mla_decode_with_kv_cache entry points to the corresponding paged decode wrappers.
What's Changed
- feat(comm): copy-engine based kernel for the PCIe IPC all-reduce by @qsang-nv in #4870
- feat(cake_warp_decode): Add SM100 NVFP4 warp decode and standalone SiLU by @yyihuang in #5036
- feat(cake_nvfp4_svdquant): add NVFP4 SVDQuant GEMM SM100 and SM103 backend by @yyihuang in #5076
- fix(moe): bound routing_replay_out dim0 from below in both validators by @aleozlx in #5072
- fix: guard BF16 warp Split-K availability and prune tactics by @jiahanc in #5089
- refactor(kda): remove the unreleased recurrent KDA training/backward family by @YangXu1990uiuc in #4965
- fix(activation): pick a vector width that divides hidden_size in act_and_mul (fixes d=3420) by @Wint3rNight in #5013
- docs: document MiniMax-H3 prepared run parameters by @kangbintNV in #5087
- perf(moe): improve SM90 mixed-input CUTLASS MoE backend performance by @StudyingShao in #5005
- refactor(kda): stop 0.7 advertising unreleased KDA surface (mark prefill wrapper experimental, trim kda_kernels.all) by @kahyunnam in #5040
- bench: compare DeepSeek MoE against pure TRTLLM BF16 by @zianglih in #4985
- perf(topk_varlen): gvr_2 dispatch rungs for 4K-8K rows, SM-count-aware register band, hint-free gvr_2 in auto by @dhiraj113 in #4986
- feat(cake_mega_moe): add SM100 BF16 rank-major MegaMoE backend by @yyihuang in #4810
- fix(attention): default skip_all_rows_active_check to true by @saltyminty in #5039
- feat(cake_fmha): add sm100/103 paged-attention by @yyihuang in #4980
- fix: preserve both generated-source pre-commit exclusions by @saltyminty in #5113
- fix(fp4_gemm autotune) SM107 kernel family excluded from autotune candidates and heuristics update by @Victor49152 in #4856
- fix(mla): halve sparse-MLA cpb L2 guard-rail window on integrated GPUs (Spark) by @bkryu in #5097
- fix(cake_sage_attention): support non-contiguous Sage attention with partial KV tiles by @yyihuang in #5083
- Fix MoE Tile Mapping Zeroing by @Vinnie6167 in #4735
- fix(test): pin the GDN disk-cache round-trip test to the flashinfer backend by @bkryu in #5104
- feat(cake_kda): add fused decode backend for B200 and B300 by @yyihuang in #4967
- feat(gemm): cuTile alpha-beta / masked-bmm / ragged-bmm / block-scaled FP8 GEMM by @yifeis-nv in #4020
- feat(cake_comm): add opt-in Blackwell MNNVL MoE all-to-all backend by @yyihuang in #4784
- feat(cake_diffusion): add SM103a BF16 pre-attention by @yyihuang in #4690
- fix(moe): Fix per-token quantization
fast-math INF scale when activation contains all-zero row by @xuantengh in #5031
- Fix bmm_fp8 cublasLt handle usage in autotuned cublas runner #26381 by @baonudesifeizhai in #2808
- build: raise the nvidia-cudnn-frontend floor to 1.28.0 by @Anerudhan in #5131
- fix(moe): avoid FP8 scale overflow in the Humming MXFP4 path by @yilin-void in #5024
- chore: add codebase-wise approvers to every CODEOWNERS rule by @bkryu in #5146
- Add CUTLASS Primitives and Task Scheduling Blackwell MoE kernels by @nekorobov in #4361
- fix[mla]: add Q8 floor for tile selection in trtllm-gen sparse mla kernels by @jimmyzho in #5106
- feat(quantization): support fp32 input in the CuTe-DSL mxfp8_quantize kernel by @bkryu in #5112
- feat(sm120): runtime KV row stride + canonical 528B GLM53_NOPE rows + NaN-safe masked gathers in sparse MLA by @lucifer1004 in #5075
- Revert "build: raise the nvidia-cudnn-frontend floor to 1.28.0" by @aleozlx in #5159
- feat(cake_diffusion): add SM100a support to MiniMax-H3 BF16 pre-attention by @yyihuang in #5137
- fix(moe): accept mxfp8's flat linear activation-scale layout in the MoE autotuner by @aleozlx in #4943
- fix(comm): make TorchDistBackend.Split() synchronizing, like MPI_Comm_split by @aleozlx in #4206
- fix(sampling): preserve softmax normalization at low temperatures by @Stelath in #5088
- fix(infra): isolate distributed rendezvous ports by test worker by @yongwww in #5153
- feat(cake_bf16_fp4_gemm): add SM100/SM103 BF16 x FP4 GEMM kernels by @yyihuang in #5138
- feat(cake_mega_moe): add reusable MegaMoE workspaces and an SM100a reducer by @yyihuang in #4819
- perf(cake_mega_moe): optimize BF16 x MXFP8 MegaMoE EP16 backend by @hzfan in #5148
- Add cuDNN backend for linear attention by @jhjpark in #4968
- feat(moe): remove QuantVariant and dispatch QuantConfig by MMA pair by @feih-nv in #5061
- fix(cake_sage_attn): cover Sage combinations and pass tensor maps by value by @yyihuang in #5127
- fix(cutile): stabilize autotune config selection in select GEMMs by @elwhyjay in #4459
- test(comm): report every rank's fate when a dist worker dies by @aleozlx in #4205
- fix[GEMM]: WAR for specific cuTile autotune candidate in mm_bf16 by @jimmyzho in #5155
- ci: apply --prerelease to rc in release workflow by @aleozlx in #4472
- fix(moe): validate TMA warp-specialized configs in queryOccupancyForConfig by @SSHdotCodes in #4847
- test(cake_comm): initialize NVSHMEM for BF16 rank-major GPU regression by @yyihuang in #5126
- feat(cake_comm): add Cake MoE finalize all-reduce backend by @yyihuang in #4875
- feat(cake_alpha_moe): add optimized Blackwell W8A8 expert up/down compute by @xslingcn in #4287
- perf(cake_comm): reduce PCIe CE ring publication overhead on SM120 by @yyihuang in #5169
- feat(prims-ts): add non-absorbed MLA prefill by @PerkzZheng in #5059
- BF16 cutedsl prune [2] + cache bf16 cutedsl kernels by @jiahanc in #5140
- docs: fix 0.7.0 documentation check findings by @kangbintNV in #5199
- build: raise the nvidia-cudnn-frontend floor to 1.29.0 by @Anerudhan in #5182
- fix(cake_attention): enable DCP on SM107 and skip unsupported VSA tests by @saltyminty in #5111
- feat(cake_comm): support SM100 and SM103 TP8 packed-QKV all-gather matmul by @yyihuang in #5116
- fix: add discoverable MLA prefill API by @saltyminty in #5158
- feat(moe): tag the unified MoE API as official with @flashinfer_api by @aleozlx in #5200
- perf(attention): specialize paged FA2 equal-stride KV by @saltyminty in #4736
- feat(comm): add allocation-stable head-chunk Ulysses primitives by @tiffany940107 in #5027
- test(kda): xfail CuTe DSL recurrent prefill tests on nvidia-cutlass-dsl<4.7 by @bkryu in #5219
- Paged-MQA logits: Rubin (SM107) enablement, FP4 next_n=4, and DKG epilogue/scheduler optimizations by @dhiraj113 in #4737
- feat(gemm): add SM10x contiguous grouped FP8 CuTe backend by @samodi-nv in #4734
- feat: enable large-head attention on SM80, for VLLM to run FP8 kv by @jhaotingc in #5044
- feat(cake_bf16_bmm): add optimized SM100a and SM103a backend by @yyihuang in #4370
- fix(comm): fence DCP receiver unpack before async output copy by @klshuster in #5172
- feat(prims-ts): add QToken-KvBlock-Sparse-Attention by @PerkzZheng in #4996
- ci(jit-cache): enable provider-only wheel releases by @dierksen in #5096
- fix(autotuner): recover from builtin memory errors by @ZenAlexa in #4920
- feat(mla): add planned TRTLLM-GEN, XQA, and CuTe DSL backends by @saltyminty in #5070
- fix(topk): handle padded short rows in CUB transforms by @zianglih in #5187
- fix(comm): match the library filename when scanning /proc/self/maps by @JacobHelwig in #5198
- feat: add low-latency Frost DSv4.1 sparse MLA decode by @YangXu1990uiuc in #5119
- fix(ci): support installed BF16 rank-major session tests by @XFDG in #5171
- fix: restore default GDN prefill dispatch to CuTe by @yyihuang in #5225
- refactor(moe): update and refactor the SM90 CuTe-DSL MoE tactics by @Aneureka in #5004
- feat(prefill): let backend="auto" reach cuDNN/CUTLASS on Blackwell ragged prefill (up to 3.3x) by @Anerudhan in #5133
- feat(prims-ts): support QK-BF16/PV-FP8 in FMHA context by @harrisonzhy in #4879
- ci: cache checksum-verified cubin downloads by @dierksen in #5240
- fix(jit): refresh stale arch detection and allow SM120 on CUDA 12.8 by @zhougit86 in #3633
- fix(gdn): compile and tune CuTe-DSL decode kernels for the operand device by @kahyunnam in #4507
- chore: add @jhjpark as CODEOWNER for GDN, Mamba, and KDA by @kahyunnam in #5247
- fix(topk): fall back to CUB for graph-safe page-table transforms on SM120 by @bkryu in #5226
- feat(moe): add cuTile MXFP4 and W4A16 support by @bkryu in #5099
- perf(cake_gdn): optimize prefill kernels for SM100 and SM103 by @yyihuang in #5243
- perf(cake_alpha_moe): accelerate AlphaMoE fused router on Blackwell by @yyihuang in #4339
- fix(gdn): force min_blocks_per_mp=1 on SM107 to avoid DSL auto-carveout fault by @Vinnie6167 in #5227
- perf(gdn): optimize SM100 CP kernels by @guangyunh-nv in #4917
- chore: remove deprecated APIs eligible at the 0.7.0 boundary by @aleozlx in #5218
- fix(moe): write fused-shared routing replay as routed-only [T, K] by @feih-nv in #4894
- refactor(attention): use fixed page tables for paged block-sparse attention by @heyuhhh in #5144
- feat(cake_comm): add optimized Blackwell MoE all-reduce fusion backend by @yyihuang in
Note truncated.