PyPI · #3319 most downloaded on PyPI

flashinfer-cubin

0.6.131.5M downloads/moflashinfer-ai/flashinfer ↗New releases in your daily digest, changelog included.

Pre-compiled cubins for FlashInfer

Last release 3 months ago
25 Jun 2026
Ships on a steady schedule
a new release about every 2 weeks
Nearly every release is documented
notes for 27 of 28 stable releases
Nothing withdrawn
no release was ever pulled
12 months old
37 releases · first in 2025

Release timeline

37 releases since 2025

20252026

One column per month.

Releases

0.6.1325 Jun 2026Deprecated
replace deprecated APIs: cute.make_fragment and cute.core.ThrMma by @brandon-yujie-sun in https://github.com/flashinfer-ai/flashinfer/pull/3430
What's Changed

Run high-likelihood OOM culprits separately, record memory usage and test duration for analysis by @dierksen in https://github.com/flashinfer-ai/flashinfer/pull/2961

fix(autotuner): differentiate file cache entries by runner specific kernel parameters by @qiching in https://github.com/flashinfer-ai/flashinfer/pull/3367

feat: integrate cute-dsl Blackwell GQA decode into BatchDecodeWithPagedKVCacheWrapper by @richardmcai in https://github.com/flashinfer-ai/flashinfer/pull/3360

Fix returning reference to temporary in moe gemm by @benbarsdell in https://github.com/flashinfer-ai/flashinfer/pull/3332

Enable compression of GPU device binaries by @benbarsdell in https://github.com/flashinfer-ai/flashinfer/pull/2949

feat: MNNVL Allreduce quant fusion and performance optimization by @timlee0212 in https://github.com/flashinfer-ai/flashinfer/pull/3385

fix(norm): widen address arithmetic to int64 for large contiguous inputs > 2**31 elements by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/3392

MLA Decode Autotuning Across TRTLLM-Gen and CuTe Backends by @Vinnie6167 in https://github.com/flashinfer-ai/flashinfer/pull/3355

profiler: group perfetto traces by SM, one row per block by @Edenzzzz in https://github.com/flashinfer-ai/flashinfer/pull/3038

Make cute dsl mxfp8/nvfp4 quantizer bitwise exact by @zianglih in https://github.com/flashinfer-ai/flashinfer/pull/3387

Extend autotuner delay kernel length by @yanqinz2 in https://github.com/flashinfer-ai/flashinfer/pull/3373

Enable smaller tile N for SM100 Cute-DSL NVFP4 GEMM by @b8zhong in https://github.com/flashinfer-ai/flashinfer/pull/3403

test: align test_fmha_v2_prefill SM gating with is_sm12x_supported by @leonardHONG in https://github.com/flashinfer-ai/flashinfer/pull/3182

feat(autotuner): enable per-op autotune bypass for faster framework warmup by @qiching in https://github.com/flashinfer-ai/flashinfer/pull/3396

bench: Unify PDL behavior, add missing norm routines, and misc improvements by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/3435

feat(kda): add recurrent KDA decode kernel with per-K gating by @djmmoss in https://github.com/flashinfer-ai/flashinfer/pull/2572

Add mHC post mapping and pre big-fuse kernels by @jmydurant in https://github.com/flashinfer-ai/flashinfer/pull/3285

[Feat] Add num_heads < 128 support for mla decode kernel by @Observer007 in https://github.com/flashinfer-ai/flashinfer/pull/3309

Fix cross-warp race in checkpointing SSU kernel (mamba) by @ishovkun in https://github.com/flashinfer-ai/flashinfer/pull/3439

feat(trace): add check callbacks to trace templates by @yyihuang in https://github.com/flashinfer-ai/flashinfer/pull/3330

Add BGMV MoE CUDA kernels for multi-LoRA by @taehokim20 in https://github.com/flashinfer-ai/flashinfer/pull/3249

replace deprecated APIs: cute.make_fragment and cute.core.ThrMma by @brandon-yujie-sun in https://github.com/flashinfer-ai/flashinfer/pull/3430

[ci feat] Support /bot run tests/<dir> to scope CI test runs by @kahyunnam in https://github.com/flashinfer-ai/flashinfer/pull/3422

perf(attention): Speed up FP8 KV-cache prefill (FA2 BatchPrefill) by repacking K/V to BF16 in shared memory by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/3485

Ep api design -- Adding the actual code and tests by @Anerudhan in https://github.com/flashinfer-ai/flashinfer/pull/3453

Optimize mxfp8 quantization on sm100 by @IwakuraRein in https://github.com/flashinfer-ai/flashinfer/pull/3289

Fix silent no-op autotuning for cuBLAS bmm_fp8 and cuDNN bmm_fp8/mm_fp4 by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/3437

fix(quantization): nvfp4_quantize(backend='cuda') silently corrupts scale factors when global_scale is not float32 by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/3497

feat(moe): add SWIGLUSTEP activation to CUTLASS fused MoE by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/3492

Support LSE buffers in TRTLLM API by @saltyminty in https://github.com/flashinfer-ai/flashinfer/pull/3410

NFC: replace deprecated API: cute.make_fragment by @brandon-yujie-sun in https://github.com/flashinfer-ai/flashinfer/pull/3473

feat(moe): write routing_replay_out from custom routing kernels by @jdebache in https://github.com/flashinfer-ai/flashinfer/pull/3382

Add CuTe DSL NVFP4 quantization with 4over6 FP16 scoring by @zianglih in https://github.com/flashinfer-ai/flashinfer/pull/3448

fix intermittent exit 141 (SIGPIPE) in test resource summary by @yongwww in https://github.com/flashinfer-ai/flashinfer/pull/3498

add_cudnn_mxfp8 by @yanqinz2 in https://github.com/flashinfer-ai/flashinfer/pull/3489

fix: use routedScalingFactor to initialize mRouteScale by @yweng0828 in https://github.com/flashinfer-ai/flashinfer/pull/3499

fix: Simple approach to restore support for bias for fp4 block scale types by @djns99 in https://github.com/flashinfer-ai/flashinfer/pull/3416

docs(infra): document env vars, refresh SM list, fix stale paths in CLAUDE.md by @kangbintNV in https://github.com/flashinfer-ai/flashinfer/pull/3440

docs(quant/sampling/activation): canonicalize quantization.rst, stub fp4_quantization.rst by @kangbintNV in https://github.com/flashinfer-ai/flashinfer/pull/3447

docs(gemm): document GEMM + grouped_mm public surface; canonicalize aliases by @kangbintNV in https://github.com/flashinfer-ai/flashinfer/pull/3442

Optimize MoE routing top-k reduction and non-power-of-two sorting by @jiahanc in https://github.com/flashinfer-ai/flashinfer/pull/3476

[bugfix] TorchDistBackend.bcast uses global rank instead of local rank by @xuanyu-mistral in https://github.com/flashinfer-ai/flashinfer/pull/3418

docs(comm): structural RST refactor for MoeAlltoAll/DCP/Mixed comm by @kangbintNV in https://github.com/flashinfer-ai/flashinfer/pull/3445

[docs] Backfill missing docstrings and decorators across kernels by @kangbintNV in https://github.com/flashinfer-ai/flashinfer/pull/3456

docs(attention): backfill missing/stale Attention/POD/cuDNN/CuteDSL APIs; restore single_prefill_with_kv_cache_return_lse by @kangbintNV in https://github.com/flashinfer-ai/flashinfer/pull/3441

docs(moe): close fused_moe / trtllm_*_moe / CuteDSL MoE doc gaps by @kangbintNV in https://github.com/flashinfer-ai/flashinfer/pull/3443

docs(comm): NumPy-style docstrings + Deprecated leads for 21 STALE comm APIs (no decorator changes) by @kangbintNV in https://github.com/flashinfer-ai/flashinfer/pull/3444

bump version to 0.6.13 by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/3513

New Contributors

@richardmcai made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/3360

@jmydurant made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/3285

@taehokim20 made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/3249

@brandon-yujie-sun made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/3430

@xuanyu-mistral made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/3418

Full Changelog: https://github.com/flashinfer-ai/flashinfer/compare/v0.6.12rc3...v0.6.13

From the same repository

flashinfer-ai/flashinfer

flashinfer-pythonPyPIFlashInfer: Kernel Library for LLM Serving

0.6.13rc110 Jun 2026pre-releaseDeprecated

replace deprecated APIs: cute.make_fragment and cute.core.ThrMma by @brandon-yujie-sun in https://github.com/flashinfer-ai/flashinfer/pull/3430

What's Changed

Run high-likelihood OOM culprits separately, record memory usage and test duration for analysis by @dierksen in https://github.com/flashinfer-ai/flashinfer/pull/2961
fix(autotuner): differentiate file cache entries by runner specific kernel parameters by @qiching in https://github.com/flashinfer-ai/flashinfer/pull/3367
feat: integrate cute-dsl Blackwell GQA decode into BatchDecodeWithPagedKVCacheWrapper by @richardmcai in https://github.com/flashinfer-ai/flashinfer/pull/3360
Fix returning reference to temporary in moe gemm by @benbarsdell in https://github.com/flashinfer-ai/flashinfer/pull/3332
Enable compression of GPU device binaries by @benbarsdell in https://github.com/flashinfer-ai/flashinfer/pull/2949
feat: MNNVL Allreduce quant fusion and performance optimization by @timlee0212 in https://github.com/flashinfer-ai/flashinfer/pull/3385
fix(norm): widen address arithmetic to int64 for large contiguous inputs > 2**31 elements by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/3392
MLA Decode Autotuning Across TRTLLM-Gen and CuTe Backends by @Vinnie6167 in https://github.com/flashinfer-ai/flashinfer/pull/3355
profiler: group perfetto traces by SM, one row per block by @Edenzzzz in https://github.com/flashinfer-ai/flashinfer/pull/3038
Make cute dsl mxfp8/nvfp4 quantizer bitwise exact by @zianglih in https://github.com/flashinfer-ai/flashinfer/pull/3387
Extend autotuner delay kernel length by @yanqinz2 in https://github.com/flashinfer-ai/flashinfer/pull/3373
Enable smaller tile N for SM100 Cute-DSL NVFP4 GEMM by @b8zhong in https://github.com/flashinfer-ai/flashinfer/pull/3403
test: align test_fmha_v2_prefill SM gating with is_sm12x_supported by @leonardHONG in https://github.com/flashinfer-ai/flashinfer/pull/3182
feat(autotuner): enable per-op autotune bypass for faster framework warmup by @qiching in https://github.com/flashinfer-ai/flashinfer/pull/3396
bench: Unify PDL behavior, add missing norm routines, and misc improvements by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/3435
feat(kda): add recurrent KDA decode kernel with per-K gating by @djmmoss in https://github.com/flashinfer-ai/flashinfer/pull/2572
Add mHC post mapping and pre big-fuse kernels by @jmydurant in https://github.com/flashinfer-ai/flashinfer/pull/3285
[Feat] Add num_heads < 128 support for mla decode kernel by @Observer007 in https://github.com/flashinfer-ai/flashinfer/pull/3309
Fix cross-warp race in checkpointing SSU kernel (mamba) by @ishovkun in https://github.com/flashinfer-ai/flashinfer/pull/3439
feat(trace): add check callbacks to trace templates by @yyihuang in https://github.com/flashinfer-ai/flashinfer/pull/3330
Add BGMV MoE CUDA kernels for multi-LoRA by @taehokim20 in https://github.com/flashinfer-ai/flashinfer/pull/3249
replace deprecated APIs: cute.make_fragment and cute.core.ThrMma by @brandon-yujie-sun in https://github.com/flashinfer-ai/flashinfer/pull/3430
[ci feat] Support /bot run tests/<dir> to scope CI test runs by @kahyunnam in https://github.com/flashinfer-ai/flashinfer/pull/3422
perf(attention): Speed up FP8 KV-cache prefill (FA2 BatchPrefill) by repacking K/V to BF16 in shared memory by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/3485
Ep api design -- Adding the actual code and tests by @Anerudhan in https://github.com/flashinfer-ai/flashinfer/pull/3453
Optimize mxfp8 quantization on sm100 by @IwakuraRein in https://github.com/flashinfer-ai/flashinfer/pull/3289
Fix silent no-op autotuning for cuBLAS bmm_fp8 and cuDNN bmm_fp8/mm_fp4 by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/3437
fix(quantization): nvfp4_quantize(backend='cuda') silently corrupts scale factors when global_scale is not float32 by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/3497
feat(moe): add SWIGLUSTEP activation to CUTLASS fused MoE by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/3492
Support LSE buffers in TRTLLM API by @saltyminty in https://github.com/flashinfer-ai/flashinfer/pull/3410
NFC: replace deprecated API: cute.make_fragment by @brandon-yujie-sun in https://github.com/flashinfer-ai/flashinfer/pull/3473
feat(moe): write routing_replay_out from custom routing kernels by @jdebache in https://github.com/flashinfer-ai/flashinfer/pull/3382
Add CuTe DSL NVFP4 quantization with 4over6 FP16 scoring by @zianglih in https://github.com/flashinfer-ai/flashinfer/pull/3448
fix intermittent exit 141 (SIGPIPE) in test resource summary by @yongwww in https://github.com/flashinfer-ai/flashinfer/pull/3498
add_cudnn_mxfp8 by @yanqinz2 in https://github.com/flashinfer-ai/flashinfer/pull/3489
fix: use routedScalingFactor to initialize mRouteScale by @yweng0828 in https://github.com/flashinfer-ai/flashinfer/pull/3499
fix: Simple approach to restore support for bias for fp4 block scale types by @djns99 in https://github.com/flashinfer-ai/flashinfer/pull/3416
docs(infra): document env vars, refresh SM list, fix stale paths in CLAUDE.md by @kangbintNV in https://github.com/flashinfer-ai/flashinfer/pull/3440
docs(quant/sampling/activation): canonicalize quantization.rst, stub fp4_quantization.rst by @kangbintNV in https://github.com/flashinfer-ai/flashinfer/pull/3447
docs(gemm): document GEMM + grouped_mm public surface; canonicalize aliases by @kangbintNV in https://github.com/flashinfer-ai/flashinfer/pull/3442
Optimize MoE routing top-k reduction and non-power-of-two sorting by @jiahanc in https://github.com/flashinfer-ai/flashinfer/pull/3476
[bugfix] TorchDistBackend.bcast uses global rank instead of local rank by @xuanyu-mistral in https://github.com/flashinfer-ai/flashinfer/pull/3418
docs(comm): structural RST refactor for MoeAlltoAll/DCP/Mixed comm by @kangbintNV in https://github.com/flashinfer-ai/flashinfer/pull/3445
[docs] Backfill missing docstrings and decorators across kernels by @kangbintNV in https://github.com/flashinfer-ai/flashinfer/pull/3456
docs(attention): backfill missing/stale Attention/POD/cuDNN/CuteDSL APIs; restore single_prefill_with_kv_cache_return_lse by @kangbintNV in https://github.com/flashinfer-ai/flashinfer/pull/3441
docs(moe): close fused_moe / trtllm_*_moe / CuteDSL MoE doc gaps by @kangbintNV in https://github.com/flashinfer-ai/flashinfer/pull/3443
docs(comm): NumPy-style docstrings + Deprecated leads for 21 STALE comm APIs (no decorator changes) by @kangbintNV in https://github.com/flashinfer-ai/flashinfer/pull/3444
bump version to 0.6.13 by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/3513

New Contributors

@richardmcai made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/3360
@jmydurant made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/3285
@taehokim20 made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/3249
@brandon-yujie-sun made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/3430
@xuanyu-mistral made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/3418

Full Changelog: https://github.com/flashinfer-ai/flashinfer/compare/v0.6.12rc3...v0.6.13rc1

Open source →

0.6.1229 May 2026Deprecated

fix deprecation warnings from cute-dsl by @b8zhong in https://github.com/flashinfer-ai/flashinfer/pull/3333

What's Changed

Loosened trtllm_ragged_attention_deepseek shape assertion by @nvjullin in https://github.com/flashinfer-ai/flashinfer/pull/3064
Update moe gemm by @IwakuraRein in https://github.com/flashinfer-ai/flashinfer/pull/3239
perf: optimize per-token nvfp4 quantization kernel. by @IwakuraRein in https://github.com/flashinfer-ai/flashinfer/pull/3237
build: add sccache-backed jit-cache builds and AOT diagnostics by @dierksen in https://github.com/flashinfer-ai/flashinfer/pull/3205
non-override tactic control by @yanqinz2 in https://github.com/flashinfer-ai/flashinfer/pull/3260
ci(jit-cache): limit sm110 builds to aarch64 by @dierksen in https://github.com/flashinfer-ai/flashinfer/pull/3275
feat(moe): add SM120 W4A16 b12x kernels by @lukealonso in https://github.com/flashinfer-ai/flashinfer/pull/3271
Add dynamic tokens-per-page TRTLLM-GEN GQA kernels by @PerkzZheng in https://github.com/flashinfer-ai/flashinfer/pull/3259
fix(cute_dsl/moe): unbias autotuner profiling for tile_size enumeration by @leejnau in https://github.com/flashinfer-ai/flashinfer/pull/3252
Support Kimi K2.5 H64 CuTe DSL MLA decode by @saltyminty in https://github.com/flashinfer-ai/flashinfer/pull/3235
feat: FP8 output support for CUTLASS MLA paged attention by @carlyou in https://github.com/flashinfer-ai/flashinfer/pull/2779
fix(jit): propagate -DNDEBUG to host-side cflags by @arpera in https://github.com/flashinfer-ai/flashinfer/pull/3278
feat: add SM120 fmha_v2 kernels to AOT pip wheel builds by @blake-snc in https://github.com/flashinfer-ai/flashinfer/pull/2885
bench(moe_deepseek): fix moe benchmark (supersedes #2886) by @leejnau in https://github.com/flashinfer-ai/flashinfer/pull/3292
fix(gdn_decode): widen pool indices to Int64 to prevent int32 element-offset overflow by @vadiklyutiy in https://github.com/flashinfer-ai/flashinfer/pull/3230
[chore] Add guard to blackwell GDN prefill by @jiahanc in https://github.com/flashinfer-ai/flashinfer/pull/3267
fix: remove over-strict K%4 assert in get_shuffle_matrix_sf_a_row_indices by @jimmyzho in https://github.com/flashinfer-ai/flashinfer/pull/3163
ci: isolate nightly package tests from source tree by @dierksen in https://github.com/flashinfer-ai/flashinfer/pull/3274
Fix [Spark unit test CI]: defer torch._dynamo.disable to avoid import-time crash in CI by @kahyunnam in https://github.com/flashinfer-ai/flashinfer/pull/3290
bench(moe_deepseek): scope autotune(True) to pre-warm only by @leejnau in https://github.com/flashinfer-ai/flashinfer/pull/3301
Improved simple mamba SSU kernel by @ishovkun in https://github.com/flashinfer-ai/flashinfer/pull/2962
add cuda tile dependency for cuda 13.0 by @nv-yunzheq in https://github.com/flashinfer-ai/flashinfer/pull/3305
[Fix] Fix XQA V tile reading from wrong page when nbVItersPerXIter > 1 by @qsang-nv in https://github.com/flashinfer-ai/flashinfer/pull/3022
fix: MNNVL Allreduce uses bitwise sentinel checking to avoid subnormal value issue (#3053) by @timlee0212 in https://github.com/flashinfer-ai/flashinfer/pull/3304
Fix: remove nvfp4 llama4 blocker by @IwakuraRein in https://github.com/flashinfer-ai/flashinfer/pull/3313
[chore] add mamba codeowners list by @jimmyzho in https://github.com/flashinfer-ai/flashinfer/pull/3318
Modify release deletion command in workflow by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/3307
Add to code owners by @dhiraj113 in https://github.com/flashinfer-ai/flashinfer/pull/3326
feat: Add CuTe DSL grouped-gemm + combine fusion support by @nvcastet in https://github.com/flashinfer-ai/flashinfer/pull/2944
fix(gdn): allow importing gdn_decode without a CUDA device by @kahyunnam in https://github.com/flashinfer-ai/flashinfer/pull/3293
feat: enable glm5 router gemm by @b8zhong in https://github.com/flashinfer-ai/flashinfer/pull/3185
fix(fmha_v2): fix FP8 V-scratch pipeline and varlen scheduler on SM90 by @jimmyzho in https://github.com/flashinfer-ai/flashinfer/pull/3276
fix typo llama routing issue in trtllm-gen moe by @IwakuraRein in https://github.com/flashinfer-ai/flashinfer/pull/3303
feat(logging,trace): cuda-graph-compatible level-5/10 logging + fi_trace template additions/fixes by @yyihuang in https://github.com/flashinfer-ai/flashinfer/pull/3172
Use cudnn 9.23 new API to query workspace with override shape by @yanqinz2 in https://github.com/flashinfer-ai/flashinfer/pull/3291
feat: Expose unpacked topk weights for routed moe (fp4) by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/2425
Reland support lse in trtllm paged attn kernels by @murphymatt in https://github.com/flashinfer-ai/flashinfer/pull/3116
fix(CI unit tests, cute_dsl, spark): set USER env var before torch._dynamo import for unmapped UIDs by @kahyunnam in https://github.com/flashinfer-ai/flashinfer/pull/3314
feat(trace): embed runnable init() in every TraceTemplate by @yyihuang in https://github.com/flashinfer-ai/flashinfer/pull/3221
feat(cute_dsl/moe): deterministic balanced autotune profile inputs by @leejnau in https://github.com/flashinfer-ai/flashinfer/pull/3286
feat(cute_dsl/moe): add moe_output_memset_inplace dense memset wrapper by @leejnau in https://github.com/flashinfer-ai/flashinfer/pull/3328
Fix/3170 dense blockscaled sm12x by @leonardHONG in https://github.com/flashinfer-ai/flashinfer/pull/3180
test: enable bmm_mxfp8 cutlass backend coverage on SM12x by @leonardHONG in https://github.com/flashinfer-ai/flashinfer/pull/3183
Ep api design - Build Infra dependencies by @Anerudhan in https://github.com/flashinfer-ai/flashinfer/pull/3315
[feat] Add gemma RMS AR fusion by @jiahanc in https://github.com/flashinfer-ai/flashinfer/pull/3322
checkpointing_ssu kernel: fused replay + conditional state-write for Mamba2 by @ishovkun in https://github.com/flashinfer-ai/flashinfer/pull/3324
Ameyn/gdn bf16 dispatcher and 4d pool by @ameynaik-hub in https://github.com/flashinfer-ai/flashinfer/pull/3268
Update trtllm FMHA cubins by @djmmoss in https://github.com/flashinfer-ai/flashinfer/pull/3317
fix(trace): repair TGV and XQA MLA reference tests by @yyihuang in https://github.com/flashinfer-ai/flashinfer/pull/3365
feat: Add 8x4 swizzle layout support to MXFP4 and MXFP8 CuTe-DSL kernels by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/3357
Add AGENTS.md shim by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/3342
Add list_api script by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/3341
Support 4over6 nvfp4 for quantizer and fused MoE by @zianglih in https://github.com/flashinfer-ai/flashinfer/pull/3264
Add DeepSeek V4 sparse MLA TRTLLM-GEN kernels by @PerkzZheng in https://github.com/flashinfer-ai/flashinfer/pull/3269
Reject EP configurations in b12x MoE with a clear error by @kahyunnam in https://github.com/flashinfer-ai/flashinfer/pull/3302
fix(cute_dsl): avoid MoE wrapper runner reference cycle by @leejnau in https://github.com/flashinfer-ai/flashinfer/pull/3340
feat: Add support for LoRa delta in MOE mxint4 x bf16, MXFP8 & BF16 to trtllm backend by @djns99 in https://github.com/flashinfer-ai/flashinfer/pull/3153
Restore monolithic CuTe-DSL MLA decode alongside modular, gated by cute_dsl_impl= by @pgera in https://github.com/flashinfer-ai/flashinfer/pull/3296
feat: RMSNorm + RoPE fusion for WAN: flashinfer.diffusion_ops.fused_qk_rmsnorm_rope by @kahyunnam in https://github.com/flashinfer-ai/flashinfer/pull/3148
fix deprecation warnings from cute-dsl by @b8zhong in https://github.com/flashinfer-ai/flashinfer/pull/3333
feat(cute_dsl/moe): re-enable use_cold_l2_cache in CuteDslMoEWrapper TuningConfig by @leejnau in https://github.com/flashinfer-ai/flashinfer/pull/3384
Add torch.compile-compatible custom op for fp4_quantize by @Kh4L in https://github.com/flashinfer-ai/flashinfer/pull/3081
Replace SM120 W4A16 MoE kernels by @lukealonso in https://github.com/flashinfer-ai/flashinfer/pull/3336
bump version to 0.6.12 by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/3388

New Contributors

@carlyou made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2779
@nvcastet made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2944
@Kh4L made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/3081

Full Changelog: https://github.com/flashinfer-ai/flashinfer/compare/v0.6.11rc1...v0.6.12

Open source →

0.6.12rc126 May 2026pre-releaseDeprecated

fix deprecation warnings from cute-dsl by @b8zhong in https://github.com/flashinfer-ai/flashinfer/pull/3333

What's Changed

Loosened trtllm_ragged_attention_deepseek shape assertion by @nvjullin in https://github.com/flashinfer-ai/flashinfer/pull/3064
Update moe gemm by @IwakuraRein in https://github.com/flashinfer-ai/flashinfer/pull/3239
perf: optimize per-token nvfp4 quantization kernel. by @IwakuraRein in https://github.com/flashinfer-ai/flashinfer/pull/3237
build: add sccache-backed jit-cache builds and AOT diagnostics by @dierksen in https://github.com/flashinfer-ai/flashinfer/pull/3205
non-override tactic control by @yanqinz2 in https://github.com/flashinfer-ai/flashinfer/pull/3260
ci(jit-cache): limit sm110 builds to aarch64 by @dierksen in https://github.com/flashinfer-ai/flashinfer/pull/3275
feat(moe): add SM120 W4A16 b12x kernels by @lukealonso in https://github.com/flashinfer-ai/flashinfer/pull/3271
Add dynamic tokens-per-page TRTLLM-GEN GQA kernels by @PerkzZheng in https://github.com/flashinfer-ai/flashinfer/pull/3259
fix(cute_dsl/moe): unbias autotuner profiling for tile_size enumeration by @leejnau in https://github.com/flashinfer-ai/flashinfer/pull/3252
Support Kimi K2.5 H64 CuTe DSL MLA decode by @saltyminty in https://github.com/flashinfer-ai/flashinfer/pull/3235
feat: FP8 output support for CUTLASS MLA paged attention by @carlyou in https://github.com/flashinfer-ai/flashinfer/pull/2779
fix(jit): propagate -DNDEBUG to host-side cflags by @arpera in https://github.com/flashinfer-ai/flashinfer/pull/3278
feat: add SM120 fmha_v2 kernels to AOT pip wheel builds by @blake-snc in https://github.com/flashinfer-ai/flashinfer/pull/2885
bench(moe_deepseek): fix moe benchmark (supersedes #2886) by @leejnau in https://github.com/flashinfer-ai/flashinfer/pull/3292
fix(gdn_decode): widen pool indices to Int64 to prevent int32 element-offset overflow by @vadiklyutiy in https://github.com/flashinfer-ai/flashinfer/pull/3230
[chore] Add guard to blackwell GDN prefill by @jiahanc in https://github.com/flashinfer-ai/flashinfer/pull/3267
fix: remove over-strict K%4 assert in get_shuffle_matrix_sf_a_row_indices by @jimmyzho in https://github.com/flashinfer-ai/flashinfer/pull/3163
ci: isolate nightly package tests from source tree by @dierksen in https://github.com/flashinfer-ai/flashinfer/pull/3274
Fix [Spark unit test CI]: defer torch._dynamo.disable to avoid import-time crash in CI by @kahyunnam in https://github.com/flashinfer-ai/flashinfer/pull/3290
bench(moe_deepseek): scope autotune(True) to pre-warm only by @leejnau in https://github.com/flashinfer-ai/flashinfer/pull/3301
Improved simple mamba SSU kernel by @ishovkun in https://github.com/flashinfer-ai/flashinfer/pull/2962
add cuda tile dependency for cuda 13.0 by @nv-yunzheq in https://github.com/flashinfer-ai/flashinfer/pull/3305
[Fix] Fix XQA V tile reading from wrong page when nbVItersPerXIter > 1 by @qsang-nv in https://github.com/flashinfer-ai/flashinfer/pull/3022
fix: MNNVL Allreduce uses bitwise sentinel checking to avoid subnormal value issue (#3053) by @timlee0212 in https://github.com/flashinfer-ai/flashinfer/pull/3304
Fix: remove nvfp4 llama4 blocker by @IwakuraRein in https://github.com/flashinfer-ai/flashinfer/pull/3313
[chore] add mamba codeowners list by @jimmyzho in https://github.com/flashinfer-ai/flashinfer/pull/3318
Modify release deletion command in workflow by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/3307
Add to code owners by @dhiraj113 in https://github.com/flashinfer-ai/flashinfer/pull/3326
feat: Add CuTe DSL grouped-gemm + combine fusion support by @nvcastet in https://github.com/flashinfer-ai/flashinfer/pull/2944
fix(gdn): allow importing gdn_decode without a CUDA device by @kahyunnam in https://github.com/flashinfer-ai/flashinfer/pull/3293
feat: enable glm5 router gemm by @b8zhong in https://github.com/flashinfer-ai/flashinfer/pull/3185
fix(fmha_v2): fix FP8 V-scratch pipeline and varlen scheduler on SM90 by @jimmyzho in https://github.com/flashinfer-ai/flashinfer/pull/3276
fix typo llama routing issue in trtllm-gen moe by @IwakuraRein in https://github.com/flashinfer-ai/flashinfer/pull/3303
feat(logging,trace): cuda-graph-compatible level-5/10 logging + fi_trace template additions/fixes by @yyihuang in https://github.com/flashinfer-ai/flashinfer/pull/3172
Use cudnn 9.23 new API to query workspace with override shape by @yanqinz2 in https://github.com/flashinfer-ai/flashinfer/pull/3291
feat: Expose unpacked topk weights for routed moe (fp4) by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/2425
Reland support lse in trtllm paged attn kernels by @murphymatt in https://github.com/flashinfer-ai/flashinfer/pull/3116
fix(CI unit tests, cute_dsl, spark): set USER env var before torch._dynamo import for unmapped UIDs by @kahyunnam in https://github.com/flashinfer-ai/flashinfer/pull/3314
feat(trace): embed runnable init() in every TraceTemplate by @yyihuang in https://github.com/flashinfer-ai/flashinfer/pull/3221
feat(cute_dsl/moe): deterministic balanced autotune profile inputs by @leejnau in https://github.com/flashinfer-ai/flashinfer/pull/3286
feat(cute_dsl/moe): add moe_output_memset_inplace dense memset wrapper by @leejnau in https://github.com/flashinfer-ai/flashinfer/pull/3328
Fix/3170 dense blockscaled sm12x by @leonardHONG in https://github.com/flashinfer-ai/flashinfer/pull/3180
test: enable bmm_mxfp8 cutlass backend coverage on SM12x by @leonardHONG in https://github.com/flashinfer-ai/flashinfer/pull/3183
Ep api design - Build Infra dependencies by @Anerudhan in https://github.com/flashinfer-ai/flashinfer/pull/3315
[feat] Add gemma RMS AR fusion by @jiahanc in https://github.com/flashinfer-ai/flashinfer/pull/3322
checkpointing_ssu kernel: fused replay + conditional state-write for Mamba2 by @ishovkun in https://github.com/flashinfer-ai/flashinfer/pull/3324
Ameyn/gdn bf16 dispatcher and 4d pool by @ameynaik-hub in https://github.com/flashinfer-ai/flashinfer/pull/3268
Update trtllm FMHA cubins by @djmmoss in https://github.com/flashinfer-ai/flashinfer/pull/3317
fix(trace): repair TGV and XQA MLA reference tests by @yyihuang in https://github.com/flashinfer-ai/flashinfer/pull/3365
feat: Add 8x4 swizzle layout support to MXFP4 and MXFP8 CuTe-DSL kernels by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/3357
Add AGENTS.md shim by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/3342
Add list_api script by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/3341
Support 4over6 nvfp4 for quantizer and fused MoE by @zianglih in https://github.com/flashinfer-ai/flashinfer/pull/3264
Add DeepSeek V4 sparse MLA TRTLLM-GEN kernels by @PerkzZheng in https://github.com/flashinfer-ai/flashinfer/pull/3269
Reject EP configurations in b12x MoE with a clear error by @kahyunnam in https://github.com/flashinfer-ai/flashinfer/pull/3302
fix(cute_dsl): avoid MoE wrapper runner reference cycle by @leejnau in https://github.com/flashinfer-ai/flashinfer/pull/3340
feat: Add support for LoRa delta in MOE mxint4 x bf16, MXFP8 & BF16 to trtllm backend by @djns99 in https://github.com/flashinfer-ai/flashinfer/pull/3153
Restore monolithic CuTe-DSL MLA decode alongside modular, gated by cute_dsl_impl= by @pgera in https://github.com/flashinfer-ai/flashinfer/pull/3296
feat: RMSNorm + RoPE fusion for WAN: flashinfer.diffusion_ops.fused_qk_rmsnorm_rope by @kahyunnam in https://github.com/flashinfer-ai/flashinfer/pull/3148
fix deprecation warnings from cute-dsl by @b8zhong in https://github.com/flashinfer-ai/flashinfer/pull/3333
feat(cute_dsl/moe): re-enable use_cold_l2_cache in CuteDslMoEWrapper TuningConfig by @leejnau in https://github.com/flashinfer-ai/flashinfer/pull/3384
Add torch.compile-compatible custom op for fp4_quantize by @Kh4L in https://github.com/flashinfer-ai/flashinfer/pull/3081
Replace SM120 W4A16 MoE kernels by @lukealonso in https://github.com/flashinfer-ai/flashinfer/pull/3336
bump version to 0.6.12 by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/3388

New Contributors

@carlyou made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2779
@nvcastet made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2944
@Kh4L made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/3081

Full Changelog: https://github.com/flashinfer-ai/flashinfer/compare/v0.6.11rc1...v0.6.12rc1

Open source →

0.6.119 May 2026

trying this one character fix for main branch by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/3213

What's Changed

trying this one character fix for main branch by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/3213
Add git submodule update to build_backend.py by @kahyunnam in https://github.com/flashinfer-ai/flashinfer/pull/3190
fix(cute_dsl/moe): correct tile_size=256 gemm2 tactic enumeration by @leejnau in https://github.com/flashinfer-ai/flashinfer/pull/3171
Fix trace-bmm-fp8 test: B should be K-major for subword types by @xrq-phys in https://github.com/flashinfer-ai/flashinfer/pull/3184
feat: Add DiT-oriented kernels where Qk (Bmm1) type can be reinterpreted into Int8 or BFloat16 by @xrq-phys in https://github.com/flashinfer-ai/flashinfer/pull/2711
[fmha-v2] Support HND and NHD paged KV cache layouts with conditional stride handling by @zhou-yuxin in https://github.com/flashinfer-ai/flashinfer/pull/2799
[feat] Trtllm-gen Per-token Nvfp4 MoE by @IwakuraRein in https://github.com/flashinfer-ai/flashinfer/pull/3027
feat: Add cuBLASLt backend for mm_bf16 and enable multi-tactic autotuning for FP8/MXFP8 runners by @vadiklyutiy in https://github.com/flashinfer-ai/flashinfer/pull/2914
trtllm non causal support by @saltyminty in https://github.com/flashinfer-ai/flashinfer/pull/3020
feat: DiT layer norm fusions for WAN: flashinfer.diffusion_ops by @kahyunnam in https://github.com/flashinfer-ai/flashinfer/pull/3157
Refactor Part 3- Add block-per-token feature in the customized routing method by @ChristinaZ in https://github.com/flashinfer-ai/flashinfer/pull/3166
fix(cute_dsl/moe): correct off-by-one in get_max_num_tiles to match TRT-LLM by @leejnau in https://github.com/flashinfer-ai/flashinfer/pull/3198
Yanqinz/fix cudnn sm120 nan by @yanqinz2 in https://github.com/flashinfer-ai/flashinfer/pull/3192
bump version to 0.6.10 by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/3179
fix: align is_sm120f_supported with SM12x family semantics by @leonardHONG in https://github.com/flashinfer-ai/flashinfer/pull/3175
fix: add sm_121 to TMEM column fallback map by @leonardHONG in https://github.com/flashinfer-ai/flashinfer/pull/3173
Include TinyGEMM into BF16 autotuner by @askliar in https://github.com/flashinfer-ai/flashinfer/pull/3203
fix(dcp_alltoall): require MNNVL workspace, drop broken plain-memory path by @davidjpyu in https://github.com/flashinfer-ai/flashinfer/pull/3210
Integrate CUTLASS Small Tile N Blockscaled GEMMs/Grouped GEMMs for SM120 and SM121 by @depaulmillz in https://github.com/flashinfer-ai/flashinfer/pull/3152
Fix bf16 cudnn override-shape test call signature by @Vinnie6167 in https://github.com/flashinfer-ai/flashinfer/pull/3215
Ameyn/wide vec t1 by @ameynaik-hub in https://github.com/flashinfer-ai/flashinfer/pull/3147
[Perf] Add FMHAv2 to flashinfer_benchmark.py and eliminate unnecessary H2D by @jimmyzho in https://github.com/flashinfer-ai/flashinfer/pull/2841
Fix multi-instances using same random seed by @guyuankan in https://github.com/flashinfer-ai/flashinfer/pull/3102
add_grouped_mm_operation_directory by @yanqinz2 in https://github.com/flashinfer-ai/flashinfer/pull/3052
Support Allreduce + Norm + Per-token Group Fp8 Quant Fusion by @wzhao18 in https://github.com/flashinfer-ai/flashinfer/pull/3059
[Bugfix] Fix fused MoE autotuning correctness issues by filtering clusterDimZ by @wzhao18 in https://github.com/flashinfer-ai/flashinfer/pull/3227
fix: add jitter to cubin download backoff by @pluh-nv in https://github.com/flashinfer-ai/flashinfer/pull/3169
cute_dsl/moe: drop redundant Python-side moe_sort buffer init by @leejnau in https://github.com/flashinfer-ai/flashinfer/pull/3226
Support Sigmoid (sigmoid+topk) routing function by @EdalatiAli in https://github.com/flashinfer-ai/flashinfer/pull/2869
cute-dsl fmha prefill (cubin integration): remove front-padding, add attention_sink, and pdl support by @limin2021 in https://github.com/flashinfer-ai/flashinfer/pull/3181
fix(mla): widen page index to int64_t to avoid 32-bit overflow by @Tracin in https://github.com/flashinfer-ai/flashinfer/pull/3136
bump version to 0.6.11 by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/3245
fix(cute_dsl/moe): make autotuner bucket configuration adapt to runtime input by @leejnau in https://github.com/flashinfer-ai/flashinfer/pull/3216
Fix: skip git submodule update when submodules are already populated by @kahyunnam in https://github.com/flashinfer-ai/flashinfer/pull/3248
Fix 10 bugs in BF16 XQA MLA kernel for SM120/SM121 by @blake-snc in https://github.com/flashinfer-ai/flashinfer/pull/2689
Tweak grouped_mm api to make backend specific argument keyword-only by @yanqinz2 in https://github.com/flashinfer-ai/flashinfer/pull/3253
perf(moe): optimize SM120 b12x MoE short decode by @lukealonso in https://github.com/flashinfer-ai/flashinfer/pull/3193
feat: Enable FP8 (E4M3/E5M2) in concat_mla_k for optimize long-context prefill performance and refactor type dispatch for BF16/FP16 by @qiching in https://github.com/flashinfer-ai/flashinfer/pull/3129
fix hang in allreduce comms in SGL by @b8zhong in https://github.com/flashinfer-ai/flashinfer/pull/3247
fix(sm12x): fix micro-kernel workspace sizing when routed_rows > num_local_experts by @meena-at-work in https://github.com/flashinfer-ai/flashinfer/pull/3191
Issue #3047: Handle empty KV in MLA chunked-prefill by @saltyminty in https://github.com/flashinfer-ai/flashinfer/pull/3251
Cutlass dsl 4.5 bump by @kahyunnam in https://github.com/flashinfer-ai/flashinfer/pull/3246

New Contributors

@xrq-phys made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/3184
@zhou-yuxin made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2799
@leonardHONG made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/3175
@guyuankan made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/3102
@pluh-nv made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/3169
@EdalatiAli made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2869
@Tracin made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/3136
@lukealonso made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/3193

Full Changelog: https://github.com/flashinfer-ai/flashinfer/compare/v0.6.10rc1...v0.6.11

Open source →

0.6.104 May 2026

Vendor CCCL v3.3.2 from GitHub instead of relying on CTK-bundled copy by @kahyunnam in https://github.com/flashinfer-ai/flashinfer/pull/3091

What's Changed

Vendor CCCL v3.3.2 from GitHub instead of relying on CTK-bundled copy by @kahyunnam in https://github.com/flashinfer-ai/flashinfer/pull/3091
[Fmha] Add head_dim=512 support for trtllm attention kernels by @djmmoss in https://github.com/flashinfer-ai/flashinfer/pull/2959
perf: optimize MXFP4xBF16 & INT4xFP8 CUTLASS MoE backend for SM90 by @samuellees in https://github.com/flashinfer-ai/flashinfer/pull/3084
Add support for the combinations of allreduce, allgather, and reducescatter by @jinyangyuan-nvidia in https://github.com/flashinfer-ai/flashinfer/pull/2563
[Fmha] update trtllm-gen FMHA cubins and sync headers for context SWA fix by @PerkzZheng in https://github.com/flashinfer-ai/flashinfer/pull/3089
Report unit test files with no result by @dierksen in https://github.com/flashinfer-ai/flashinfer/pull/3105
autotuner: check cache before synthesizing profile input tensors by @leejnau in https://github.com/flashinfer-ai/flashinfer/pull/3126
fix: pass skip_softmax_threshold_scale_factor to prefill wrapper in test by @PerkzZheng in https://github.com/flashinfer-ai/flashinfer/pull/3154
feat: Add DCP All-to-All kernel for context-parallel attention reduction by @davidjpyu in https://github.com/flashinfer-ai/flashinfer/pull/2951
perf(autotuner): replace power-of-2 token buckets with hybrid spacing & fix missing routing_replay_out arg by @StudyingShao in https://github.com/flashinfer-ai/flashinfer/pull/3115
feat: Integrate CuTe DSL FMHA prefill kernels by loading cubin by @limin2021 in https://github.com/flashinfer-ai/flashinfer/pull/3039
unifying all reduce memory allocation for single-node and multi-node nvlink by @Amir-19 in https://github.com/flashinfer-ai/flashinfer/pull/2955
Add all-gather matmul by @kwen2501 in https://github.com/flashinfer-ai/flashinfer/pull/2665
Add examples of calling FlashInfer from JAX via jax-tvm-ffi by @katjasrz in https://github.com/flashinfer-ai/flashinfer/pull/3092
bump version to 0.6.9 by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/3123
chore: Address non-blocking review feedback for #3051 / #3080 by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/3128
perf: Add no-bias path for tinygemm_bf16 by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/3151
feat: Add row_starts and dsa_graph_safe to topk by @zianglih in https://github.com/flashinfer-ai/flashinfer/pull/3133
fix: guard MXFP8 fc1 weight shape check for non-gated activations by @ianliuy in https://github.com/flashinfer-ai/flashinfer/pull/3082
[fix] fix blackwell gdn accuracy issue by @Observer007 in https://github.com/flashinfer-ai/flashinfer/pull/3156
fix: fix OOB issue for vLLM by @nv-yunzheq in https://github.com/flashinfer-ai/flashinfer/pull/2762
Build mnnvl_moe_alltoall with logger and stringUtils by @tiran in https://github.com/flashinfer-ai/flashinfer/pull/2807
CICD bug fix: ensure data/ symlinks exist before jit-cache AOT compilation by @kahyunnam in https://github.com/flashinfer-ai/flashinfer/pull/3158
feat: add get flashinfer-trace interface .fi_trace by @yyihuang in https://github.com/flashinfer-ai/flashinfer/pull/2931
fix(gdn): use physical SM count for SM100 persistent prefill kernel by @arpera in https://github.com/flashinfer-ai/flashinfer/pull/3155
fix(gdn): address remaining CodeRabbit feedback from #3001 by @arpera in https://github.com/flashinfer-ai/flashinfer/pull/3165
Support NVFP4 KV for prefill and batch attention kernels by @Tom-Zheng in https://github.com/flashinfer-ai/flashinfer/pull/3097
fix: skip version check for editable/source installs (0.0.0+unknown) by @ianliuy in https://github.com/flashinfer-ai/flashinfer/pull/3061

New Contributors

@davidjpyu made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2951
@StudyingShao made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/3115
@kwen2501 made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2665
@katjasrz made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/3092
@ianliuy made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/3082
@arpera made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/3155

Full Changelog: https://github.com/flashinfer-ai/flashinfer/compare/v0.6.9rc1...v0.6.10

Open source →

0.6.10rc130 Apr 2026pre-release

Vendor CCCL v3.3.2 from GitHub instead of relying on CTK-bundled copy by @kahyunnam in https://github.com/flashinfer-ai/flashinfer/pull/3091

What's Changed

Vendor CCCL v3.3.2 from GitHub instead of relying on CTK-bundled copy by @kahyunnam in https://github.com/flashinfer-ai/flashinfer/pull/3091
[Fmha] Add head_dim=512 support for trtllm attention kernels by @djmmoss in https://github.com/flashinfer-ai/flashinfer/pull/2959
perf: optimize MXFP4xBF16 & INT4xFP8 CUTLASS MoE backend for SM90 by @samuellees in https://github.com/flashinfer-ai/flashinfer/pull/3084
Add support for the combinations of allreduce, allgather, and reducescatter by @jinyangyuan-nvidia in https://github.com/flashinfer-ai/flashinfer/pull/2563
[Fmha] update trtllm-gen FMHA cubins and sync headers for context SWA fix by @PerkzZheng in https://github.com/flashinfer-ai/flashinfer/pull/3089
Report unit test files with no result by @dierksen in https://github.com/flashinfer-ai/flashinfer/pull/3105
autotuner: check cache before synthesizing profile input tensors by @leejnau in https://github.com/flashinfer-ai/flashinfer/pull/3126
fix: pass skip_softmax_threshold_scale_factor to prefill wrapper in test by @PerkzZheng in https://github.com/flashinfer-ai/flashinfer/pull/3154
feat: Add DCP All-to-All kernel for context-parallel attention reduction by @davidjpyu in https://github.com/flashinfer-ai/flashinfer/pull/2951
perf(autotuner): replace power-of-2 token buckets with hybrid spacing & fix missing routing_replay_out arg by @StudyingShao in https://github.com/flashinfer-ai/flashinfer/pull/3115
feat: Integrate CuTe DSL FMHA prefill kernels by loading cubin by @limin2021 in https://github.com/flashinfer-ai/flashinfer/pull/3039
unifying all reduce memory allocation for single-node and multi-node nvlink by @Amir-19 in https://github.com/flashinfer-ai/flashinfer/pull/2955
Add all-gather matmul by @kwen2501 in https://github.com/flashinfer-ai/flashinfer/pull/2665
Add examples of calling FlashInfer from JAX via jax-tvm-ffi by @katjasrz in https://github.com/flashinfer-ai/flashinfer/pull/3092
bump version to 0.6.9 by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/3123
chore: Address non-blocking review feedback for #3051 / #3080 by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/3128
perf: Add no-bias path for tinygemm_bf16 by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/3151
feat: Add row_starts and dsa_graph_safe to topk by @zianglih in https://github.com/flashinfer-ai/flashinfer/pull/3133
fix: guard MXFP8 fc1 weight shape check for non-gated activations by @ianliuy in https://github.com/flashinfer-ai/flashinfer/pull/3082
[fix] fix blackwell gdn accuracy issue by @Observer007 in https://github.com/flashinfer-ai/flashinfer/pull/3156
fix: fix OOB issue for vLLM by @nv-yunzheq in https://github.com/flashinfer-ai/flashinfer/pull/2762
Build mnnvl_moe_alltoall with logger and stringUtils by @tiran in https://github.com/flashinfer-ai/flashinfer/pull/2807
CICD bug fix: ensure data/ symlinks exist before jit-cache AOT compilation by @kahyunnam in https://github.com/flashinfer-ai/flashinfer/pull/3158
feat: add get flashinfer-trace interface .fi_trace by @yyihuang in https://github.com/flashinfer-ai/flashinfer/pull/2931
fix(gdn): use physical SM count for SM100 persistent prefill kernel by @arpera in https://github.com/flashinfer-ai/flashinfer/pull/3155
fix(gdn): address remaining CodeRabbit feedback from #3001 by @arpera in https://github.com/flashinfer-ai/flashinfer/pull/3165
Support NVFP4 KV for prefill and batch attention kernels by @Tom-Zheng in https://github.com/flashinfer-ai/flashinfer/pull/3097
fix: skip version check for editable/source installs (0.0.0+unknown) by @ianliuy in https://github.com/flashinfer-ai/flashinfer/pull/3061

New Contributors

@davidjpyu made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2951
@StudyingShao made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/3115
@kwen2501 made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2665
@katjasrz made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/3092
@ianliuy made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/3082
@arpera made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/3155

Full Changelog: https://github.com/flashinfer-ai/flashinfer/compare/v0.6.9rc1...v0.6.10rc1

Open source →

0.6.924 Apr 2026

feat: Add backend="b12x" for mm_fp4 on SM120 by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/3051

What's Changed

feat: Add backend="b12x" for mm_fp4 on SM120 by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/3051
docs: document MAX_JOBS env var and its interaction with FLASHINFER_N… by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/3060
PR #2772 might have introduced a device side compilation regression by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/3056
[feat] Add routing_replay_out support to MoE kernels and Python API by @TomerBN-Nvidia in https://github.com/flashinfer-ai/flashinfer/pull/3024
fused_moe: pre-filter SM89 tactics with zero occupancy on SM120 Blackwell (fix review feedback on #2764) by @aniskumar-nv in https://github.com/flashinfer-ai/flashinfer/pull/3032
feat: Add b12x CuTe DSL fused MoE for SM120 by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/3066
CuTe DSL FP4 GEMM Heuristic by @Vinnie6167 in https://github.com/flashinfer-ai/flashinfer/pull/2940
Support lse in trtllm paged attn kernels by @murphymatt in https://github.com/flashinfer-ai/flashinfer/pull/3058
Revert "Support lse in trtllm paged attn kernels" by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/3079
docs(gdn): document -1 padding index semantics for pool+indices path by @kaixih in https://github.com/flashinfer-ai/flashinfer/pull/3019
feat(gdn): separate input and output pool indices by @feldsherov in https://github.com/flashinfer-ai/flashinfer/pull/2905
[CICD fix] Adjust CICD MAX_JOBS to fix OOM on H100 tests by @kahyunnam in https://github.com/flashinfer-ai/flashinfer/pull/3078
Add qiching as code owner for autotuner files by @sricketts in https://github.com/flashinfer-ai/flashinfer/pull/3104
Route the missing parameter for trtllm_fp8_per_tensor_scale_moe_op by @pavanimajety in https://github.com/flashinfer-ai/flashinfer/pull/3094
Fix: Extend b12x FP4 GEMM support to SM121 (GB10/DGX Spark) by @meena-at-work in https://github.com/flashinfer-ai/flashinfer/pull/3113
Add parallel attention by @xueweilnvidia in https://github.com/flashinfer-ai/flashinfer/pull/2630
[feat] Faster topk algorithm by @Aalanli in https://github.com/flashinfer-ai/flashinfer/pull/3009
feat: Add b12x_fused_moe / B12xMoEWrapper SM120 APIs with micro kernel and ReLU2 by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/3080
[fmhav2] skip fp8 tests and add warning by @jimmyzho in https://github.com/flashinfer-ai/flashinfer/pull/3050
feat: implement configurable tie_break for filtered topk by @zianglih in https://github.com/flashinfer-ai/flashinfer/pull/3095
Add custom tuning buckets and rounding direction to autotune() by @vadiklyutiy in https://github.com/flashinfer-ai/flashinfer/pull/2958
[CuTe DSL] Fix FP8 MLA persistent perf regression and ProxyKind cu13 wheel breakage by @pgera in https://github.com/flashinfer-ai/flashinfer/pull/3132

New Contributors

@TomerBN-Nvidia made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/3024
@aniskumar-nv made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/3032
@Vinnie6167 made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2940
@meena-at-work made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/3113
@xueweilnvidia made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2630
@Aalanli made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/3009
@vadiklyutiy made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2958

Full Changelog: https://github.com/flashinfer-ai/flashinfer/compare/v0.6.8rc1...v0.6.9

Open source →

0.6.9rc123 Apr 2026pre-release

feat: Add backend="b12x" for mm_fp4 on SM120 by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/3051

What's Changed

feat: Add backend="b12x" for mm_fp4 on SM120 by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/3051
docs: document MAX_JOBS env var and its interaction with FLASHINFER_N… by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/3060
PR #2772 might have introduced a device side compilation regression by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/3056
[feat] Add routing_replay_out support to MoE kernels and Python API by @TomerBN-Nvidia in https://github.com/flashinfer-ai/flashinfer/pull/3024
fused_moe: pre-filter SM89 tactics with zero occupancy on SM120 Blackwell (fix review feedback on #2764) by @aniskumar-nv in https://github.com/flashinfer-ai/flashinfer/pull/3032
feat: Add b12x CuTe DSL fused MoE for SM120 by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/3066
CuTe DSL FP4 GEMM Heuristic by @Vinnie6167 in https://github.com/flashinfer-ai/flashinfer/pull/2940
Support lse in trtllm paged attn kernels by @murphymatt in https://github.com/flashinfer-ai/flashinfer/pull/3058
Revert "Support lse in trtllm paged attn kernels" by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/3079
docs(gdn): document -1 padding index semantics for pool+indices path by @kaixih in https://github.com/flashinfer-ai/flashinfer/pull/3019
feat(gdn): separate input and output pool indices by @feldsherov in https://github.com/flashinfer-ai/flashinfer/pull/2905
[CICD fix] Adjust CICD MAX_JOBS to fix OOM on H100 tests by @kahyunnam in https://github.com/flashinfer-ai/flashinfer/pull/3078
Add qiching as code owner for autotuner files by @sricketts in https://github.com/flashinfer-ai/flashinfer/pull/3104
Route the missing parameter for trtllm_fp8_per_tensor_scale_moe_op by @pavanimajety in https://github.com/flashinfer-ai/flashinfer/pull/3094
Fix: Extend b12x FP4 GEMM support to SM121 (GB10/DGX Spark) by @meena-at-work in https://github.com/flashinfer-ai/flashinfer/pull/3113
Add parallel attention by @xueweilnvidia in https://github.com/flashinfer-ai/flashinfer/pull/2630
[feat] Faster topk algorithm by @Aalanli in https://github.com/flashinfer-ai/flashinfer/pull/3009
feat: Add b12x_fused_moe / B12xMoEWrapper SM120 APIs with micro kernel and ReLU2 by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/3080
[fmhav2] skip fp8 tests and add warning by @jimmyzho in https://github.com/flashinfer-ai/flashinfer/pull/3050
feat: implement configurable tie_break for filtered topk by @zianglih in https://github.com/flashinfer-ai/flashinfer/pull/3095
Add custom tuning buckets and rounding direction to autotune() by @vadiklyutiy in https://github.com/flashinfer-ai/flashinfer/pull/2958
[CuTe DSL] Fix FP8 MLA persistent perf regression and ProxyKind cu13 wheel breakage by @pgera in https://github.com/flashinfer-ai/flashinfer/pull/3132

New Contributors

@TomerBN-Nvidia made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/3024
@aniskumar-nv made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/3032
@Vinnie6167 made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2940
@meena-at-work made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/3113
@xueweilnvidia made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2630
@Aalanli made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/3009
@vadiklyutiy made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2958

Full Changelog: https://github.com/flashinfer-ai/flashinfer/compare/v0.6.8rc1...v0.6.9rc1

Open source →

0.6.816 Apr 2026

Add to CODEOWNER by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/2875

What's Changed

Add to CODEOWNER by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/2875
fix: int32 overflow in trtllm_fp4_block_scale_moe causing "Unsupported hidden state scale shape" for EP32+ configs by @qiching in https://github.com/flashinfer-ai/flashinfer/pull/2853
feat: bump nvidia-cutlass-dsl to >=4.4.2 by @limin2021 in https://github.com/flashinfer-ai/flashinfer/pull/2833
fix: add cute dsl moe utils to AOT by @nv-yunzheq in https://github.com/flashinfer-ai/flashinfer/pull/2872
fix: fix cute dsl swap_ab tactic failure by @nv-yunzheq in https://github.com/flashinfer-ai/flashinfer/pull/2870
[gdn] support non-contiguous state for decoding by @ZJY0516 in https://github.com/flashinfer-ai/flashinfer/pull/2727
chore: fix the python dependency override by @yongwww in https://github.com/flashinfer-ai/flashinfer/pull/2651
backinteg: nvidia-nvshmem-cu12 3.6.5 seems broken by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/2893
Yanqinz/gemm cudnn autotune fix by @yanqinz2 in https://github.com/flashinfer-ai/flashinfer/pull/2863
feat: Add CuTe-DSL backend for NVFP4 quantization by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2838
Add cute dsl mla decode op by @limin2021 in https://github.com/flashinfer-ai/flashinfer/pull/2743
Support for MXFP4 and NVFP4 group GEMMs on GeForce and Spark by @depaulmillz in https://github.com/flashinfer-ai/flashinfer/pull/2738
feat: add pdl support for cute dsl mla decode kernel support by @Observer007 in https://github.com/flashinfer-ai/flashinfer/pull/2901
feat: expose swizzled_input_sf parameter for CUTLASS fused MOE by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/2330
fix: support fp32 logits for fp8_per_tensor and fp8_block by @yweng0828 in https://github.com/flashinfer-ai/flashinfer/pull/2534
Fix autotuner crash when input tensor is None by @he-yufeng in https://github.com/flashinfer-ai/flashinfer/pull/2756
Support in-place update for trtllm_fp8_block_scale_moe by @wzhao18 in https://github.com/flashinfer-ai/flashinfer/pull/2739
[fix] bugfix 2856: Fix pre-allocated out shape check in trtllm_batch_decode_with_kv_cache_mla for q_len_per_req > 1 by @qsang-nv in https://github.com/flashinfer-ai/flashinfer/pull/2876
PR auto-labelling by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/2827
fix test error regarding logits_types by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/2918
Use 6-hour timeout for flashinfer-jit-cache wheel build (release + nightly) by @yongwww in https://github.com/flashinfer-ai/flashinfer/pull/2880
fix: expose trigger_completion_at_end through unified API by @nv-yunzheq in https://github.com/flashinfer-ai/flashinfer/pull/2894
fix: clamp enable_pdl=True to False on SM < 90 to prevent PDL PTX on Ampere by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2928
feat: add Relu2 (squared ReLU) activation support in CUTLASS MoE backend by @askliar in https://github.com/flashinfer-ai/flashinfer/pull/2926
docker: upgrade cuDNN to latest version in CI install script by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2930
[NVIDIA] fix(jit): enable GDC for CUTLASS fused MoE PDL — prevent random crashes on SM12x by @johnnynunez in https://github.com/flashinfer-ai/flashinfer/pull/2913
fix: Fix autotuner crash on meta-device tensor in trtllm_fp4_block_scale_routed_moe by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2916
Yanqinz/dynamic shape unified api by @yanqinz2 in https://github.com/flashinfer-ai/flashinfer/pull/2910
doc: add CI triggering guide to CONTRIBUTING.md by @yongwww in https://github.com/flashinfer-ai/flashinfer/pull/2924
read real strides for kv and block scale by @sychen52 in https://github.com/flashinfer-ai/flashinfer/pull/2844
perf: Optimize CuTe-DSL fp4 and fp8 quantization kernels by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2904
fix: vectorize get_shuffle_matrix_a_row_indices to eliminate CPU contention by @youkaichao in https://github.com/flashinfer-ai/flashinfer/pull/2935
feat: implement deterministic topk by @jiangyinzuo in https://github.com/flashinfer-ai/flashinfer/pull/2661
feat(gdn): add BF16 state kernel with MTP support beyond T>4 with intermediate caching. by @ameynaik-hub in https://github.com/flashinfer-ai/flashinfer/pull/2679
ci: remove 1gpu label from H100 runner selector by @yongwww in https://github.com/flashinfer-ai/flashinfer/pull/2946
perf: Optimize GDN MTP decode kernel (v15) — eliminate ilp=1 fallback… by @ameynaik-hub in https://github.com/flashinfer-ai/flashinfer/pull/2842
feat: add MXFP8 GEMM support for SM120 by @samuellees in https://github.com/flashinfer-ai/flashinfer/pull/2902
fix: avoid re-downloading BMM export headers when flashinfer-cubin is installed by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/2903
test: xfail cuDNN FP8 prefill on Blackwell with CUDA <= 12.9 by @dierksen in https://github.com/flashinfer-ai/flashinfer/pull/2963
test: skip unsupported mm_mxfp8 configurations on SM12x by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2974
[Fmha] revert blackwell ultra optimization that causes deadlocks. by @PerkzZheng in https://github.com/flashinfer-ai/flashinfer/pull/2956
feat: SM121 (GB10) tile filtering and autotuner robustness by @askliar in https://github.com/flashinfer-ai/flashinfer/pull/2927
Mamba SSU: horizontal MTP kernel (+ DSTATE=96 support) by @ishovkun in https://github.com/flashinfer-ai/flashinfer/pull/2865
fix: use float instead of double in sampling binary search to avoid FP64 bottleneck on SM103 by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2945
Refactor the routing part by @ChristinaZ in https://github.com/flashinfer-ai/flashinfer/pull/2803
fix: snap weight_scale_vec_size to handle block_scale_interleave padding for SM120 by @samuellees in https://github.com/flashinfer-ai/flashinfer/pull/2898
Add filelock to ensure_symlink by @wzhao18 in https://github.com/flashinfer-ai/flashinfer/pull/2979
Update NVSHMEM interface to use NVSHMEM4Py instead of custom bindings by @benhg in https://github.com/flashinfer-ai/flashinfer/pull/2960
docs: document replay command in CLI reference by @ooooo-create in https://github.com/flashinfer-ai/flashinfer/pull/2919
[Chore] add missing MOE code part by @jiahanc in https://github.com/flashinfer-ai/flashinfer/pull/2998
enable_pdl_and_bias_for_cudnn_backend by @yanqinz2 in https://github.com/flashinfer-ai/flashinfer/pull/2948
bench: Enable microbenchmarking on SM121 by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/3002
fix: tinygemm2 hang issue due to barrier sync by @jimmyzho in https://github.com/flashinfer-ai/flashinfer/pull/2996
[Perf] Refactor MoE autotuning to set valid topk ids in routed MoE tuning by @wzhao18 in https://github.com/flashinfer-ai/flashinfer/pull/2942
fix: restore SM120 CUTLASS MoE tile candidate removed by #2927 (test_trtllm_cutlass_fused_moe.py) by @samuellees in https://github.com/flashinfer-ai/flashinfer/pull/2984
misc: Update gemm/batched gemm cubins from trtllm-gen, gemm header refactor by @jimmyzho in https://github.com/flashinfer-ai/flashinfer/pull/2740
fix: use sym_int64 for strides in rmsnorm CuTe DSL kernels to prevent int32 overflow by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/3007
Add SM 103 as one of supported capabilities for mm_M1_16_K7168_N256 by @harrisonlimh in https://github.com/flashinfer-ai/flashinfer/pull/2991
feat: add PDL support to rmsnorm_fp4quant and add_rmsnorm_fp4quant CuTe DSL kernels by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/3008
[Fmha] support nvfp4 output keepsMmaAb generation kernels by @PerkzZheng in https://github.com/flashinfer-ai/flashinfer/pull/2988
Only swizzle on v block scale; rename kv_block_scales to kv_cache_sf by @sychen52 in https://github.com/flashinfer-ai/flashinfer/pull/2954
feat(gdn): state checkpointing in chunk_gated_delta_rule by @feldsherov in https://github.com/flashinfer-ai/flashinfer/pull/2908
[chore] Install nvidia-cutlass-dsl[cu13] for cu130+ by @jiahanc in https://github.com/flashinfer-ai/flashinfer/pull/3017
Add flashinfer.fused_rmsnorm_silu() with native kernel backend by @kahyunnam in https://github.com/flashinfer-ai/flashinfer/pull/2965
feat: Support padding tokens with seqlen=0 for rope+quant+kv cache update fusion kernel by @elvischenv in https://github.com/flashinfer-ai/flashinfer/pull/2792
Update README.md: Jetson Thor compute capability by @qiching in https://github.com/flashinfer-ai/flashinfer/pull/3012
[fix] bugfix 1044: Auto-inject well-known JIT additional tensor buffers in prefill and decode run() APIs by @qsang-nv in https://github.com/flashinfer-ai/flashinfer/pull/2855
Update Docker CI tags to 20260408-4cce866 by @flashinfer-bot in https://github.com/flashinfer-ai/flashinfer/pull/3018
perf: Port TRT-LLM SM120/SM121 FP4 CUTLASS GEMM optimizations. Add PDL by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/3026
perf: Optimize CUTLASS MoE helper kernels for small-batch decode workloads by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/3014
[fix] bugfix 541: Make single_prefill/decode compatible with torch.compile CUDA graphs by @qsang-nv in https://github.com/flashinfer-ai/flashinfer/pull/2857
Prevent MoE autotuner buffer overflow on large token buckets by @leejnau in https://github.com/flashinfer-ai/flashinfer/pull/3025
Fused moe all-reduce routed scaling factor + quant support by @murphymatt in https://github.com/flashinfer-ai/flashinfer/pull/2966
fix: check for ptr before calling close_mnnvl_memory by @jdebache in https://github.com/flashinfer-ai/flashinfer/pull/2892
Second part of refactoring the routing part by @ChristinaZ in https://github.com/flashinfer-ai/flashinfer/pull/2993
feat(comm): add MOE Finalize/Reduction patterns to unified allreduce_fusion API by @samuellees in https://github.com/flashinfer-ai/flashinfer/pull/2982
Fix compilation error: add missing <optional> header by @he-yufeng in https://github.com/flashinfer-ai/flashinfer/pull/2772
[chore] Fix CI pre-commit mypy error by @jiahanc in https://github.com/flashinfer-ai/flashinfer/pull/3040
Add support for Relu2 in BF16 fused MoE by @amitz-nv in https://github.com/flashinfer-ai/flashinfer/pull/2864
fix: extend moe alltoall top-k specializations by @bobboli in https://github.com/flashinfer-ai/flashinfer/pull/3021
Fix MXFP4/MXFP8 failures in SM120 FAST_BUILD and expand all_tiles[] by @askliar in https://github.com/flashinfer-ai/flashinfer/pull/2994
[feat] Add blackwell GDN prefill kernel by @jiahanc in https://github.com/flashinfer-ai/flashinfer/pull/3001
Fix silent bug with FP8 per tensor non-gated MoE by @danisereb in https://github.com/flashinfer-ai/flashinfer/pull/2882
Add @qsang-nv as a code owner for attention by @sricketts in https://github.com/flashinfer-ai/flashinfer/pull/3055
[CuTe DSL] Add modular FMHA prefill and MLA decode attention kernels by @pgera in https://github.com/flashinfer-ai/flashinfer/pull/2805
bump version to 0.6.8 by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/3042

New Contributors

@qiching made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2853
@ZJY0516 made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2727
@depaulmillz made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2738
@Observer007 made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2901
@yweng0828 made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2534
@he-yufeng made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2756
@wzhao18 made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2739
@askliar made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2926
@benhg made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2960
@ooooo-create made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2919
@harrisonlimh made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2991
@feldsherov made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2908
@pgera made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2805

Full Changelog: https://github.com/flashinfer-ai/flashinfer/compare/v0.6.7.post3...v0.6.8

Open source →

0.6.8rc114 Apr 2026pre-release

Add to CODEOWNER by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/2875

What's Changed

Add to CODEOWNER by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/2875
fix: int32 overflow in trtllm_fp4_block_scale_moe causing "Unsupported hidden state scale shape" for EP32+ configs by @qiching in https://github.com/flashinfer-ai/flashinfer/pull/2853
feat: bump nvidia-cutlass-dsl to >=4.4.2 by @limin2021 in https://github.com/flashinfer-ai/flashinfer/pull/2833
fix: add cute dsl moe utils to AOT by @nv-yunzheq in https://github.com/flashinfer-ai/flashinfer/pull/2872
fix: fix cute dsl swap_ab tactic failure by @nv-yunzheq in https://github.com/flashinfer-ai/flashinfer/pull/2870
[gdn] support non-contiguous state for decoding by @ZJY0516 in https://github.com/flashinfer-ai/flashinfer/pull/2727
chore: fix the python dependency override by @yongwww in https://github.com/flashinfer-ai/flashinfer/pull/2651
backinteg: nvidia-nvshmem-cu12 3.6.5 seems broken by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/2893
Yanqinz/gemm cudnn autotune fix by @yanqinz2 in https://github.com/flashinfer-ai/flashinfer/pull/2863
feat: Add CuTe-DSL backend for NVFP4 quantization by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2838
Add cute dsl mla decode op by @limin2021 in https://github.com/flashinfer-ai/flashinfer/pull/2743
Support for MXFP4 and NVFP4 group GEMMs on GeForce and Spark by @depaulmillz in https://github.com/flashinfer-ai/flashinfer/pull/2738
feat: add pdl support for cute dsl mla decode kernel support by @Observer007 in https://github.com/flashinfer-ai/flashinfer/pull/2901
feat: expose swizzled_input_sf parameter for CUTLASS fused MOE by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/2330
fix: support fp32 logits for fp8_per_tensor and fp8_block by @yweng0828 in https://github.com/flashinfer-ai/flashinfer/pull/2534
Fix autotuner crash when input tensor is None by @he-yufeng in https://github.com/flashinfer-ai/flashinfer/pull/2756
Support in-place update for trtllm_fp8_block_scale_moe by @wzhao18 in https://github.com/flashinfer-ai/flashinfer/pull/2739
[fix] bugfix 2856: Fix pre-allocated out shape check in trtllm_batch_decode_with_kv_cache_mla for q_len_per_req > 1 by @qsang-nv in https://github.com/flashinfer-ai/flashinfer/pull/2876
PR auto-labelling by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/2827
fix test error regarding logits_types by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/2918
Use 6-hour timeout for flashinfer-jit-cache wheel build (release + nightly) by @yongwww in https://github.com/flashinfer-ai/flashinfer/pull/2880
fix: expose trigger_completion_at_end through unified API by @nv-yunzheq in https://github.com/flashinfer-ai/flashinfer/pull/2894
fix: clamp enable_pdl=True to False on SM < 90 to prevent PDL PTX on Ampere by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2928
feat: add Relu2 (squared ReLU) activation support in CUTLASS MoE backend by @askliar in https://github.com/flashinfer-ai/flashinfer/pull/2926
docker: upgrade cuDNN to latest version in CI install script by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2930
[NVIDIA] fix(jit): enable GDC for CUTLASS fused MoE PDL — prevent random crashes on SM12x by @johnnynunez in https://github.com/flashinfer-ai/flashinfer/pull/2913
fix: Fix autotuner crash on meta-device tensor in trtllm_fp4_block_scale_routed_moe by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2916
Yanqinz/dynamic shape unified api by @yanqinz2 in https://github.com/flashinfer-ai/flashinfer/pull/2910
doc: add CI triggering guide to CONTRIBUTING.md by @yongwww in https://github.com/flashinfer-ai/flashinfer/pull/2924
read real strides for kv and block scale by @sychen52 in https://github.com/flashinfer-ai/flashinfer/pull/2844
perf: Optimize CuTe-DSL fp4 and fp8 quantization kernels by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2904
fix: vectorize get_shuffle_matrix_a_row_indices to eliminate CPU contention by @youkaichao in https://github.com/flashinfer-ai/flashinfer/pull/2935
feat: implement deterministic topk by @jiangyinzuo in https://github.com/flashinfer-ai/flashinfer/pull/2661
feat(gdn): add BF16 state kernel with MTP support beyond T>4 with intermediate caching. by @ameynaik-hub in https://github.com/flashinfer-ai/flashinfer/pull/2679
ci: remove 1gpu label from H100 runner selector by @yongwww in https://github.com/flashinfer-ai/flashinfer/pull/2946
perf: Optimize GDN MTP decode kernel (v15) — eliminate ilp=1 fallback… by @ameynaik-hub in https://github.com/flashinfer-ai/flashinfer/pull/2842
feat: add MXFP8 GEMM support for SM120 by @samuellees in https://github.com/flashinfer-ai/flashinfer/pull/2902
fix: avoid re-downloading BMM export headers when flashinfer-cubin is installed by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/2903
test: xfail cuDNN FP8 prefill on Blackwell with CUDA <= 12.9 by @dierksen in https://github.com/flashinfer-ai/flashinfer/pull/2963
test: skip unsupported mm_mxfp8 configurations on SM12x by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2974
[Fmha] revert blackwell ultra optimization that causes deadlocks. by @PerkzZheng in https://github.com/flashinfer-ai/flashinfer/pull/2956
feat: SM121 (GB10) tile filtering and autotuner robustness by @askliar in https://github.com/flashinfer-ai/flashinfer/pull/2927
Mamba SSU: horizontal MTP kernel (+ DSTATE=96 support) by @ishovkun in https://github.com/flashinfer-ai/flashinfer/pull/2865
fix: use float instead of double in sampling binary search to avoid FP64 bottleneck on SM103 by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2945
Refactor the routing part by @ChristinaZ in https://github.com/flashinfer-ai/flashinfer/pull/2803
fix: snap weight_scale_vec_size to handle block_scale_interleave padding for SM120 by @samuellees in https://github.com/flashinfer-ai/flashinfer/pull/2898
Add filelock to ensure_symlink by @wzhao18 in https://github.com/flashinfer-ai/flashinfer/pull/2979
Update NVSHMEM interface to use NVSHMEM4Py instead of custom bindings by @benhg in https://github.com/flashinfer-ai/flashinfer/pull/2960
docs: document replay command in CLI reference by @ooooo-create in https://github.com/flashinfer-ai/flashinfer/pull/2919
[Chore] add missing MOE code part by @jiahanc in https://github.com/flashinfer-ai/flashinfer/pull/2998
enable_pdl_and_bias_for_cudnn_backend by @yanqinz2 in https://github.com/flashinfer-ai/flashinfer/pull/2948
bench: Enable microbenchmarking on SM121 by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/3002
fix: tinygemm2 hang issue due to barrier sync by @jimmyzho in https://github.com/flashinfer-ai/flashinfer/pull/2996
[Perf] Refactor MoE autotuning to set valid topk ids in routed MoE tuning by @wzhao18 in https://github.com/flashinfer-ai/flashinfer/pull/2942
fix: restore SM120 CUTLASS MoE tile candidate removed by #2927 (test_trtllm_cutlass_fused_moe.py) by @samuellees in https://github.com/flashinfer-ai/flashinfer/pull/2984
misc: Update gemm/batched gemm cubins from trtllm-gen, gemm header refactor by @jimmyzho in https://github.com/flashinfer-ai/flashinfer/pull/2740
fix: use sym_int64 for strides in rmsnorm CuTe DSL kernels to prevent int32 overflow by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/3007
Add SM 103 as one of supported capabilities for mm_M1_16_K7168_N256 by @harrisonlimh in https://github.com/flashinfer-ai/flashinfer/pull/2991
feat: add PDL support to rmsnorm_fp4quant and add_rmsnorm_fp4quant CuTe DSL kernels by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/3008
[Fmha] support nvfp4 output keepsMmaAb generation kernels by @PerkzZheng in https://github.com/flashinfer-ai/flashinfer/pull/2988
Only swizzle on v block scale; rename kv_block_scales to kv_cache_sf by @sychen52 in https://github.com/flashinfer-ai/flashinfer/pull/2954
feat(gdn): state checkpointing in chunk_gated_delta_rule by @feldsherov in https://github.com/flashinfer-ai/flashinfer/pull/2908
[chore] Install nvidia-cutlass-dsl[cu13] for cu130+ by @jiahanc in https://github.com/flashinfer-ai/flashinfer/pull/3017
Add flashinfer.fused_rmsnorm_silu() with native kernel backend by @kahyunnam in https://github.com/flashinfer-ai/flashinfer/pull/2965
feat: Support padding tokens with seqlen=0 for rope+quant+kv cache update fusion kernel by @elvischenv in https://github.com/flashinfer-ai/flashinfer/pull/2792
Update README.md: Jetson Thor compute capability by @qiching in https://github.com/flashinfer-ai/flashinfer/pull/3012
[fix] bugfix 1044: Auto-inject well-known JIT additional tensor buffers in prefill and decode run() APIs by @qsang-nv in https://github.com/flashinfer-ai/flashinfer/pull/2855
Update Docker CI tags to 20260408-4cce866 by @flashinfer-bot in https://github.com/flashinfer-ai/flashinfer/pull/3018
perf: Port TRT-LLM SM120/SM121 FP4 CUTLASS GEMM optimizations. Add PDL by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/3026
perf: Optimize CUTLASS MoE helper kernels for small-batch decode workloads by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/3014
[fix] bugfix 541: Make single_prefill/decode compatible with torch.compile CUDA graphs by @qsang-nv in https://github.com/flashinfer-ai/flashinfer/pull/2857
Prevent MoE autotuner buffer overflow on large token buckets by @leejnau in https://github.com/flashinfer-ai/flashinfer/pull/3025
Fused moe all-reduce routed scaling factor + quant support by @murphymatt in https://github.com/flashinfer-ai/flashinfer/pull/2966
fix: check for ptr before calling close_mnnvl_memory by @jdebache in https://github.com/flashinfer-ai/flashinfer/pull/2892
Second part of refactoring the routing part by @ChristinaZ in https://github.com/flashinfer-ai/flashinfer/pull/2993
feat(comm): add MOE Finalize/Reduction patterns to unified allreduce_fusion API by @samuellees in https://github.com/flashinfer-ai/flashinfer/pull/2982
Fix compilation error: add missing <optional> header by @he-yufeng in https://github.com/flashinfer-ai/flashinfer/pull/2772
[chore] Fix CI pre-commit mypy error by @jiahanc in https://github.com/flashinfer-ai/flashinfer/pull/3040
Add support for Relu2 in BF16 fused MoE by @amitz-nv in https://github.com/flashinfer-ai/flashinfer/pull/2864
fix: extend moe alltoall top-k specializations by @bobboli in https://github.com/flashinfer-ai/flashinfer/pull/3021
Fix MXFP4/MXFP8 failures in SM120 FAST_BUILD and expand all_tiles[] by @askliar in https://github.com/flashinfer-ai/flashinfer/pull/2994
[feat] Add blackwell GDN prefill kernel by @jiahanc in https://github.com/flashinfer-ai/flashinfer/pull/3001
Fix silent bug with FP8 per tensor non-gated MoE by @danisereb in https://github.com/flashinfer-ai/flashinfer/pull/2882
Add @qsang-nv as a code owner for attention by @sricketts in https://github.com/flashinfer-ai/flashinfer/pull/3055
[CuTe DSL] Add modular FMHA prefill and MLA decode attention kernels by @pgera in https://github.com/flashinfer-ai/flashinfer/pull/2805
bump version to 0.6.8 by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/3042

New Contributors

@qiching made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2853
@ZJY0516 made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2727
@depaulmillz made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2738
@Observer007 made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2901
@yweng0828 made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2534
@he-yufeng made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2756
@wzhao18 made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2739
@askliar made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2926
@benhg made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2960
@ooooo-create made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2919
@harrisonlimh made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2991
@feldsherov made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2908
@murphymatt made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2966
@pgera made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2805

Full Changelog: https://github.com/flashinfer-ai/flashinfer/compare/v0.6.7.post3...v0.6.8rc1

Open source →

0.6.725 Mar 2026Breaking

bump version to 0.6.7 & fix api breaking changes by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/2832

What's Changed

perf(gdn): optimize MTP kernel with ILP rows and SMEM v caching by @ameynaik-hub in https://github.com/flashinfer-ai/flashinfer/pull/2618
Feat/gdn decode pooled by @xutizhou in https://github.com/flashinfer-ai/flashinfer/pull/2521
fix(jit): GEMM kernels produce NaN under concurrency — missing GDC flags cause PDL synchronization barriers to compile as no-ops by @voipmonitor in https://github.com/flashinfer-ai/flashinfer/pull/2716
Support NVFP4 KV cache decode on SM120 by @Tom-Zheng in https://github.com/flashinfer-ai/flashinfer/pull/2520
feat: Add TRTLLM fmha_v2 library for SM90 attention with Skip-Softmax by @jimmyzho in https://github.com/flashinfer-ai/flashinfer/pull/2446
bump version to 0.6.6 by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/2724
[benchmark] Add All Reduce benchmark by @jiahanc in https://github.com/flashinfer-ai/flashinfer/pull/2696
Revert "fix(jit): GEMM kernels produce NaN under concurrency — missing GDC flags cause PDL synchronization barriers to compile as no-ops" by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/2737
refactor: refactoring cuda code to cute-dsl (part 1) by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/2428
Added missing padding by @nvjullin in https://github.com/flashinfer-ai/flashinfer/pull/2726
docker: add CUDA 13.1 Dockerfiles with cuda-tile by @yongwww in https://github.com/flashinfer-ai/flashinfer/pull/2774
[BugFix] guard against uint32 underflow in multi-CTA TopK chunk calculation by @LopezCastroRoberto in https://github.com/flashinfer-ai/flashinfer/pull/2592
fix: guard CUTLASS FMHA against SM12x and fix fmha_v2 SM121a check by @blake-snc in https://github.com/flashinfer-ai/flashinfer/pull/2560
fix: fix illegal memory access for NaN input in sampling kernels by @zack041 in https://github.com/flashinfer-ai/flashinfer/pull/2456
Add cuda-tile to package dependencies by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/2758
tests: skip sliding window + fp8 to prevent hang in fmha_v2 unit tests by @jimmyzho in https://github.com/flashinfer-ai/flashinfer/pull/2781
feat: Add autotuner config caching, thread safety, and documentation by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2554
fix: block PR merge when CI is skipped due to pending authorization by @yongwww in https://github.com/flashinfer-ai/flashinfer/pull/2761
[feat] Add air top-p algorithm by @qsang-nv in https://github.com/flashinfer-ai/flashinfer/pull/2752
[chore] Add jiahanc to moe related code owner by @jiahanc in https://github.com/flashinfer-ai/flashinfer/pull/2748
fix: Fix cute dsl moe failure with nvidia-cutlass-dsl >= 4.4.0 by @nv-yunzheq in https://github.com/flashinfer-ai/flashinfer/pull/2735
[Spark unit test debugging] Fix for tests/attention/test_trtllm_gen_mla.py by @kahyunnam in https://github.com/flashinfer-ai/flashinfer/pull/2750
[Spark unit test debugging] Fix for tests/gemm/test_groupwise_scaled_gemm_fp8.py by @kahyunnam in https://github.com/flashinfer-ai/flashinfer/pull/2751
[feat] Add 2048 experts and 32 Top K by @jiahanc in https://github.com/flashinfer-ai/flashinfer/pull/2744
perf: Performance tune cute dsl RMSNorm variants by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2777
feat: Add FP4 KV cache quant/dequant kernels by @samuellees in https://github.com/flashinfer-ai/flashinfer/pull/2757
Add cute-dsl backends to mxfp[8,4]_quantization for future refactor by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2443
feat: FP32 dtype output for BF16 matmuls (CUTLASS & cuDNN) by @raayandhar in https://github.com/flashinfer-ai/flashinfer/pull/2644
Create separate cuDNN handle per GPU by @dhiraj113 in https://github.com/flashinfer-ai/flashinfer/pull/2688
CuteDSL MoE fix redundant output buffer zeroing by @leejnau in https://github.com/flashinfer-ai/flashinfer/pull/2811
Add NVFP4 KV cache quantization support for SM100 by @sychen52 in https://github.com/flashinfer-ai/flashinfer/pull/2702
[fix] Bugfix 1367: fix VariableBlockSparseAttention buffer overflow by dynamically resizing kv_lens_buffer by @qsang-nv in https://github.com/flashinfer-ai/flashinfer/pull/2802
fix: Workaround org teams perm issue for approval purposes by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/2816
Implement override shape support for cuDNN GEMM operations by @yanqinz2 in https://github.com/flashinfer-ai/flashinfer/pull/2790
feat: Add support for TRTLLM MXFP8 non-gated MoE with ReLU2 by @danisereb in https://github.com/flashinfer-ai/flashinfer/pull/2707
Upgrade cutlass 4.2.1 -> 4.4.2 by @kahyunnam in https://github.com/flashinfer-ai/flashinfer/pull/2798
chore: cute dsl nvfp4 moe clean up by @nv-yunzheq in https://github.com/flashinfer-ai/flashinfer/pull/2775
fix: Add SM120 (RTX Blackwell desktop) support for NVFP4 MoE kernels by @brandonmmusic-max in https://github.com/flashinfer-ai/flashinfer/pull/2725
Protect against null clusterUuid in mnnvl.py by @akshaver in https://github.com/flashinfer-ai/flashinfer/pull/2626
Deprecation for gated_delta_rule_mtp's intermediate_states_buffer=True by @kahyunnam in https://github.com/flashinfer-ai/flashinfer/pull/2730
fix: Autotuner _find_nearest_profile non-power-of-2 num_tokens, create launchers for all supported tileN in trtllm fused MoE by @amitz-nv in https://github.com/flashinfer-ai/flashinfer/pull/2821
fix(jit): enable GDC for CUTLASS GEMM PDL — SM100 flag only by @voipmonitor in https://github.com/flashinfer-ai/flashinfer/pull/2780
[Fmha] Sparse MLA decode kernel selection heuristics by @PerkzZheng in https://github.com/flashinfer-ai/flashinfer/pull/2836
fix: add missing re-exports for rmsnorm quant and fused_add_rmsnorm q… by @DevashishLal-CB in https://github.com/flashinfer-ai/flashinfer/pull/2783
Add varlen and speculative decoding support to selective state update by @roikoren755 in https://github.com/flashinfer-ai/flashinfer/pull/2700
[feat] trtllm-gen mxfp8 gemm by @IwakuraRein in https://github.com/flashinfer-ai/flashinfer/pull/2653
[Spark bug] Fix arch 12.1 -> "sm120a" flag for Spark, CUDA 12.9 by @kahyunnam in https://github.com/flashinfer-ai/flashinfer/pull/2839
skip per-pr for draft PRs by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/2831
feat(gdn): add padding index guard for bf16 decode kernel by @kaixih in https://github.com/flashinfer-ai/flashinfer/pull/2810
docker: Add CUDA 13.2 Docker containers by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2843
[fix] bugfix 1419: Add batch size shape validation in decode and prefill run() APIs by @qsang-nv in https://github.com/flashinfer-ai/flashinfer/pull/2801
Update Docker CI tags to 20260322-ff86ea0 by @flashinfer-bot in https://github.com/flashinfer-ai/flashinfer/pull/2854
feat: Expose TRT-LLM FMHA style paged KV Cache and page table layout by @DomBrown in https://github.com/flashinfer-ai/flashinfer/pull/2770
[Spark unit test] Adjust tolerance for test_xqa, test_logits_processor by @kahyunnam in https://github.com/flashinfer-ai/flashinfer/pull/2828
Mamba2 SSD Combined Forward Pass (Blackwell CuTe DSL Kernel) by @ishovkun in https://github.com/flashinfer-ai/flashinfer/pull/2709
bump version to 0.6.7 & fix api breaking changes by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/2832
[Spark unit test debugging] Fix for tests/autotuner/test_autotuner_core.py by @kahyunnam in https://github.com/flashinfer-ai/flashinfer/pull/2867
fix: use current CUDA device instead of tp_rank for SymmDeviceMemory allocation by @fzyzcjy in https://github.com/flashinfer-ai/flashinfer/pull/2662

New Contributors

@voipmonitor made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2716
@dhiraj113 made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2688
@leejnau made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2811
@sychen52 made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2702
@yanqinz2 made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2790
@brandonmmusic-max made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2725
@akshaver made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2626
@DevashishLal-CB made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2783
@roikoren755 made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2700

Full Changelog: https://github.com/flashinfer-ai/flashinfer/compare/v0.6.6...v0.6.7

Open source →

0.6.54 Mar 2026

feat: BF16 GEMM benchmarking support by @raayandhar in https://github.com/flashinfer-ai/flashinfer/pull/2525

What's Changed

feat: BF16 GEMM benchmarking support by @raayandhar in https://github.com/flashinfer-ai/flashinfer/pull/2525
[bugfix]Correct chunk_end calculation in multi-CTA collaboration when max_len > length by @huangzhilin-hzl in https://github.com/flashinfer-ai/flashinfer/pull/2489
test: Skip test_decode_delta_rule.py by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2600
feat: add issue self-claim workflow for external contributors by @jwu1980 in https://github.com/flashinfer-ai/flashinfer/pull/2586
ci: add cleanup step to nightly release self-hosted runner jobs by @yongwww in https://github.com/flashinfer-ai/flashinfer/pull/2510
ci: fix H100 cleanup by @yongwww in https://github.com/flashinfer-ai/flashinfer/pull/2590
tests: add bias testing to nvfp4 moe by @jimmyzho in https://github.com/flashinfer-ai/flashinfer/pull/2585
feat: cute dsl mmfp4 for blackwell by @nv-yunzheq in https://github.com/flashinfer-ai/flashinfer/pull/2540
fix: correct #pragma unoll typo to #pragma unroll in vec_dtypes.cuh by @Bias92 in https://github.com/flashinfer-ai/flashinfer/pull/2611
fix: get tensors by const ref to not rely on deleted move constructor for TensorView by @hypdeb in https://github.com/flashinfer-ai/flashinfer/pull/2602
Mamba SSU: better automatic kernel selection + algorithm selection optionally exposed to the user. by @ishovkun in https://github.com/flashinfer-ai/flashinfer/pull/2591
chore/feat: Add do_finalize to trtllm-gen fp8/f16 MoE APIs by @IwakuraRein in https://github.com/flashinfer-ai/flashinfer/pull/2548
docs: Document setuptools upgrade requirement for editable installs with --no-build-isolation by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2541
docs: resolve TODO by documenting log2f vs logf performance rationale in sampling by @Bias92 in https://github.com/flashinfer-ai/flashinfer/pull/2609
Ameyn/gdn bf16 tolerance parallel reduction by @ameynaik-hub in https://github.com/flashinfer-ai/flashinfer/pull/2610
feat: trtllm tinygemm2 in flashinfer as bf16 routergemm by @jimmyzho in https://github.com/flashinfer-ai/flashinfer/pull/2587
fix: cute dsl nvfp4 moe routing index error by @nv-yunzheq in https://github.com/flashinfer-ai/flashinfer/pull/2629
[bugfix] Fix FilteredTopK overflow correctness by @jiangyinzuo in https://github.com/flashinfer-ai/flashinfer/pull/2605
fix: add SM121 support to SM120 version guards by @Yuening-wa in https://github.com/flashinfer-ai/flashinfer/pull/2631
benchmark: Enable speculative decode microbenchmarking for paged decode by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2628
feat: add is_sm12x_supported() helper for SM12x family detection by @blake-snc in https://github.com/flashinfer-ai/flashinfer/pull/2574
benchmark: Add MXFP4/MXFP8 quantization mode support to FP4 MoE benchmark by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2635
fix: duplicate username bug in codeowners_analyzer.py by @sricketts in https://github.com/flashinfer-ai/flashinfer/pull/2637
Perf: Optimize GDN decode pretranspose kernel for all batch sizes by @ameynaik-hub in https://github.com/flashinfer-ai/flashinfer/pull/2588
support qk_nope_head_dim for 192 check for GLM-5 by @rainj-me in https://github.com/flashinfer-ai/flashinfer/pull/2607
fix: trtllm_mxint4_block_scale_moe unit test to index output list by @jimmyzho in https://github.com/flashinfer-ai/flashinfer/pull/2627
chore: Update CODEOWNERS by @flashinfer-bot in https://github.com/flashinfer-ai/flashinfer/pull/2286
fix: Add fused MOE and GEMM AOT modules for SM121 by @blake-snc in https://github.com/flashinfer-ai/flashinfer/pull/2654
refactor: pull trtllm-gen batch-gemm/gemm headers from artifactory; update tma descriptor shape init by @jimmyzho in https://github.com/flashinfer-ai/flashinfer/pull/2235
fix: Add tests for the AutoTuner and fix bug in _find_nearest_profile by @danisereb in https://github.com/flashinfer-ai/flashinfer/pull/2617
Bf16 routed moe by @IwakuraRein in https://github.com/flashinfer-ai/flashinfer/pull/2594
perf: Update trtllm-gen batched GEMM kernels - faster, more NVFP4 tile dims, MXFP8 with relu2 act by @amitz-nv in https://github.com/flashinfer-ai/flashinfer/pull/2667
Add code owner for scripts/codeownder_overrides.js by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/2656
feat: Autotuner support CUDA graph and cold L2 cache by @amitz-nv in https://github.com/flashinfer-ai/flashinfer/pull/2663
benchmarks: Add FP8 input / BF16 output in ragged prefill benchmark by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2666
Fix ImportError in AllReduceFusionWorkspace destructor during Python shutdown by @chaunceyjiang in https://github.com/flashinfer-ai/flashinfer/pull/2659
Version bump to 0.6.5 by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/2668

New Contributors

@jwu1980 made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2586
@Bias92 made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2611
@jiangyinzuo made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2605
@chaunceyjiang made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2659

Full Changelog: https://github.com/flashinfer-ai/flashinfer/compare/v0.6.4...v0.6.5

Open source →

0.6.419 Feb 2026

perf: add fp4 GEMM tile configs and streamK scheduler for SM120 by @Yuening-wa in https://github.com/flashinfer-ai/flashinfer/pull/2460

What's Changed

perf: add fp4 GEMM tile configs and streamK scheduler for SM120 by @Yuening-wa in https://github.com/flashinfer-ai/flashinfer/pull/2460
refactor: Port upstream CUTLASS fixes and refactor grouped_gemm_nt_masked GEMM module location by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2503
feat: cuteDSL fp4 moe for better DSR1 performance. by @nv-yunzheq in https://github.com/flashinfer-ai/flashinfer/pull/2398
ci: refactor PR tests to hide failed spot jobs from PR status by @yongwww in https://github.com/flashinfer-ai/flashinfer/pull/2500
Enable setting user in CI containers by @dierksen in https://github.com/flashinfer-ai/flashinfer/pull/2515
perf: cache cudaGetDeviceProperties in gdn_prefill to avoid per-call overhead by @xutizhou in https://github.com/flashinfer-ai/flashinfer/pull/2509
Revert "ci: refactor PR tests to hide failed spot jobs from PR status… by @yongwww in https://github.com/flashinfer-ai/flashinfer/pull/2524
feat: Add TRTLLM-Gen Skip-Softmax kernels for prefill and decode by @DomBrown in https://github.com/flashinfer-ai/flashinfer/pull/2477
add salyminty (me) to authorized_codeowners, fix alphabetical ordering by @saltyminty in https://github.com/flashinfer-ai/flashinfer/pull/2537
chore: update benchmark scripts; fix trtllm-gen moe comments by @IwakuraRein in https://github.com/flashinfer-ai/flashinfer/pull/2412
Add sm90 guard to fence.acquire by @jhalabi-nv in https://github.com/flashinfer-ai/flashinfer/pull/2535
feat: Add MXFP8 GEMM mm_mxfp8 (cutlass) by @danisereb in https://github.com/flashinfer-ai/flashinfer/pull/2464
fallback to fa2 (instead of fa3) for unsupported configuration (bf16 Q, Fp8 KV) by @saltyminty in https://github.com/flashinfer-ai/flashinfer/pull/2536
misc: point triton blackwell-ptxas to local cuda ptxas by @jimmyzho in https://github.com/flashinfer-ai/flashinfer/pull/2543
tests: bmm_fp8 for SM110 by @jimmyzho in https://github.com/flashinfer-ai/flashinfer/pull/2538
Add parallel testing to unit test script by @dierksen in https://github.com/flashinfer-ai/flashinfer/pull/2531
Add gen_gemm_sm100_module_cutlass_mxfp8 to jit-cache by @yongwww in https://github.com/flashinfer-ai/flashinfer/pull/2549
fix: Sampling: CUDA Graph fix by @IzzyPutterman in https://github.com/flashinfer-ai/flashinfer/pull/2432
fix: include fp8_blockscale_gemm_90 in AOT jit-cache by @Edward-lyz in https://github.com/flashinfer-ai/flashinfer/pull/2533
bugfix: fix the enum/int type mismatch mentioned in #2507 by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/2508
Add test case for Qwen3N by @samuellees in https://github.com/flashinfer-ai/flashinfer/pull/2532
Chore: Cute dsl moe update (TMA.RED implementation) by @nv-yunzheq in https://github.com/flashinfer-ai/flashinfer/pull/2529
benchmarks: Add microbenchmark support for Mamba selective_state_update by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2512
Update Docker CI tags to 20260209-a2d3b39 by @flashinfer-bot in https://github.com/flashinfer-ai/flashinfer/pull/2528
Ameyn/gdn decode cutedsl kernel by @ameynaik-hub in https://github.com/flashinfer-ai/flashinfer/pull/2498
[Bugfix][comm] Fix FP4 one-shot launch config instability in trtllm_allreduce_fusion by @baonudesifeizhai in https://github.com/flashinfer-ai/flashinfer/pull/2557
pick fa2 for BatchDecodeWithPagedKVCacheWrapper auto backend by @saltyminty in https://github.com/flashinfer-ai/flashinfer/pull/2530
Feat: Trtllm-gen MxFP8 MoE integration by @IwakuraRein in https://github.com/flashinfer-ai/flashinfer/pull/2505
[Bug] Fix spark unit test failures for test_add_rmsnorm_fp4_quant_cute_dsl by @kahyunnam in https://github.com/flashinfer-ai/flashinfer/pull/2573
fix: W4A8 autotune crash in cutlass_fused_moe profiler workspace by @ipnon in https://github.com/flashinfer-ai/flashinfer/pull/2564
Add Hopper to CI by @yongwww in https://github.com/flashinfer-ai/flashinfer/pull/2552
fix: allow fmha_v2_prefill_deepseek on SM121 (DGX Spark) by @blake-snc in https://github.com/flashinfer-ai/flashinfer/pull/2559
feat: Enable TRTLLM-Gen Skip-Softmax attention for MLA by @DomBrown in https://github.com/flashinfer-ai/flashinfer/pull/2547
docs: Add note on feature support for compute capabilities by @sricketts in https://github.com/flashinfer-ai/flashinfer/pull/2578
bump version to 0.6.4 by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/2565

New Contributors

@Yuening-wa made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2460
@xutizhou made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2509
@DomBrown made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2477
@saltyminty made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2537
@IzzyPutterman made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2432
@Edward-lyz made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2533
@ameynaik-hub made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2498
@baonudesifeizhai made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2557
@ipnon made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2564
@blake-snc made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2559

Full Changelog: https://github.com/flashinfer-ai/flashinfer/compare/v0.6.3...v0.6.4

Open source →

0.6.36 Feb 2026

ci: add permission control for public ci tests by @yongwww in https://github.com/flashinfer-ai/flashinfer/pull/2397

What's Changed

ci: add permission control for public ci tests by @yongwww in https://github.com/flashinfer-ai/flashinfer/pull/2397
Remove cudaMalloc/Free in GDN prefill kernel by @KevinZeng08 in https://github.com/flashinfer-ai/flashinfer/pull/2415
Update cudnn prefill to use correct sequence strides by @vedaanta in https://github.com/flashinfer-ai/flashinfer/pull/2414
perf: mm_fp4 heuristic prioritizes CUTLASS over cuDNN on SM103 by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2404
test: add coverage for all cli commands by @sricketts in https://github.com/flashinfer-ai/flashinfer/pull/1848
feat: BF16 GEMM using cuDNN backend by @raayandhar in https://github.com/flashinfer-ai/flashinfer/pull/2376
refactor: simplify fp4 rmsnorm by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/2421
feat: update trtllm-gen MoE cubins by @nekorobov in https://github.com/flashinfer-ai/flashinfer/pull/2416
chore/feat: A2A + MoE benchmark; add routed counterpart for trtllm_gen_fp8_fused_moe by @rosenrodt in https://github.com/flashinfer-ai/flashinfer/pull/2379
[CI] Add on-demand rerun for spot-terminated jobs by @yongwww in https://github.com/flashinfer-ai/flashinfer/pull/2403
fix: Fix NaN output in mxfp8_quantize for very small input values by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2441
feat: Support Fused MoE non gated Relu2 NVFP4 & FP8 and support Nemotron by @amitz-nv in https://github.com/flashinfer-ai/flashinfer/pull/2304
infra: add manual code owner override support in codeowner_analyzer.py by @sricketts in https://github.com/flashinfer-ai/flashinfer/pull/2418
fix: improve numerical stability of Gumbel sampling by @ixlmar in https://github.com/flashinfer-ai/flashinfer/pull/2438
ci: CI build workflow should always pull fresh and do not cache by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2454
Update Docker CI tags to 20260131-a52eff1 by @flashinfer-bot in https://github.com/flashinfer-ai/flashinfer/pull/2457
Revert "feat: Support Fused MoE non gated Relu2 NVFP4 & FP8 and support Nemotron" by @nv-yunzheq in https://github.com/flashinfer-ai/flashinfer/pull/2451
Skip trtllm_alltoall tests on Thor by @dierksen in https://github.com/flashinfer-ai/flashinfer/pull/2448
Fix argument type error in _cudnn_gemm_fp4_requirement by @Kangyan-Zhou in https://github.com/flashinfer-ai/flashinfer/pull/2450
fix: set_log_level now properly sets logger level to enable DEBUG logs by @kahyunnam in https://github.com/flashinfer-ai/flashinfer/pull/2449
bugfix: fix stub generation directory in fused_moe module by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/2445
[Perf][Feature] Add SM103-specific schedulers for NVFP4 CUTLASS kernels by @LopezCastroRoberto in https://github.com/flashinfer-ai/flashinfer/pull/2303
ci: set LD_LIBRARY_PATH in Docker images for correct cuBLAS detection by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2468
add sgl_kernel.fast_topk_v2 to top_k benchmark by @huangzhilin-hzl in https://github.com/flashinfer-ai/flashinfer/pull/2461
Update Docker CI tags to 20260203-9b5901e by @flashinfer-bot in https://github.com/flashinfer-ai/flashinfer/pull/2475
MTP for mamba by @ishovkun in https://github.com/flashinfer-ai/flashinfer/pull/2444
Add sm90 guard to fence ptx by @jhalabi-nv in https://github.com/flashinfer-ai/flashinfer/pull/2439
perf: improve gdn decode cute-dsl kernels by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/2405
ci: migrate release workflows to ci-infra runners by @yongwww in https://github.com/flashinfer-ai/flashinfer/pull/2467
fix: blockscale moe routine supports non-DS routing by @hypdeb in https://github.com/flashinfer-ai/flashinfer/pull/2476
Fix autotuner oom by @zack041 in https://github.com/flashinfer-ai/flashinfer/pull/2442
refactor: reduce hopper's gdn prefill compilation time and fix docstring. by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/2422
fix: Fix memory bandwidth calculation in MLA benchmarks by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2479
fix: Rename tests/mamba/test_utils.py to tests/mamba/utils.py to fix CI test discovery by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2481
Add/update multi node/multi GPU test scripts by @dierksen in https://github.com/flashinfer-ai/flashinfer/pull/2410
feat: Support Fused MoE non gated Relu2 NVFP4 & FP8 and support Nemotron, fixed by @amitz-nv in https://github.com/flashinfer-ai/flashinfer/pull/2462
ci: fix permission errors in release workflow on ci-infra runner by @yongwww in https://github.com/flashinfer-ai/flashinfer/pull/2488
benchmarks: Expand microbenchmark harness to include sampling and RoPe APIs by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2484
fix: add support check for gemm config for cutlass moe by @nv-yunzheq in https://github.com/flashinfer-ai/flashinfer/pull/2495
Allow non-DeepSeekV3 routing with one group by @dbari in https://github.com/flashinfer-ai/flashinfer/pull/2502
bump version to 0.6.3 by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/2497

New Contributors

@KevinZeng08 made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2415
@vedaanta made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2414
@ixlmar made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2438
@Kangyan-Zhou made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2450
@LopezCastroRoberto made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2303
@huangzhilin-hzl made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2461
@zack041 made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2442

Full Changelog: https://github.com/flashinfer-ai/flashinfer/compare/v0.6.2...v0.6.3

Open source →

0.6.223 Jan 2026

chore: MoE benchmark effective BW fix for trtllm_block_scale_moe by @rosenrodt in https://github.com/flashinfer-ai/flashinfer/pull/2341

What's Changed

chore: MoE benchmark effective BW fix for trtllm_block_scale_moe by @rosenrodt in https://github.com/flashinfer-ai/flashinfer/pull/2341
Update Docker CI tags to 20260114-cc1a362 by @flashinfer-bot in https://github.com/flashinfer-ai/flashinfer/pull/2351
[perf] Improve gemm_fp8_nt_groupwise (cutlass backend) by 10-40% for batch sizes <= 32 by @aidando73 in https://github.com/flashinfer-ai/flashinfer/pull/2327
feat: Add auto-fixing pre-commit to Claude Code workflows by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/2331
tiny support glm routing by @b8zhong in https://github.com/flashinfer-ai/flashinfer/pull/2313
fix: Handle zeros in Mistral Large 3 MoE inference by @dbari in https://github.com/flashinfer-ai/flashinfer/pull/2238
benchmarks: Add norm and quantization routines to microbenchmark harness. by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2362
[CI] Add support for testing dependency commits before release by @yongwww in https://github.com/flashinfer-ai/flashinfer/pull/2353
feat: introduce GitHub Actions workflow for PR testing by @yongwww in https://github.com/flashinfer-ai/flashinfer/pull/2326
chore: Add TRTLLM MoE A2A benchmark by @rosenrodt in https://github.com/flashinfer-ai/flashinfer/pull/2354
Added the cudnn backend Ragged KV Cache wrapper by @Anerudhan in https://github.com/flashinfer-ai/flashinfer/pull/2352
Enable fp16/bf16/f32 support for selective_state_update (mamba) by @ishovkun in https://github.com/flashinfer-ai/flashinfer/pull/2366
ci: increase nightly release build timeout by @yongwww in https://github.com/flashinfer-ai/flashinfer/pull/2371
chore: fix claude git actions by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/2384
chore: add script to run unittests/benchmarks on Modal GPU runners by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/2377
bugfix: hotfix of PR 2366 (mamba kernel) by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/2378
ci: add docker cleanup before running tests by @yongwww in https://github.com/flashinfer-ai/flashinfer/pull/2386
chore: Refactor benchmark imports to be lazy-loaded by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2388
fix: ensure each CTA processes full numHeadsQPerKv for trtllm decode kernel by @dongjiyingdjy in https://github.com/flashinfer-ai/flashinfer/pull/2380
ci: add Docker Hub authentication to mitigate pull rate limits by @yongwww in https://github.com/flashinfer-ai/flashinfer/pull/2393
A Blackwell-optimized version of selective_state_update (mamba) by @ishovkun in https://github.com/flashinfer-ai/flashinfer/pull/2387
fix: In-place Residual Update for add_rmsnorm_fp4quant by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2385
hotfix: remove uv.lock and add it to .gitignore by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/2399
feat: [Qwen3-Next] Add Cute DSL GDN decode kernel and tests by @HongliMi in https://github.com/flashinfer-ai/flashinfer/pull/2370
Update Mamba selective_state_scan API signature by @shaharmor98 in https://github.com/flashinfer-ai/flashinfer/pull/2392
Optimize quantization function in large problem size by @Shunkangz in https://github.com/flashinfer-ai/flashinfer/pull/2343
feat: Add output_both_sf_layouts option to add_rmsnorm_fp4quant API by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2395
release: bump version to 0.6.2 by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/2411

New Contributors

@rosenrodt made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2341
@aidando73 made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2327
@HongliMi made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2370
@shaharmor98 made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2392
@Shunkangz made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2343

Full Changelog: https://github.com/flashinfer-ai/flashinfer/compare/v0.6.1...v0.6.2

Open source →

0.6.08 Jan 2026

feat: Add backend='auto' to mm_fp4 and enable autotune for backend='cudnn' by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/1979

What's Changed

feat: Add backend='auto' to mm_fp4 and enable autotune for backend='cudnn' by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/1979
fix: Fix bench_mm_fp8.py by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2129
feat: Enable API Logging for Better Debugging POC by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2108
fix: add a check for int32 indices in sampling.py by @raayandhar in https://github.com/flashinfer-ai/flashinfer/pull/2127
update autotuner input tensor random range by @jiahanc in https://github.com/flashinfer-ai/flashinfer/pull/2116
enable xqa speculative decoding by @qsang-nv in https://github.com/flashinfer-ai/flashinfer/pull/2105
Add custom communicator for trtllm_mnnvl_ar by @wenscarl in https://github.com/flashinfer-ai/flashinfer/pull/2056
fix: DeepSeek activation uninitialized data by @nekorobov in https://github.com/flashinfer-ai/flashinfer/pull/2128
chore: Update CODEOWNERS by @flashinfer-bot in https://github.com/flashinfer-ai/flashinfer/pull/2135
bugfix: fix unittest error introduced in #2056 by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/2136
fix flaky xqa test by @qsang-nv in https://github.com/flashinfer-ai/flashinfer/pull/2126
fix: some bugs of headDim 256 trtllm-gen fmha kernels. by @PerkzZheng in https://github.com/flashinfer-ai/flashinfer/pull/2137
fix(trtllm): reset negative strideBatch to 0 for ragged KV layout to … by @YAMY1234 in https://github.com/flashinfer-ai/flashinfer/pull/2134
feat: add trtllm-gen per-tensor sparseMla kernels. by @PerkzZheng in https://github.com/flashinfer-ai/flashinfer/pull/2138
Use global TuningConfig, to fix memory leak caused by AutoTuner LRU cache and dynamic lambda TuningConfig by @juju812 in https://github.com/flashinfer-ai/flashinfer/pull/2140
feat: add seed offset args to sampler to allow cuda graph support by @ksukrit in https://github.com/flashinfer-ai/flashinfer/pull/2132
ci: Reduce test time by moving compilation off-line by @kahyunnam in https://github.com/flashinfer-ai/flashinfer/pull/2089
feat: TRTLLM FMHAv2 backend for ctx attention by @jimmyzho in https://github.com/flashinfer-ai/flashinfer/pull/2142
refactor: pass hopper deepgemm include directory through python by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/2090
bugfix: add driver support to CUPTI benchmark function, issue #2145 by @nv-yunzheq in https://github.com/flashinfer-ai/flashinfer/pull/2154
Bump tvm ffi version to 0.1.4 by @cyx-6 in https://github.com/flashinfer-ai/flashinfer/pull/2155
Update Docker CI tags to 20251202-23ff744 by @flashinfer-bot in https://github.com/flashinfer-ai/flashinfer/pull/2158
misc: Label APIs for Logging by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2153
Update nvidia-cutlass-dsl version to 4.3.1 by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/2161
chore: Update CODEOWNERS by @flashinfer-bot in https://github.com/flashinfer-ai/flashinfer/pull/2152
feat: C++ side tensor validation by @raayandhar in https://github.com/flashinfer-ai/flashinfer/pull/2160
Update Docker CI tags to 20251203-4efb7bb by @flashinfer-bot in https://github.com/flashinfer-ai/flashinfer/pull/2164
ci: Install CUDA version specified torch first during container building. by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2167
fix xqa mha_sm90.cu by @qsang-nv in https://github.com/flashinfer-ai/flashinfer/pull/2157
Update Docker CI tags to 20251203-1e15fed by @flashinfer-bot in https://github.com/flashinfer-ai/flashinfer/pull/2172
enable sm103 moe dsl backend by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/2149
ci: Use stable Torch Release for cu130 by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2174
tiny upd mm_fp4 docstring by @b8zhong in https://github.com/flashinfer-ai/flashinfer/pull/2177
fix: compile flags for trtllm fmha_v2 by @jimmyzho in https://github.com/flashinfer-ai/flashinfer/pull/2175
Fix/dsl smem query by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/2178
Update Docker CI tags to 20251204-cdc5fb7 by @flashinfer-bot in https://github.com/flashinfer-ai/flashinfer/pull/2176
feat: MxInt4 x Bf16 TRT-LLM Gen MoE support by @nekorobov in https://github.com/flashinfer-ai/flashinfer/pull/2159
refactor: Move mla code from decode.py to mla.py and add to documentation by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2163
Fix gemm allreduce two shot by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/2171
Update Docker CI tags to 20251205-54c1678 by @flashinfer-bot in https://github.com/flashinfer-ai/flashinfer/pull/2179
Rename noauxtc to fused_topk_deepseek by @nv-yunzheq in https://github.com/flashinfer-ai/flashinfer/pull/2181
refactor: update fa3 codebase and fix hopper unittest [part 1] by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/2111
Add data type check for deepseek fp4 moe by @samuellees in https://github.com/flashinfer-ai/flashinfer/pull/2165
benchmark: Make use_cupti the default in microbenchmarks. by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2180
ci: Specify MPI implementation to mpich by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2182
Update Docker CI tags to 20251206-185d63a by @flashinfer-bot in https://github.com/flashinfer-ai/flashinfer/pull/2184
test: Skip sm90 test in test_jit_warmup.py if not on sm90 by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2189
ci: Update sm12X minimum cuda capability to 12.9 in aot.py by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2188
Super tiny fix version by @fzyzcjy in https://github.com/flashinfer-ai/flashinfer/pull/2199
docs: Document CUDA version support in README and installation page by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2197
docs: Fix inaccurate API docstrings for attention prefill by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2196
feat: unit-test and api change, w4a8 grouped-gemm fused MoE for SM90 by @jimmyzho in https://github.com/flashinfer-ai/flashinfer/pull/2193
Permute page table in benchmarking by @jhjpark in https://github.com/flashinfer-ai/flashinfer/pull/2194
Fix for moe on sm110 by @jhalabi-nv in https://github.com/flashinfer-ai/flashinfer/pull/2190
chore: update authorized codeowners by @jimmyzho in https://github.com/flashinfer-ai/flashinfer/pull/2210
perf: bunch of features and optimizations for top-k (sampling + sparse attention) by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/2119
Refactor trtllm_mnnvl_allreduce by @timlee0212 in https://github.com/flashinfer-ai/flashinfer/pull/2118
chore: Update CODEOWNERS by @flashinfer-bot in https://github.com/flashinfer-ai/flashinfer/pull/2186
feat: support more head dim in RoPE kernel by @raayandhar in https://github.com/flashinfer-ai/flashinfer/pull/2109
Port TRT-LLM communication kernels to flashinfer by @djns99 in https://github.com/flashinfer-ai/flashinfer/pull/2102
cicd: Add sanity test script by @kahyunnam in https://github.com/flashinfer-ai/flashinfer/pull/2212
feat: add memcpy and memset to CUPTI timing method by @nv-yunzheq in https://github.com/flashinfer-ai/flashinfer/pull/2223
Added an initial implementation of Q and KV Cache in fp8 and to use t… by @Anerudhan in https://github.com/flashinfer-ai/flashinfer/pull/2035
feat: Support unpadded output hidden size for trtllm_fp4_block_scale_moe by @elvischenv in https://github.com/flashinfer-ai/flashinfer/pull/2217
fix: Eliminate the usage of CUDA ARCH macro in host function. by @timlee0212 in https://github.com/flashinfer-ai/flashinfer/pull/2228
misc: support checks for gemm by @jimmyzho in https://github.com/flashinfer-ai/flashinfer/pull/2214
feat: Cold L2 Cache Benchmarking with Rotating Buffers by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2213
Move the run function definition out of BatchedGemmInterface by @jhalabi-nv in https://github.com/flashinfer-ai/flashinfer/pull/2211
make DeepGEMM swapAB available for linear gemm SM90 by @katec846 in https://github.com/flashinfer-ai/flashinfer/pull/2131
misc: upgrade tvm-ffi dependency to 0.1.6 by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/2229
A unified API for the MNNVL and single-node/multi-GPU AllReduce kernels. by @nvmbreughe in https://github.com/flashinfer-ai/flashinfer/pull/2130
Update Docker CI tags to 20251217-f059241 by @flashinfer-bot in https://github.com/flashinfer-ai/flashinfer/pull/2231
Rebase FP8 SM100 Cutlass FMHA Attention to main (original PR#1238) by @pavanimajety in https://github.com/flashinfer-ai/flashinfer/pull/2047
[feat] Integrate SGLang concat_mla_k kernel into flashinfer by @jiahanc in https://github.com/flashinfer-ai/flashinfer/pull/2237
fix: add DeepSeek routing for Bf16xBf16 and MxIntxBf16 TRT-LLM Gen MoE by @nekorobov in https://github.com/flashinfer-ai/flashinfer/pull/2234
fix: Fix compilation with GCC 11 by @dbari in https://github.com/flashinfer-ai/flashinfer/pull/2242
feat: RMSNorm/Fused RMSNorm + FP8 Quantization kernels by @BLaZeKiLL in https://github.com/flashinfer-ai/flashinfer/pull/2243
feat: further optimize top-k and add fused top-k page construction kernels for DSA by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/2215
test: Fix MNNVL tests to skip when container lacks SYS_PTRACE capability by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2245
Remove cudaStreamSynchronize from gemm_groupwise_sm120.cuh for CUDA graph compatibility by @Copilot in https://github.com/flashinfer-ai/flashinfer/pull/2244
feat: support variable sequence length in decode kernel of trtllm-gen attention by @yaoyaoding in https://github.com/flashinfer-ai/flashinfer/pull/2125
feat: Fused RMSNorm + FP4 Quantization Kernels in CuTe-DSL by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2233
Allreduce auto backend improvements by @nvmbreughe in https://github.com/flashinfer-ai/flashinfer/pull/2239
cicd / testing: Add xfails tracker script by @kahyunnam in https://github.com/flashinfer-ai/flashinfer/pull/2227
chore: export compile commands for better IDE integration by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/2253
feat: support non-contiguous query for trtllm-gen attention backend by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/2254
Fp8 attention are now part of cuDNN 9.17.1 by @Anerudhan in https://github.com/flashinfer-ai/flashinfer/pull/2241
feat: Support numLocalTokens=0 for moe All-to-all by @trevor-m in https://github.com/flashinfer-ai/flashinfer/pull/2247
feat: support inplace update output for get_batch_indices_positions by @elvischenv in https://github.com/flashinfer-ai/flashinfer/pull/2257
Fix CUTLASS FP8 gemm correctness issue on SM120/SM121 for shapes where N is not divisible by ScaleGranularityN. by @yongwww in https://github.com/flashinfer-ai/flashinfer/pull/2261
fix: support int64 IdType for RoPE part argument in rope_quantize_fp8_append_paged_kv_cache by @elvischenv in https://github.com/flashinfer-ai/flashinfer/pull/2255
[Minor] Reduce num blocks of qknorm in small batch size by @DarkSharpness in https://github.com/flashinfer-ai/flashinfer/pull/2264
test: use .float() in in F.cosine_similarity() in bmm_fp8 test by @yongwww in https://github.com/flashinfer-ai/flashinfer/pull/2266
feat: Add support for bmm mxfp8 by @danisereb in https://github.com/flashinfer-ai/flashinfer/pull/2256
[performance]optimize for nvfp4 by @Bruce-x-1997 in https://github.com/flashinfer-ai/flashinfer/pull/2268
chore: Update CODEOWNERS by @flashinfer-bot in https://github.com/flashinfer-ai/flashinfer/pull/2218
agent: add CLAUDE.md and claude skills by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/2240
bugfix: fix claude skills by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/2275
fix: Add global scale support and optional output allocation for RMSNorm+FP4Quant fusion kernels by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2260
cicd: add a github workflow for xfails report script by @kahyunnam in https://github.com/flashinfer-ai/flashinfer/pull/2273
feat: IdType indices in sampling kernels by @raayandhar in https://github.com/flashinfer-ai/flashinfer/pull/2281
feat: add GDN Attention by @guangyunh-nv in https://github.com/flashinfer-ai/flashinfer/pull/2276
chore: update documentation and notice year to 2026 by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/2285
Tiny fix bench tgv gemm by @vincentzed in https://github.com/flashinfer-ai/flashinfer/pull/2277
dependency: update nvidia-cutlass-dsl by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/2288
Enable Hopper FA3 FP8 attention in decode.py by @nvpohanh in https://github.com/flashinfer-ai/flashinfer/pull/2148
Update Docker CI tags to 20260105-a97b5d7 by @flashinfer-bot in https://github.com/flashinfer-ai/flashinfer/pull/2289
[WIP] Refactor: simplify torch -> cute-dsl boilerplate and enable tvm-ffi for cute-dsl kernels by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/2279
fix: Decode benchmark's fa2_tc uses backend=fa2 in wrapper by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2302
bugfix: use torch cached default generators by @cyx-6 in https://github.com/flashinfer-ai/flashinfer/pull/2295
[TRTLLM-Gen Fmha] add optimized trtllm-gen decode kernels for high throughput + speculative decoding by @PerkzZheng in https://github.com/flashinfer-ai/flashinfer/pull/2265
update version to 0.6.0 by @nv-yunzheq in https://github.com/flashinfer-ai/flashinfer/pull/2300

New Contributors

@YAMY1234 made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2134
@juju812 made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2140
@ksukrit made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2132
@samuellees made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2165
@jhjpark made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2194
@jhalabi-nv made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2190
@djns99 made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2102
@katec846 made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2131
@dbari made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2242
@BLaZeKiLL made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2243
@Copilot made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2244
@yaoyaoding made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2125
@DarkSharpness made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2264
@danisereb made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2256
@Bruce-x-1997 made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2268
@guangyunh-nv made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2276
@vincentzed made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2277

Full Changelog: https://github.com/flashinfer-ai/flashinfer/compare/v0.5.3...v0.6.0

Open source →

0.5.324 Nov 2025Deprecated

[API change] deprecate tile_token_dim in trtllm_moe by @jiahanc in https://github.com/flashinfer-ai/flashinfer/pull/2086

What's Changed

perf: improve sampling/mask/softmax performance (part 1/2) by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/2044
misc: Add XQA decode to microbenchmark for sm90 and sm120 by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2055
test: Skip unsupported SM Archs for newly added trtllm MoE test by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2060
feat: suitable_auto_backends to prune auto backends, bmm_fp8 refactor, heuristic_func intake by @jimmyzho in https://github.com/flashinfer-ai/flashinfer/pull/2029
update trtllm cutlass moe by @nv-yunzheq in https://github.com/flashinfer-ai/flashinfer/pull/2020
perf: Optimize helper max/minmax function in sampling.cuh by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2058
[DSV3] Optimized Router Gemm by @nvmbreughe in https://github.com/flashinfer-ai/flashinfer/pull/2019
Fix moe fp8 failure for sm121 by @yongwww in https://github.com/flashinfer-ai/flashinfer/pull/2061
perf: TRT-LLM MoE Block-FP8 activation optimization by @nekorobov in https://github.com/flashinfer-ai/flashinfer/pull/2063
[feat] Refactor trtllmgen MOE and add Bf16 trtllmgen moe by @jiahanc in https://github.com/flashinfer-ai/flashinfer/pull/2014
Fix: several bugs/issues with trtllm-gen attention kernels. by @PerkzZheng in https://github.com/flashinfer-ai/flashinfer/pull/2062
refactor: remove MetaInfoHash class by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/2064
chore: Update CODEOWNERS by @flashinfer-bot in https://github.com/flashinfer-ai/flashinfer/pull/2067
feat: add xqa mla backend by @qsang-nv in https://github.com/flashinfer-ai/flashinfer/pull/2053
Enable renormalize(naive) routing for fp8 per-tensor by @IwakuraRein in https://github.com/flashinfer-ai/flashinfer/pull/2030
unittest: improve the efficiency of xqa unittests by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/2075
minor: canonicalize TFLOPS calculation by @Edenzzzz in https://github.com/flashinfer-ai/flashinfer/pull/2069
fix: fix test_trtllm_gen_attention when max_seq_len < page_size by @dongjiyingdjy in https://github.com/flashinfer-ai/flashinfer/pull/2076
enable xqa fp8 output by @qsang-nv in https://github.com/flashinfer-ai/flashinfer/pull/2081
chore: update requires-python in pyproject.toml by @raayandhar in https://github.com/flashinfer-ai/flashinfer/pull/2080
[Test] Optimize test_trtllm_gen_fused_moe.py by @jiahanc in https://github.com/flashinfer-ai/flashinfer/pull/2072
test: Change incorrect inputs in test_hopper.py by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2083
[NVIDIA] Thor & Spark Support by @johnnynunez in https://github.com/flashinfer-ai/flashinfer/pull/2028
[API change] deprecate tile_token_dim in trtllm_moe by @jiahanc in https://github.com/flashinfer-ai/flashinfer/pull/2086
[Feature] Support batch prefill for POD Attention by @AKKamath in https://github.com/flashinfer-ai/flashinfer/pull/2079
Patch sm103 for 3xfp4 moe generation by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/2082
MNNVL All Reduce for large number of tokens by @nvmbreughe in https://github.com/flashinfer-ai/flashinfer/pull/2074
perf: TRT-LLM Gen finalize kernel optimization by @nekorobov in https://github.com/flashinfer-ai/flashinfer/pull/2092
refactor: update dpsk fused_moe test [1] by @yyihuang in https://github.com/flashinfer-ai/flashinfer/pull/2088
chore: update thor cuda arch (from 110f to 110a) by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/2096
perf: enable pdl for cutlass fp4 gemm by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/2095
chore: Update CODEOWNERS by @flashinfer-bot in https://github.com/flashinfer-ai/flashinfer/pull/2098
feat: Add flashinfer.rope.rope_quantize_fp8_append_paged_kv_cache (fused RoPE + Q + KV cache, supports MLA/GQA/MHA) by @kahyunnam in https://github.com/flashinfer-ai/flashinfer/pull/2037
[API change] Allow using torch.Tensor for scales for trtllm-gen attention by @IwakuraRein in https://github.com/flashinfer-ai/flashinfer/pull/2084
refactor: update dpsk fused_moe test [2] by @yyihuang in https://github.com/flashinfer-ai/flashinfer/pull/2097
hotfix: rename moe/test_utils.py to moe/utils.py by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/2106
[DSR1] Added MLA test by @nvmbreughe in https://github.com/flashinfer-ai/flashinfer/pull/2100
test: Enable testing for trtllm-gen decode bs1 by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2103
[DSV3] Optimized routing kernels dsv3 by @nv-yunzheq in https://github.com/flashinfer-ai/flashinfer/pull/2099
feature: make the LSE returned by MLA support base 2 or e #2113 by @staugust in https://github.com/flashinfer-ai/flashinfer/pull/2114
update xqa license by @qsang-nv in https://github.com/flashinfer-ai/flashinfer/pull/2117
add tensor scale input for xqa by @qsang-nv in https://github.com/flashinfer-ai/flashinfer/pull/2110
hotfix: add 9.0a to README and installation doc by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/2112
ci/cd: add nvidia-ml-py to requirments of build-system of flashinfer-cubin by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/2123

New Contributors

@nekorobov made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2063
@dongjiyingdjy made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2076
@johnnynunez made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2028
@staugust made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2114

Full Changelog: https://github.com/flashinfer-ai/flashinfer/compare/v0.5.2...v0.5.3

Open source →

0.5.02 Nov 2025

fix vllm graph register and add test by @NVShreyas in https://github.com/flashinfer-ai/flashinfer/pull/1894

What's Changed

fix vllm graph register and add test by @NVShreyas in https://github.com/flashinfer-ai/flashinfer/pull/1894
Support checks PoC by @nvmbreughe in https://github.com/flashinfer-ai/flashinfer/pull/1809
chore: Restore FLASHINFER_LOCAL_VERSION environment variable by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/1934
chore: fix wheel license packaging issues by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/1932
Add layernorm op for inputs of mixed dtype by @akhilg-nv in https://github.com/flashinfer-ai/flashinfer/pull/1926
fixbug: fix devcontainer context by @cyx-6 in https://github.com/flashinfer-ai/flashinfer/pull/1938
MLA RoPE + quantization fused kernel: shape generalization for MHA / GQA by @kahyunnam in https://github.com/flashinfer-ai/flashinfer/pull/1924
ci: limit max_jobs for arm64 jit wheel cache build on CI by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/1943
Add realistic bench for persistent kernel by @Edenzzzz in https://github.com/flashinfer-ai/flashinfer/pull/1942
Add junit xml flags back to reorganized test script by @dierksen in https://github.com/flashinfer-ai/flashinfer/pull/1940
fix get max_q_len in page prefill plan by @ZhuJiaqi9905 in https://github.com/flashinfer-ai/flashinfer/pull/1930
ci: Create Github Action to Automate CODEOWNER update by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/1870
chore: Update CODEOWNERS by @github-actions[bot] in https://github.com/flashinfer-ai/flashinfer/pull/1871
chore: use flashinfer-bot account to create auto pull requests in release-ci-docker workflow by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/1944
Update Docker CI tags to 20251018-dbdf533 by @flashinfer-bot in https://github.com/flashinfer-ai/flashinfer/pull/1945
Fix #1641: Use /usr/local/cuda as default CUDA_HOME if possible, like torch.utils.cpp_extension.CUDA_HOME by @netanel-haber in https://github.com/flashinfer-ai/flashinfer/pull/1948
Fix bias dtype by @wenscarl in https://github.com/flashinfer-ai/flashinfer/pull/1876
chore: rename FLASHINFER_JIT_VERBOSE to FLASHINFER_JIT_DEBUG for clarity by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/1946
fix: Fix trtllm-gen prefill IMA when batch_size==1 by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/1912
Feature: Support Relu2 activation in fused MoE by @amirkl94 in https://github.com/flashinfer-ai/flashinfer/pull/1954
fix: Add cutlass as an mm_fp4 backend in compute capability 12.0 in benchmark code by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/1959
Update the routing for TRTLLMGEN to support kimi k2 and qwen by @ChristinaZ in https://github.com/flashinfer-ai/flashinfer/pull/1831
unittest: fix deepgemm sha256 by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/1953
misc: Update artifacts docstring and MetaInfoHash by @jimmyzho in https://github.com/flashinfer-ai/flashinfer/pull/1967
silu_and_mul nvfp4 quanization fusion rework by @wenscarl in https://github.com/flashinfer-ai/flashinfer/pull/1927
unittest: fix test_artifacts.py by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/1950
chore: update the list of authorized codeowners by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/1970
Added heuristic for trtllm_allreduce_fusion by @nvjullin in https://github.com/flashinfer-ai/flashinfer/pull/1972
Bump tvm ffi to stable version 0.1.0 by @cyx-6 in https://github.com/flashinfer-ai/flashinfer/pull/1960
Update Docker CI tags to 20251024-0e48aaf by @flashinfer-bot in https://github.com/flashinfer-ai/flashinfer/pull/1975
fix: Make attention microbenchmark correctly use page table by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/1976
fix: Skipping attention sink Blackwell test outside of Blackwell by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/1978
feat: enable deepgemm jit for fp8 block-scale on SM90 by @djmmoss in https://github.com/flashinfer-ai/flashinfer/pull/1969
chore: Update CODEOWNERS by @flashinfer-bot in https://github.com/flashinfer-ai/flashinfer/pull/1949
fix: correct PDL parameter handling in RopeQuantize kernel by @cicirori in https://github.com/flashinfer-ai/flashinfer/pull/1982
Fix: Verify scales are not None for Cutlass FP8 FusedMoE by @amirkl94 in https://github.com/flashinfer-ai/flashinfer/pull/1961
feat: add xqa fp8 mha and fp8 kv cache by @qsang-nv in https://github.com/flashinfer-ai/flashinfer/pull/1769
unittest: fix failed unittest on hopper by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/1952
docs: Update documented versioning scheme to right-shifted semver by @sricketts in https://github.com/flashinfer-ai/flashinfer/pull/1990
Bugfix: Change get() -> GetDLTensorPtr() in cutlass FusedMoE validations by @amirkl94 in https://github.com/flashinfer-ai/flashinfer/pull/1995
unittest: Add SM arch checks to skip unsupported tests on Hopper by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/1998
Added workspace check and reflected this in test by @nvmbreughe in https://github.com/flashinfer-ai/flashinfer/pull/1991
minor fix for xqa by @qsang-nv in https://github.com/flashinfer-ai/flashinfer/pull/1994
Feature: Add support for L40 FusedMoE in cutlass path by @amirkl94 in https://github.com/flashinfer-ai/flashinfer/pull/1973
unittest: Add head dim 256 test cases and mark as xfail by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/1999
feat: autotune tile_tokens_dim in trtllm-gen MOE by @jiahanc in https://github.com/flashinfer-ai/flashinfer/pull/1980
Fix trtllm-gen attention illegal memory access by @Tom-Zheng in https://github.com/flashinfer-ai/flashinfer/pull/2002
release: Bump version for v0.5.0rc1 release; by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2008
bugfix: fix regex in update wheel index script by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/2009
fix: Enable SM121 for mm_fp4 by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/2012
fix: ensure SM120/121 SFA/SFB contiguity by @yongwww in https://github.com/flashinfer-ai/flashinfer/pull/1963
More realistic bench for POD Attn by @Edenzzzz in https://github.com/flashinfer-ai/flashinfer/pull/2013
Feature: Support non-gated activation in cutlass fused MoE nvfp4 by @omera-nv in https://github.com/flashinfer-ai/flashinfer/pull/2011
feat: add xqa backend and completes NHD/HND coverage for trtllm-gen/xqa backend by @qsang-nv in https://github.com/flashinfer-ai/flashinfer/pull/2001

New Contributors

@NVShreyas made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/1894
@akhilg-nv made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/1926
@ZhuJiaqi9905 made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/1930
@flashinfer-bot made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/1945
@netanel-haber made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/1948
@ChristinaZ made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/1831
@cicirori made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/1982
@Tom-Zheng made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2002
@omera-nv made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/2011

Full Changelog: https://github.com/flashinfer-ai/flashinfer/compare/v0.4.1...v0.5.0

Open source →

0.4.116 Oct 2025

fix: fix the failed sampling unittest on 5090 by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/1886

What's Changed

fix: fix the failed sampling unittest on 5090 by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/1886
Updated to latest docker tag by @nvmbreughe in https://github.com/flashinfer-ai/flashinfer/pull/1889
Fix: Prevent race condition in cubin loader when file is being consumed by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/1852
Improve graph caching of cudnn graph by @Anerudhan in https://github.com/flashinfer-ai/flashinfer/pull/1887
misc: Various Updates to Attention Microbenchmark Suite by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/1891
docs: Fix installation instructions for CUDA-specific package URLs by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/1893
docker image improvements by @nvmbreughe in https://github.com/flashinfer-ai/flashinfer/pull/1890
tests: Add batch size 1 cases to test_trtllm_gen_attention.py that fail, marked xfail by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/1897
Ensure docker installs the torch version we need by @nvmbreughe in https://github.com/flashinfer-ai/flashinfer/pull/1901
bugfix: exclude tests/utils/test_load_cubin_compile_race_condition.py from pytest by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/1907
ci: use self-hosted runner for building docker containers by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/1908
feat: Add FP4 TRTLLM-Gen throughput MOE batched gemms by @jiahanc in https://github.com/flashinfer-ai/flashinfer/pull/1882
Update Docker CI tags to 20251010-8d072e6 by @github-actions[bot] in https://github.com/flashinfer-ai/flashinfer/pull/1915
ci/cd: consolidate release workflow by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/1910
bugfix: fix cli error when cuda toolkit is not installed by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/1905
feat: trtrllm-gen global scaled FP8 GEMMs by @hypdeb in https://github.com/flashinfer-ai/flashinfer/pull/1829
feat:enable fp8 blockscale moe for fused cultass for sm90 by @djmmoss in https://github.com/flashinfer-ai/flashinfer/pull/1819
use ffi::TensorView instead of ffi::Tensor by @cyx-6 in https://github.com/flashinfer-ai/flashinfer/pull/1844
Minor updates to cubin_loader.py download_file to avoid race condition on temporary file by @nvjullin in https://github.com/flashinfer-ai/flashinfer/pull/1918
chore: make cache directory flashinfer-version specific by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/1920
misc: checksum check when downloading artifacts by @jimmyzho in https://github.com/flashinfer-ai/flashinfer/pull/1761
release: bump version v0.4.1 by @yzh119 in https://github.com/flashinfer-ai/flashinfer/pull/1921

New Contributors

@jiahanc made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/1882

Full Changelog: https://github.com/flashinfer-ai/flashinfer/compare/v0.4.0...v0.4.1

Open source →