NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
NuGet, #1988 most downloaded on NuGet
This package contains native shared library artifacts for all supported platforms of ONNX Runtime.
Latest stable
1.31.1
Last release 24 days ago
28 Oct 2026
Ships fairly regularly
a new release about every 3 weeks
Nearly every release is documented
notes for 58 of the last 60 stable releases
4 versions withdrawn
withdrawn after publishing
127 years old
70 releases · first in 1900
Drag across the ruler to compare, or tap a release to open it.
As noted in the deprecation notice in ORT 1.9, InferenceSession now requires the providers parameters to be set when enabling Execution Providers othe…
Contributors to ONNX Runtime include members across teams at Microsoft, along with our community members: snnn, gineshidalgo99, fs-eire, gwang-msft, edgchen1, hariharans29, skottmckay, jeffdaily, baijumeswani, fdwr, smk2007, suffiank, souptc, RyanUnderhill, iK1D, yuslepukhin, chilo-ms, satyajandhyala, hanbitmyths, thiagocrepaldi, wschin, tianleiwu, pengwa, xadupre, zhanghuanrong, SherlockNoMad, wangyems, RandySheriffH, ashbhandare, tiagoshibata, yufenglee, mindest, sumitsays, MaajidKhan, gramalingam, tracysh, georgen117, jywu-msft, sfatimar, martinb35, nkreeger, ytaous, ashari4, stevenlix, chandru-r, jingyanwangms, mosdav, raviskolli, faxu, liqunfu, kit1980, weixingzhang, pranavsharma, jcwchen, chenfucn, BowenBao, jeffbloo
Your coding agent can read these notes before it upgrades. Set up the MCP server →
SessionOptionsAppendExecutionProvider_TensorRT API is deprecated; use SessionOptionsAppendExecutionProvider_TensorRT_V2
uname -m output of your target device.--_use_extensions option to run models with custom operators implemented in onnxruntime-extensionsCUDA EP
TensorRT EP
OpenVINO EP
DirectML EP
Contributors to ONNX Runtime include members across teams at Microsoft, along with our community members: edgchen1, gwang-msft, tianleiwu, fs-eire, hariharans29, skottmckay, baijumeswani, RyanUnderhill, iK1D, souptc, nkreeger, liqunfu, pengwa, SherlockNoMad, wangyems, chilo-ms, thiagocrepaldi, KeDengMS, suffiank, oliviajain, chenfucn, satyajandhyala, yuslepukhin, pranavsharma, tracysh, yufenglee, hanbitmyths, ytaous, YUNQIUGUO, zhanghuanrong, stevenlix, jywu-msft, chandru-r, duli2012, smk2007, wschin, MaajidKhan, tiagoshibata, xadupre, RandySheriffH, ashbhandare, georgen117, Tixxx, harshithapv, Craigacp, BowenBao, askhade, zhangxiang1993, gramalingam, weixingzhang, natke, tlh20, codemzs, ryanlai2, raviskolli, pranav-prakash, faxu, adtsai, fdwr, wenbingl, jcwchen, neginraoof, cschreib-ibex
This release contains fixes and key updates for 1.8.0. For all package installation details, please refer to https://www.onnxruntime.ai.
This release contains fixes and key updates for 1.8.0. For all package installation details, please refer to https://www.onnxruntime.ai.
Performance updates for ONNX Runtime for PyTorch (training acceleration for PyTorch models)
Building onnxruntime from source now requires a C++ compiler with full C++14 support.
--enable_type_reduction flag, and perform minimal build --enable_reduced_operator_type_support flagMemory footprint reduction through shared pre-packed weights for shared initializers
Memory footprint reduction through arena shrinkage
Quantization
Transformers
Contributors to ONNX Runtime include members across teams at Microsoft, along with our community members:
snnn, gwang-msft, baijumeswani, fs-eire, edgchen1, zhanghuanrong, yufenglee, thiagocrepaldi, hariharans29, skottmckay, weixingzhang, tianleiwu, SherlockNoMad, ashbhandare, tracysh, satyajandhyala, liqunfu, iK1D, RandySheriffH, suffiank, hanbitmyths, wangyems, askhade, stevenlix, chilo-ms, smk2007, kit1980, codemzs, raviskolli, pranav-prakash, chenfucn, xadupre, gramalingam, harshithapv, oliviajain, xzhu1900, ytaous, MaajidKhan, RyanUnderhill, mrry, orilevari, jingyanwangms, sfatimar, KeDengMS, jywu-msft, souptc, adtsai, tlh20, yuslepukhin, duli2012, pranavsharma, faxu, georgen117, jeffbloo, Tixxx, wschin, YUNQIUGUO, tiagoshibata, martinb35, alberto-magni, ryanlai2, Craigacp, suryasidd, fdwr, jcwchen, neginraoof, natke, BowenBao
Deprecated support for OpenVINO 2020.2
Starting from this release, all ONNX Runtime CPU packages are now built without OpenMP. A version with OpenMP is available on Nuget (Microsoft.ML.OnnxRuntime.OpenMP) and PyPi (onnxruntime-openmp). Please report any issues in GH Issues.
Note: The 1.7.0 GPU package is uploaded on this Azure DevOps Feed due to the size limit on Nuget.org. Please use 1.7.1 for the GPU package through Nuget.
Contributors to ONNX Runtime include members across teams at Microsoft, along with our community members: edgchen1, snnn, skottmckay, gwang-msft, hariharans29, tianleiwu, xadupre, yufenglee, ryanlai2, wangyems, suffiank, liqunfu, orilevari, baijumeswani, weixingzhang, pranavsharma, RandySheriffH, ashbhandare, oliviajain, smk2007, tracysh, stevenlix, fs-eire, Craigacp, faxu, mrry, codemzs, chilo-ms, jcwchen, zhanghuanrong, SherlockNoMad, iK1D, askhade, zhangxiang1993, yuslepukhin, tlh20, MaajidKhan, wschin, smkarlap, wenbingl, pengwa, duli2012, natke, alberto-magni, Tixxx, HectorSVC, jingyanwangms, jstoecker, kit1980, suryasidd, RandyShuai, sfatimar, jywu-msft, liuziyue, mosdav, thiagocrepaldi, souptc, fdwr
OpenMP will be disabled in future official builds (build option will still be available). A NoOpenMP version of ONNX Runtime is now available with thi
_mm_pause() for spin loop to improve performance and power consumptionOrtValue data structure
OrtValue (check Scenario 4 in the IOBinding section of the documentation for an example)OrtValue instances can be used to bind inputs/outputs. This is in addition to existing interfaces that allows binding a piece of memory directly/using numpy arrays that can be bound and may be particularly useful when binding ORT allocated device memory.ReleaseEnv from DLLMain or put OrtEnv in global variables. It is not safe to call FreeLibrary from DllMain. - referenceContributors to ONNX Runtime include members across teams at Microsoft, along with our community members: gwang-msft, snnn, skottmckay, edgchen1, hariharans29, wangyems, yufenglee, yuslepukhin, tianleiwu, SherlockNoMad, tracysh, ryanlai2, askhade, xadupre, liqunfu, RandySheriffH, jywu-msft, KeDengMS, pranavsharma, mrry, ashbhandare, iK1D, RyanUnderhill, MaajidKhan, wenbingl, kit1980, weixingzhang, tlh20, suffiank, Craigacp, smkarlap, stevenlix, zhanghuanrong, sfatimar, ytaous, tiagoshibata, fdwr, oliviajain, alberto-magni, jcwchen, mosdav, xzhu1900, wschin, codemzs, duli2012, smk2007, natke, zhijxu-MS, manashgoswami, zhangxiang1993, faxu, HectorSVC, take-cheeze, jingyanwangms, chilo-ms, YUNQIUGUO, jgbradley1, jessebenson, martinb35, Andrews548, souptc, pengwa, liuziyue, orilevari, BowenBao, thiagocrepaldi, jeffbloo
This is a minor patch release on 1.5.1 with the following changes:
This is a minor patch release on 1.5.1 with the following changes:
Reduced Operator Kernel build allows ORT binaries to be built with only required operators in the model(s) - learn more
MacOS binaries now rely on openmp to be installed. See this for reference.
Contributors to ONNX Runtime include members across teams at Microsoft, along with our community members:
gwang-msft, snnn, skottmckay, hariharans29, thiagocrepaldi, tianleiwu, wangyems, RandySheriffH, yufenglee, SherlockNoMad, smk2007, jywu-msft, liqunfu, edgchen1, yuslepukhin, tiagoshibata, fdwr, ashbhandare, iK1D, wschin, BowenBao, zhanghuanrong, RyanUnderhill, ryanlai2, askhade, pranavsharma, martinb35, suffiank, ytaous, KeDengMS, rayankrish, natke, YUNQIUGUO, range4life, smkarlap, zhangxiang1993, xzhu1900, codemzs, weixingzhang, stevenlix, tracysh, mosdav, jingyanwangms, tlh20, souptc, orilevari, kit1980, yangchen-MS, faxu, fs-eire, wenbingl, chilo-ms, xkszltl, Andrews548, yuzawa-san, MaximKalininMS, jgbradley1, nickfeeney, zhijxu-MS, Tixxx, suryasidd, Craigacp, duli2012, jeffbloo
Performance optimizations for Transformer models
New Execution Providers available for preview:
Contributors to ONNX Runtime include members across teams at Microsoft, along with our community members:
snnn, tianleiwu, edgchen1, hariharans29, skottmckay, tracysh, yufenglee, fs-eire, codemzs, tiagoshibata, yuslepukhin, gwang-msft, wschin, smk2007, prabhat00155, liuziyue, liqunfu, ytaous, iK1D, BowenBao, askhade, pranavsharma, faxu, jywu-msft, ryanlai2, xzhu1900, KeDengMS, tlh20, smkarlap, weixingzhang, jeffbloo, RyanUnderhill, mrry, jgbradley1, stevenlix, zhanghuanrong, suffiank, Andrews548, pengwa, SherlockNoMad, orilevari, duli2012, yangchen-MS, yan12125, jornt-xilinx, ashbhandare, neginraoof, Tixxx, thiagocrepaldi, Craigacp, mayeut, chilo-ms, prasanthpul, martinb35, manashgoswami, zhangxiang1993, suryasidd, wangyems, kit1980, RandySheriffH, fdwr
Function expansion support that enables several new ONNX 1.7 ops such as NegativeLogLikelihoodLoss, GreaterOrEqual, LessOrEqual, Celu to run without a
Please note that this version of onnxruntime depends on Visual C++ 2019 runtime. Previous versions depended on Visual C++ 2017. Please also refer https://github.com/microsoft/onnxruntime/tree/rel-1.3.0#system-requirements for the full set of system requirements.
We'd like to recognize our community members across various teams at Microsoft and other companies for all their valuable contributions. Our community contributors in this release include: Adam Pocock, pranavm-nvidia, Andrew Kane, Takeshi Watanabe, Jianhao Zhang, Colin Jermain, Andrews548, Jan Scholz, Pranav Prakash, suryasidd, and S. Manohar Karlapalem.
The ONNX Runtime Training code was originally developed internally at Microsoft, before being ported to Github. We’d like to recognize the original contributors: Aishwarya Bhandare, Ashwin Kumar, Cheng Tang, Du Li, Edward Chen, Ethan Tao, Fanny Nina Paravecino, Ganesan Ramalingam, Harshitha Parnandi Venkata, Jesse Benson, Jorgen Thelin, Ke Deng, Liqun Fu, Li-Wen Chang, Peng Wang, Sergii Dymchenko, Sherlock Huang, Stuart Schaefer, Tao Qin, Thiago Crepaldi, Tianju Xu, Weichun Wang, Wei Zuo, Wei-Sheng Chin, Weixing Zhang, Xiaowan Dong, Xueyun Zhu, Zeeshan Siddiqui, and Zixuan Jiang.
[Preview] Availability of Windows Machine Learning (WinML) APIs in Windows builds of ONNX Runtime, with DirectML for GPU acceleration
We'd like to recognize our community members across various teams at Microsoft and other companies for all their valuable contributions. Our community contributors in this release include: Eric Cousineau (Toyota Research Institute), Adam Pocock (Oracle), tinchi, Changyoung Koh, Andrews548, Jianhao Zhang, nicklas-mohr-jas, James Yuzawa, William Tambellini, Maher Jendoubi, Mina Asham, Saquib Nadeem Hashmi, Sanster, and Takeshi Watanabe.
This is a minor patch release on 1.1.1.
This is a minor patch release on 1.1.1.
This fixes the a minor issue that some logging in execution_frame.cc cannot be controlled by SessionLogVerbosityLevel in SessionOptions. PR #3043
This is a minor patch release on 1.1.0.
This is a minor patch release on 1.1.0.
Performance improvements to accelerate BERT model inference latency on both GPU and CPU. Updates include:
We'd like to thank our community members across various teams at Microsoft and other companies for all the valuable contributions.
We'd like to extend special recognition to these individuals for their contributions in this release: Jianhao Zhang (JD AI), Adam Pocock (Oracle), nihui (Tencent), and Nick Groszewski. From the Intel teams, we'd like to thank Patrick Foley, Akhila Vidiyala, Ilya Lavrenov, Manohar Karlapalem, Surya Siddharth Pemmaraju, Sreekanth Yalachigere, Michal Karzynski, Thomas V Trimeloni, Tomasz Dolbniak, Amy Zhuang, Scott Cyphers, Alexander Slepko and other team members on their valuable work to support the Intel Execution Providers for ONNX Runtime.
ONNX 1.6 compatibility - operator support for all opset11 ops on CPU, including Sequence ops.
Fix in C# API marshalling for InferenceSession.Run()
Bug Fixes
Only NuGet packages are released for this patch release, because only the C# API users are impacted
MKL-DNN provider (subgraph based execution) for improved performance
This release brings kernel performance improvements, expanded operator support, and reliability fixes to the ONNX Runtime WebGPU Plugin EP.
This release brings kernel performance improvements, expanded operator support, and reliability fixes to the ONNX Runtime WebGPU Plugin EP.
MatMulNBits wide-tile execution using subgroup shuffle (#31703)Split when every output segment is vec4-aligned (#32251)EngramGate, NGramHashMapping) (#32106, #32268)MatMulNaiveProgram's pipeline cache key cover everything it bakes into WGSL (#32048)copy_tensors misuse instead of terminating the process, and rejected foreign GPU handles in the built-in data transfer (#32315, #32317)onnxruntime_providers compilation from its archive (#32162)wgsl_template Python tests in CI (#32214)Thanks to everyone who contributed to this release:
@4n4ny4, @daijh, @edgchen1, @hariharans29, @jchen10, @kunal-vaishnavi, @qjia7, @rvandermeulen, @sushraja-msft, @tianleiwu, @xhcao, @xiaofeihan1
This release covers commits affecting WebGPU Plugin EP code and packaging.
This summary was drafted with AI assistance from commit history and PR metadata, and reviewed before publishing.
This is a patch release for 0.3.0.
This is a patch release for 0.3.0.
ONNX Runtime WebGPU Plugin EP 0.3.0 expands model and data-type coverage, improves generative-model performance, and strengthens configuration, reliab
ONNX Runtime WebGPU Plugin EP 0.3.0 expands model and data-type coverage, improves generative-model performance, and strengthens configuration, reliability, and release tooling.
These release notes were drafted with AI assistance.
int64 for Add, Cast, Clip, Concat, Equal, Gather, Min, Max, ReduceSum, Reshape, Sub, Tile, and Where; uint8 for Cast, Expand, Gather, and Reshape; and int32/uint32 for CumSum and Tile. (#28804, #29392, #29830, #29834, #29839, #29844, #29847, #29854, #29861, #29897, #31049, #31702, #31709, #31714)GatherBlockQuantized support and integrated ONNX 1.22 with opset 27. (#29054, #28754)GatherBlockQuantized, and fixed WebGPU data-transfer callbacks on Windows x86. (#28704, #29030, #29255, #29595, #31568)Thank you to everyone who contributed to this release:
@AngelGalindo7, @daijh, @danielsongmicrosoft, @edgchen1, @fanchenkong1, @feich-ms, @guschmue, @haoxli, @hariharans29, @Honry, @huningxin, @jchen10, @Jiawei-Shao, @miaobin, @mingmingtasd, @mirounga, @mustjab, @nicholascelestin, @prathikr, @qjia7, @Reranko05, @Shivani767, @skottmckay, @ssam18, @sushraja-msft, @tairenpiao, @tianleiwu, @titaiwangms, @wuisabel-gif, @xhcao, and @xiaofeihan1.
Scope: commits affecting ONNX Runtime WebGPU Plugin EP code, tests, build integration, and packaging since plugin-ep-webgpu/v0.2.1.
This release brings sparse attention and packed continuous-batching support, INT2 and mixed-width QMoE execution,
decode-kernel tuning, and reliability fixes to the ONNX Runtime CUDA Plugin EP. Highlights include new QSA/CSA
attention operators, expanded paged speculative decoding, opt-in MatMul auto-tuning, and improved MXFP4 support
on Blackwell consumer GPUs.
The minimum ONNX Runtime core version remains 1.24.4, but meeting that minimum alone does not guarantee
compatibility for contributed operators. Core and the plugin must use the same contributed-operator schemas;
building both from the same ONNX Runtime revision is recommended. Schema mismatches can cause incorrect execution
or crashes and may not be detected when registering the plugin (#32855).
The matched ONNX Runtime version is 1.31.0 for this release.
GroupQueryAttention workspace recipes for KV-cache preparation and XQA, FlashAttention, memory-efficient, and unfused routes, plus conservative workspace estimation for resource-constrained graph partitioning. cuDNN workspace is reported as unavailable rather than incorrectly estimated as zero (#32446, #32453, #32454, #32617).fpA_intB split-K semaphores, paged XQA semaphores, and compact convolution state updates, reducing CUDA Graph dependency overhead (#32884).Gemm and DecoderAttention with direct broadcasts on the consuming CUDA stream, removed the constant-ones cache, and corrected zero-K Gemm bias scaling (#32852).PagedAttention for eligible FP16/BF16 causal workloads with unquantized KV caches. It is selected ahead of FlashAttention when XQA is not the winner, automatically considered on SM90+, and can be disabled with ORT_ENABLE_CUDNN_FLASH_ATTENTION=0. Follow-up fixes stabilize default-scale cache keys and backend selection during CUDA Graph execution (#32493, #32624).PagedAttention beyond FlashAttention, including portable paths for small cache pages and supported XQA speculative paths; added coverage for head size 128 with 16/32/64-token pages and ragged batches (#32619).Attention when the outputs alias the external KV-cache inputs (#31150).SparsePagedAttention and DynamicSparseAttention for externally selected attention indices over paged and contiguous KV caches. These support Qwen4-Exp QSA selected-main-cache attention and DeepSeek V4 CSA joint local/selected-auxiliary attention, with one softmax across participating entries and optional attention sinks (#32524, #32525).SparseAttentionIndexer for QSA token selection and CSA compressed-block selection, including rotary preparation, scoring, deterministic TopK, and cache updates (#32526).PackedSparseAttentionIndexer for packed, variable-length continuous batching, with per-request metadata, fixed-capacity state, and selection outputs compatible with SparsePagedAttention (#32618).DynamicSparseAttention across selected-token tiles, bounded split workspace, replaced quadratic duplicate validation with a GPU bitmap, and added single-split fast paths for large-context inference (#32671).EngramGate and NGramHashMapping for Qwen4-Exp, including normalized gated-value output, autoregressive history, per-head offsets, and EOS/segment resets (#32285).VarlenNGramHashMapping for packed DeepSeek Engram batching, preserving request-local history and preventing n-gram mixing across request boundaries, then extended it with Qwen4-Exp hashing semantics (#32358, #32467).activation attribute to GatedRMSNorm for SiLU, its Swish alias, and Sigmoid gating. SiLU remains the default (#32512).BranchwiseRMSNorm, ScaledSiLU, HyperConnectionPreMix, and HyperConnectionPostMix (#32687).GatherBlockQuantized to FP8 and FP4 data, including full-axis blocks and scale broadcasting. Behavior change: out-of-range indices now produce zero output slices for both integer and floating-point quantized formats, rather than following ONNX Gather error semantics (#32480).(2,2), (2,4), and (4,2), with packed decode paths and a scratch-bounded dequantization fallback for supported configurations (#32697, #32743, #32761).FLOAT8E8M0 type mapping, and accelerated FP4 conversion in decode and grouped GEMM (#33067).MatMul auto-tuning via ep.cuda.enable_gemm_auto_tune=1 or ORT_CUDA_GEMM_AUTO_TUNE=1, choosing among cuBLAS, small-N GEMV, and eligible tinygemm2 kernels on SM90+. cuBLAS remains the default. Graph-enabled sessions validate candidates with CUDA Graph replay before caching the winner; warm-up inference is required before capture (#32876, #32901, #33085).MatMulNBits small-M GEMV tiling, biased near-tie tactic selection toward GEMV for small decode batches, and enabled the W8 M=6-8 fast path for the qualified SM121 FP16 shape K=3584, N=200064, block size 64 (#32876, #32884, #32742).MatMulBlockQuantizedFp8Weight workloads with a_scale and block size 128, avoiding the FP16/BF16 weight-dequantization buffer. Enable with ORT_FP8_MATMUL_DEEPGEMM=1 in a supported non-Windows DeepGEMM build; it is disabled by default and native FP8 rounding requires model-accuracy validation (#32599).MatMulNBits GEMV experiments: ep.cuda.fpa_intb_gemv_wave_aware=1 for eligible SM12x FP16/INT4 M=8 tiles, and ep.cuda.fpa_intb_gemv_paired_k=1 (or ORT_FPA_INTB_GEMV_PAIRED_K=1) for eligible FP16/INT4 block-32 M=5-8 profiling buckets. Both are disabled by default. The paired-K tactic is experimental: FP16 partial sums can overflow even when the default GEMV remains finite, so numerical validation is required; tactic profiling checks speed, not numerical safety (#33186, #33188).Split fast path for 2-4 outputs, passing metadata by value and eliminating per-invocation pinned-memory allocations and metadata uploads (#32706).MatMulNBits optional-input detection, including omitted edges and empty placeholders, and restored zero-point/scale type comparisons (#32970).MatMulInteger, Conv/FusedConv/ConvTranspose, Squeeze, and GemmFloat8, and corrected the GemmFloat8 C-operand descriptor type (#32971, #31967).GatherND index bounds checks (#31641, #31646).PackedMultiHeadAttention and RestorePadding token offsets before invalid values can participate in device address calculations (#33098, #33099).Split dimension products and rejected generation-buffer expansion when the input sequence exceeds the maximum sequence length (#32551, #32553).Resize handling for out-of-range coordinates and bounded coefficient windows (#33056, #33095).GatedDeltaNet shape inference for rank-2/rank-3 packed QKV with omitted key/value inputs, while preserving separate-Q/K/V behavior (#32708).onnxruntime_USE_FLASH_ATTENTION=OFF by excluding FlashAttention CUDA sources, and fixed builds with memory-efficient attention disabled (#32580, #32588).Thanks to the 17 human contributors who contributed to this release:
@apsonawane,
@baijumeswani,
@bheu,
@chilo-ms,
@edgchen1,
@eserscor,
@gramalingam,
@hanbitmyths,
@hariharans29,
@jiafatom,
@justinchuby,
@kunal-vaishnavi,
@SIDDARTHAREDDY8,
@skottmckay,
@tianleiwu,
@titaiwangms,
@xadupre
This release covers changes from origin/plugin-ep-cuda/rel-0.2 to origin/plugin-ep-cuda/rel-0.3 affecting CUDA Plugin
EP code, shared CUDA kernels, build integration, and packaging. The highlights focus on user-facing changes rather
than listing every infrastructure-only update.
This summary was drafted with AI assistance from commit history and PR metadata.
ONNX 1.4 compatibility
New build packages
Custom op registration via C API
Non-Tensor type support for input/output for C and C# API
Major performance work for attention-heavy LLMs.
Major performance work for attention-heavy LLMs.
max_k_step was enabled for NVIDIA (#28511).Qwen3 and Gemma 4 model-path improvements.
LinearAttention and quantized-path optimizations.
Reliability and hardening fixes.
past_state == present_state buffer handling (#28753).Graph-capture and buffer-management improvements.
Note: This section was AI-generated. It may have inaccuracies.
Thanks to everyone who contributed to the WebGPU EP (human contributors, alphabetical):
@apsonawane, @daijh, @edgchen1, @feich-ms, @GopalakrishnanN, @guschmue, @hariharans29, @HectorSVC, @jchen10, @qjia7, @tianleiwu, @xiaofeihan1, @xenova, @yuslepukhin.
Note: This list was compiled on a best-effort basis from PRs that touched WebGPU EP-specific paths and
intentionally includes human contributors only, so it may not capture every contribution. If yours was
missed, the omission is unintentional. Your work is no less appreciated.
This release brings expanded quantized inference and attention support, kernel performance improvements, and reliability fixes to the ONNX Runtime CUD
This release brings expanded quantized inference and attention support, kernel performance improvements, and
reliability fixes to the ONNX Runtime CUDA Plugin EP. Highlights include 2-bit MatMulNBits, INT4 paged KV caches,
speculative decoding improvements, compact recurrent-state updates, and safer CUDA Graph execution across sessions.
Please refer to the CUDA Plugin EP Quick Start for usage and installation guidance. The matched ONNX Runtime version is 1.30.0.
GatherND capture-safe and kept CudaAsyncBuffer staging buffers alive across graph replay (#32120, #32121).GroupQueryAttention support and an is_causal attribute to PagedAttention (#31704, #32225).onnxruntime_USE_INT4_KV_CACHE build option, which defaults to enabled (#32515).PagedAttention dispatch diagnostics (#32102, #32099).session.gqa_value_layout session option for applications using a BNHS GroupQueryAttention Value cache, with graph-inserted layout conversions; BNSH remains the default (#32139).GatedDeltaNet with transition capture for replaying accepted speculative prefixes, plus BF16 support (#32282, #32307).VarlenCausalConvWithState for packed, variable-length continuous batching, and compact state updates that avoid duplicating full convolution-state checkpoints for speculative prefixes (#32168, #32290).EngramGate and NGramHashMapping (#32268).ReduceMean kernels (#32326).MatMulNBits support, including dequantization and specialized GEMV/batched paths, and fused floating-point-activation/integer-weight (fpA_intB) GEMM/GEMV kernels for eligible FP16/BF16 workloads (#32693, #32699).fpA_intB kernels the default build configuration, extended compact MatMulNBits to BF16, and refined GEMV eligibility checks. The complete kernel matrix remains available through onnxruntime_USE_FPA_INTB_GEMM_FULL (#32324, #32721, #32338).ORT_QMOE_FP4_DEEPGEMM and disabled by default. This path is disabled on Windows because of upstream header requirements (#32122, #32485).fpA_intB tactic-profiling work and added optional, size-gated M-row chunking for large MatMulNBits workloads. Chunking is disabled by default and can be configured with ep.cuda.matmul_nbits_m_chunk_size or ORT_MATMULNBITS_M_CHUNK_SIZE (#32758, #32810).ArgMax/ArgMin over wide last axes, optimized wide-last-axis TopK, and accelerated low-lane INT64 CumSum (#32092, #32404, #32238).Slice fast path for contiguous subregions (#28902).Concat per-input metadata by value and avoided pinned buffers in CUDA Split and Concat fast paths (#32119, #32410).ScatterElements reduction dispatch by element type and Abs signed-zero handling (#29879, #31477).BiasGelu and FastGelu as no-ops and zero-sized outputs in CUDA random generator kernels (#31698, #31997).Split sizes, 8-bit MatMulNBits g_idx, GatherElements counts, and ScatterND index depth (#29461, #31643, #32030, #32034).ImageScaler bias, and Crop inputs (#32014, #32029, #32002, #32157).RemovePadding sequence-token counts and Whisper beam-search cross-QK layer/head indices (#31994, #31998).RotaryEmbedding, SparseAttention, CUDA reduction scans, and softmax offsets (#31995, #31996, #32137, #32330).PagedAttention builds without FlashAttention and added the CUDA 13 CCCL include path to plugin builds (#32327, #32392).Thanks to the 17 human contributors who contributed to this release:
@apsonawane, @arnej27959, @baijumeswani, @chilo-ms, @danfiedler-msft, @DKAIN-py, @edgchen1, @eserscor, @javier-intel, @jiafatom, @justinchuby, @kunal-vaishnavi, @MohamedElashri, @Noperi0r, @sanaa-hamel-microsoft,
@tianleiwu, @titaiwangms
This release covers changes since CUDA Plugin EP v0.1.0 affecting CUDA Plugin EP code, CUDA kernels, build integration,
and packaging. The highlights focus on user-facing changes rather than listing every infrastructure-only update.
This summary was drafted with AI assistance from commit history and PR metadata.
Full Changelog: plugin-ep-cuda/v0.1.0...plugin-ep-cuda/v0.2.0
This is just a minor patch to the previous 0.1.4 release.
This is just a minor patch to the previous 0.1.4 release.
One column per quarter.