npm · #4595 most downloaded on npm

onnxruntime-common

1.30.040M downloads/moMicrosoft/onnxruntime ↗New releases in your daily digest, changelog included.

ONNXRuntime JavaScript API library

Last release 20 days ago
14 Sep 2026
Release timing varies
gaps range from 9 days to 3 months
Nearly every release is documented
notes for 34 of 35 stable releases
Nothing withdrawn
no release was ever pulled
5 years old
133 releases · first in 2021

Release timeline

133 releases since 2021

202120222023202420252026

20212026

One column per quarter.

Releases

1.31.0-dev.20260911-2a43ec07e11 Sept 2026pre-release
Nothing published for this version
1.30.014 Sept 2026
ONNX Runtime 1.30.0 expands generative AI inference, improves CPU and GPU performance, adds Go bindings, and strengthens runtime reliability. These no
ONNX Runtime 1.30.0 expands generative AI inference, improves CPU and GPU performance, adds Go bindings, and strengthens runtime reliability. These notes cover changes since ONNX Runtime 1.29.1.

Highlights

Expanded CUDA inference support with variable-length causal convolution for continuous batching, speculative decoding in paged XQA, and INT4 paged KV caches with per-channel scales (#32168, #32340, #32515).

Improved WebGPU PagedAttention, added GPT-OSS support and INT8 KV-cache block quantization, and extended convolution optimizations (#31727, #32277, #32284, #32420).

Added fused CPU LinearAttention kernels for AVX-512, Arm64 NEON, and SVE, plus AVX2 LayerNorm/RMSNorm acceleration (#31674, #31973, #32178, #32356).

Added Go bindings for the ONNX Runtime C API and DeepSeek Engram contrib operators (#29615, #32268).

Announcements & Compatibility

FP4 QMoE kernels are now enabled by default in CUDA builds, with Windows build support added in this release. Source builds can opt out with -Donnxruntime_USE_FP4_QMOE=OFF (#32096, #32163).

CUDA fpA-intB builds now default to a compact kernel set for FP16 activations, INT4/INT8 weights, scale-only quantization, and block_size=32. Set -Donnxruntime_USE_FPA_INTB_GEMM_FULL=ON when building from source to retain the full kernel set, including BF16, zero-point, bias, larger-block-size, and native Hopper variants (#32324).

CPU FP16 Gemm and MatMul execution is gated on hardware acceleration. CPU-assigned FP16 nodes without a matching kernel now fall back to FP32 (#32301, #32197).

WebGPU plugin EP packaging now supports Linux AArch64. Plugin versions were advanced to WebGPU 0.4.0 and CUDA 0.2 (#32287, #31960, #31970).

Security & Reliability

Model Loading, Memory, and Input Validation

Limited nested model-graph depth and canonicalized external-data locations to harden model loading (#32344, #32135).

Added checked rounding for BFC arena allocations and fixed prepacked-weight reference lifetimes (#32010, #32040).

Strengthened shape, rank, and parameter validation for Split, Scan, GatherND, ScatterND, SpaceToDepth/DepthToSpace, Crop, Conv, Normalizer, and pooling (#29461, #31668, #32034, #32039, #32076, #32157, #32160, #32161, #32345, #32349).

Hardened generation and attention input handling, including attention-attribute narrowing, BifurcationDetector inputs, generation subgraph shapes, and QEmbed segment inputs. BeamSearch buffer expansion now uses dynamic shape storage (#31648, #31701, #32009, #32078, #32144).

Validated TreeEnsemble node references and bounded subtree comparison, rejected non-finite CPU RoiAlign coordinates, and required ImageScaler bias to match the channel count (#32031, #32043, #32011, #32002).

Added an allowlist of safe LoRA adapter parameter data types, validated MatMulFpQ4 shape inputs, and checked MLAS blockwise quantization/dequantization index ranges (#31682, #32032, #32007).

GPU Bounds and Resource Lifetimes

Hardened CUDA indexing and buffer-size arithmetic in MatMulNBits, RemovePadding, RotaryEmbedding, SparseAttention, Whisper beam search, NMS, QDQ, and GatherElements (#31643, #31994, #31995, #31996, #31998, #32014, #32029, #32030).

Fixed overflow in CUDA reduction scans and Softmax offset arithmetic, and handled zero-sized outputs in CUDA random-generator kernels (#32137, #32330, #31997).

Fixed CUDA MultiHeadAttention shared-cache scratch lifetimes and kept CudaAsyncBuffer staging storage alive across CUDA graph replay (#31968, #32121).

Fixed WebGPU out-of-bounds subgroup-matrix loads for partial tiles, zero-initialized writable device-allocator buffers, and rejected foreign GPU handles in built-in data transfers (#32364, #32063, #32317).

Dependencies and Tooling

Upgraded Protobuf to 33.6 and refreshed Python documentation dependencies, including an ONNX security-related update (#29906, #32190, #32424).

Updated JavaScript dependencies including js-yaml, joi, fast-uri, and the Next.js end-to-end fixture (#32397, #32486, #32488, #32505, #32508).

Pinned GitHub Actions to full-length commit SHAs and strengthened packaging infrastructure with authenticated package feeds and NPM network isolation (#32176, #32005, #32440).

New Features

Core APIs & Runtime

Added Go bindings for the ONNX Runtime C API (#29615).

Extended memory importing with host-pointer support and added access to preallocated outputs through KernelContext::GetPreallocatedOutput (#29726, #32089).

Added packed-attention workspace recipes and estimates, and made workspace input-shape handling aware of optional inputs (#32283, #32321, #32312).

Added DeepSeek Engram contrib operators, EngramGate and NGramHashMapping, and expanded kernel coverage for Qwen-3.5 operators (#32268, #32106).

Plugin Execution Providers

Added LoRA adapter support with plugin EP allocators, preserved custom allocators during EP registration, and reset plugin stream chunks before release (#32221, #32272, #31983).

Fixed CUDA plugin device discovery on WSL and improved Windows ARM64 packaging (#32517, #32355).

Execution Provider Updates

CUDA EP

Attention and Decoding

Added INT4 paged KV caches with per-channel scales and an is_causal attribute to PagedAttention (#32515, #32225).

Extended paged XQA with speculative decoding, query-to-KV head group size 6, head size 256 including FP16 caches, and native block tables for 128-token pages (#32340, #32108, #32229, #32263, #32127).

Enabled split-KV for paged FlashAttention decode and improved PagedAttention dispatch diagnostics (#32102, #32099).

Added VarlenCausalConvWithState for continuous batching and compact variable-length causal-convolution state updates (#32168, #32290).

Added a compact GatedDeltaNet operator and BFloat16 support for CUDA GatedDeltaNet (#32282, #32307).

MoE and Quantized Matrix Multiplication

Added an opt-in FP8 DeepGEMM MoE decode path for supported fixed-shape QMoE workloads on Hopper GPUs (ORT_QMOE_FP4_DEEPGEMM=1, default off). This path is disabled on Windows (#32122, #32485).

Bounded QMoE workspace with configurable row tiling and FP8 weight-dequantization scratch memory by tiling over output columns (#32097, #32129).

Vectorized NVFP4 weight dequantization for prefill, tuned NVFP4 GEMV tiling for Qwen multi-token prediction, and extended speculative-decode GEMVs to 64 rows (#32128, #32140, #32289).

Tuned FP4 and FP8 GEMV scheduling for 48-SM SM121 GPUs, including FP8 KSplit32 scheduling, and improved FP8 GEMV residency for grids just beyond two blocks per SM (#32408, #32409, #32433).

Added an opt-in split-K GEMV path for small-N FP16 MatMul shapes and refined fpA-intB GEMV support checks (#31478, #32338).

Operators

Improved TopK, ArgMax, and ArgMin performance for wide last axes, and accelerated low-lane INT64 CumSum (#32404, #32092, #32238).

Added a single-memcpy Slice fast path for contiguous subregions and removed pinned-buffer use from Split and Concat fast paths (#28902, #32410).

Registered BF16 ReduceMean kernels and fixed ScatterElements reduction dispatch by element type and signed-zero handling in Abs (#32326, #29879, #31477).

WebGPU EP

Improved PagedAttention and added PagedAttention metadata, GPT-OSS support, and INT8 KV-cache block quantization (#31727, #32277, #32284).

Added INT64 Gather support and optimized MatMulNBits wide tiles with subgroup shuffle (#31714, #31703).

Extended convolution fusion with eight additional activations, fused activation handling in the im2col path, and uniform-based activation parameters for Conv/MatMul (#32117, #32185, #32116).

Reused subgroup-matrix MatMul for pointwise convolution and added convolution-weight prepacking for the im2col-matmul path (#32304, #32420).

Enabled GELU and BiasGELU fusion, added transpose-optimizer handling for Elu and contrib GELU variants, and enabled LayerNorm fusion to fix FP16 inference correctness (#32053, #32118, #32294).

Added subgroup-size control, selected subgroup size 32 for subgroup-matrix MatMul/Gemm, and enabled the subgroup-matrix path in WASM builds (#32056, #32306, #32269).

Scaled Dawn pipeline-compilation workers with CPU count, vectorized Split when all output segments are vec4-aligned, and selected pooling paths based on occupancy (#29820, #32251, #32313).

Exposed safe graph-capture I/O in Python, added a robustness provider option, and enabled forwarding of onnxruntime_perf_test -i options to WebGPU (#32074, #31971, #32316).

Fixed synchronization for user-provided GPUDevice instances, corrected MatMul pipeline-cache keys and the 1D-dispatch shader fast path, and changed copy_tensors misuse to report errors instead of terminating the process (#32259, #32048, #32343, #32315).

WebNN EP

Added SkipLayerNormalization support and corrected output-rank validation and fallback data-type support checks (#32377, #31708, #32067, #32293).

TensorRT & DirectML

Fixed TensorRT shape-value handling (#32415).

Validated kernel_shape and output_padding lengths during DirectML kernel setup (#31999).

CPU & Core Optimizations

MLAS and CPU Kernels

Added fused LinearAttention kernels for AVX-512, Arm64 NEON, and SVE (#31674, #32178, #32356).

Added AVX2 LayerNorm/RMSNorm kernels and registered BFloat16 LayerNorm/RMSNorm kernels on the CPU EP (#31973, #31974).

Added Arm SVE i8mm INT8 QGEMM kernels and enabled the SBGemm fast-math path on Darwin Arm64 (#31146, #32152).

Improved NCHWc convolution thread utilization, added HardSwish fusion for MobileNetV3 models, and introduced an AVX-512 16-wide Erf kernel and NCHWc reorder improvements for MobileCLIP-S0 (#31660, #31957, #31958).

Optimized INT4 weight prepacking on CPU, rejected KleidiAI Q4 prepacking with dynamic scales, and fixed Arm64 SymmQgemm INT16 overflow (#31690, #32068, #32057).

Fixed FP16 QuantizeLinear rounding, prevented CPU TensorScatter index overflow, serialized ScatterND string updates, and widened Compress loop counters (#32452, #32012, #32033, #32008).

Improved edge-case handling for empty LpNormalization inputs, zero-element BiasGelu/FastGelu, extreme Trilu diagonals, and empty reduction axes (#32020, #31698, #32013, #32156).

Graph, Optimizer, and Runtime

Hardened graph optimizers against model-supplied indices and invalid fusion inputs, including MatMulNBits dequantization block sizes, GQA projection shapes, and Slice starts rank (#31670, #31678, #32018, #32044).

Prevented overlapping MatMulIntegerToFloat fusions, skipped overridable-initializer fusion, fixed FuseInitializersTransformer consumer-input lookup, and avoided fusing identity transposes into Gemm (#32038, #32143, #32426, #32435).

Fixed optional zero-point input handling in QDQ, empty-initializer axis scaling, and NodeAttrHelper string-default lifetimes (#32051, #32138, #32019).

Released external-data loaders after graph initialization, added portable random-access file reads to Env, and clarified how external-initializer paths interact with EP context paths (#32502, #32503, #32442).

Avoided redundant ONNX schema registration when static registration is enabled, updated cpuinfo with thread-safe deinitialization, and prevented telemetry crashes in shell-less containers (#32353, #32300, #32226).

Reduced the active-session mutex scope around ETW callback registration and filtered out display adapters using the Microsoft Basic Render Driver (#32000, #32006).

Language Bindings & Web

Retained Python asynchronous-run resources and pinned C# RunAsync arguments until completion (#32041, #32015).

Fixed Java provider-option value cleanup, validated Rust tensor element types, and made Rust string-tensor output extraction safe (#31828, #32035, #32045).

Avoided duplicate N-API cleanup-hook registration in Node.js and added FP16 support to the JavaScript WebGPU Conv3DNaive shader (#32469, #32357).

Fixed WinML image-dimension overflow (#32046).

Build, Packaging & CI

Upgraded CUTLASS to 4.7 and cuDNN Frontend to 1.27, and enabled compact fpA-intB kernel builds by default (#32111, #32324).

Fixed Windows CUDA 12.9 SM120 compilation, CUDA 13 CCCL include paths in plugin builds, Windows DeepGEMM build errors, and PagedAttention builds without FlashAttention (#32114, #32392, #32485, #32327).

Added D3D12 Agility SDK support for Dawn WebGPU builds while excluding WebGPU plugin EP builds from Agility SDK use (#32253, #32380).

Added BTI support to MLAS AArch64 assembly, restored runtime vector-extension checks on RISC-V, and disabled POWER MLAS kernels when VSX is unavailable (#32070, #32406, #32389).

Improved CUDA plugin ARM64 build parallelism and packaging tests, and based plugin development versions on commit timestamps (#32165, #32399, #32072, #32095).

Separated provider compilation from archive creation, corrected Python-binding delay-load linkage, and fixed Linux minimal-build dependencies and spurious Windows Spectre-mitigation warnings (#32162, #31637, #32491, #32516).

Expanded PagedAttention CUDA test coverage and head-size-256 paged XQA coverage, corrected GatedDeltaNet test verification, and removed a Hugging Face download dependency from a dynamic-quantization test (#31687, #32262, #32329, #32443).

Added WGSL template tests to CI and enabled WebGPU CI on WebGPU plugin EP release branches (#32214, #32090).

Contributors

Thanks to our 57 human contributors for this release!

@4n4ny4, @apsonawane, @arnej27959, @baijumeswani, @bmehta001, @chilo-ms, @crvineeth97, @daijh, @danfiedler-msft, @danielsongmicrosoft, @dannyota, @DKAIN-py, @edgchen1, @ericcraw, @eserscor, @fanchenkong1, @hanbitmyths, @hariharans29, @hdharpure9922, @Honry, @jambayk, @javier-intel, @jchen10, @jiafatom, @jnagi-intel, @justinchuby, @kadyrbekovhamit-cyber, @kunal-vaishnavi, @Lapis0x0, @LOGO127, @Manogna-Sree, @martin-klacer-arm, @mei1127, @miaobin, @mirounga, @miyanyan, @MohamedElashri, @mustjab, @Nikhi00718, @Noperi0r, @Novestars, @pkubaj, @preetha-intel, @qjia7, @rvandermeulen, @sanaa-hamel-microsoft, @skottmckay, @sushraja-msft, @swetha097, @sylvesterkaczmarek, @tianleiwu, @titaiwangms, @toothache, @xadupre, @xhcao, @xiaofeihan1, @Zestion

Full Changelog: rel-1.29.1...rel-1.30.0

Release highlights were prepared with AI assistance.

From the same repository

Microsoft/onnxruntime

1.29.024 Aug 2026Security

Fixed a path traversal vulnerability in TensorRT and NvTensorRTRTX engine refitting by making external-data path validation unconditional ( #29396 ).

Announcements & Breaking Changes

onnxruntime-web has announced the deprecation of WebGL and JSEP. The native WebGPU EP is the recommended path going forward. See the deprecation and migration plans for details (#29716, #31683).
POSIX telemetry is now available on Linux, macOS, Android, and iOS when ONNX Runtime is built with telemetry enabled. It does not change the public ABI, WebAssembly remains telemetry-free, and setting ORT_DISABLE_TELEMETRY=1 before initialization disables non-Windows telemetry for the process (#27379, #29872).
The unused internal onnxruntime/python/tools/tensorrt dashboard tooling was removed. This does not affect the TensorRT Execution Provider APIs (#29395).

Security Fixes

Path, bounds, and input validation

Fixed a path traversal vulnerability in TensorRT and NvTensorRTRTX engine refitting by making external-data path validation unconditional (#29396).
Validated the CPU MoE k attribute against the number of experts and fixed a CPU TensorScatter security issue (#29907, #29916).
Added missing rank, shape, and parameter validation for pooling, LSTM and DynamicQuantizeLSTM, Sampling, FeatureVectorizer, SkipLayerNorm, QLinearConv, Whisper decoding, RNN activations, GridSample, contrib Range, and CropAndResize (#29254, #29255, #29265, #29579, #29595, #29605, #29871, #31636, #31671, #31675, #31676, #31684).
Hardened CUDA indexing and buffer handling in GridSample, transpose, GatherBlockQuantized, InstanceNormalization, LayerNorm/RMSNorm, BeamSearch, DeformConv, AveragePool, and MaxPool (#29581, #29631, #29638, #31640, #31642, #31644, #31645, #31647, #31650).
Fixed packed sub-byte tensor over-copying in OrtApi::GetValue and validated DML constant tensor byte sizes (#29157, #31665).

Supply chain and tooling

Updated npm lockfiles, refreshed the Next.js end-to-end fixture lockfile for security advisories, and upgraded adm-zip for onnxruntime-node (#29827, #29926, #31192).

New Features

Core APIs & Runtime

Default intra-op and inter-op thread-pool sizes can now be set with ORT_INTRA_OP_NUM_THREADS and ORT_INTER_OP_NUM_THREADS. Explicit thread settings still take precedence, and 0 preserves machine-sized defaults (#29688).
Added weightless-model support for all initializer types, allowed zero-input EpContext nodes, and wired maximum-shape inference into workspace estimation (#29607, #29799, #31613).
Added ONNX-domain support for rotary embedding and a fused MRotaryEmbedding contrib operator for Qwen mRoPE variants (#29261, #31728).
Added multi-shape profiling to onnxruntime_perf_test through --data_shape, plus verbose graph-transformer tracing and broader inference-session error-path coverage (#29555, #29558, #29569, #29571).

Execution Provider ABI & Plugin EPs

WebGPU now supports device-free compile-only sessions for offline graph transformation (#29681).
Expanded CUDA plugin EP packaging and testing, including Windows ARM64 package and size options, updated package outputs, and aligned architecture selections across Python, C API, TensorRT, Node.js, and plugin packages (#31635, #31722, #31992).
Improved plugin lifecycle handling by unloading failed EP library loads and fixing allocator-deleter lifetime (#29634, #29770).

Execution Provider Updates

NVIDIA CUDA EP

Attention and decoding

Added PagedAttention with quantized KV cache, XQA decode, MLA, QK-Norm, and head-sink support (#29912).
Extended quantized KV-cache support with attention sinks, independent and per-channel scales, sliding-window cache support, and a fused K/V dequantization launch (#29900, #29904, #31480).
Added a cuDNN SDPA decode tier to the standard ONNX Attention CUDA kernel and enabled cuDNN SDPA for contrib Attention (#29715, #29717).
Added attention_bias support to the GroupQueryAttention unfused path and state_window support to LinearAttention and CausalConvWithState for MTP (#29525, #31157).
Fixed LinearAttention on GPUs with limited shared memory (#31982).

MoE and quantized GEMM

Added NVFP4 QMoE, including native FP4xFP4 prefill on SM120, faster decode GEMV, fused routing/finalization paths, and reduced activation and weight-dequantization overhead (#29697, #29824, #29887, #29919, #31156, #31159, #31349, #31479).
Added MatMulBlockQuantizedFp4Weight and MatMulBlockQuantizedFp8Weight, plus block-scaled tensor-core/GEMV decode paths, packed FP4 decode, M-tiling, and folded W8A8 activation QDQ (#29818, #29850, #29896, #31155, #31481).
Improved MatMulNBits and QMoE robustness and efficiency by optimizing 8-bit dequantization, releasing raw MXFP4 initializers after prepack, and fixing subgraph prepacking and mixed FP8/FP4 build failures (#29852, #31141, #31154, #31350).

Operators and collectives

Added LinearAttentionGate, GatedRMSNorm, and GatedAdd contrib operators (#31158, #31835).
Added bfloat16 support to AllReduce, AllGather, and AllToAll (#31571).
Fixed the default zero point in CUDA GatherBlockQuantized (#31693).

WebGPU EP

Added DFT, HardSwish, Max/Min, Trilu, GRU, PRelu, MatMulBnb4, and MRotaryEmbedding support (#29454, #29587, #29828, #29833, #29840, #29845, #30512, #31976).
Expanded integer support across Clip, Reshape, Cast, Add, Tile, Concat, Expand, Gather, CumSum, Max, and Min (#29830, #29834, #29835, #29839, #29844, #29847, #29854, #29861, #29897, #29918, #31049, #31702, #31709).
Added the initial WebGPU PagedAttention implementation and moved Softmax and non-flash Attention to online algorithms (#29694, #29724, #31611).
Improved MatMulNBits wide-tile accumulation precision (#29611).
Added and extended Intel subgroup-matrix MatMul/Gemm kernels, including f16, batched-B, and odd-N support (#29592, #29749, #29813, #29893).
Reduced cold-start and upload overhead with deferred dispatch and staging-buffer improvements; tuned FlashAttention, subgroup Gemm/MatMul, Split-K on Panther Lake, and Xe im2col-matmul (#29271, #29505, #29557, #29586, #29846, #30514).
Upgraded Dawn and improved reliability by avoiding exceptions in Dawn callbacks, fixing a Linux adapter-failure self-deadlock, correcting Windows x86 transfer callbacks, and fixing TurboQuant batched sequence lengths (#29389, #29591, #29625, #29752, #31568).

WebNN EP

Added uint8-packed 4-bit GatherBlockQuantized and LpNormalization, reused the shared WASM loader for Blob-backed external data, and fixed per-axis QDQ and MatMulNBits edge cases (#29475, #29801, #31151, #31152, #31197).

OpenVINO / QNN / DML / XNNPACK / TensorRT

OpenVINO fixed float16 constant-output corruption and output-name routing, added dot-separated KV-cache names to the stateful transform, and corrected raw-data-backed float initializer handling (#29729, #29882, #29895, #31138).
QNN added a reshape handler for split-axis reshapes (#29660).
DML fixed wide-string handling and made fused graph kernels own their model paths (#31656, #31664).
XNNPACK now reads dynamic Gemm M from the input tensor at compute time (#31189).
TensorRT deduplicated context-path handling and added a build option for fused-attention cubins (#29640, #31632).

CPU & Core Optimizations

MLAS

Added Arm64 half-precision GEMM and convolution support through KleidiAI, including FP16 MatMul/Gemm/Conv paths and asymmetric Q4 and SME2 MatMulNBits kernels (#28786, #29654, #29709, #29898).
Added a RISC-V RVV QNBitGemm backend, an Arm64 NEON fp32 RoPE kernel, portable SVE elementwise kernels with FEXPA exp, and Arm64 UDOT routing for S8U8 QGEMM (#29537, #29787, #29836, #31145).
Added AVX2/VNNI 2-bit weight kernels and vectorized 2-bit dequantization, and improved fp16 MatMulNBits paths by avoiding fp32 temporaries and writing fp16 output directly across 2-, 4-, and 8-bit paths (#29619, #29766, #29791, #29842, #29864, #29901).

CPU Attention & Kernels

Improved masked Attention performance, enabled CPU FlashAttention on Linux Arm64 through L2-cache detection, and added FP16 GQA with quantized KV cache (#29621, #29719, #29825).
Added double support to CPU Cos and int32 support to CPU Trilu, and fixed int8 QLinearSoftmax saturation and AvgPool ceil_mode/count_include_pad behavior (#28975, #29476, #29629, #29728).
Fixed TfIdfVectorizer weight indexing and skipped MinLength logits-processor construction when eos_token_id is negative (#29604, #31649).
Tightened K/V and cache-indirection shape contracts in CPU Attention and MultiHeadAttention, and fixed LinearAttention output shape inference for grouped-query attention (#29892, #31190, #31634).

Graph, Optimizer, and Runtime

Extended reshape fusion, fixed double recursion in subgraph type/shape inference, and made constant-folding output deterministic (#29027, #29617, #29789).
Fixed in-memory external initializer loading, memory-pattern allocation stream selection, and a leak in GetOverridableInitializerNames() (#29349, #29589, #29616).
Reduced small MatMul batch allocations and redundant LUT initialization (#29085, #29690).
Fixed static-initialization-order crashes when importing ONNX Runtime and reduced eager runtime initialization (#29880, #31964).
Negative CPU Split axes now produce an error instead of being accepted (#31149).

Web & JavaScript

Added on-demand loading of Blob-backed external data in JSPI builds (#29477).
Fixed JSEP pooling output shape for ceil_mode, allowed DFT to ignore excess input data, and fixed a webpack/Terser release-build crash (#29627, #29680, #31652).

Build, Packaging & CI

CUDA package architecture selections are now aligned across plugin EP, Python, C API, TensorRT, and Node.js pipelines. Windows arm64 is only available in CUDA plugin EP (#31992):

OS	CUDA	CUDA architectures (all in `-real` form)
Linux x64	12.8	60;70;75;80;86;89;90a;120a
Linux x64	13.x	75;80;86;89;90a;120a
Linux aarch64	13.x	89;90a;120a;121a
Windows x64	12.8	61;75;86;89;120a
Windows x64	13.x	75;80;86;89;120a
Windows arm64	13.x	120a;121a

Reduced CUDA compilation time and memory usage by splitting generated SM80 MoE, fpA_intB, and MatMulNBits translation units and adding two-level workspace estimation (#29614, #29699, #29811, #31834, #31837).
Fixed CUDA 13 plugin and packaging builds on Windows, Linux, and Windows ARM64, including MSVC/TMA compatibility and CI memory limits (#31608, #31609, #31615, #31616, #31617, #31622, #31729, #31748).
Fixed MLAS AVX2 builds on toolchains without AVX-VNNI assembler support, GCC 15 -Werror builds, and an MSVC C1001 issue in the W2 AVX-512-VNNI dispatch path (#28767, #29679, #29885).
Improved Windows compatibility by skipping DXGI discovery when Win32k system calls are unavailable and delay-loading shell32 (#29755, #30889).
Fixed Dawn parallel-build races, and GPU discovery in build/test environments (#29858, #29866).

Contributors

Thanks to our 63 contributors for this release!

@adrastogi, @ahsan-ca, @AngelGalindo7, @ankitm3k, @apsonawane, @blazingphoenix7, @bmehta001, @chilo-ms, @claude, @daijh, @ducviet00, @edgchen1, @elwhyjay, @eserscor, @GopalakrishnanN, @guptaishaan, @hariharans29, @Honry, @huningxin, @jchen10, @jiafatom, @jiangzhuo, @Jiawei-Shao, @JonathanC-ARM, @justinchuby, @kjg0724, @kunal-vaishnavi, @kylo5aby, @Laan33, @martin-klacer-arm, @mastryukov1990, @mcollinswisc, @miaobin, @mingmingtasd, @mirounga, @mustjab, @n1harika, @namgyu-youn, @neilmsft, @nenad1002, @nicholascelestin, @OscarFree, @prathikr, @qjia7, @quic-muchhsu, @Sammy-Dabbas, @sanaa-hamel-microsoft, @shiyi9801, @skottmckay, @tairenpiao, @TedThemistokleous, @the0cp, @tianleiwu, @titaiwangms, @velonica0, @wangw-1991, @wuisabel-gif, @xadupre, @xhcao, @xiaofeihan1, @xiaoyu-work, @yen-shi, @zlma7001

Full Changelog: v1.28.0...v1.29.0

Open source →

Additional notes

ONNX Runtime v1.29.0

Compare

Choose a tag to compare

Open source →

1.27.019 Jun 2026Deprecated

CUDA 12 packages are deprecated, please move to CUDA 13 ASAP.

n.b. This release is targeting ONNX 1.21. ONNX 1.22 will be supported in ORT 1.28. n.b. This changelog was generated via LLM. Only the contributor list has been verified. As always, only trust the commit history.

Announcements & Breaking Changes

CUDA 12 package files are now explicitly named as such.
CUDA 12 packages are deprecated, please move to CUDA 13 ASAP.

Security Fixes

Fixed out-of-bounds read in SoftmaxCrossEntropyLoss via label bounds validation (#28004)
Hardened OneHot input validation and output-size computation (#28014)
Added SafeInt overflow protection in Expand and capped constant-folding output sizes (#28055)
Bounded total output allocation size in Tile kernel (#28070)
Added mask/input shape consistency checks in MaxpoolWithMask::Compute (#28223)
Fixed BitShift UB for shift amounts greater than or equal to bit width (#28272)
Validated sequence bounds in GQA (seqlens_k vs cos_cache) (#28277)
Validated conv bias shape in WordConvEmbedding to prevent OOB reads (#28279)
Fixed int32 overflow in CUDA Cast and UnaryElementWise kernels for very large tensors (#28386)
Fixed out-of-bounds read in CropBase scale handling (#28399)
Fixed rank-underflow bug in Inverse kernel trailing-dimension indexing (#28400)
Added sparse tensor external file path validation and additional external-path hardening (#28408, #28709, #28725)
Switched remaining torch.load() calls to weights_only=True (#28421)
Added CPU cache-indirection beam-index validation (#28486)
Added additional overflow/bounds checks and test coverage in runtime buffers (#28713, #28747)

New Features

Execution Provider Plugin API

Added zero-copy I/O for plugin EPs with HOST_ACCESSIBLE memory (#28037)
Added OrtEp::OnSessionInitializationEnd() callback (#28319)
Added plugin EP session-options getters (#28377)
Added CUDA Plugin EP provider options for streams and external allocators (#28603)

Core APIs & Runtime

Added support for ONNX overloaded functions (IR v10+) (#28275)
Added FLOAT8E8M0 datatype support in ONNX Runtime (#28381)
Added CPU Cast support for FLOAT8E8M0 (#28435)
Added kOrtEpDevice_EpMetadataKey_OSDriverVersion example and docs (#28282)

Quantization & Training Tooling

Added calibration cache support to quantize_static (#28221)
Added ActivationRestrictedAsymmetric quantization option (#28237)
Added opset-21 block_size attribute support to QDQ quantization (#28522)
Added CPU fallback for FusedAdam optimizer in ORT Training (#28233)

Execution Provider Updates

NVIDIA CUDA EP

Added ConvTranspose-22 support (#27710)
Filled CUDA opset gaps for LSTM, RNN, Reshape, Cast, Round/Equal, ReduceMax/ReduceMin, Sin/Cos, and Random* ops (#27737, #27743, #27742, #27744, #27754, #27755, #27756, #27759)
Added LpNormalization support for CUDA EP (#28724)
Added chunked dequant+GEMM for MatMulNBits to reduce peak GPU memory (#28712)
Added QMoE tests for standard swiglu and improved decode-path routing/softmax kernels (#28741, #29026)
Fixed CUDA Attention dispatch mismatch for GQA head-size cases (#28358)
Fixed CUTLASS FMHA bias-loader alignment on unaligned kernel path (#28369)

WebGPU EP

Added LSTM support on WebGPU (#27881)
Added per-graph buffer manager for multi-graph capture (#28260)
Added QKV and MLP layer fusions for Qwen3-style models (#28280)
Added QKV bias support in FlashAttention for MultiHeadAttention (#28380)
Added shader dump-to-file environment variable and nightly validation checks (#28674)
Added opset-24 + KV-shared decoder support (Gemma 4) (#28501)
Performance improvements: FlashAttention M4 Max optimization and subgroup-based LinearAttention tuning (#27780, #28412, #28519, #28520)
Fixed numerical and correctness issues in QMoE, LayerNorm/SkipLayerNorm, and MatMul bias indexing (#28427, #28434, #28475)

CoreML EP

Added support for pre-opset-13 Split (split attribute path) and scalar Gather indices (#28270, #28278)
Added FusedConv, Identity, Ceil, Tile, Cast(bool), Sin, Cos, and GatherND support (#28289, #28293, #28595, #28596, #28598)

TensorRT / DML / QNN

Improved TensorRT RTX compatibility (multi-GPU tests, API guards, and subgraph fixes) (#27837, #28361, #28611, #28503)
Added diagnostics for DML failure paths (#28495)
Updated QNN ETW log level rule handling (#27593)

Web & JavaScript

JavaScript / Node.js

Updated JavaScript dependencies (next, postcss, tmp, qs, body-parser, and other npm packages) (#27705, #27894, #28304, #28547, #28644, #28683, [#28694]

CPU & Core Optimizations

MLAS / Quantization / Attention

Added NHWC convolution path in MLAS to reduce transpose overhead (#26834)
Added CPU QMoE 2-bit support and LUT GEMM fast path (#28185)
Added quantized KV-cache support for CPU GroupQueryAttention with SIMD optimizations and tiled compute (#28576, #28578, #28606, #28695)
Added RVV-optimized NCHWc convolution/pooling and LLM operators for RISC-V (#28411, #28518)
Parallelized CPU ScatterElements and optimized MatMulNBits 2-bit float-zero-point path (#28588, #28589)
Added optional bias support in MatMulNBits CPU LUT GEMM path (#28742)

Graph, Optimizer, and Fusion

Added DiT attention fusion for F5-TTS and diffusion transformer models (#27999)
Improved ONNX Attention dispatch path and removed legacy unfused MHA path (#27992)
Added support for optional present_key/present_value outputs in GQA and Gemma4 support (#28242)
Added implicit-input handling for partitioning/fusion around control-flow nodes (#28608, #28690)

Language Bindings

Python

Fixed runtime-unresolvable type annotations in Session and InferenceSession (#27802)
Made sympy an optional runtime dependency (#28141)
Added PEP 561 py.typed marker to the onnxruntime package (#28438)

C#

Added EP tests for CUDA Plugin EP (#28375)

Bug Fixes

Critical & Correctness Fixes

Fixed session use-after-free when UserLoggingFunction is used (#28314)
Prevented double-free in OrtModelEditorApi ownership transfer (#28123)
Fixed plugin EP provider-library load refcount leak and added regression test (#28396, #28430)
Fixed dangling pointer from temporary return value (#28419)
Fixed PRelu returning NaN for infinite inputs on CPU EP (#28750)
Fixed CPU Attention softcap/attn_mask ordering and added CUDA spec coverage consolidation (#28379)
Fixed Reshape allowzero=1 handling for chained zero-size tensors (#28455)
Fixed CPU QLinearConv per-channel weight zero-point handling for distinct values (#28456)
Fixed Unicode-path handling issues on Windows and AppContainer path canonicalization (#28390, #28509)

Portability Fixes

Build and portability fixes across AIX, GCC-15, S390x, libc++/Clang, FreeBSD, and cross-compilation flows (#26704, #27191, #28016, #28049, #28074, #28362, #28567, #28507)
Fixed CPUIDInfo bounds handling for unknown ARM vendors (#28344)
Fixed Resize nearest-mode rounding bug for negative halfway values (#28345)

Contributors

Thanks to our 106 contributors for this release!

@adrastogi, @adrianlizarraga, @AIFrameworksIntegration, @AlekseiNikiforovIBM, @angelser, @angelserMS, @ankitm3k, @anzzraju1997-glitch, @apsonawane, @arajendra, @ayappanec, @bachelor-dou, @badranX, @baijumeswani, @BoarQing, @bopeng1234, @bsosnader, @cbourjau, @chilo-ms, @chunghow-qti, @chwarr, @Craigacp, @daijh, @derdeljan-msft, @dparikh79, @edgchen1, @elwhyjay, @ericcraw, @eserscor, @feich-ms, @fs-eire, @gaugarg-nv, @gblong1, @GopalakrishnanN, @gramalingam, @guschmue, @hariharans29, @HectorSVC, @intbf, @ishwar-raut1, @Jaswanth51, @jatinwadhwa921, @javier-intel, @jchen10, @jiafatom, @Jiawei-Shao, @jnagi-intel, @jocelyn-stericker, @JonathanC-ARM, @justinchuby, @jwludzik, @kevinch-nv, @Kotomi-Du, @kpkbandi, @KV2773, @Laan33, @lhrios, @maxwbuckley, @MayureshV1, @mdvoretc-intel, @mingyueliuh, @mklimenk, @mustjab, @n1harika, @nazanin-beheshti, @orlmon01, @prathikr, @preetha-intel, @psakhamoori, @qiurui144, @qjia7, @qti-ashwshan, @qti-hungjuiw, @qti-yuduo, @rajatmonga, @RajeevSekar, @Rishi-Dave, @rvandermeulen, @RyanMetcalfeInt8, @SamuelLess, @sanaa-hamel-microsoft, @sfatimar, @sgbihu, @shiyi9801, @simonbyrne, @skottmckay, @susbhere, @sushraja-msft, @tairenpiao, @TejalKhade28, @theHamsta, @tianleiwu, @titaiwangms, @umangb-09, @velonica0, @vraspar, @vthaniel, @wenqinI, @xadupre, @xenova, @xhan65, @xiaofeihan1, @xieofxie, @yuslepukhin, @ZackyLake, @zejianzhang1982, @zz002

Full Changelog: https://github.com/microsoft/onnxruntime/compare/v0.1.4...v1.27.0

Open source →

Additional notes

ONNX Runtime v1.27.0

Compare

Choose a tag to compare

Open source →

1.26.08 May 2026Security

SVM and TreeEnsemble bounds/security fixes ( #27950 , #27951 , #27952 , #27989 ).

n.b. The following was generated via LLM from Git history. Only the contributor list has been verified.

Announcement - Breaking Changes

Support for CUDA 12 will be removed in 1.27.0.
- CUDA 13 will continue to be published as onnxruntime-<os>-<arch>-gpu_cuda13-<version>.<ext>
CUDA runtime will be moving soon to a dedicated Execution Provider (EP) instead of a published package from ORT core.

Highlights

Added optional memory mapping for .ort model loads (#28164).
Added RISC-V Vector (RVV) support for CPU EP (#28261).
OpenVINO EP upgraded for 1.26.0 development release (#28297).
WebGPU gained GridSample support (#28264) and Split-K improvements (#28151).
CUDA plugin EP gained graph support (#28002), profiling API (#28216).

Security and Reliability Hardening

Replaced unrestricted Python setattr configuration with an allowlist (#28083).
Hardened multiple OOB and overflow scenarios across ML and core ops:
- Attention mask index OOB write (#27789).
- MaxPoolGrad indices bounds validation (#27903).
- SVM and TreeEnsemble bounds/security fixes (#27950, #27951, #27952, #27989).
- RNN sequence_lens OOB read and integer overflow handling (#28052, #28003).
- GroupQueryAttention seqlens_k bounds validation and compatibility follow-up (#28031, #28259).
- MatMulBnb4 and ML coefficient SafeInt checks (#27995, #28001).
- CUDA Gather int32 overflow fix (#28108).
- GridSample float->int64 cast hardening for NaN/Inf/out-of-range coords (#28302).
Fixed session logger use-after-free during EP teardown under verbose logging (#28274).

CUDA, Attention, and MLAS

Filled CUDA opset/operator gaps and extended support:
- Transpose opset 23 -> 25 (#27740).
- QuantizeLinear/DequantizeLinear opset 25 (#28046).
- CUDA TopK INT8/INT16/UINT8 support (#27862).
- LabelEncoder CUDA support for numeric types (#28045).
Attention/GQA improvements:
- Fixed ONNX Attention min-bias alignment crash on SM<80 and masked-batch NaN behavior (#27831).
- Added FP32 QK accumulation path for unfused GQA attention (#28198).
- Added CUDART_VERSION reduction compatibility in GQA attention (#28296).
- Fixed CUDA 13 build error in GQA unfused attention (#28309).
- PagedAttention fallback for SM<80 fp16 (#28200).
MLAS updates:
- FP16 Gelu enablement (#26815).
- Arm64 BF16 fast-math conv kernels for NCHW/NCHWc paths (#27878).

WebGPU, WebNN, and JavaScript

WebGPU feature and correctness updates:
- Added GridSample (#28264).
- Split-K support for batch size > 1 (#28151).
- MatMulNBits refactor and batching improvements (#28109, #28197).
- MHA correctness fix when present outputs are not requested (#28027).
- Buffer upload overflow fix (#27948).
- Position ID bounds validation in WebGPU/JS RotaryEmbedding (#28214).
WebNN change:
- Renamed pool2d property roundingType -> outputShapeRounding (#28172).
JavaScript ecosystem maintenance:
- Multiple dependency bumps.

Plugin EP and EP Ecosystem

CUDA plugin EP:
- Graph capture/replay support ported and expanded (#27958, #28002).
- Sync support for IOBinding (#27919).
- Profiling API implementation (#28216).
- Resource accounting integration (#28028).
WebGPU plugin EP:
- Pipeline updates and API init error handling fixes (#28121, #28211).
Other EP updates:
- CoreML: HardSigmoid and QuickGelu support; Pad reflect support/fixes (#28182, #28184, #28073, #28062).
- NvTensorRTRTX compatibility and diagnostics updates (#28263, #27577).
- QNN file-mapping guard improvements (#27871).

Contributors

@tianleiwu, @yuslepukhin, @edgchen1, @vraspar, @hariharans29, @skottmckay, @eserscor, @xadupre, @sanaa-hamel-microsoft, @elwhyjay, @Rishi-Dave, @titaiwangms, @adrianlizarraga, @jatinwadhwa921, @jchen10, @Jiawei-Shao, @maxwbuckley, @preetha-intel, @qjia7, @qti-hungjuiw, @RajeevSekar, @umangb-09, @adrastogi, @akote123, @amd-genmingz, @ankitm3k, @apsonawane, @bachelor-dou, @baijumeswani, @bopeng1234, @chilo-ms, @chwarr, @Craigacp, @dccarmo, @derdeljan-msft, @ericcraw, @fdwr, @fs-eire, @gaugarg-nv, @gblong1, @GopalakrishnanN, @Honry, @intbf, @ishwar-raut1, @Jaswanth51, @javier-intel, @JonathanC-ARM, @julia-thorn, @justinchuby, @jwludzik, @Kevin-Taha, @Kotomi-Du, @MayureshV1, @mdvoretc-intel, @miaobin, @milpuz01, @mingyueliuh, @mklimenk, @n1harika, @prathikr, @psakhamoori, @qti-yuduo, @quic-calvnguy, @RyanMetcalfeInt8, @sfatimar, @sgbihu, @ShirasawaSama, @ssam18, @susbhere, @sushraja-msft, @TejalKhade28, @theHamsta, @TomCrypto, @TsofnatMaman, @velonica0, @vthaniel, @wenqinI, @xhan65, @xhcao

Open source →

Additional notes

ONNX Runtime v1.26.0

Compare

Choose a tag to compare

Open source →

1.24.15 Feb 2026Security

DoS vulnerability in FuseReluClip

📢 Announcements & Breaking Changes

Platform Support Changes

Python 3.10 wheels are no longer published — Please upgrade to Python 3.11+
Python 3.14 support added
Free-threaded Python (PEP 703) — Added support for Python 3.13t and 3.14t in Linux (#26786)
x86_64 binaries for macOS/iOS are no longer provided and minimum macOS is raised to 14.0

API Version

ORT_API_VERSION updated to 24 (#26418)

✨ New Features

🤖 Execution Provider (EP) Plugin API

A major infrastructure enhancement enabling plugin-based EPs with dynamic loading:

Initial kernel-based EP support (#26206)
Weight pre-packing support for plugin EPs (#26754)
EP Context model support (#25124)
Control flow kernel APIs (#26927)
OrtKernelInfo APIs for kernel-based plugin EPs (#26803)

🔧 Core APIs

OrtApi::CreateEnvWithOptions() and OrtEpApi::GetEnvConfigEntries() (#26971)
EP Device Compatibility APIs (#26922)
External Resource Importer API for D3D12 shared resources (#26828)
Session config access from KernelInfo (#26589)

📊 Dependencies & Integration

ONNX upgraded to 1.20.1 (#26579)
Protobuf updated from 3.20.3 → 4.25.8 (#26910)
CUDA Graph enabled by default (#26929)

🖥️ Execution Provider Updates

NVIDIA

CUDA EP: Flash Attention updates, GQA kernel fusion, BF16 support for MoE/qMoE/MatMulNBits, CUDA 13.0 support
TensorRT EP: Upgraded to TensorRT 10.14, automatic plugin loading, NVFP4 custom ops
TensorRT RTX EP: RTX runtime caching, CUDA graph support, BFloat16, memory-mapped engines

Qualcomm QNN EP

QNN SDK upgraded to 2.42.0 with new ops (RMSNorm, ScatterElements, GatherND, STFT, RandomUniformLike)
Gelu pattern fusion, LPBQ quantization support, ARM64 wheel builds, v81 device support

Intel & AMD

OpenVINO EP: Upgraded to 2025.4.1
VitisAI EP: External EP loader, compiled model compatibility API
MIGraphX EP: QuickGelu, multihead attention, QLinear pooling ops

ArmNN EP

Arm is formally deprecating the Arm NN Execution Provider (EP) in ONNX Runtime. The Arm NN EP is still experimental and depends on technology that is no longer actively maintained. Keeping it available now only adds complexity and potential confusion for users.

What to expect:

Effective immediately, the Arm NN EP is deprecated and will no longer be maintained
All build options, documentation, and examples referencing ArmNN will be removed once the upstream change merges; the removal will appear in the first ONNX Runtime release that includes that change. We will confirm the release number as soon as it is known
Builds that still rely on Arm NN-specific options (for example --use_armnn) will fail after the change lands, so please adjust configurations in advance

🌐 Web & JavaScript

WebGPU EP: Flash Attention optimizations, graph capture, Split-K MatMul, qMoE support, WGSL templates
WebNN EP: GQA local attention, GatherBlockQuantized, ConvInteger/MatMulInteger
Node.js/React Native: Node.js v22, JSI for React Native, JSPI build support

🧠 CPU Improvements

KleidiAI: SME1/SME2 Convolution and SGemm kernels, FP32 Gemv, Windows/Arm support
New ops: MoE/qMoE kernels, RotaryEmbeddings opset 23, LayerNorm/RMSNorm broadcasting
Platform support: S390x SIMD, LoongArch64 4-bit quantization, FP16 inference improvements
ARM NCHWc layout support: NCHWc layout support for potential performance improvement of Conv models. Needs building from source with --enable_arm_neon_nchwc to enable this feature (#25580 #26838 #26691 #26171). This feature may be turned ON by default in a future release based on community feedback.
ARM perf improvements: Dedicated depthwise conv kernel (#26688) and SiLU activation perf improvement (#26753)

🔌 Language Bindings

C#

.NET 9.0 MAUI targets (#26463)

Python

add_external_initializers_from_files (#26012)

Java

Auto EP and compile model support (#25131)
OrtCompiledModelCompatibility (#26028)

🐛 Bug Fixes

Critical Fixes

DoS vulnerability in FuseReluClip (#26878)
Security issue loading arbitrary files as external data (#26776)
Memory leak fix for KernelContext_GetAllocator (#26883)
Local Attention off-by-1 bug (#25927)

EP-Specific Fixes

[QNN] Clip op with min/max from QDQ (#26601)
[CoreML] Gather fp16 support (#26442)

🙏 Contributors

Thanks to our 170 contributors for this release!

@fs-eire, @tianleiwu, @edgchen1, @qjia7, @yuslepukhin, @hariharans29, @Honry, @qti-yuduo, @adrianlizarraga, @snnn, @eserscor, @vraspar, @xiaofeihan1, @guschmue, @daijh, @quic-muchhsu, @qti-jkilpatrick, @tirupath-qti, @Jiawei-Shao, @qti-hungjuiw, @quic-ashwshan, @titaiwangms, @qti-mattsinc, @chilo-ms, @jchen10, @xhcao, @skottmckay, @quic-calvnguy, @JonathanC-ARM, @Rohanjames1997, @sushraja-msft, @jambayk, @adrastogi, @xenova, @quic-tirupath, @justinchuby, @HectorSVC, @kunal-vaishnavi, @wenqinI, @prathikr, @baijumeswani, @preetha-intel, @jatinwadhwa921, @umangb-09, @qti-ashwshan, @carzh, @bachelor-dou, @ranjitshs, @gedoensmax, @xadupre, @nenad1002, @TedThemistokleous, @keshavv27, @zpye, @jnagi-intel, @jiafatom, @mingyueliuh, @Colm-in-Arm, @borg323, @chunghow-qti, @Craigacp, @BODAPATIMAHESH, @AlekseiNikiforovIBM, @hans00, @thevishalagarwal, @MaanavD, @qti-kromero, @damdoo01-arm, @BoarQing, @naomiOvad, @yuhuchua-qti, @hadiFute, @vishalpandya1990, @rivkastroh, @minfhong-qti, @kuanyul-qti, @xieofxie, @ankitm3k, @RyanMetcalfeInt8, @MayureshV1, @bopeng1234, @vthaniel, @mdvoretc-intel, @ericcraw, @javier-intel, @saurabhkale17, @sfatimar, @Kotomi-Du, @intbf, @n1harika, @TejalKhade28, @gupta-pallavi, @cbourjau, @nieubank, @r-devulap, @wszqkzqk, @sanketkaleoss, @amancini-N, @fanchenkong1, @meakbiyik, @hisham-hchowdhu, @shaoboyan091, @Stonesjtu, @qwu16, @wangw-1991, @bonktree, @naetherm, @nikhilfujitsu, @Panxuefeng-loongson, @selenayang888, @moyo1997, @chwarr, @patryk-kaiser-ARM, @fdwr, @SavaLione, @shiyi9801, @mcost45, @aciddelgado, @prudhvi-qti, @Jonahcb, @lifang-zhang, @zhaoxul-qti, @gaugarg-nv, @cocotdf, @WangFengtu1996, @orlmon01, @weidu-tpvision, @theHamsta, @kevinch-nv, @XXXXRT666, @movedancer, @melkap01-Arm, @KingSora, @urpetkov-amd, @junchao-loongson, @jixiongdeng, @wcy123, @GrigoryEvko, @anujj, @peishenyan, @quic-ankus, @jchen351, @yihonglyu, @satyajandhyala, @co63oc, @mschofie, @quic-ashigarg, @asoldano, @nproshun, @jiangzhaoming, @seungtaek94, @liqunfu, @jaholme, @hanbitmyths, @quic-boyuc, @rM-planet, @qti-vaiskv, @AndreyOrb, @pkubaj, @xhan65, @Jaswanth51, @quic-hungjuiw, @jywu-msft, @mklimenk, @derdeljan-msft, @ianfhunter, @NingW101, @feich-ms, @Akupadhye, @wschin

Full Changelog: v1.23.2...rel-1.24.1

Open source →

1.22.09 May 2025

This release introduces new API's for Model Editor, Auto EP infrastructure, and AOT Compile

Announcements

This release introduces new API's for Model Editor, Auto EP infrastructure, and AOT Compile
OnnxRuntime GPU packages require CUDA 12.x , packages built for CUDA 11.x are no longer published.
The min supported Windows version is now 10.0.19041.

GenAI & Advanced Model Features

Constrained Decoding: Introduced new capabilities for constrained decoding, offering more control over generative AI model outputs.

Execution & Core Optimizations

Core

Auto EP Selection Infrastructure: Added foundational infrastructure to enable automatic selection of Execution Providers via selection policies, aiming to simplify configuration and optimize performance. (Pull Request #24430)
Compile API: Introduced new APIs to support explicit compilation of ONNX models.
- See: OrtCompileApi Struct Reference (Assuming a similar link structure for future documentation)
- See: EP Context Design (Assuming a similar link structure for future documentation)
Model Editor API api's for creating or editing ONNX models
- See: OrtModelEditorApi

Execution Provider (EP) Updates

CPU EP/MLAS

KleidiAI Integration: Integrated KleidiAI into ONNX Runtime/MLAS for enhanced performance on Arm architectures.
MatMulNBits Support: Added support for MatMulNBits, enabling matrix multiplication with weights quantized to 8 bits.
GroupQueryAttention optimizations and enhancements

OpenVINO EP

Added support up to OpenVINO 2025.1
Introduced Intel compiler level optimizations for QDQ models.
Added support to select Intel devices based on LUID
Load_config feature improvement to support AUTO, HETERO and MULTI plugin.
misc bugfixes/optimizations
For detailed updates, refer to Pull Request #24394: ONNXRuntime OpenVINO - Release 1.22

QNN EP

SDK Update: Added support for QNN SDK 2.33.2.
operator updates/support to Sum, Softmax, Upsample, Expand, ScatterND, Einsum
QNN EP can be built as shared or static library.
enable QnnGpu backend
For detailed updates refer to recent QNN tagged PR's

TensorRT EP

TensorRT Version: Added support for TensorRT 10.9.
- Note for onnx-tensorrt open-source parser users: Please check here for specific requirements (Referencing 1.21 link as a placeholder, this should be updated for 1.22).
New Features:
- EP option to enable TRT Preview Feature
- Support to load TensorRT V3 plugin
Bug Fixes:
- Resolved an issue related to multithreading scenarios.
- Fixed incorrect GPU usage that affected both TensorRT EP and CUDA EP.

NV TensorRT RTX EP

New Execution Provider: Introduced a new Execution Provider specifically for Nvidia RTX GPUs, leveraging TensorRT for optimized performance.

CUDA EP

MatMulNBits Enhancement: Added support for 8-bit weight-only quantization in MatMulNBits.
Bug Fixes:
- Fixed incorrect GPU usage (also mentioned under TensorRT EP).

VitisAI EP

Miscellaneous bug fixes and improvements.

Infrastructure & Build Improvements

Build System & Packages

QNN Nuget Package: The QNN Nuget package is now built as ARM64x.

Dependencies / Version Updates

CUDA Version Update: This release includes an update to the CUDA version. Users should consult the documentation for specific version requirements. CUDA 11 based GPU packages no longer published.

Web

WebGPU Expansion:
- Added WebGPU support to the node.js package (Windows and macOS).
- Enabled WebGPU when building from source for macOS, Linux, and Windows.

Mobile

No major updates of note this release.

Contributions

Contributors to ONNX Runtime include members across teams at Microsoft, along with our community members:

Yulong Wang, Jian Chen, Changming Sun, Satya Kumar Jandhyala, Hector Li, Prathik Rao, Adrian Lizarraga, Jiajia Qin, Scott McKay, Jie Chen, Tianlei Wu, Edward Chen, Wanming Lin, xhcao, vraspar, Dmitri Smirnov, Jing Fang, Yifan Li, Caroline Zhu, Jianhui Dai, Chi Lo, Guenther Schmuelling, Ryan Hill, Sushanth Rajasankar, Yi-Hong Lyu, Ankit Maheshkar, Artur Wojcik, Baiju Meswani, David Fan, Enrico Galli, Hans, Jambay Kinley, John Paul, Peishen Yan, Yateng Hong, amarin16, chuteng-quic, kunal-vaishnavi, quic-hungjuiw, Alessio Soldano, Andreas Hussing, Ashish Garg, Ashwath Shankarnarayan, Chengdong Liang, Clément Péron, Erick Muñoz, Fanchen Kong, George Wu, Haik Silm, Jagadish Krishnamoorthy, Justin Chu, Karim Vadsariya, Kevin Chen, Mark Schofield, Masaya, Kato, Michael Tyler, Nenad Banfic, Ningxin Hu, Praveen G, Preetha Veeramalai, Ranjit Ranjan, Seungtaek Kim, Ti-Tai Wang, Xiaofei Han, Yueqing Zhang, co63oc, derdeljan-msft, genmingz@AMD, jiangzhaoming, jing-bao, kuanyul-quic, liqun Fu, minfhong-quic, mingyue, quic-tirupath, quic-zhaoxul, saurabh, selenayang888, sfatimar, sheetalarkadam, virajwad, zz002, Ștefan Talpalaru

Open source →

1.21.08 Mar 2025Breaking

Chat mode introduced breaking changes in the API (see migration guide).

Announcements

No large announcements of note this release! We've made a lot of small refinements to streamline your ONNX Runtime experience.

GenAI & Advanced Model Features

Enhanced Decoding & Pipeline Support

Added "chat mode" support for CPU, GPU, and WebGPU.
Provided support for decoder model pipelines.
Added support for Java API for MultiLoRA.

API & Compatibility Updates

Chat mode introduced breaking changes in the API (see migration guide).

Bug Fixes for Model Output

Fixed Phi series garbage output issues with long prompts.
Resolved gibberish issues with top_k on CPU.

Execution & Core Optimizations

Core Refinements

Reduced default logger usage for improved efficiency(#23030).
Fixed a visibility issue in theadpool (#23098).

Execution Provider (EP) Updates

General

Removed TVM EP from the source tree(#22827).
Marked NNAPI EP for deprecation (following Google's deprecation of NNAPI).
Fixed a DLL delay loading issue that impacts WebGPU EP and DirectML EP's usability on Windows (#23111, #23227)

TensorRT EP Improvements

Added support for TensorRT 10.8.
- onnx-tensorrt open-source parser user: please check here for requirement.
Assigned DDS ops (NMS, RoiAlign, NonZero) to TensorRT by default.
Introduced option trt_op_types_to_exclude to exclude specific ops from TensorRT assignment.

CUDA EP Improvements

Added a python API preload_dlls to coexist with PyTorch.
Miscellaneous enhancements for Flux model inference.

QNN EP Improvements

Introduced QNN shared memory support.
Improved performance for AI Hub models.
Added support for QAIRT/QNN SDK 2.31.
Added Python 3.13 package.
Miscellaneous bug fixes and enhancements.
QNN EP is now built as a shared library/DLL by default. To retain previous build behavior, use build option --use_qnn static_lib.

DirectML EP Support & Upgrades

Updated DirectML version from 1.15.2 to 1.15.4(#22635).

OpenVINO EP Improvements

Introduced OpenVINO EP Weights Sharing feature.
Added support for various contrib Ops in OVEP:
- SkipLayerNormalization, MatMulNBits, FusedGemm, FusedConv, EmbedLayerNormalization, BiasGelu, Attention, DynamicQuantizeMatMul, FusedMatMul, QuickGelu, SkipSimplifiedLayerNormalization
Miscellaneous bug fixes and improvements.

VitisAI EP Improvements

Miscellaneous bug fixes and improvements.

Mobile Platform Enhancements

CoreML Updates

Added support for caching generated CoreML models.

Extensions & Tokenizer Improvements

Expanded Tokenizer Support

Now supports more tokenizer models, including ChatGLM, Baichuan2, Phi-4, etc.
Added full Phi-4 pre/post-processing support for text, vision, and audio.
Introduced RegEx pattern loading from tokenizer.json.

Image Codec Enhancements

ImageCodec now links to native APIs if available; otherwise, falls back to built-in libraries.

Unified Tokenizer API

Introduced a new tokenizer op schema to unify the tokenizer codebase.
Added support for loading tokenizer data from a memory blob in the C API.

Infrastructure & Build Improvements

Runtime Requirements

All the prebuilt Windows packages now require VC++ Runtime version >= 14.40(instead of 14.38). If your VC++ runtime version is lower than that, you may see a crash when ONNX Runtime was initializing. See https://github.com/microsoft/STL/wiki/Changelog#vs-2022-1710 for more details.

Updated minimum iOS and Android SDK requirements to align with React Native 0.76:

iOS >= 15.1
Android API >= 24 (Android 7)

All macOS packages now require macOS version >= 13.3.

CMake File Changes

CMake Version: Increased the minimum required CMake version from 3.26 to 3.28. Added support for CMake 4.0. Python Version: Increased the minimum required Python version from 3.8 to 3.10 for building ONNX Runtime from source. Improved VCPKG support

Added the following cmake options for WebGPU EP

onnxruntime_USE_EXTERNAL_DAWN
onnxruntime_CUSTOM_DAWN_SRC_PATH
onnxruntime_BUILD_DAWN_MONOLITHIC_LIBRARY
onnxruntime_ENABLE_PIX_FOR_WEBGPU_EP
onnxruntime_ENABLE_DAWN_BACKEND_VULKAN
onnxruntime_ENABLE_DAWN_BACKEND_D3D12

Added cmake option onnxruntime_BUILD_QNN_EP_STATIC_LIB for building with QNN EP as a static library. Removed cmake option onnxruntime_USE_PREINSTALLED_EIGEN.

Fixed a build issue with Visual Studio 2022 17.3 (#23911)

Modernized Build Tools

Now using VCPKG for most package builds.
Upgraded Gradle from 7.x to 8.x.
Updated JDK from 11 to 17.
Enabled onnxruntime_USE_CUDA_NHWC_OPS by default for CUDA builds.
Added support for WASM64 (build from source; no package published).

Dependency Cleanup

Removed Google’s nsync from dependencies.

Others

Updated Node.js installation script to support network proxy usage (#23231)

Web

No updates of note.

Contributors

Contributors to ONNX Runtime include members across teams at Microsoft, along with our community members:

Changming Sun, Yulong Wang, Tianlei Wu, Jian Chen, Wanming Lin, Adrian Lizarraga, Hector Li, Jiajia Qin, Yifan Li, Edward Chen, Prathik Rao, Jing Fang, shiyi, Vincent Wang, Yi Zhang, Dmitri Smirnov, Satya Kumar Jandhyala, Caroline Zhu, Chi Lo, Justin Chu, Scott McKay, Enrico Galli, Kyle, Ted Themistokleous, dtang317, wejoncy, Bin Miao, Jambay Kinley, Sushanth Rajasankar, Yueqing Zhang, amancini-N, ivberg, kunal-vaishnavi, liqun Fu, Corentin Maravat, Peishen Yan, Preetha Veeramalai, Ranjit Ranjan, Xavier Dupré, amarin16, jzm-intel, kailums, xhcao, A-Satti, Aleksei Nikiforov, Ankit Maheshkar, Javier Martinez, Jianhui Dai, Jie Chen, Jon Campbell, Karim Vadsariya, Michael Tyler, PARK DongHa, Patrice Vignola, Pranav Sharma, Sam Webster, Sophie Schoenmeyer, Ti-Tai Wang, Xu Xing, Yi-Hong Lyu, genmingz@AMD, junchao-zhao, sheetalarkadam, sushraja-msft, Akshay Sonawane, Alexis Tsogias, Ashrit Shetty, Bilyana Indzheva, Chen Feiyue, Christian Larson, David Fan, David Hotham, Dmitry Deshevoy, Frank Dong, Gavin Kinsey, George Wu, Grégoire, Guenther Schmuelling, Indy Zhu, Jean-Michaël Celerier, Jeff Daily, Joshua Lochner, Kee, Malik Shahzad Muzaffar, Matthieu Darbois, Michael Cho, Michael Sharp, Misha Chornyi, Po-Wei (Vincent), Sevag H, Takeshi Watanabe, Wu, Junze, Xiang Zhang, Xiaoyu, Xinpeng Dou, Xinya Zhang, Yang Gu, Yateng Hong, mindest, mingyue, raoanag, saurabh, shaoboyan091, sstamenk, tianf-fff, wonchung-microsoft, xieofxie, zz002

Open source →

1.20.031 Oct 2024Deprecated

All ONNX Runtime Training packages have been deprecated. ORT 1.19.2 was the last release for which onnxruntime-training (PyPI), onnxruntime-training-c…

Release Manager: @apsonawane

Announcements

All ONNX Runtime Training packages have been deprecated. ORT 1.19.2 was the last release for which onnxruntime-training (PyPI), onnxruntime-training-cpu (PyPI), Microsoft.ML.OnnxRuntime.Training (Nuget), onnxruntime-training-c (CocoaPods), onnxruntime-training-objc (CocoaPods), and onnxruntime-training-android (Maven Central) were published.
ONNX Runtime packages will stop supporting Python 3.8 and Python 3.9. This decision aligns with NumPy Python version support. To continue using ORT with Python 3.8 and Python 3.9, you can use ORT 1.19.2 and earlier.
ONNX Runtime 1.20 CUDA packages will include new dependencies that were not required in 1.19 packages. The following dependencies are new: libcudnn_adv.so.9, libcudnn_cnn.so.9, libcudnn_engines_precompiled.so.9, libcudnn_engines_runtime_compiled.so.9, libcudnn_graph.so.9, libcudnn_heuristic.so.9, libcudnn_ops.so.9, libnvrtc.so.12, and libz.so.1.

Build System & Packages

Python 3.13 support is included in PyPI packages.
ONNX 1.17 support will be delayed until a future release, but the ONNX version used by ONNX Runtime has been patched to include a shape inference change to the Einsum op.
DLLs in the Maven build are now digitally signed (fix for issue reported here).
(Experimental) vcpkg support added for the CPU EP. The DML EP does not yet support vcpkg, and other EPs have not been tested.

Core

MultiLoRA support.
Reduced memory utilization.
- Fixed alignment that was causing mmap to fail for external weights.
- Eliminated double allocations when deserializing external weights.
- Added ability to serialize pre-packed weights so that they don’t cause an increase in memory utilization when the model is loaded.
Support bfloat16 and float8 data types in python I/O binding API.

Performance

INT4 quantized embedding support on CPU and CUDA EPs.
Miscellaneous performance improvements and bug fixes.

EPs

CPU

FP16 support for MatMulNbits, Clip, and LayerNormalization ops.

CUDA

Cudnn frontend integration for convolution operators.
Added support of cuDNN Flash Attention and Lean Attention in MultiHeadAttention op.

TensorRT

TensorRT 10.4 and 10.5 support.

QNN

QNN HTP support for weight sharing across multiple ORT inference sessions. (See ORT QNN EP documentation for more information.)
Support for QNN SDK 2.27.

OpenVINO

Added support up to OpenVINO 2024.4.1.
Compile-time memory optimizations.
Enhancement of ORT EPContext Session option for optimized first inference latency.
Added remote tensors to ensure direct memory access for inferencing on NPU.

DirectML

DirectML 1.15.2 support.

Mobile

Improved Android QNN support, including a pre-built Maven package and various performance improvements.
FP16 support for ML Program models with CoreML EP.
FP16 XNNPACK kernels to provide a fallback option if CoreML is not available at runtime.
Initial support for using the native WebGPU EP on Android and iOS. _Note: The set of initial operators is limited, and the code is available from the main branch, not ORT 1.20 packages. See #22591 for more information.

Web

Quantized embedding support.
On-demand weight loading support (offloads Wasm32 heap and enables 8B-parameter LLMs).
Integrated Intel GPU performance improvements.
Opset-21 support (Reshape, Shape, Gelu).

GenAI

MultiLoRA support.
Generations can now be terminated mid-loop.
Logit soft capping support in Group Query Attention (GQA).
Additional model support, including Phi-3.5 Vision Multi-Frame, ChatGLM3, and Nemotron-Mini.
Python package now available for Mac.
Mac / iOS now available in NuGet packages.

Full release notes for ONNX Runtime generate() API v0.5.0 can be found here.

Extensions

Tokenization performance improvements.
Support for latest Hugging Face tokenization JSON format (transformers>=4.45).
Unigram tokenization model support.
OpenCV dependency removed from C API build.

Full release notes for ONNX Runtime Extensions v0.13 can be found here.

Olive

Olive command line interface (CLI) now available with support to execute well-defined, concrete workflows without the need to create or edit configs manually.
Additional improvements, including support for YAML-based workflow configs, streamlined DataConfig management, simplified workflow configuration, and more.
Llama and Phi-3 model updates, including an updated MultiLoRA example using the ORT generate() API. Full release notes for Olive v0.7.0 can be found here.

Contributors

Big thank you to the release manager @apsonawane, as well as @snnn, @jchen351, @sheetalarkadam, and everyone else who made this release possible!

Tianlei Wu, Yi Zhang, Yulong Wang, Scott McKay, Edward Chen, Adrian Lizarraga, Wanming Lin, Changming Sun, Dmitri Smirnov, Jian Chen, Jiajia Qin, Jing Fang, George Wu, Caroline Zhu, Hector Li, Ted Themistokleous, mindest, Yang Gu, jingyanwangms, liqun Fu, Adam Pocock, Patrice Vignola, Yueqing Zhang, Prathik Rao, Satya Kumar Jandhyala, Sumit Agarwal, Xu Xing, aciddelgado, duanshengliu, Guenther Schmuelling, Kyle, Ranjit Ranjan, Sheil Kumar, Ye Wang, kunal-vaishnavi, mingyueliuh, xhcao, zz002, 0xdr3dd, Adam Reeve, Arne H Juul, Atanas Dimitrov, Chen Feiyue, Chester Liu, Chi Lo, Erick Muñoz, Frank Dong, Jake Mathern, Julius Tischbein, Justin Chu, Xavier Dupré, Yifan Li, amarin16, anujj, chenduan-amd, saurabh, sfatimar, sheetalarkadam, wejoncy, Akshay Sonawane, AlbertGuan9527, Bin Miao, Christian Bourjau, Claude, Clément Péron, Emmanuel, Enrico Galli, Fangjun Kuang, Hann Wang, Indy Zhu, Jagadish Krishnamoorthy, Javier Martinez, Jeff Daily, Justin Beavers, Kevin Chen, Krishna Bindumadhavan, Lennart Hannink, Luis E. P., Mauricio A Rovira Galvez, Michael Tyler, PARK DongHa, Peishen Yan, PeixuanZuo, Po-Wei (Vincent), Pranav Sharma, Preetha Veeramalai, Sophie Schoenmeyer, Vishnudas Thaniel S, Xiang Zhang, Yi-Hong Lyu, Yufeng Li, goldsteinn, mcollinswisc, mguynn-intc, mingmingtasd, raoanag, shiyi, stsokolo, vraspar, wangshuai09

Full changelog: v1.19.2...v1.20.0

Open source →

1.19.015 Aug 2024Deprecated

Remove calls to deprecated api’s

Announcements

Note that the wrong commit was initially tagged with v1.19.0. The final commit has since been correctly tagged: https://github.com/microsoft/onnxruntime/commit/26250ae74d2c9a3c6860625ba4a147ddfb936907. This shouldn't effect much, but sorry for the inconvenience!

Build System & Packages

Numpy support for 2.x has been added
Qualcomm SDK has been upgraded to 2.25
ONNX has been upgraded from 1.16 → 1.16.1
Default GPU packages use CUDA 12.x and Cudnn 9.x (previously CUDA 11.x/CuDNN 8.x) CUDA 11.x/CuDNN 8.x packages are moved to the aiinfra VS feed.
TensorRT 10.2 support added
Introduced Java CUDA 12 packages on Maven.
Discontinued support for Xamarin. (Xamarin reached EOL on May 1, 2024)
Discontinued support for macOS 11 and increasing the minimum supported macOS version to 12. (macOS 11 reached EOL in September 2023)
Discontinued support for iOS 12 and increasing the minimum supported iOS version to 13.

Core

Implemented DeformConv
Fixed big-endian and support build on AIX

Performance

Added QDQ support for INT4 quantization in CPU and CUDA Execution Providers
Implemented FlashAttention on CPU to improve performance for GenAI prompt cases
Improved INT4 performance on CPU (X64, ARM64) and NVIDIA GPUs

Execution Providers

TensorRT
- Updated to support TensorRT 10.2
- Remove calls to deprecated api’s
- Enable refittable embedded engine when ONNX model provided as byte stream
CUDA
- Upgraded cutlass to 3.5.0 for performance improvement of memory efficient attention.
- Updated MultiHeadAttention and Attention operators to be thread-safe.
- Added sdpa_kernel provider option to choose kernel for Scaled Dot-Product Attention.
- Expanded op support - Tile (bf16)
CPU
- Expanded op support - GroupQueryAttention, SparseAttention (for Phi-3 small)
QNN
- Updated to support QNN SDK 2.25
- Expanded op support - HardSigmoid, ConvTranspose 3d, Clip (int32 data), Matmul (int4 weights), Conv (int4 weights), prelu (fp16)
- Expanded fusion support – Conv + Clip/Relu fusion
OpenVINO
- Added support for OpenVINO 2024.3
- Support for enabling EpContext using session options
DirectML
- Updated DirectML from 1.14.1 → 1.15.1
- Updated ONNX opset from 17 → 20
- Opset 19 and Opset 20 are supported with known caveats:
  - Gridsample 20: 5d not supported
  - DeformConv not supported

Mobile

Additional CoreML ML Program operators were added
- See supported operators list here
Fixed packaging issue with macOS framework in onnxruntime-c cocoapod
Removed Xamarin support
- Xamarin EOL was May 1, 2024
- Xamarin official support policy | .NET (microsoft.com)

Web

Updated JavaScript packaging to align with best practices, including slight incompatibilities when apps bundle onnxruntime-web
Improved CPU operators coverage for WebNN (now supported by Chrome)

Training

No specific updates

GenAI

Support for new models Qwen, Llama 3.1, Gemma 2, phi3 small
Support to build quantized models with method AWQ and GPTQ
Performance improvements for Intel and Arm CPU
Packing and language binding
- Added Java bindings (build from source)
- Separate OnnxRuntime.dll and directml.dll out of GenAI package to improve usability
- Publish packages for Win Arm
- Support for Android (build from source)
Bug fixes, like the long prompt correctness issue for phi3.

Extensions

Added C APIs for language, vision and audio processors including new FeatureExtractor for Whisper
Support for Phi-3 Small Tokenizer and new OpenAI tiktoken format for fast loading of BPE tokenizers
Added new CUDA custom operators such as MulSigmoid, Transpose2DCast, ReplaceZero, AddSharedInput and MulSharedInput
Enhanced Custom Op Lite API on GPU and fused kernels for DORT
Bug fixes, including null bos_token for Qwen2 tokenizer and SentencePiece converted FastTokenizer issue on non-ASCII characters, as well as necessary updates for MSVC 19.40 and numpy 2.0 release

Contributors

Changming Sun, Baiju Meswani, Scott McKay, Edward Chen, Jian Chen, Wanming Lin, Tianlei Wu, Adrian Lizarraga, Chester Liu, Yi Zhang, Yulong Wang, Hector Li, kunal-vaishnavi, pengwa, aciddelgado, Yifan Li, Xu Xing, Yufeng Li, Patrice Vignola, Yueqing Zhang, Jing Fang, Chi Lo, Dmitri Smirnov, mingyueliuh, cloudhan, Yi-Hong Lyu, Ye Wang, Ted Themistokleous, Guenther Schmuelling, George Wu, mindest, liqun Fu, Preetha Veeramalai, Justin Chu, Xiang Zhang, zz002, vraspar, kailums, guyang3532, Satya Kumar Jandhyala, Rachel Guo, Prathik Rao, Maximilian Müller, Sophie Schoenmeyer, zhijiang, maggie1059, ivberg, glen-amd, aamajumder, Xavier Dupré, Vincent Wang, Suryaprakash Shanmugam, Sheil Kumar, Ranjit Ranjan, Peishen Yan, Frank Dong, Chen Feiyue, Caroline Zhu, Adam Louly, Ștefan Talpalaru, zkep, winskuo-quic, wejoncy, vividsnow, vivianw-amd, moyo1997, mcollinswisc, jingyanwangms, Yang Gu, Tom McDonald, Sunghoon, Shubham Bhokare, RuomeiMS, Qingnan Duan, PeixuanZuo, Pavan Goyal, Nikolai Svakhin, KnightYao, Jon Campbell, Johan MEJIA, Jake Mathern, Hans, Hann Wang, Enrico Galli, Dwayne Robinson, Clément Péron, Chip Kerchner, Chen Fu, Carson M, Adam Reeve, Adam Pocock.

Big thank you to everyone who contributed to this release!

Full Changelog: https://github.com/microsoft/onnxruntime/compare/v1.18.1...v1.19.0

Open source →

1.18.017 May 2024Deprecated

…iOS cocoapods are being deprecated. Please use the onnxruntime-android Android package, and onnxruntime-c/onnxruntime-objc cocoapods, which support ON…

Announcements

Windows ARM32 support has been dropped at the source code level.
Python version >=3.8 is now required for build.bat/build.sh (previously >=3.7). Note: If you have Python version <3.8, you can bypass the tools and use CMake directly.
The onnxruntime-mobile Android package and onnxruntime-mobile-c/onnxruntime-mobile-objc iOS cocoapods are being deprecated. Please use the onnxruntime-android Android package, and onnxruntime-c/onnxruntime-objc cocoapods, which support ONNX and ORT format models and all operators and data types. Note: If you require a smaller binary size, a custom build is required. See details on creating a custom Android or iOS package on Custom build | onnxruntime.

Build System & Packages

CoreML execution provider now depends on coremltools.
Flatbuffers has been upgraded from 1.12.0 → 23.5.26.
ONNX has been upgraded from 1.15 → 1.16.
EMSDK has been upgraded from 3.1.51 → 3.1.57.
Intel neural_speed library has been upgraded from v0.1.1 → v0.3 with several important bug fixes.
There is a new onnxruntime_CUDA_MINIMAL CMake option for building ONNX Runtime CUDA execution provider without any operations apart from memcpy ops.
Added support for Catalyst for macOS build support.
Added initial support for RISC-V and three new build options for it: --rv64, --riscv_toolchain_root, and --riscv_qemu_path.
Now you can build TensorRT EP with protobuf-lite instead of the full version of protobuf.
Some security-related compile/link flags have been moved from the default setting → new build option: --use_binskim_compliant_compile_flags. Note: All our release binaries are built with this flag, but when building ONNX Runtime from source, this flag is default OFF.
Windows ARM64 build now depends on PyTorch CPUINFO library.
Windows OneCore build now uses “Reverse forwarding” apisets instead of “Direct forwarding”, so onnxruntime.dll in our Nuget packages will depend on kernel32.dll. Note: Windows systems without kernel32.dll need to have reverse forwarders (see API set loader operation - Win32 apps | Microsoft Learn for more information).

Core

Added ONNX 1.16 support.
Added additional optimizations related to Dynamo-exported models.
Improved testing infrastructure for EPs developed as shared libraries.
Exposed Reserve() in OrtAllocator to allow custom allocators to work when session.use_device_allocator_for_initializers is specified.
Improved lock contention due to memory allocations.
Improved session creation time (graph and graph transformer optimizations).
Added new SessionOptions config entry to disable specific transformers and rules.
[C# API] Exposed SessionOptions.DisablePerSessionThreads to allow sharing of threadpool between sessions.
[Java API] Added CUDA 12 Java support.

Performance

Improved 4bit quant support:
- Added HQQ quantization support to improve accuracy.
- Implemented general GEMM kernel and improved GEMV kernel performance on GPU.
- Improved GEMM kernel quality and performance on x64.
- Implemented general GEMM kernel and improved GEMV performance on ARM64.
Improved MultiheadAttention performance on CPU.

Execution Providers

TensorRT
- Added support for TensorRT 10.
- Finalized support for DDS ops.
- Added Python support for user provided CUDA stream.
- Fixed various bugs.
CUDA
- Added support of multiple CUDA graphs.
- Added a provider option to disable TF32.
- Added Python support for user provided CUDA stream.
- Extended MoE to support of Tensor Parallelism and int4 quantization.
- Fixed bugs in BatchNorm and TopK kernel.
QNN
- Added support for up to QNN SDK 2.22.
- Upgraded support from A16W8 → mixed 8/16-bit precision configurability per layer.
- Added fp16 execution support via enable_htp_fp16 option.
- Added multiple partition support for QNN context binary.
- Expanded operator support and fixed various bugs.
- Added support for per-channel quantized weights for Conv.
- Integration with Qualcomm’s AIHub.
OpenVINO
- Added support for up to OpenVINO 2024.1.
- Added support for importing pre-compiled blob as EPContext blob.
- Separated device and precision as inputs by removing support for device_id in provider options and adding precision as separate CLI option.
- Deprecated CPU_FP32 and GPU_FP32 terminology and introduced CPU and GPU terminology.
- AUTO:GPU,CPU will only create GPU blob, not CPU blob.
DirectML
- Additional ONNX operator support: Resize-18 and Resize-19, Col2Im-18, InNaN-20, IsInf-20, and ReduceMax-20.
- Additional contrib op support: SimplifiedLayerNormalization, SkipSimplifiedLayerNormalization, QLinearAveragePool, MatMulIntegerToFloat, GroupQueryAttention, DynamicQuantizeMatMul, and QAttention.

Mobile

Improved performance of ARM64 4-bit quantization.
Added support for building with QNN on Android.
Added MacCatalyst support.
Added visionOS support.
Added initial support for creating ML Program format CoreML models.
Added support for 1D Conv and ConvTranspose to XNNPACK EP.

Web

Added WebNN EP preview.
Improved WebGPU performance (MHA, ROE).
Added more WebGPU and WebNN examples.
Increased generative model support.
Optimized Buffer management to reduce memory footprint.

Training

Large Model Training
- Added optimizations for Dynamo-exported models.
- Added Mixtral integration using ORT backend.
On-Device Training
- Added support for models >2GB to enable SLM training on edge devices.

GenAI

Added additional model support: Phi-3, Gemma, LLama-3.
Added DML EP support.
Improved tokenizer quality.
Improved sampling method and ORT model performance.

Extensions

Created Java packaging pipeline and published to Maven repository.
Added support for conversion of Huggingface FastTokenizer into ONNX custom operator.
Unified the SentencePiece tokenizer with other Byte Pair Encoding (BPE) based tokenizers.
Fixed Whisper large model pre-processing bug.
Enabled eager execution for custom operator and refactored the header file structure.

Contributors

Yi Zhang, Yulong Wang, Adrian Lizarraga, Changming Sun, Scott McKay, Tianlei Wu, Peng Wang, Hector Li, Edward Chen, Dmitri Smirnov, Patrice Vignola, Guenther Schmuelling, Ye Wang, Chi Lo, Wanming Lin, Xu Xing, Baiju Meswani, Peixuan Zuo, Vincent Wang, Markus Tavenrath, Lei Cao, Kunal Vaishnavi, Rachel Guo, Satya Kumar Jandhyala, Sheil Kumar, Yifan Li, Jiajia Qin, Maximilian Müller, Xavier Dupré, Yi-Hong Lyu, Yufeng Li, Alejandro Cid Delgado, Adam Louly, Prathik Rao, wejoncy, Zesong Wang, Adam Pocock, George Wu, Jian Chen, Justin Chu, Xiaoyu, guyang3532, Jingyan Wang, raoanag, Satya Jandhyala, Hariharan Seshadri, Jiajie Hu, Sumit Agarwal, Peter Mcaughan, Zhijiang Xu, Abhishek Jindal, Jake Mathern, Jeff Bloomfield, Jeff Daily, Linnea May, Phoebe Chen, Preetha Veeramalai, Shubham Bhokare, Wei-Sheng Chin, Yang Gu, Yueqing Zhang, Guangyun Han, inisis, ironman, Ivan Berg, Liqun Fu, Yu Luo, Rui Ren, Sahar Fatima, snadampal, wangshuai09, Zhenze Wang, Andrew Fantino, Andrew Grigorev, Ashwini Khade, Atanas Dimitrov, AtomicVar, Belem Zhang, Bowen Bao, Chen Fu, Dhruv Matani, Fangrui Song, Francesco, Frank Dong, Hans Chen, He Li, Heflin Stephen Raj, Jambay Kinley, Masayoshi Tsutsui, Matttttt, Nanashi, Phoebe Chen, Pranav Sharma, Segev Finer, Sophie Schoenmeyer, TP Boudreau, Ted Themistokleous, Thomas Boby, Xiang Zhang, Yongxin Wang, Zhang Lei, aamajumder, danyue, Duansheng Liu, enximi, fxmarty, kailums, maggie1059, mindest, mo-ja, moyo1997 Big thank you to everyone who contributed to this release!

Open source →

1.17.031 Jan 2024Deprecated

Updated calls to deprecated TensorRT APIs (e.g., enqueue_v2 → enqueue_v3).

Announcements

In the next release, we will totally drop support for Windows ARM32.

General

Added support for new ONNX 1.15 opsets: IsInf-20, IsNaN-20, DFT-20, ReduceMax-20, ReduceMin-20, AffineGrid-20, GridSample, ConstantOfShape-20, RegexFullMatch, StringConcat, StringSplit, and ai.onnx.ml.LabelEncoder-4.
Updated C/C++ libraries: abseil, date, nsync, googletest, wil, mp11, cpuinfo, safeint, and onnx.
Added vector optimization code for loongarch architecture.

Build System and Packages

Dropped CentOS 7 support. All Linux binaries now require glibc version >=2.28, but users can still build the source code for a lower glibc version.
Added CUDA 12 packages for Python and Nuget.
Added Python 3.12 packages for ONNX Runtime Inference. ONNX Runtime Training Python 3.12 packages cannot be provided at this time since training packages depend on PyTorch, which does not support Python 3.12 yet.
Linux binaries (except those in AMD GPU packages) are built in a more secure way that is compliant with BinSkim's default policy (e.g., the binaries no longer have an executable stack).
Added support for Windows ARM64X for users who build ONNX Runtime from source. No prebuilt package provided yet.
Removed Windows ARM32 binaries from official packages. Users who still need these binaries can build them from source.
Added AMD GPU package with ROCm and MiGraphX (Python + Linux only).
Split ONNX Runtime GPU Nuget package into two packages.
When building the source code for Linux ARM64 or Android, the C/C++ compiler must support BFloat16. Support for Android NDK 24.x has been removed. Please use NDK 25.x or 26.x instead.
Link time code generation (LTCG or LTO) is now disabled by default when building from source. To re-enable it, users can add "--enable_lto" to the build command. All prebuilt binaries are still built with LTO.

Core

Optimized graph inlining.
Allow custom op to invoke internal thread-pool for parallelism.
Added support for supplying a custom logger at the session level.
Added new logging and tracing of session and execution provider options.
Added new dynamic ETW provider that can trace/diagnose ONNX internals while maintaining great performance.

Performance

Added 4bit quant support on NVIDIA GPU and ARM64.

EPs

TensorRT EP

Added support for direct load of precompiled TensorRT engines and customizable engine prefix.
Added Python support for TensorRT plugins via ORT custom ops.
Fixed concurrent Session::Run bugs.
Updated calls to deprecated TensorRT APIs (e.g., enqueue_v2 → enqueue_v3).
Fixed various memory leak bugs.

QNN EP

Added support for QNN SDK 2.18.
Added context binary caching and model initialization optimizations.
Added mixed precision (8/16 bit) quantization support.
Add device-level session options (soc_model, htp_arch, device_id), extreme_power_saver for htp_performance_mode, and vtcm_mb settings.
Fixed multi-threaded inference bug.
Fixed various other bugs and added performance improvements.
QNN profiling of the NPU can be enabled dynamically with ETW or write out to CSV.

OpenVINO EP

Added support for OpenVINO 2023.2.
Added AppendExecutionProvider_OpenVINO_V2 API for supporting new OpenVINO EP options.

DirectML EP

Updated to DirectML 1.13.1.
Updated operators LpPool-18 and AveragePool-19 with dilations.
Improved Python I/O binding support.
Added RotaryEmbedding.
Added support for fusing subgraphs into DirectML execution plans.
Added new Python API to choose a specific GPU on multi-GPU devices with the DirectML EP.

Mobile

Added initial support for 4bit quantization on ARM64.
Extended CoreML/NNAPI operator coverage.
Added support for YOLOv8 pose detection pre/post processing.
Added support for macOS in CocoaPods package.

Web

Added support for external data format.
Added support for I/O bindings.
Added support for training.
Added WebGPU optimizations.
Transitioned WebGPU out of experimental.
Added FP16 support for WebGPU.

Training

Large Model Training

Enabled support for QLoRA (with support for BFloat16).
Added symbolic shape support for Triton codegen (see PR).
Made improvements to recompute optimizer with easy ON/OFF to allow layer-wise recompute (see PR).
Enabled memory-efficient gradient management. For Mistral, we see ~10GB drop in memory consumption when this feature is ON (see PR).
Enabled embedding sparsity optimizations.
Added support for Aten efficient attention and Triton Flash Attention (see PR).
Packages now available for CUDA 11.8 and 12.1.

On Device Training

On-Device training will now support training on the web. This release focuses on federated learning and developer exploration scenarios. More features coming soon in future releases.

Extensions

Modified gen_processing_model tokenizer model to output int64, unifying output datatype of all tokenizers.
Implemented support for post-processing of YOLO v8 within the Python extensions package.
Introduced 'fairseq' flag to enhance compatibility with certain Hugging Face tokenizers.
Incorporated 'added_token' attribute into the BPE tokenizer to improve CodeGen tokenizer functionality.
Enhanced the SentencePiece tokenizer by integrating token indices into the output.
Added support for the custom operator implemented with CUDA kernels, including two example operators.
Added more tests on the Hugging Face tokenizer and fixed identified bugs.

Known Issues

The onnxruntime-training package is not yet available in PyPI but can be accessed in ADO as follows:

python -m pip install cerberus flatbuffers h5py numpy>=1.16.6 onnx packaging protobuf sympy setuptools>=41.4.0
pip install -i https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ORT/pypi/simple/ onnxruntime-training
pip install torch-ort
python -m torch_ort.configure

Installation instructions can also be accessed here.

For models with int4 kernel only:
- Crash may occur when int4 is applied on Intel CPUs with hybrid core if the E-cores are disabled in BIOS. Fix is in progress to be patched.
- The "neural-speed" library used by int4 kernels has a bug that could lead to out-of-bounds memory read/write.
- Performance regression on the int4 kernel on x64 makes the op following MatMulNBits much slower. Fix is in progress to be patched.
Current bug in BeamSearch implementation of T5, GPT, and Whisper may break models under heavy inference load using BeamSearch on CUDA. See #19345. Fix is in progress to be patched.
Full support of ONNX 1.15 opsets is still in progress. A list of new ONNX 1.15 opset support that has been included in this release can be found above in the 'General' section.
Some Cast nodes will not be removed (see https://github.com/microsoft/onnxruntime/pull/17953): Cast node from higher precision to lower precision (like fp32 to fp16) will be kept. If model result is different from ORT 1.16 and 1.17, check whether some Cast nodes was removed in 1.16 but kept in 1.17.
When running ONNX Runtime's python 3.12 package on Windows 11, you may see a warning like: “Unsupported Windows version (11). ONNX Runtime supports Windows 10 and above, only.” You may safely ignore it.

Contributions

Contributors to ONNX Runtime include members across teams at Microsoft, along with our community members: Changming Sun, Yulong Wang, Tianlei Wu, Yi Zhang, Jian Chen, Jiajia Qin, Adrian Lizarraga, Scott McKay, Wanming Lin, pengwa, Hector Li, Chi Lo, Dmitri Smirnov, Edward Chen, Xu Xing, satyajandhyala, Rachel Guo, PeixuanZuo, RandySheriffH, Xavier Dupré, Patrice Vignola, Baiju Meswani, Guenther Schmuelling, Jeff Bloomfield, Vincent Wang, cloudhan, zesongw, Arthur Islamov, Wei-Sheng Chin, Yifan Li, raoanag, Caroline Zhu, Sheil Kumar, Ashwini Khade, liqun Fu, xhcao, aciddelgado, kunal-vaishnavi, Aditya Goel, Hariharan Seshadri, Ye Wang, Adam Pocock, Chen Fu, Jambay Kinley, Kaz Nishimura, Maximilian Müller, Yang Gu, guyang3532, mindest, Abhishek Jindal, Justin Chu, Numfor Tiapo, Prathik Rao, Yufeng Li, cao lei, snadampal, sophies927, BoarQing, Bowen Bao, George Wu, Jiajie Hu, MistEO, Nat Kershaw (MSFT), Sumit Agarwal, Ted Themistokleous, ivberg, zhijiang, Christian Larson, Frank Dong, Jeff Daily, Nicolò Lucchesi, Pranav Sharma, Preetha Veeramalai, Cheng Tang, Xiang Zhang, junchao-loongson, petermcaughan, rui-ren, shaahji, simonjub, trajep, Adam Louly, Akshay Sonawane, Artem Shilkin, Atanas Dimitrov, AtanasDimitrovQC, BODAPATIMAHESH, Bart Verhagen, Ben Niu, Benedikt Hilmes Brian Lambert, David Justice, Deoksang Kim, Ella Charlaix, Emmanuel Ferdman, Faith Xu, Frank Baele, George Nash, hans00, computerscienceiscool, Jake Mathern, James Baker, Jiangzhuo, Kevin Chen, Lennart Hannink, Lukas Berbuer, Mike Guo, Milos Puzovic, Mustafa Ateş Uzun, Peishen Yan, Ran Gal, Ryan Hill, Steven Roussey, Suryaprakash Shanmugam, Vadym Stupakov, Yiming Hu, Yueqing Zhang, Yvonne Chen, Zhang Lei, Zhipeng Han, aimilefth, gunandrose4u, kailums, kushalpatil07, kyoshisuki, luoyu-intel, moyo1997, tbqh, weischan-quic, wejoncy, winskuo-quic, wirthual, yuwenzho

Open source →

onnxruntime-common

Release timeline

Releases

Highlights

Announcements & Compatibility

Security & Reliability

Model Loading, Memory, and Input Validation

GPU Bounds and Resource Lifetimes

Dependencies and Tooling

New Features

Core APIs & Runtime

Plugin Execution Providers

Execution Provider Updates

CUDA EP

Attention and Decoding

MoE and Quantized Matrix Multiplication

Operators

WebGPU EP

WebNN EP

TensorRT & DirectML

CPU & Core Optimizations

MLAS and CPU Kernels

Graph, Optimizer, and Runtime

Language Bindings & Web

Build, Packaging & CI

Contributors

From the same repository

Choose a tag to compare

Announcements & Breaking Changes

Security Fixes

Path, bounds, and input validation

Supply chain and tooling

New Features

Core APIs & Runtime

Execution Provider ABI & Plugin EPs

Execution Provider Updates

NVIDIA CUDA EP

Attention and decoding

MoE and quantized GEMM

Operators and collectives

WebGPU EP

WebNN EP

OpenVINO / QNN / DML / XNNPACK / TensorRT

CPU & Core Optimizations

MLAS

CPU Attention & Kernels

Graph, Optimizer, and Runtime

Web & JavaScript

Build, Packaging & CI

Contributors

Choose a tag to compare

Announcements & Breaking Changes

Security Fixes

New Features

Execution Provider Plugin API

Core APIs & Runtime

Quantization & Training Tooling

Execution Provider Updates

NVIDIA CUDA EP

WebGPU EP

CoreML EP

TensorRT / DML / QNN

Web & JavaScript

JavaScript / Node.js

CPU & Core Optimizations

MLAS / Quantization / Attention

Graph, Optimizer, and Fusion

Language Bindings

Python

C#

Bug Fixes

Critical & Correctness Fixes

Portability Fixes

Contributors

Choose a tag to compare

Announcement - Breaking Changes

Highlights

Security and Reliability Hardening

CUDA, Attention, and MLAS

WebGPU, WebNN, and JavaScript