This release is meant to fix the following regressions:

• Wheels from PyPI are unusable out of the box on PRM-based Linux distributions: https://github.com/pytorch/pytorch/issues/138324
• PyPI arm64 distribution logs cpuinfo error on import: https://github.com/pytorch/pytorch/issues/138333
• Crash When Using torch.compile with Math scaled_dot_product_attention in AMP Mode: https://github.com/pytorch/pytorch/issues/133974
• [MPS] Internal crash due to the invalid buffer size computation if sliced API is used: https://github.com/pytorch/pytorch/issues/137800
• Several issues related to CuDNN Attention: https://github.com/pytorch/pytorch/pull/138522

Besides the regression fixes, the release includes several documentation updates.

See release tracker https://github.com/pytorch/pytorch/issues/132400 for additional information.

PyTorch 2.5 Release Notes

• Highlights
• Backwards Incompatible Change
• Deprecations
• New Features
• Improvements
• Bug fixes
• Performance
• Documentation
• Developers
• Security

Highlights

We are excited to announce the release of PyTorch® 2.5! This release features a new CuDNN backend for SDPA, enabling speedups by default for users of SDPA on H100s or newer GPUs. As well, regional compilation of torch.compile offers a way to reduce the cold start up time for torch.compile by allowing users to compile a repeated nn.Module (e.g. a transformer layer in LLM) without recompilations. Finally, TorchInductor CPP backend offers solid performance speedup with numerous enhancements like FP16 support, CPP wrapper, AOT-Inductor mode, and max-autotune mode. This release is composed of 4095 commits from 504 contributors since PyTorch 2.4. We want to sincerely thank our dedicated community for your contributions. As always, we encourage you to try these out and report any issues as we improve 2.5. More information about how to get started with the PyTorch 2-series can be found at our Getting Started page. As well, please check out our new ecosystem projects releases with TorchRec and TorchFix.

Beta	Prototype
CuDNN backend for SDPA	FlexAttention
torch.compile regional compilation without recompilations	Compiled Autograd
TorchDynamo added support for exception handling & MutableMapping types	Flight Recorder
TorchInductor CPU backend optimization	Max-autotune Support on CPU with GEMM Template
	TorchInductor on Windows
	FP16 support on CPU path for both eager mode and TorchInductor CPP backend
	Autoload Device Extension
	Enhanced Intel GPU support

*To see a full list of public feature submissions click here.

BETA FEATURES

[Beta] CuDNN backend for SDPA

The cuDNN "Fused Flash Attention" backend was landed for torch.nn.functional.scaled_dot_product_attention. On NVIDIA H100 GPUs this can provide up to 75% speed-up over FlashAttentionV2. This speedup is enabled by default for all users of SDPA on H100 or newer GPUs.

[Beta] torch.compile regional compilation without recompilations

Regional compilation without recompilations, via torch._dynamo.config.inline_inbuilt_nn_modules which default to True in 2.5+. This option allows users to compile a repeated nn.Module (e.g. a transformer layer in LLM) without recompilations. Compared to compiling the full model, this option can result in smaller compilation latencies with 1%-5% performance degradation compared to full model compilation.

See the tutorial for more information.

[Beta] TorchInductor CPU backend optimization

This feature advances Inductor’s CPU backend optimization, including CPP backend code generation and FX fusions with customized CPU kernels. The Inductor CPU backend supports vectorization of common data types and all Inductor IR operations, along with the static and symbolic shapes. It is compatible with both Linux and Windows OS and supports the default Python wrapper, the CPP wrapper, and AOT-Inductor mode.

Additionally, it extends the max-autotune mode of the GEMM template (prototyped in 2.5), offering further performance gains. The backend supports various FX fusions, lowering to customized kernels such as oneDNN for Linear/Conv operations and SDPA. The Inductor CPU backend consistently achieves performance speedups across three benchmark suites—TorchBench, Hugging Face, and timms—outperforming eager mode in 97.5% of the 193 models tested.

PROTOTYPE FEATURES

[Prototype] FlexAttention

We've introduced a flexible API that enables implementing various attention mechanisms such as Sliding Window, Causal Mask, and PrefixLM with just a few lines of idiomatic PyTorch code. This API leverages torch.compile to generate a fused FlashAttention kernel, which eliminates extra memory allocation and achieves performance comparable to handwritten implementations. Additionally, we automatically generate the backwards pass using PyTorch's autograd machinery. Furthermore, our API can take advantage of sparsity in the attention mask, resulting in significant improvements over standard attention implementations.

For more information and examples, please refer to the official blog post and Attention Gym.

[Prototype] Compiled Autograd

Compiled Autograd is an extension to the PT2 stack allowing the capture of the entire backward pass. Unlike the backward graph traced by AOT dispatcher, Compiled Autograd tracing is deferred until backward execution time, which makes it impervious to forward pass graph breaks, and allows it to record backward hooks into the graph.

Please refer to the tutorial for more information.

[Prototype] Flight Recorder

Flight recorder is a new debugging tool that helps debug stuck jobs. The tool works by continuously capturing information about collectives as they run. Upon detecting a stuck job, the information can be used to quickly identify misbehaving ranks/machines along with code stack traces.

For more information please refer to the following tutorial.

[Prototype] Max-autotune Support on CPU with GEMM Template

Max-autotune mode for the Inductor CPU backend in torch.compile profiles multiple implementations of operations at compile time and selects the best-performing one. This is particularly beneficial for GEMM-related operations, using a C++ template-based GEMM implementation as an alternative to the ATen-based approach with oneDNN and MKL libraries. We support FP32, BF16, FP16, and INT8 with epilogue fusions for x86 CPUs. We’ve seen up to 7% geomean speedup on the dynamo benchmark suites and up to 20% boost in next-token latency for LLM inference.

For more information please refer to the tutorial.

[Prototype] TorchInductor CPU on Windows

Inductor CPU backend in torch.compile now works on Windows. We support MSVC (cl), clang (clang-cl) and Intel compiler (icx-cl) for Windows inductor currently.

See the tutorial for more details.

[Prototype] FP16 support on CPU path for both eager mode and TorchInductor CPP backend

Float16 is a commonly used reduced floating point type for performance improvement in neural network inference/training. Since this release, float16 for both eager and TorchInductor is supported on the CPU path.

[Prototype] Autoload Device Extension

PyTorch now supports autoloading for out-of-tree device extensions, streamlining integration by eliminating the need for manual imports. This feature, enabled through the torch.backends entrypoint, simplifies usage by ensuring seamless extension loading, while allowing users to disable it via an environment variable if needed.

See the tutorial for more information.

[Prototype] Enhanced Intel GPU support

Intel GPUs support enhancement is now available for both Intel® Data Center GPU Max Series and Intel® Client GPUs (Intel® Core™ Ultra processors with built-in Intel® Arc™ graphics and Intel® Arc™ Graphics for dGPU parts), which is to make it easier to accelerate your Machine Learning workflows on Intel GPUs in PyTorch 2.5 release. We also enabled the initial support of PyTorch on Windows for Intel® Client GPUs in this release.

• Expanded PyTorch hardware backend support matrix to include both Intel Data Center and Client GPUs.
• The implementation of SYCL* kernels to enhance coverage and execution of Aten operators on Intel GPUs to boost performance in PyTorch eager mode.
• Enhanced Intel GPU backend of torch.compile to improve inference and training performance for a wide range of deep learning workloads.

These features are available through PyTorch preview and nightly binary PIP wheels. For more information regarding Intel GPU support, please refer to documentation.

Backwards Incompatible changes

Distributed

• [c10d] Remove Option for ProcessGroup and Expose backend Options to reflect the correct code structure (#132931)

  • We released Dispatchable collectives in 2.0 and we will use Backend Option for Backend initialization and the PG options are not needed any more.
  • In 2.4 and before, users can do:

  # Users can pass in a basic option when creating an instance of ProcessGroup
  base_pg_options = ProcessGroup.Options(backend=str(backend))
  base_pg_options._timeout = timeout
  
  pg: ProcessGroup = ProcessGroup(
    store, rank, group_size, base_pg_options
  )
  
  # Users then need to create a backend option to create the comm backend (e.g., ProcessGroupNCCL)
  pg_options = ProcessGroupNCCL.Options()
  backend = ProcessGroupNCCL(
    store, rank, group_size, pg_options
  )
  

  • But from 2.5 onwards, users don’t need to pass in an option to create an instance of ProcessGroup and user can still set default backend for the pg since users still try to get default backend in the code:

  # No basic option is passed in when creating a instance of ProcessGroup
  pg: ProcessGroup = ProcessGroup(store, rank, group_size)
  pg._set_default_backend(Backend.backend_type_map[backend])
  # Users then need to create a backend option to create the comm backend (e.g., ProcessGroupNCCL)
  pg_options = ProcessGroupNCCL.Options()
  backend = ProcessGroupNCCL(
    store, rank, group_size, pg_options
  )
  

Export

• Remove dynamic_dim() (#134211)
  • The dynamic_dim() method for specifying dynamic shapes in torch.export() has been removed. Please refer to the export tutorial for using Dims to specify dynamic shapes.

Inductor

• [Torch] Support meta device in checkpoint (#132684)

• Switch to internal benchmarking and update benchmarking path (#132827)

  This change moves from using triton’s benchmarking utils to the internal inductor utils at torch._inductor.runtime.benchmarking. To update your benchmarking code:

  # before
  from torch._inductor.runtime.runtime_utils import do_bench_gpu
  # ...
  do_bench_gpu(kernel, rep=40, fast_flush=True)
  
  # after
  from torch._inductor.runtime.benchmarking import benchmarker
  # ...
  benchmarker.benchmark_gpu(kernel_call, rep=40, fast_flush=True)
  

mps

• [MPS][BE] Delete MacOS-12.3 specific checks (#133141)

nn

• Update fused kernels and call _safe_softmax from SDPA (#131863)

  Before this PR, fully masked rows in the attn_mask passed to nn.functional.scaled_dot_product_attention would yield NANs in the output, after this PR, fully masked rows yield 0s.

  Example:

  2.4.0

  B, S, D = 1, 1, 128
  
  q = torch.randn(B, S, D, device='cuda')  
  k = torch.randn(B, S, D, device='cuda')  
  v = torch.randn(B, S, D, device='cuda')
  
  attn_mask = torch.tensor([False], device='cuda')
  
  F.scaled_dot_product_attention(q, k, v, attn_mask=attn_mask)  
  tensor([[[nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,  
            nan, nan, nan, nan, nan, nan, nan, nan, nan]]], device='cuda:0')  
  

  2.5.0

  B, S, D = 1, 1, 128
  
  q = torch.randn(B, S, D, device='cuda')  
  k = torch.randn(B, S, D, device='cuda')  
  v = torch.randn(B, S, D, device='cuda')
  
  attn_mask = torch.tensor([False], device='cuda')
  
  F.scaled_dot_product_attention(q, k, v, attn_mask=attn_mask)  
  tensor([[[-0., -0., 0., -0., 0., 0., -0., -0., -0., -0., 0., -0., -0., -0., 0., -0., 0., 0., -0., -0., 0., -0., -0.,  
            0., 0., -0., 0., -0., -0., 0., -0., -0.]]], device='cuda:0')  
  

Optimizer Frontend

• Add support to GradScaler for respecting an already set grad_scale value (#123429)

Python Frontend

• No more CPython 3.8 support and removal from binary (#132138)
  CPython 3.8 is now EOL and PyTorch 2.4 is the last version that is supported.
  See https://devguide.python.org/versions/ for CPython EOL timelines and https://github.com/pytorch/pytorch/blob/main/RELEASE.md#python for PyTorch’s support CPython version policy.

ONNX

Options to torch.onnx.export (except for the first three arguments) are now keyword-only (#131501)

Options can be supplied by keywords only to allow for future addition and evolution of the torch.onnx.export API.

Example:
Version 2.4

torch.onnx.export(model, input, f, True, False)  

Version 2.5:

torch.onnx.export(model, input, f, export_params=True, verbose=False)  

Deprecated internal API torch.onnx._export has been removed (133824)

torch.onnx._export is an internal API which is not meant for public consumption. Use the public torch.onnx.export instead.

Example:
Version 2.4

torch.onnx._export(...)  

Version 2.5:

torch.onnx.export(...)  

The op_level_debug option from torch.onnx.ExportOptions has been removed (#134961)

This option, designed to identify operator discrepancies, proved unreliable and has been removed. Instead, use torch.onnx.export(..., report=True, verify=True) option to validate exported models.

The ONNXProgramSerializer class has been removed (#135261)

The ONNX model in torch.onnx.ONNXProgram is now maintained and serialized by ONNX IR.
textproto, onnxtext, and json formats are supported by default when calling ONNXProgram.save() with a corresponding file extension.

The SymbolicContext class has been removed (#132184)

The deprecated torch.onnx.SymbolicContext class has been removed. (Non-dynamo) custom symbolic functions can no longer take ctx: torch.onnx.SymbolicContext as the first argument.

Support for caffe2 has been removed (#129021)

• Remove Caffe2 handling from onnx_unpack_quantized_weights (#129021)
• Remove is_caffe2_aten_fallback in torch.onnx.symbolic_helper

Some errors classes are removed

CheckerError and InvalidExportOptionsError are removed. Users can always catch RuntimeError to handle torch.onnx export errors.

Deprecations

Dynamo

• Remove torch._dynamo.utils.CompileProfiler (#135133)

Export

• Deprecate None for specifying static dimensions in dynamic_shapes (#134877)<br> The use of None at the dimension-level for specifying dynamic shapes is now deprecated, and a user warning will be raised, so please use Dim.STATIC in its place. Specifying None for an entire input, or an entire program, is still supported.

Inductor

• aot_autograd: copy metadata from fw to bw nodes (#126573)
• deprecate search_autotune_cache (#133628)

Releng

• Deprecate Python 3.8 support from CI/CD (#133621, #133624, #135245)

ONNX

Supplying model keyword arguments to torch.onnx.export is deprecated (#131501)

The ability to supply model keyword arguments as a final dictionary is deprecated. Users should use the kwargs parameter instead.

Deprecated:

torch.onnx.export(model, (arg1, arg2, {“kwarg1”: …}))

Future:

torch.onnx.export(model, (arg1, arg2), kwargs={“kwarg1”: …})  

torch.onnx.OperatorExportTypes is deprecated (#131501)

The ability to supply operator_export_type in torch.onnx.export() is deprecated. Exported ONNX graphs will always use the ONNX opset domain. Options ONNX_FALLTHROUGH, ONNX_ATEN and ONNX_ATEN_FALLBACK are no longer supported. The OperatorExportTypes class will be removed in a future release.

The training option in torch.onnx.export is deprecated

Set the model training mode first before exporting instead.

Deprecated:

torch.onnx.export(model, inputs, path, training=torch.onnx.TrainingMode.EVAL)  

Future:

model = model.eval()  
torch.onnx.export(model, inputs, path)  

New features

Autograd frontend

• Add selective activation checkpoint support to torch.utils.checkpoint (#125795, #129262)

Distributed

Flight Recorder with an analyzer

• Flight Recorder captures diagnostics information as collectives run- right now only for NCCL collectives. The captured diagnostic information is used to help root cause issues when jobs get stuck or timeout. An available analyzer script runs known heuristics using the collected data and attempts to automatically identify the underlying issue that caused the job to stall. (#110960, #113678, #114615, #114651, #114810, #114817, #115090, #115139, #115176, #115358, #115851, #118044, #118046, #118047, #119249, #119748, #119837, #120063, #120262, #120724, #120975, #122731, #126581, #126581, #126726, #128190, #128781, #128948, #129505, #130764, #131268, #133150, #133237, #133933, #133412, #134383, #134528, #134780, #134794)

c10d

• Enabled symmetricMemory-based, low contention intra-node all-gather and reduce-scatter (#130583)

Dynamo

• Introduce torch._dynamo.config.enable_compiler_collectives for syncing compilation across ranks (#130935)

Export

• export_for_training [WIP/unstable] (#129092, #130062, #134677, #135549)
• Automatic dynamic shapes (#133620, #134486, #134702)

Inductor

• [inductor] Add Triton template for Conv3D (#129518)
• Autoheuristic: add config options for specifying for which optimizations to collect data, and for which optimizations to use learned heuristics (#130245)
• Automatic horizontal fusion for Inductor ComboKernels (#131675)
• Mode to emulate amp numerics (#131595)
• [halide]The issue Add GPU support for the Halide backend adds the necessary functionality to enable GPU acceleration in PyTorch's Halide backend, improving compatibility with GPU-based computation and enhancing performance for specific workloads.(#127506)
• [halide]The issue Enable bfloat16 support for the Halide backend introduces support for bfloat16 (bf16) data types in the Halide backend of PyTorch, expanding its capability to handle lower-precision computations and improving performance for models that benefit from mixed-precision training.(#129036)
• [halide]The issue Support scan kernels in the Halide backend adds support for scan operations in PyTorch's Halide backend, enabling efficient reductions across multiple axes in tensors(#129035)
• The issue Support adding a new inductor backend using PrivateUse1 enables the registration of a custom backend using the PrivateUse1 device type in PyTorch's inductor, facilitating backend extensions and new device types for specialized hardware.(#129953)
• [halide]The issue Random number generation for the Halide backend introduces support for random number generation in PyTorch's Halide backend, enabling randomized operations for certain tensor computations that previously lacked support.(#130211)
• [aoti] The issue Add packaging solution introduces a packaging solution for AOTInductor, allowing AOT-generated files to be packaged into a zipfile and loaded in Python. This feature supports the compilation and loading of precompiled models, enabling a more efficient workflow for distributed models (#129895)
• Adds support for matrix decompositions when working with tensors that have unbacked sizes.(#128655)
• Adds support for Intel GPUs by splitting reduction operations. (#129120)
• Introduces a benchmark flag for inductor configuration, enhancing test workflows. (#129034)
• Adds support for nested kernels in Triton when using indirect indexing. (#129223)
• Introduces a composable kernel backend for ROCm-enabled devices in PyTorch's inductor. (#125453)
• Adds support for mutating input tensors within CUDAGraph trees. (#129184)
• Enables vectorization for bitwise operations in the C++ backend. (#129733)
• Adds support for quantized linear GEMM templates with FP32 outputs. (#128825)
• Extends GEMM template support to INT8 output with unary post-operation support. (#129048)
• Adds support for binary fusion in GEMM templates for quantized linear operations. (#129103)
• Adds support for AMX micro-GEMM kernel with int8 data type for quantized linear operations. (#129220)
• Extends UserDefinedTritonKernel to support multiple outputs. (#129325)
• Enables support for handling multiple outputs in FlexAttention operations. (#129344)
• Introduces visualization methods for block masks in FlexAttention. (#129950)
• Introduces the initial implementation of the B2B-GEMM pass with accompanying tests. (#129995)
• Adds support for FX graph caching on AMD GPUs. (#130463)
• Adds a GroupedSchedulerNode to handle nodes that need to be scheduled together in FSDP2. (#128568)
• Adds support for flex decoding in FlexAttention's high-order programming (HOP). (#129415)
• Adds partial masking support to FlexAttention. (#130415)
• Adds a DeferredCudaGridLine wrapper for CUDA grid operations. (#129268)
• Adds support for folding conv_bn with mixed data types in the post-grad phase. (#133968)
• Enables OpenMP support in the inductor when using the Intel compiler on Linux. (#134973)
• Enables the use of CUDA graphs even when there are unused CPU inputs. (#134749)
• Adds support for MKLDNN convolution operations in the C++ wrapper for inductor. (#134475)
• Introduces support for generalized linear operations using MKLDNN in the C++ wrapper. (#134783)
• Extends support for quantized convolution operations in MKLDNN via the C++ wrapper. (#134795)
• Adds support for unbacked symbolic integer (symint) divisors in variables and sizes. (#130595)

nn

• Made FlexAttention API public (#130755)
• Add nn.Modules.set_submodule() like get_submodule (#127714)
• Add nn.Buffer like nn.Parameter (#125971)

Optim

• Add an Adafactor impl (forloop and foreach) (#129905, #132336)
• Add support for capturable optimizers on hpu and xpu (#132119)

Optimizer Frontend

• Disable expandable segments checkpointing internally (#132048)

Profiler

• [Profiler] Collect observer traces from C++ child threads (#128743)
• [Profiler][XPU] Introduce kineto-based XPU profiler (#130811)
• [Profiler] Add API for Dynamic Activity Toggling [2/n] (#133035)
• [Memory Snapshot][Viz] Show event timestamps if collected (#132523)
• [Memory Snapshot][Viz] Add Allocator Settings Tab (#132518)
• [Profiler] Add kwargs to Kineto Traces (#130373)

Python Frontend

• Add support for device extension autoloading (#127074)
• Added support for sharing tensors on the meta device between processes (#129520)
• add __torch_function__ handler to Tensor.get_device cpp (#132567)
• Add torch.serialization.skip_data context manager to create a metadata-only checkpoint (#134504)
• Add torch.serialization.safe_globals context manager to work with weights_only serialization (#127939)

Quantization

PT2E Numeric Debugger

• Preserve _numeric_debug_handle throguh deepcopy and re-export (#129287)
• Add numeric_debugger top level APIs (#130643)
• Update pt2e numeric debugger to use node.meta["custom"] field (#134040)
• Fix output node's meta (#131706)

Releng

• Split Build - Create a distribution of pytorch which is composed of a c++ component and mostly python component similar to jax and jaxlib (#129088, #127934, #126813, #129011, #129270, #129253, #129269, #129774, #132537, #124995, #134624)
• Intel GPU enablement in CI/CD. Add prototype Linux Manywheel binary builds with better ATen operation coverage and improved torch.compile support (#129730, #129560, #128486, #130742, #130922, #133069, #132854, #129847, #134074, #134204, #134214, #134461, #134464, #134455, #134312, #133151, #124147)
• Add prototype Linux Manywheel Python 3.13 binary builds (#130030, #132984, #133670)

XPU

• Improve ATen operation coverage and support Intel Client GPUs in addition to Intel Data Center GPUs (#135833)
• Enable Windows support and enable PyTorch wheel build for Windows (#135833, #133151, #134312)
• Enable deterministic support for Intel GPU operations (#127277, #129864)
• Support _GLIBCXX_USE_CXX11_ABI both 0 and 1 mode (#130110)

Sparse Frontend

• Add pinned memory support to COO/CSR/CSC/BSR/BSC tensors (#129645)
• Add MaskedTensor support to _is_any_true (#128574)
• Add MaskedTensor support to *_like API, example: empty_like, etc. ops (#128637)
• Add MaskedTensor passthrough: unfold, F.Unfold, F.Fold, stack (#125262)

ONNX

The dynamo=True option and new export logic (#132530, #133743, #134304, #134782, #135378, #135399, #135786, #136162, #135134, #134976, #135367, #135418, #135591, #135520)

We introduce the dynamo=True option in torch.onnx.export(). This is recommended as a replacement for torch.onnx.dynamo_export starting in PyTorch 2.5.

Version 2.5:

onnx_program = torch.onnx.export(model, inputs, kwargs=kwargs, dynamo=True)  
# Use the external_data option to save weights as external data  
onnx_program.save(“model.onnx”, external_data=True)  
# To save without initializers  
onnx_program.save(“model.onnx”, include_initializers=False, keep_initializers_as_inputs=True)  

torch.onnx.export(model, args, dynamo=True, report=True, verify=True) leverages torch.export and ONNX IR to convert captured ExportedPrograms to ONNX efficiently and robustly. This new process reduces memory consumption by half compared to dynamo_export in 2.4, while preserving rich tensor shape and stack trace information in the ONNX graph. You can leverage the report=True option to obtain a conversion report in markdown format to diagnose any conversion issues. Set verify=True to verify the ONNX model numerically with ONNX Runtime.

When using external_data=True to save model weights as external data to the .onnx file, weights larger than 1 MB are now aligned at 64 KB addresses. This allows runtimes to memory-map weights for better memory efficiency during inference.

[NOTE]
The dynamo=True option currently supports only ONNX opset 18. Future releases will expand support to newer opsets.

[NOTE]
The dynamo=True option requires the latest versions of onnxscript and onnx packages.

Improvements

Autograd frontend

• Support GradientEdge as output for torch.autograd.grad (#127766)
• torch.autograd.graph.increment_version accept List[Tensor] (#132652)
• Hooks registered via torch.autograd.graph.Node.register_hook during Node execution are run (#134728)

Compostability

Custom ops:

• Improvements to torch.library.custom_op
  • Supported calling from a multithreaded context (#128547)
  • Improved overhead when autograd is unnecessary (#127976)
  • Made it compatible with from __future__ import annotations (#128809)
  • Supported string default values in schema (#129179)
  • Added better suggestions for unsupported types (#129417)
  • Added mutates_args=”unknown” for if you didn’t know the alias information of your custom op (#129614)
  • Added ability to add default values for device types (#129792)
  • Added ability to construct factory functions (#129978)
  • Added ability to temporarily disable kernels (#130190, #130406)
  • [docs] Redirect custom ops landing page to the correct place (#129177)
  • Prevented Dynamo from peeking into torch.library.{custom_op, register_kernel} (#133125)
  • Improve aliasing error message (#134688)
• Miscellaneous
  • register_flop_formula now actually does something for custom ops (#131777)
  • Improved torch.library.opcheck docs. (#134692)

Dynamic shapes:

• User facing features offering more control over dynamism
  • [dynamic] config to disable duck sizing (#129804)
  • Trigger dynamism on stride changes (#130232)
  • Add mark_unbacked (#128638)
• Unbacked SymInt support
  • Fix set_unbacked_bindings when list of Tensors is returned (#133585)
  • Compute and do renamings even when ignoring fresh unbacked symbols (#134407)
  • Also preserve unbacked SymInts when partitioning as backward inputs (#128338)
  • Improve unbacked reasoning involving has internal overlap (#128332)
  • Make are_strides_like_channels_last size oblivious (#129677)
  • Correctly put mark_unbacked symbols in shape_env_to_source_to_symbol_cache (#129869)
  • Don't attempt to compute hints for unbacked expressions (#132060)
• Export related bug fixes
  • Fix ConstraintViolationError exception string when exprs are int (#129271)
  • suggested fix for data-dependent error (#125378)
  • carry cond in data-dependent error (#131932)
  • add src map to data-dependent errors (#132393)
  • check unsupported sympy functions for runtime asserts (#132457)
  • fix silly error when printing diff (#133345)
  • remove dead code for suggesting legacy dynamic shapes fixes (#133700)
• Improved symbolic reasoning, including performance improvements
  • Remove some implications from the static_eval pattern matcher (#128500)
  • Replace sympy Min/Max with reimplementations (#133319)
  • Fixed dynamic shape inference (#128807)
  • remove redundant upper bound check at runtime (#133627)
  • Remove dead expect_rational (#135105)
  • Stop updating hints (#129893)
  • Don't constrain range on the replacement for a symbol (#129907)
  • Make sym_node log more useful (#130436)
  • When translation validation is enabled, assert that hint is consistent (#130478)
  • Add trace_shape_events artifact tracing for ShapeEnv events (#130473)
  • FakeTensor cache SymInt support (#127596)
• Dynamic shapes improvements to specific operators
  • [PT2] Resolve PT2 compatility issue in slice and diff (#133740)
  • Use integer divison in arange length calculation when start/end/step are integral (#134296)
  • Remove unnecessary expect_true in split_with_sizes (#133439)

Decompositions, FakeTensor and meta tensors

Operator decompositions, FakeTensors and meta tensors are used to trace out a graph in torch.compile and torch.export. They received several improvements:

Decompositions:

• Fixes to existing decompositions:
  • rot90 (#129097)
  • aten.slice_scatter (#123744)
  • bucketize (#133652)
  • torch.istft (#135234)
  • torch.exp (#129154)
  • aten._to_copy (#130381)
  • aten.masked_fill_ (#127871)
  • const_pad_nd (#132679)
• New operator decompositions:
  • aten.channel_shuffle (#118775)
  • aten.nll_loss2d (#133534)
  • aten.reflection_pad{1,2,3}d_backward (#130299)
  • aten._unsafe_index_put (#133365)

Meta tensors:

• Fixes to existing meta tensor op implementation
• aten._scaled_mm (#129521)
• _convert_weight_to_int4pack (#130707)
• New meta tensor op impls
  • fused_adamw (#133728)
  • poisson (#134103)

Misc fixes:

• Infer prim tags from equivalent aten ones (#130367)
• Fix dim order calculation for channels_last in decomps (#131366)
• Allow cross-device copies for cpu scalars in refs (#135140)
• Update fake tensor error checks for bool tensor subtraction (#128492)

Cpp frontend

• Add padding_side to pad_sequence with "left" and "right" options ("right" as default) (#131884)
• Add out variants to avg_pool1d and adaptive_avg_pool1d (#135051)

Cuda

• [BE] Improve CUDA UpSample error message (#131252)
• Reduce number of guards introduced by check_cudnn_tensor_shapes when cudnn version is higher enough (#132384)
• [CUDA]: Add frexp CUDA bfloat16 support (#133313)
• Allow torch.cuda.memory.mem_get_info to take a device str argument with an unspecified device index. (#132616)
• Change index_put on GPU to accept FP8 inputs (#128758)

Distributed

Activation Checkpointing (AC)

• Added kwargs to composable AC API to enable full capabilities (#128516)
• Made ActivationWrapper an abstract class (#129808)

c10d

• Applied somaxconn and enabled TCP_NODELAY to TCPStoreLibUvBackend (#128739)
• Improved connect and retry logic for TCPStore (#129261)
• Added pings to verify network connectivity on connect for TCPStore (#129985)
• Made new_group eager when used with comm_split (#129284)
• Exposed running handlers from Python for control plane (#130149)
• Added new control plane handler (#129712)
• Added a new Pytorch API split_group to create a process group (#130507)
• Add bfloat16 support for NAN check (#131131)
• Enabled custom work registration from python in the Functional Collective (#130354)
• Not removed collective ops in dce since they have side-effect in the Functional Collective (#131023)
• Used float tensor for ProcessGroupNCCL barrier all-reduce (#132701)
• Set a shorter heartbeat detect timeout to avoid race with ProcessGroupNCCL watchdog timeout (#133028)
• Made it not call ncclCommAbort if comm is not initialized (#133630)
• Made it not broadcast uniqueId during a split (#133962)
• Reconciled barrier and NaN checker (#134707)
• Releases gil lock during eager init (#134779)
• Improved logic to infer device for barrier (#134617)

DeviceMesh

• Added supports for non-continuous slicing (#132310)

Dtensor

• Moved DTensor to public namespace (#134203)
• Made slice_backward to use op strategy (#130287)
• Improved from_local API with run_check (#130289)
• Added a few dunder methods to pointwise ops (#130754)
• Added naive support for nn.init.orthogonal_ (#132104)
• Added support for custom op registration (#131108)
• Added more foreach ops to supported sharding prop list (#132066)
• Added naive replicate strategy for more diagonal ops (#132201)
• Rewrote redistribute algorithm for multi-dim mesh (#131210)
• Added missing all to public modules (#133305)
• Made DTensor sharding propagation for scaled_dot_product_efficient_attention and scaled_dot_product_flash_attention more conservatively cached (#134146)
• Extended implicit replication to replicate DTensor for foreach ops so model doesn't have to be fully tp-ed when using 2D (#134551)
• Added gradient scaler for DTensor (#132816)

DistributedStateDict (DSD)

• Kept 'exp_avg' as DTensor after torch.distributed.checkpoint.state_dict.set_optimizer_state_dict (#128004)
• Correctly handled shared parameters for optimizer state_dict (#128685)

FullyShardedDataParallel (FSDP)

• Integrated device agnostic APIs in FSDP library (#134337)
• Made clip_grad_norm_ norm compute order deterministic (#134673)
• Casted input args with dataclass(frozen=True) (#135067)
• Avoided GPU syncs by reusing Pre-allocated Zero Tensor (#128069)

fully_shard (FSDP2)

• Included module FQN in FSDPParamGroup record_functions (#128624)
• Added APIs for explicit fwd/bwd prefetching (#128884)
• Added set_post_optim_event (#128975)
• Ran reduce-scatter copy-in in default stream (#129721)
• Relaxed contract to allow Sequence[nn.Module] (#127773) (#130947)
• Allowed List[nn.Module] as arg (#130949)
• Preserved fsdp.set_ op through lowering (#130786)
• Added set_reduce_scatter_divide_factor (#129286)
• Added hpu device to _get_remote_device_str (#132120)
• Added repr to FSDPParamGroup and FSDPParam (#132350)
• Added missing event wait (for future) in FSDP2 (#132568)
• Let fsdp.set_ convey to functionalization that it mutates storage (#132322)
• Enabled autoselect default device in FSDP construction. (#127609)
• Reset FSDPParam.sharded_param in lazy_init (#132954)
• Enabled HSDP + TP in FSDP2 (#133335)
• Added eager fast-path for fp32->bf16 param cast (#133369)
• Kept DTensor params for replicate and fully_shard (#133059)

TorchElastic

• Shared TCPStore by default when using c10d_rendezvous_backend (#128096)
• Used wait instead of get for store barrier (#130148)
• Added missing rank tracing support in the barrier inside TorchElastic (#132818)
• Made torch elastic not have to realize TCPStore backend type and rely on c10d to decide which backend to use (#134882)
• No signal handling when off the main thread (#135088)
• Supported local_addr across all rendezvous implementations (#135262)
• Created processes in parallel in mp.start_processes for forkserver (#134629)

TensorParallel(TP)

• Improve SequenceParallel and its documentation (#131346)

Pipelining

• Supported separate dw_runner for PipelineStage (#128983)
• Supported arbitrary stage ordering on ranks (#128976)
• Supported W action for schedules (#129233)
• Added to/from CSV format and improved repr (#129264)
• Implemented flexible PP schedule (#129597)
• Reordered _Action from F1_1 to 1F1 (#129786)
• Added forward only schedule (#132177)
• Added schedule unshard/reshard pass (#129810)
• Added schedule send/recv pass (#130378)
• Added zb1p schedule (#130210)
• Added get_schedule_class util (#132768)
• Added pytorch-native input/weight grad split (#132691)
• Added ZeroBubble schedule (#133467)
• Unblocked zero bubble composability with DP (#134052)

Dynamo

• More tracing support: (non-exhaustive list) -
  • • Weakref objects (#128533) -
  • Some set methods (e.g. discard (#133317), intersection (#130672), remove (#132943))
  • Support for proxying frozen dataclasses (#134846)
  • inspect.signature.bind (#132330) and inspect.signature.Parameter attribute access (#134636)
  • random.Random objects (#133725)
  • User-defined method descriptors (#130159)
  • set on KeysView (#131389)
  • dict conversion of objects derived from MutableMapping (#131367), constrained subclasses of dict and OrderedDict (#132558)
  • __contains__ on __dict__ of user defined classes (#131378)
  • abc.MutableMapping.get (#132363)
  • frozenset (#134563)
  • reading attributes from pybind objects (#134630)
  • classes with custom __new__ (#132977), object.__new__ (#133746)
  • torch.cuda.device (#133385)
  • id(Parameter) (#130100)
  • str on UserDefinedObjectVariable (#130506)
  • out-variant custom ops that return None (#129078)
  • autograd.Function mark_non_differentiable (#134087), ctx.set_materialize_grads (#133978)
  • Better support for HuggingFace ModelOutput class (#127780)
• mark_static can now be used on nn.Modules to make int attributes static (instead of automatic dynamic) (#134713)
• Recursively skip frames when Dynamo cache limit is hit (#135144)
• Context hints for backend compilers (#132860)
• Add queue_callback() support (#126366)
• Suppress guards generated by empty_strided in ir_node_to_tensor (#130431)

Export

• Effect token support for TorchBind calls (#128397)
• Pretty printing for unflattener (#128617)
• Decomposition support for export_for_training (#128077, #129249, #134801)
• Add CSE and SymInt compute optimization to export graphs (#130380)
• Check node schema for side-effectful ops when DCE-ing (#130552)
• Experimental joint-graph export API (#128847)
• Dynamic shapes serialization (#134718)
• Suggested fixes for data-dependent errors in non-strict mode (#125378)
• Kill forced specializations, and prefer runtime asserts for dynamic shapes (#130775, #132698)
• Fully support extension operators in de/serialization (#130851)
• Fully handle preserved_ops with run_decompositions() (#130970, #131075)
• Add torch.amp.autocast_mode as a higher-order-op subgraph (#132677)
• Make ExportedProgram.validate() public-facing (#132777)
• Allow string outputs for ExportedPrograms (#132808)
• Better user error messages for dynamic_shapes mismatches (#132982)
• Add a graph & node-level “custom” metadata field (#131912)
• Add a getitem deduplication pass (#133618, #134830)
• Make move_to_device_pass public-facing (#134263)
• Make while_loop higher-order-op public-facing (#128562)
• TORCHEXPORT_EXTENDED_DEBUG_CURRENT_LOC=1 flag for line-by-line debugging in non-strict mode (#134298)
• Support aten.full.default and aten.full_like.default (#130639)
• Handle python list append, list add, aten.to.dtype + mutation_op pattern for TSConverter (#132529)
• Quantized ops to standard ops pass. (#133026)
• Add InterpreterModule to trace_rules (#132949)
• Add tracing_mode support for TorchBind (#129586)
• Inline True branch for torch.cond when predicate is a Python constant (#130493)
• Support Unbacked SymBool inputs for torch.cond (#133589)
• Support set_grad_enabled higher-order-op in dynamo to enable re-tracing (#134281)

ForEach Frontend

• Increase parity for dtype support for _foreach_sigmoid and _foreach_lgamma (#134253, #134344)

Fx

• Use to_dtype node and const folding in convert fp32 to fp16 fx pass (#127829)
• Add decomposition_table as an arg to get_isolated_graphmodule (#130886)
• Support meta["val"] that is a dict, for triton kernels and for the partitioner (#132466)
• Allow SymInt input for torch.fx reinplace pass (#133178)
• Add max_acc_splits (#133041, #133724)
• Update source matcher to use torch_fn (#133642)
• Set maximum warning count during fx.Graph.lint (#135069)
• Graph Printing:
  • Change colored logging to only be turned on if printing to interactive terminal (#128874)
  • Print float with full precision, don't truncate (#130027)
  • Save DOT file of graph instead of SVG for GraphTranformObserver (#128634)
  • Add normalize_args constructor argument to FxGraphDrawer (#130348)
  • Show size/strides for all tensors in python_code(verbose=True) (#132192)
• Fix py codegen to delete values that don't have any users (#131028)
• Propagate sparsity in fx graph (#131920)

Inductor

• Inductor windows support improvements (#134772, #133921, #131767, #131980, #132025, #132326, #132533, #132491, #132630, #132848, #132841, #132973, #133184, #134033, #134229, #134358, #134348, #134397, #134402, #134400, #134401, #134419, #134420, #134394, #134424, #134426, #134427, #134221, #132387, #132394, #132571, #134365)
• Autoheuristic improvements (#130304, #133608, #131610, #132685, #131615, #131714, #131710)
• ROCm Support: enable dynamic shapes for CK backend (#133285)
• ppc64le: VSX Support for Inductor (#132746)
• Add an option to exclude CPU overheads using do_bench_using_profiling in TORCHINDUCTOR_PROFILE (#133523)
• Add thread blocking config TORCHINDUCTOR_CPP_GEMM_THREAD_FACTORS (#132730)
• FlexAttention improvements (#133019, #132015, #133159, #134065, #134351, #133836, #133664, #132157, #131559, #132547, #131404, #131552, #130904, #134055)
• Make config.autotune_remote_cache be a three-way option (#132285)
• Add config option to force higher-dimensional tiling (#132937)
• Add torch.save() for individual intermediate tensor (#133871)
• Add inductor config: masked_vec (#134566)
• Add dynamic shapes for combo_kenel/foreach_kernel (#134477)
• Add Inductor config for default stride behavior (#135238)
• Optionally allow padding on non-GPU devices (#135280)
• Make conv template work with dynamic stride/padding (#132938)
• Adds a print_readable function to the unflattener for better debug output, improving readability of the module structure. (#128617)
• Adds logging for static input indices in CUDAGraphs to aid debugging. (#132726)
• Improves dispatch logic for vectorized instruction sets in CPU code. (#128320)
• Adds a shape property to intermediate representation nodes for better shape inference. (#127818)
• Supports comprehensive padding for tensor operations, enhancing flexibility in padding schemes. (#128555)
• Reduces superfluous mask handling in Triton code generation. (#128518)
• Improves fusion logs by making it easier to attribute nodes to the aten graph, aiding in debugging and performance tuning. (#127159)
• Ensures mixed_mm operations are only enabled when casting from a lower bitwidth type to a higher one, avoiding unnecessary type conversions. (#128899)
• Enables multiple workers to function correctly if the method being used is subprocess. (#129002)
• Adopts future annotations for better type hinting and maintainability in the inductor's scheduler and IR. (#128892)
• Moves auto-tuning for Triton kernels into a separate block for better performance isolation. (#129057)
• Improves convolution dilation by adding a size_hint to optimize memory allocation. (#129631)
• Switches the Halide code cache to the new cpp_builder framework for better compilation support. (#129441)
• Refactors GEMM template to pass weight data type explicitly. (#129221)
• Introduces helper functions to convert score_mod into block_mask in FlexAttention. (#129909)
• Updates HalideCodeCache to use the new cpp_builder framework for improved functionality. (#130146)
• Limits functions in foreach operations when dependent on multiple subkernels to improve consistency. (#130046)
• Introduces improved methods for generating runtime checks for symbolic dimensions. (#130220)
• Adds a check to verify if the FX graph returned by aot_compile is a tuple. (#129824)
• Adds support for passing a module map to the Triton make_ir API for better modularity. (#134774)
• Removes VecChecker and implements a fallback for non-supported vector operations with a scalar loop. (#134569)
• Generates a reindexer for each epilogue_node in C++ inductor operations. (#134984)
• Adds a padding_value for boundary-checked loads in FlexAttention, improving load operations in corner cases. (#134573)
• Improves the way argument names are used as keys in constant and signature dictionaries for better consistency. (#135170)
• Removes "spawn" as an option for parallel compilation to avoid issues with certain platforms. (#130746)
• Introduces a heuristic to determine whether padding should be applied before matrix multiplication operations. (#128643)
• Introduces inductor-specific counters within the FX graph cache for better debugging and performance tracking. (#130635)
• Adds a counter for num_matches_for_scatter on constant tensors to the cached metrics. (#130843)
• The issue Lift inductor lowerings for jagged <-> padded dense kernels introduces lowerings for conversions between jagged and padded dense tensors, supporting operations like _jagged_to_padded_dense_forward and _padded_dense_to_jagged_forward. This improves PyTorch's ability to handle irregular tensor structures efficiently, particularly in tasks like natural language processing where padding is common.(#125968)
• The issue Add BackendFeature gating introduces a mechanism to gate backend features in PyTorch's inductor based on the availability of specific backend capabilities. This ensures that operations requiring specialized backend support are only enabled when those features are present, improving compatibility and stability. (#128266)
• The issue Support additional lifted constants supplied to const folding enhances AOTInductor's constant folding by allowing additional lifted constants from graphs to be included. This update improves compatibility when using lifted graphs in constant folding processes, particularly with the split_const_gm function. (#130743)
• The issue Compute q_num_blocks from kv_num_blocks if q_num_blocks is not passed in updates the logic in flex_attention to compute the q_num_blocks from kv_num_blocks when q_num_blocks is not explicitly provided, ensuring consistent behavior across different input configurations. (#130809)
• The issue Remove static param counting if inlining NN modules eliminates the need for static parameter counting when inlining neural network modules in the inductor backend. This simplifies the logic for handling parameters and improves performance by skipping unnecessary counting. (#130503)
• The issue Use get_reduction_combine_fn for reduction ops updates the Halide backend by introducing a function that determines how reductions are combined during computation, improving the clarity and efficiency of reduction operations in the backend. (#130212)
• The issue Use 0D scalar inputs/outputs allows the Halide backend to handle 0-dimensional scalar inputs and outputs, fixing issues related to scalar operations in the Halide integration with PyTorch. (#130129)
• The issue Change the schema of QLinear Binary modifies the schema to better support the corresponding GEMM templates, making it easier to handle autotuning by reordering tensor inputs. (#129049)
• The issue Dimension-based indexing for the halide-backend changes the indexing in the generated Halide code to be dimension-based rather than 1D-based, which resolves multiple bugs and performance issues related to indexing in PyTorch's Halide backend. (#129026)
• The issue Use _unsafe_masked_index in masked_scatter decomposition updates the masked_scatter operation by using the _unsafe_masked_index function in its decomposition, improving performance and reliability.(#123667)
• The issue Only autotune at compile time when enabled via config ensures that autotuning in AOTInductor only occurs when explicitly enabled via configuration, preventing unintended autotuning during execution unless specified. (#129413)
• The issue Adjust efficient_conv_bn_eval_graph for inlining modifies the inlining of the efficient_conv_bn_eval_graph function to improve execution during evaluation in PyTorch, particularly for built-in neural network modules. (#128878)
• The issue Make config.fx_graph_remote_cache be a three-value switch introduces a configuration option allowing fx_graph_remote_cache to be set as True, False, or None, where None disables it for OSS (open-source software) but enables internal configurations for JK. (#128628)
• Enables GraphTransformObserver for inductor backend.(#127962)

mps

• [MPS] Add Metal implementation of exp op (#128421)
• [MPS] Add lu_factor in MPS (#99269)
• [MPS] Add support for autocast in MPS (#99272)
• [MPS] Add support for autocast in MPS (#99272)
• [MPS] Enable MPS mm from macOS >= 14.4 (#133494)
• [MPS] Add support for autocast in MPS (#99272)
• [MPS] Add int4mm weight packing mps kernel, and improved int4mm shader (#128965)
• [MPS] Fast math env var (#129007)
• [MPS] Check and error message for no support for conv with output_channels > 2^16 (#129484)
• [MPS] Add tensor_lr overloads to fused adam & adamw (#129451)
• [MPS] Add SDPA implentation (#131362)
• [MPS] Add native implementation for shift ops (#131813)
• [MPS] Add native strided API for MPSNDArray starting with macOS 15 (#128393)

nn

• Made nn.Module state_dict load_state_dict pre-hook and state_dict post-hook public (#131690)
• Add deterministic support in nn.functional.interpolate for XPU (#129864)
• Add batching rule for sdpa_math, sdpa_efficient_attention forward, cudnn, and flash attention (#133964)
• Support nn.Module.mtia() (#131499)

Optim

• Move fused optimizers’ param's device check to later in _init_group to allow better error checking in more cases (#131153)
• Update typing and 1-element check for tensor lr support on all optimizers (#131065)

Optimizer Frontend

• Fix fake_tensor w/ non-view tensor (#132050)
• [BE][optim] Make pyright recognize exported symbols (#135043)

Profiler

• [Memory Snapshot] Move user_defined annotations to Native Caching Allocator (#130964)
• [Memory Snapshot] Add recordAnnotations to capture record_function annotations (#129072)

Python Frontend

• Improve error message for weights_only torch.load (#129705)
• Add new blocklist for weights_only load to prevent some modules from being allowlisted (#131259)
• Make PyTorch argparser understand python’s “complex” type (#129580)
• Allow to register “fallback” to torch.library (#131707)
• Move module_tracker to general logging system when reporting confused hierarchy (#134467)
• Make torch.serialization.set_default_mmap_options usable as a context manager (#134371)

Quantization

PT2E quantization

• Support set_module_name_qconfig in X86InductorQuantizer (#126044)
• Enable PT2E symmetric dynamic quantization in metadata porting (#124615)
• Fix add annotation with constant (#132092)
• Fix Maxpool2d share quantization params (#132704)
• Fix getattr for quantizing constants (#132705)
• Use returned model from Quantizer.transform_for_annotation in prepare_pt2e (#132893)

Observers

• Fix edge cases for HistogramObserver (#129387)
• Corner-case fix for upscale_histogram in the new HistogramObserver (#130316)

Export IR Migration

• Fix batch norm pattern match in quantization (#134157)
• Fix getitem not exist (#134259)
• Fix tests for quantized_decomposed.quantize_per_tensor decomposition (#134525)

Others

• Enable torch.empty for float8 dtypes on cpu in deterministic mode (#128744)
• Fixing equalize with three things and improving functionality (#124632)
• Fix the warning for cat operators with same qparams (#133999)

Releng

• Update NCCL to 2.21.5 (#124014)
• Update cuSPARSELt to v0.6.2 (#134022)
• Update CI/CD to CUDA 12.4.1 (#132202, #125944)
• [ROCm] upgrade CI/CD to 6.2, triton-rocm improvements. (#132875, #133238, #128525, #131637, #129361, #129480, #128873, #127947, #129094)
• Migrate conda, manywheel, libtorch Docker images build from pytorch/builder to pytorch/pytorch. (#129022, #132410, #133699, #133709)
• Migrate nightly CD builds to ephemeral runners (#134469, #134380, #134367, #134463, #134473)

Infrastructure

• Remove usages of Rockset as storage in favor of AWS (#129503, #129594, #129544, #130153, #130168)
• Migrated pytorch/pytorch’s CI to run on hardware owned by the Linux Foundation (#129246, #129462, #129746, #133225, #133232, #133320, #133124, #133457, #134231, #134796, #134800, #129612, #131325, #131188, #128969, #128985, #129679, #129977, #131955, #132870, #131472, #134128)
• Upgraded the CI Linux runners to use the Amazon Linux 2023 AMI (#128619, #132918, #131792, #134355, #131246, #131514, #131485, #131677, #131963, #131771, #133036, #133352, #133469, #133355, #133641, #134116, #131821, #131250)

XPU

Intel GPU Backend for Inductor

• Generalize GPU type for Intel GPU and CUDA for Inductor Triton backend (#132740)
• Initialize AOTInductor support for Intel GPU by supporting store SPIR-V binary file output from Intel Triton. (#130849)
• Handle device_put op in constant folding. (#130824)
• Support reduction split. (#129120)
• Add new prop to _XpuDevicePropertie for triton gemm optimization (#131738)

Intel GPU ATen Operation

• Allow XPU device in copy, cdist, index_put_impl (#130088)
• Customized XPU behavior in indexing, group norm (#134453)
• Enable codegen for Intel GPU (#130082)
• Enable Dispatch Stub support for Intel GPU (#130019)
• Enhance the stability of the complex divide code (#134647)
• Add support for XPU accumulate type (#128579)

Intel GPU Runtime and Generalization

• Device guard codegen for XPU (#133980)
• Refactor caching device allocator utils (#130923)
• Refactor cached tensor more generic (#129359)
• Refine the logic of device construction when only device index is given (#129119)
• XPUHooksInterface inherits from AcceleratorHooksInterface (#129463)
• Add XPU memory-related APIs (#129919)
• Add xpu to getAccelerator (#129205)

Nested-Tensor Frontend

• Backward support for unbind() with NJT (#128032)
• Backward support for cat() with NJT (#132076)
• Backward support for chunk() on non-batch, non-jagged dimensions for NJTs(#132193)
• Support linear backward for NJT with dim > 3 (#129393)
• Accept min / max sequence length in nested_tensor_from_jagged() constructor (#130175)
• Support sum operator along the jagged dimension for NJTs (#130425)
• Support mean operator along the jagged dimension for NJTs (#131132)
• Support permute() for NJT (#135336)
• Support dtype conversions for NJT (#134164)
• Support for copy_() when shape is identical for NJTs (#132193)
• Implement 2D version of masked_select() for NJTs (#133889)

cuDNN

• [cuDNN][64-bit indexing] cuDNN v9.3+ supports non-batch-splittable convolutions with > 2**31 elements (#134890)
• cuDNN now supports convolutions with spatial dimensions that require 64-bit indexing. Previously this would fallback to a native im2col or vol2col style implementation that was very memory inefficient and lead to OOMs.

Sparse Frontend

• Add Half for sparse.mm reduce on CPU (#133672)
• Add partial support for COO/CSR/CSC/BSR/BSC representation in traced graph (#132690, #134037, #133371, #129983)

ONNX

• Lazy-import onnxruntime (#134662)
• Add upsample trilinear to skip decomp (#128259)
• Add onnx::Gelu support for version 20 (#128773)

ROCm

• Add warpSize to torch.cuda.get_device_properties (#128449)

Bug fixes

Autograd frontend

• Fix case where saved tensors are incorrectly shared version counter (#128545)
• Fix thread safety of torch.autograd.graph.register_multi_grad_hook (#132055)
• Fix PyObject reference counting of torch.autograd.graph.saved_tensors_hooks (#131700)
• Update torch.autograd.graph.saved_tensors_hooks to not detach when unpacking tensors that do not require grad (#127959)
• Fix device propagation for torch.utils.checkpoint (#128671)

Compostability

• Fix interaction between torch.conj() tensor subclasses and scalar tensors (#131482)

• Better complex number support for aliasing prim operators (#132699)

• AOTDispatcher is a component in the torch.compile stack responsible for capturing a graph of normalized/functionalized ATen IR as well as capture the backward. A few bugfixes this release:

  • Properly bump version counter on input mutations in inference graphs (#131665)
  • Ensure that graph input mutations from the backward remain in the backward graph after partitioning (#129130)

• Dynamic Shapes

  • Do not generate -1* in SymPy expressions when canonicalising (#128411)
  • Add FloatPow in the symbolic shape guard closure (#129857)
  • Add FloatTrueDiv and ToFloat to SYMPY_INTERP (#128418)
  • Fix symbolic nested int printing (#131916)

Cuda

• [CUDA][Pooling] Fix 64-bit indexing in avg_pool_2d backward attempt 2 (#129818)
• fixes (#124582, #128483)
• Add threadfence to 2-stage reduction for correct writes visibility (#128455)
• [cuDNN][SDPA] Limit cuDNN SDPA head-dim to 128 (#130494)
• expose host_emptyCache to python, fix a bug in freeing cudaHostRegist… (#134919)

Distributed

Distributed checkpoint

• Update _all_gather_keys utils function to derive world size based on input process group. (#135045)
• Fix non-tensor objects not being loaded correctly during checkpoint loads. (#129398)
• Fix meta tensors not being loaded during checkpoint loads. (#133256)

c10d

• Fixed commSplit bug by having every rank being called even though it is no-color (#128459)
• Made sure current device is correct in torch.distributed.barrier()'s streamSynchronize (#129908)
• Added flag to control which rank should perform NaN check (#134345)
• Set correct device to CUDA guards (#134357)
• Fixed p2p group commsplit (#128803)
• Fixed corrupt log due to uint_8 printing as char (#130184)
• Fixed an issue where ENABLE_INTRA_NODE_COMM=1 + multiple process groups leads to failure (#130269)
• Fixed an issue where input check fails when running all-reduce on sub groups (#130492)
• Fixed some issues in two-shot all-reduce (#131244)
• Fixed split_group usage when there is a single rank (#131824)
• Fixed remote address in the TCPStore (#131773) (#131913)
• Fixed a getenv segfault due to a race in getting nccl version (#133744)
• Changed collective to take in a list of tensors so it work fully for all collectives (#135049)

CPU profiler for distributed

• Fixed input/output dimension overflow (#134360)

DSD

• Disabled 2D state_dict temporarily before the feature is fully ready (#129519)

DeviceMesh

• Fixed replicate with DeviceMesh initialization (#133024)

DTensor

• Fixed foreach_norm when ord is 2 (#130753)
• Fixed _MaskPartial when multiple embeddings coexist (#131264)
• Fixed the bug where Sharding Prop cache was wrongly shared among multi-threaded ProcessGroup in tests (#134509)

FSDP2

• Fixed unshard without lazy init (#129241)

TensorParallel(TP)

• Fixed loss_parallel with BF16 logits (#130550)

RPC

• Fixed Distributed EventList usage (#132448)

Dynamo

• Do not run default saved tensor hooks during tracing (#123196)
• Make .data mutations invisible to autograd (#131403)
• Skip frame if TorchDispatchMode is enabled (#131828)
• Ensure nn_module_stack is the same when inlining inbuilt nn.Modules (#128295)
• Compiled autograd bug fixes:
  • use same graph node names as AOTDispatcher (#133148)
  • match eager behavior for post_acc_grad_hooks (#134205) and ctx.saved_variables (#134286)
  • error instead of deadlock on reentrant autograd (#134530)
  • match eager behavior for inplace detached activations (#134186)
• OptimizedModule.training flag now mirrors the wrapped module. (#131546)
• Fixes to exception handling (#131795, #131801, #132425)
• Fix indexing and slicing of ranges (#128567)
• Create list slices as new local objects (#132912)
• Handle infinite map/zip and return map/zip instead of a tuple (#135074)
• Guard correctly on tensor subclasses (#130780)
• Register all entrypoint backends on first attempt to list_backends (#132546)
• Cache attr_proxy for nn_module attribute to fix guard check failure (#130280)

Export

• Fix unflattener for unused inputs/outputs + preserve_module_call_signature (#128260)
• Preserve .requires_grad on FakeTensors in graph metadata (#128656)
• Handle deduplicated SymInt compute nodes when unflattening (#129153)
• Handle mutated FakeScriptObject state when reused in aot_export (#128844)
• Fix FakeMode mismatch for joint-graph export API (#129421)
• Don’t DCE side-effectful custom ops (#129680, #130970)
• Fix FakeMode detection for 0-input graphs (#129928, #131995)
• Allow kwargs for training IR (#130553)
• Fix constants & non-persistent buffers for training IR (#130864)
• Fix training IR for 0-input graphs (#130990, #133031)
• Improve output node metadata preservation for strict mode (#131706)
• Preserve autograd state when lowering to inference IR (#131988)
• Preserve source_fn_stack for training IR (#132033)
• Preserve .requires_grad for unflattened parameters (#134353)
• Preserve aten::to for training IR (#134622)
• Error out when exporting ScriptModule (#135302)
• Fix warning for ComposeImplicitAutograd custom ops in pre-dispatch (#130623)
• fix node.users when inlining higher-order-ops (#133144)
• Improve logging for TSConverter (#132082)
• Fix serialization of OpOverload with SymInt outputs (#132126)
• Construct empty graph when there's no tensor computation (#129541)
• Fix inline_inbuilt_nn_modules + export (#133731)

ForEach Frontend

• Perform reciprocal optimization with foreach_div (#128433)

Fx

• Fix index issues in torch.fx.interpreter (#129527)
• Recursively apply options to print_readable (#130268)
• Implement deepcopy for Proxy (#133706, #133470)
• Fix linearize(grad(...)) call by moving DCE (#133364)
• Respect find_all setting in sequential mode of minimizer (#134339)

Jit

• Validate that node TK_ASSIGN have field initialized (#127878)
• Fixes for LLVM 18 and ASAN (#130661, #133623, #134572)

Linalg Frontend

• Fix input checks for linalg.lstsq (#130612)

mps

• [MPS] Make erfinv compilable for bfloat16 (#128375)
• [MPS] Fix Clamp correctness with type promotion (#130226)
• [MPS] Fix torch.[all|any] for 5+D tensors (#130542)
• [MPS] Store philox counter as part of the RNG state (#130662)
• [MPS] Min and max NaN propagation fix in MPS backend (#130445)
• [MPS] Correct nonzero warning and fix the test (#132127)
• [MPS] Fix SDP training (#134719)
• [MPS] Fix bachnorm_2d for channels last (#134618)
• [MPS] Fix NaNs in triu op (#128575)
• [MPS] Parameterize group_size in int4_mm test, fix int4mm for group_size > 128 (#129628)
• [MPS] Fix crash when running PyTorch with Metal API validation turned on (#130377)
• [MPS] LSTM backward kernel workaround on MacOS 14.4+ to fix correctness (#130038)
• [MPS] Fix masked_fill_ in non_contiguous cases (#131957)
• [MPS] Fix relu for 0-element input case (#133191)
• [MPS] Add workaround for nonzero with large/complex inputs (#126188)
• [MPS] Enable batch matmul for sizes > 2**32 when tensor can be split along batch axis (#133430)

nn

• Use newer toAccumulateType signature in Normalization.cpp (#134540)

Optim

• Fix accidental change of step signature which affected GradScaler interaction (#129933)
• Remove an introduced Host & Device Sync In LR Scheduler (#133663)
• Fall back to slower foreach_add in optim.step() when is_compiling() to avoid untracked tensor during graph tracing (#130909)

Optimizer Frontend

• [3/N] Enable clang-tidy on torch/csrc/inductor (#132101)

Profiler

• [Profiler] Fix profiler_kineto Clang errors (#128464)
• [Profiler] Add Rank to NCCL Debug Info (#129528)
• [Profiler] Directly use end_ns to create the FunctionEvent instead of using start_ns + duration_ns in pytorch profiler post processing for checking parent-child precisely (#129554)
• [Profiler] exclude gpu_user_annotation when accumulating cuda time total (#130733)
• [Memory Snapshot] Make recordAnnotations callback initialize lazily (#129242)
• [Memory Snapshot] Fix race on alloc_trace vector - S430480 (#130180)
• [Memory Snapshot] Stop duplicating annotations to all device_traces (#130315)
• [Profiler] Allow record_function kwargs to be non-string values (#134893)
• [Profiler] Fix CPU Annotation Overlapping with Python Events (#129599)

Python Frontend

• Fix Storage.filename to not track the filename when storage was mmap-ed with MAP_PRIVATE (#128725)
• Fix allowlisting of builtins for weights_only unpickler (#129244)
• Fix div with rounding_mode="floor" when division overflows (#129536)
• Fix warning when pickle.load torch.Storage (#130246)
• Fix dtype mismatch in lobpcg eigen solver (#132762)
• Prevent an unnecessary device -> host copy for CuPy arrays when not explicitly setting a device in torch.as_tensor. (#132595)
• Fix type promotion for torch.ldexp (#133519)
• Fix 0-dim tensor of complex or bool type for torch.aminmax. (#128404)
• Fix torch.prod vectorized path for bool (#128009)

Releng

• Fix MacOS double-loading of OpenMP runtime (#129473)
• Fix exposing statically linked libstdc++ CXX11 ABI symbols in Linux binaries (#137209)
• Release engineering tooling and CI fixes. Workflows, Trymerge, Bot Labeler, Mergebot (#128840, #129924, #129500, #128924, #129291, #128842, #129720, #129987, #130570, #132681, #133143, #133350, #133372, #133861, #131475, #134047, #134711, #134785, #133869)

XPU

• Fix xpu nightly wheel test failure (#130742)
• Fix xpu nightly wheel test env (#134395)
• Fix test_sgd_weight_decay_xpu accuracy error (#134744)
• Fix windows xpu build issue (#133845)
• Fix tensor print behavior for XPU (#130523)
• Fix overriding default CMAKE_CXX_FLAGS on Windows (#135093)
• Remove duplicate XPU switch case in DispatchStub (#132480)
• Keep zero check be compatible with different sympy versions (#130729)
• Align the appearance of device_put op in fx_graph generated for CUDA and XPU, which is exposed in the issue #130823 (#132479)
• Fix patch for old llvm package error for triton xpu (#134204)
• Fix tensor print behavior for XPU (#130523)
• Fix windows xpu build issue (#133845)
• Check compilation status before query cudnn version in conv (#135332)

Nested-Tensor Frontend

• Default to input tensor device for as_nested_tensor(t) (#130050)
• Fix SDPA backward for the special case of an NJT with ragged second batch dim and constant length (#128349)

cuDNN

• [CUDNN][SDPA] Fix unsupported trivial stride-1 transpose case (#134031)
• Minor bugfix for cuDNN SDPA

ONNX

• Fix onnx conversion scaled_dot_product_attention (#133314)
• Fix scaled_dot_product_attention with float scale (#135594)
• Add assertion nodes to ignoring list (#135591)

ROCm

• [ROCm] CUDA_VISIBLE_DEVICES fallback option for device_count (#129650)
• [ROCm] Return correct AMDSMI socket_power metric (#130331)
• [ROCm] Check supported archs before setting preferred blas backend to hipblasLT (#128753)

Performance

Cuda

• [pytorch][cuda] Generate kernels for 5x5 filters on depth wise convolution backward (#129609)
• [fp8 rowwise] Retune the tile heuristics to increase perf (#134781)

Distributed

CPU profiler for distributed

• Added API for Dynamic Activity Toggling for CPU profiler (#133353)

Dynamo

Compile time improvements

• Manually implement nn.Module.__getattr__ (#129315)
• Manually implement nn.Module._call_impl (#129285)
• Manually trace torch.nn.Module.parameters (#129583)
• Optimize guard for small tuples (#130400)
• Use dict tags to skip guards on immutable dict getitems (#130654)
• Reduce overhead for PolyfilledFunctionVariable.call_function (#134842)

Fx

• Speed up fx graph iteration by implementing it in C++ (#128288)
• Remove unnecessary get_implications calls (#128410)
• Implement a fast-path to FakeTensor detach (#131899)
• Optimize Node.__update_args_kwargs (#135076)
• Remove generators in map_aggregate (#135082)
• Bypass custom setattr in Node.__init__ (#135079)

Inductor

• [Compile] Add NEON implementation for bf16->fp32 cast (#134297)
• Remove dtype check/assert for reduction vectorization and support bool for min/max (#132473)
• Support masked vectorization for the tail_loop for INT8 datatype (#131155)
• Support vectorization for torch.argmax/min(float/int64_t)-> int64_t in inductor cpp backend (#131016)
• Optimize aten.cat calls of a repeated element (#132081)
• Fix mm pad regression - more conservative estimation of plannable inputs (#128909)
• Void fallback case for custom scan op lowering (#130936)
• Move bias add to gemm epilogue (#130675)
• Add B2B-GEMM performance tuning (#130778)
• Optimize arbitrary N in cpp packed gemm template (#130690)
• Improve cache blocking with CPU info in the cpp GEM template (#129348)
• Improve thread blocking heuristics in cpp gemm (#131024)
• Support k slicing for static shapes in cpp gemm (#130821)
• Apply compute/comm reordering passes to achieve overlap (#131614)
• Support pointwise intermediate nodes in B2B-GEMM (#131685)
• Add lowering for _scaled_mm that autotunes between ATen kernels and Triton kernels (#130422)
• Save and run post compilation steps within FXGraphCache (#132294)
• Add vectorization support for doubles in inductor cpp (#131886)
• Support vectorization for torch.any(bool) -> bool (#132472)
• Performance, precision, and dependency improvements to B2B-GEMM (#132354)
• Support use_libdevice_for_f64 for pointwise ops on XPU, align with CUDA. (#132739)
• Intel GPU Support: Support codegen empty_strided_xpu, align with #118255 (#126678)
• Move GPU_TYPE(The runtime avaliable gpu type, cuda or xpu) from (#132740)
• Support _check_triton_bf16_support on XPU. (#132748)
• Moves intermediary tensors which are constructed on the cpu to XPU when safe, align with CUDA. (#132843)
• Support masked vectorization for the tail_loop of the 2d tiles kernel (#130724)
• Improve large bs perf with better cache blocking in cpp gemm (#132729)
• Add auto-tuning for sparse semi-structured MM operator (#123742)
• Support vectorization for atomic add (#131314)
• Skip cudagraph if too many distinct sizes (#131387)
• Improve sort kernel performance (#131719)
• Add unbind_stack_to_cat_pass (#132542)
• support masked vectorization for the tail_loop (#126526)
• Update unbind_cat_to_view pass to include more complicated cases (#132831)
• Extend split_stack_to_cats when split and stack have different dims (#133060)
• Add unbind_stack_to_slices pass (#133420)
• Improve compile time regression from MemoryDep.normalize (#135070)
• Reduce memory alloc overhead by allocating local acc once per thread in cpp gemm (#135277)
• Enable dynamic M for k-slicing in cpp gemm (#133447)
• Speedup int8-to-float conversion on aarch64 (#132676)
• Optimizes cache key calculation by memoizing device data for predictable behavior. (#128366)
• Adjusts backward kernel block sizes for the FlexAttention module. (#128853)
• Updates template indexing to use broadcasting instead of masks in the FlexAttention module, optimizing performance. (#128938)
• Evaluates symbolic expressions during the loading of cached entries, preventing unnecessary computations during writes. (#128997)
• Adds support for BF16 matrix multiplication (micro-gemm) using Advanced Matrix eXtension (AMX) instructions available in Intel processors for performance improvements. (#127195)
• Optimizes memory use by avoiding the materialization of large sparse matrices during backward passes in conditional execution. (#129043)
• Enhances autotuning of FlexAttention by passing fake inputs for block sparse entries. (#129915)
• Reduces unnecessary tensor metadata in the AOTAutogradCache to improve performance. (#128583)
• Introduces a caching mechanism for precompilation functions to avoid redundant compilation. (#130350)
• Updates the loop order to occur post-fusion in inductor. (#126254)
• Ensures that icx built-in math libraries are preloaded in the compilation process. (#134870)
• Converts addmm to a decomposition operation instead of a fallback for better performance. (#134823)
• Adds support for vectorization in the tail loop for dynamic shapes in inductor. (#131745)
• Optimizes template_buffer usage by introducing a local accumulator when the buffer has multiple users. (#135081)
• Maximizes the use of available bits for BF16/FP16 vectorization in CPU operations. (#126502)
• Directly sets meta.val for efficient batch fusion during aten operations. (#135078)
• Improves performance by tuning the INT8 AMX WoQ micro-kernel for CPU. (#134832)
• Adjusts the tiling factor for lower precision data types (e.g., BF16, FP16) in inductor's C++ backend to optimize performance. (#133830)
• Enhances cache blocking mechanisms for dynamic matrix multiplications. (#131306)
• Addresses issues with loop splitting optimizations in the inductor backend. (#135303)
• Adds improved cache-blocking configurations for dynamic shapes in GEMM operations. (#133538)
• Optimizes tensor core usage during matrix multiplications in FlexAttention. (#135168)
• Refactors memory usage patterns in LoopBody to improve efficiency. (#135286)
• Introduces a fast path for extracting read/write information without requiring full tracing, improving performance. (#135377)
• Directly uses empty tensors with strides in CUDA graphs during copy operations for efficiency. (#130777)
• The issue Fix compile time regression by caching get_gpu_type addresses a significant compile time regression in inductor by caching the results of GPU type queries, reducing the need for repetitive calls to nvidia-smi. This leads to substantial performance improvements, particularly for jobs involving collective operations (#128363)
• The issue Introduce heuristic for mixed_mm on A100 adds a heuristic for mixed matrix multiplications, specifically for the A100 GPU, significantly improving performance in certain configurations by tuning the heuristic to outperform existing solutions. (#128232)
• The issue Improve codegen for ops.masked in triton enhances code generation for masked operations in the Triton backend, ensuring better performance and correctness for these operations within the PyTorch inductor. (#128054)
• The issue Enable shape_padding multiplier adjustment allows for the adjustment of the shape_padding_multiplier value in PT2 to improve query-per-second (QPS) performance based on specific configurations. (#128346)
• The issue Emit strided block pointer from ModularIndexing and FloorDiv improves the efficiency of indexing in multi-dimensional tensors by emitting block pointers instead of relying on division and modulo operations. This enhancement optimizes performance, especially for workloads involving strided tensor access patterns. (#127342)
• The issue Make tl.atomic_add relaxed changes the atomic addition operation in Triton to use relaxed memory ordering rather than acquire/release synchronization, improving performance without sacrificing correctness where strict memory synchronization is unnecessary. (#129133)
• The issue Linear time dead node elimination optimizes the inductor's dead code elimination by ensuring dead nodes are removed in linear time, improving execution efficiency in PyTorch's computation graphs. (#129082)
• The issue Add lowering and codegen for aten.sort implements lowering and code generation for aten.sort in PyTorch's inductor, improving the performance of sorting operations by optimizing them for Triton kernels under specific size thresholds. (#128458)
• The issue Fallback to eager if re-record too many times ensures that if a CUDAGraph re-records a function too frequently (more than cudagraph_max_recording), PyTorch falls back to the eager execution mode, preventing performance degradation from excess

Note truncated.

This release is meant to fix the following issues (regressions / silent correctness):

Breaking Changes:

• The pytorch/pytorch docker image now installs the PyTorch package through pip and has switch its conda installation from miniconda to miniforge (#134274)

Windows:

• Fix performance regression on Windows related to MKL static linking (#130619) (#130697)
• Fix error during loading on Windows: [WinError 126] The specified module could not be found. (#131662) (#130697)

MPS:

• Fix tensor.clamp produces wrong values (#130226)
• Fix Incorrect result from batch norm with sliced inputs (#133610)

ROCM:

• Fix for launching kernel invalid config error when calling embedding with large index (#130994)
• Added a check and a warning when attempting to use hipBLASLt on an unsupported architecture (#128753)
• Fix image corruption with Memory Efficient Attention when running HuggingFace Diffusers Stable Diffusion 3 pipeline (#133331)

Distributed:

• Fix FutureWarning when using torch.load internally (#130663)
• Fix FutureWarning when using torch.cuda.amp.autocast internally (#130660)

Torch.compile:

• Fix exception with torch compile when onnxruntime-training and deepspeed packages are installed. (#131194)
• Fix silent incorrectness with torch.library.custom_op with mutable inputs and torch.compile (#133452)
• Fix SIMD detection on Linux ARM (#129075)
• Do not use C++20 features in cpu_inducotr code (#130816)

Packaging:

• Fix for exposing statically linked libstdc++ CXX11 ABI symbols (#134494)
• Fix error while building pytorch from source due to not missing QNNPACK module (#131864)
• Make PyTorch buildable from source on PowerPC (#129736)
• Fix XPU extension building (#132847)

Other:

• Fix warning when using pickle on a nn.Module that contains tensor attributes (#130246)
• Fix NaNs return in MultiheadAttention when need_weights=False (#130014)
• Fix nested tensor MHA produces incorrect results (#130196)
• Fix error when using torch.utils.flop_counter.FlopCounterMode (#134467)

Tracked Regressions:

• The experimental remote caching feature for Inductor's autotuner (enabled via TORCHINDUCTOR_AUTOTUNE_REMOTE_CACHE) is known to still be broken in this release and actively worked on in main. Following Error is generated: redis.exceptions.DataError: Invalid input of type: 'dict'. Please use nightlies if you need this feature (reported and Fixed by PR: #134032)

Release tracker #132400 contains all relevant pull requests related to this release as well as links to related issues.

PyTorch 2.4 Release Notes

• Highlights
• Tracked Regressions
• Backward incompatible changes
• Deprecations
• New features
• Improvements
• Bug Fixes
• Performance
• Documentation
• Developers
• Security

Highlights

We are excited to announce the release of PyTorch® 2.4! PyTorch 2.4 adds support for the latest version of Python (3.12) for torch.compile. AOTInductor freezing gives developers running AOTInductor more performance based optimizations by allowing the serialization of MKLDNN weights. As well, a new default TCPStore server backend utilizing libuv has been introduced which should significantly reduce initialization times for users running large-scale jobs. Finally, a new Python Custom Operator API makes it easier than before to integrate custom kernels into PyTorch, especially for torch.compile.

This release is composed of 3661 commits and 475 contributors since PyTorch 2.3. We want to sincerely thank our dedicated community for your contributions. As always, we encourage you to try these out and report any issues as we improve 2.4. More information about how to get started with the PyTorch 2-series can be found at our Getting Started page.

<table> <tr> <td>Beta </td> <td>Prototype </td> <td>Performance Improvements </td> </tr> <tr> <td>Python 3.12 support for torch.compile </td> <td>FSDP2: DTensor-based per-parameter-sharding FSDP </td> <td>torch.compile optimizations for AWS Graviton (aarch64-linux) processors </td> </tr> <tr> <td>AOTInductor Freezing for CPU </td> <td>torch.distributed.pipelining, simplified pipeline parallelism </td> <td>BF16 symbolic shape optimization in TorchInductor </td> </tr> <tr> <td>New Higher-level Python Custom Operator API </td> <td>Intel GPU is available through source build </td> <td>Performance optimizations for GenAI projects utilizing CPU devices </td> </tr> <tr> <td>Switching TCPStore’s default server backend to libuv </td> <td> </td> <td> </td> </tr> <tr> <td> </td> <td> </td> <td> </td> </tr> </table>

*To see a full list of public feature submissions click here.

Tracked Regressions

Subproc exception with torch.compile and onnxruntime-training

There is a reported issue (#131070) when using torch.compile if onnxruntime-training lib is installed. The issue will be fixed (#131194) in v2.4.1. It can be solved locally by setting the environment variable TORCHINDUCTOR_WORKER_START=fork before executing the script.

cu118 wheels will not work with pre-cuda12 drivers

It was also reported (#130684) that the new version of triton uses cuda features that are not compatible with pre-cuda12 drivers. In this case, the workaround is to set TRITON_PTXAS_PATH manually as follows (adapt the code according to the local installation path):

TRITON_PTXAS_PATH=/usr/local/lib/python3.10/site-packages/torch/bin/ptxas  python script.py

Backwards Incompatible Change

Python frontend

Default TreadPool size to number of physical cores (#125963)

Changed the default number of threads used for intra-op parallelism from the number of logical cores to the number of physical cores. This should reduce core oversubscribing when running CPU workload and improve performance. Previous behavior can be recovered by using torch.set_num_threads to set the number of threads to the desired value.

Fix torch.quasirandom.SobolEngine.draw default dtype handling (#126781)

The default dtype value has been changed from torch.float32 to the current default dtype as given by torch.get_default_dtype() to be consistent with other APIs.

Forbid subclassing torch._C._TensorBase directly (#125558)

This is an internal subclass that a user used to be able to create an object that is almost a Tensor in Python and was advertised as such in some tutorials. This is not allowed anymore to improve consistency and all users should subclass torch.Tensor directly.

Composability

Non-compositional usages of as_strided + mutation under torch.compile will raise an error (#122502)

The torch.compile flow involves functionalizing any mutations inside the region being compiled. Torch.as_strided is an existing view op that can be used non-compositionally: meaning when you call x.as_strided(...), as_strided will only consider the underlying storage size of x, and ignore its current size/stride/storage_offset when creating a new view. This makes it difficult to safely functionalize mutations on views of as_strided that are created non-compositionally, so we ban them rather than risking silent correctness issues under torch.compile.

An example of a non-compositional usage of as_strided followed by mutation that we will error on is below. You can avoid this issue by re-writing your usage of as_strided so that it is compositional (for example: either use a different set of view ops instead of as_strided, or call as_strided directly on the base tensor instead of an existing view of it).

@torch.compile
def foo(a):
    e = a.diagonal()
    # as_strided is being called on an existing view (e),
    # making it non-compositional. mutations to f under torch.compile
    # are not allowed, as we cannot easily functionalize them safely
    f = e.as_strided((2,), (1,), 0)
    f.add_(1.0)
    return a

We now verify schemas of custom ops at registration time (#124520)

Previously, you could register a custom op through the operator registration APIs, but give it a schema that contained types unknown to the PyTorch Dispatcher. This behavior came from TorchScript, where “unknown” types were implicitly treated by the TorchScript interpreter as type variables. However, calling such a custom op through regular pytorch would result in an error later. As of 2.4, we will raise an error at registration time, when you first register the custom operator. You can get the old behavior by constructing the schema with allow_typevars=true.

TORCH_LIBRARY(my_ns, m) {
  // this now raises an error at registration time: bar/baz are unknown types
  m.def("my_ns::foo(bar t) -> baz");
  // you can get back the old behavior with the below flag
  m.def(torch::schema("my_ns::foo(bar t) -> baz", /*allow_typevars*/ true));
}

Autograd frontend

Delete torch.autograd.function.traceable APIs (#122817)

The torch.autograd.function.traceable(...) API, which sets the is_traceable class attribute on a torch.autograd.Function class was deprecated in 2.3 and is now being deleted. This API does not do anything and was only meant for internal purposes. The following raised an warning in 2.3, and now errors because the API has been deleted:

@torch.autograd.function.traceable
class Func(torch.autograd.Function):
    ...

Release engineering

• Remove caffe2 db and distributed from build system (#125092)

Optim

• Remove SparseAdam weird allowance of raw Tensor input (#127081).

Distributed

DeviceMesh

Update get_group and add get_all_groups (#128097) In 2.3 and before, users can do:

mesh_2d = init_device_mesh(
    "cuda", (2, 2), mesh_dim_names=("dp", "tp")
)
mesh_2d.get_group()  # This will return all sub-pgs within the mesh
assert mesh_2d.get_group()[0] == mesh_2d.get_group(0)
assert mesh_2d.get_group()[1] == mesh_2d.get_group(1)

But from 2.4 forward, if users call get_group without passing in the dim, users will get a RuntimeError. Instead, they should use get_all_groups:

mesh_2d = init_device_mesh(
    "cuda", (2, 2), mesh_dim_names=("dp", "tp")
)
mesh_2d.get_group()  # This will throw a RuntimeError
assert mesh_2d.get_all_groups()[0] == mesh_2d.get_group(0)
assert mesh_2d.get_all_groups()[1] == mesh_2d.get_group(1)

Pipelining

Retire torch.distributed.pipeline (#127354) In 2.3 and before, users can do:

import torch.distributed.pipeline # warning saying that this will be removed and users need to migrate to torch.distributed.pipelining

But from 2.4 forward, if users write the code above, users will get a ModuleNotFound error. Instead, they should use torch.distributed.pipelining:

import torch.distributed.pipeline # -> ModuleNotFoundError
import torch.distributed.pipelining

jit

• Fix serialization/deepcopy behavior for tensors that are aliasing but not equal (#126126)

Fx

Complete revamp of float/promotion sympy handling (#126905)

ONNX

• Remove caffe2 contrib and experiments (#125038)

Deprecations

Python frontend

• User warning when using torch.load with default weights_only=False value (#129239, #129396, #129509). A warning is now raised if the weights_only value is not specified during a call to torch.load, encouraging users to adopt the safest practice when loading weights.
• Deprecate device-specific autocast API (#126062) All the autocast APIs are unified under torch.amp and it can be used as a drop-in replacement for torch.{device}.amp APIs (passing a device argument where applicable)..
• Export torch.newaxis=None for Python Array API/Numpy consistency (#125026)

Composability

• Deprecate calling FakeTensor.data_ptr in eager-mode. FakeTensors are tensors without a valid data pointer, so in general their data pointer is not safe to access. This makes it easier for torch.compile to provide a nice error message when tracing custom ops into a graph that are not written in a PT2-friendly way (because, for example, they try to directly access a tensor’s data pointer from a region of code being traced). More details on integrating custom ops with torch.compile can be found here (#123292)
• Dynamic shapes:
  • SymInt-ify mem-efficient attention forward op signature (#125418)
  • Don't call item() into torch.scalar_tensor uselessly (#125373)
  • Fix scalar type for constraint_range to Long (#121752)
  • Guard oblivious on meta registrations (#122216), vector_norm (#126772), and unbind (#124959)
  • Make expected stride test in torch._prims_common size oblivious (#122370)
  • Use torch._check for safety assert in _reshape_view_helper (#125187)
  • Add a code comment about torch._check_is_size in tensor_split (#125292)
  • Make min(stride, strides[idx]) in collapse_view_helper size oblivious (#125301)
  • Don't short circuit if shape is same (#125188)

CPP

• Refactor autocast C++ APIs to be device-agnostic (#124359)

Release Engineering

• Remove of QNNPACK third-party module (#126941)

Optim

• Deprecate LRScheduler.print_lr (#126105)

nn

• torch.nn.hardtahn allowed min_val to be greater than max_val (#121627)

Distributed

• Distributed Checkpointing (DCP) Deprecated submodules feature for distributed_state_dict (#127793) In 2.3 and before, users can do:

  model = AnyModel(device=torch.device("cuda"))
  model_state_dict = get_model_state_dict(model)
  set_model_state_dict(
      model,
      model_state_dict=new_model_state_dict,
      options=StateDictOptions(strict=False),
  )
  
  # Below way of calling API is also legit
  model_state_dict2 = get_model_state_dict(model, submodules={model.submodule})
  set_model_state_dict(
      model,
      model_state_dict={model.submodule: new_submodel_state_dict},
      options=StateDictOptions(strict=False),
  )
  

  But from 2.4 forward, if users call get_model_state_dict or set_model_state_dict with a submodule path or state_dict, users will see a warning about the feature. To achieve the same functionality, users can manually filter out the state_dict returned from get_state_dict API and preprocess the model_state_dict before calling set_state_dict API:

  model = AnyModel(device=torch.device("cuda"))
  model_state_dict = get_model_state_dict(model)
  set_model_state_dict(
      model,
      model_state_dict=new_model_state_dict,
      options=StateDictOptions(strict=False),
  )
  # Deprecating warnings thrown for the below way of calling API
  model_state_dict2 = get_model_state_dict(model, submodules={model.submodule})
  set_model_state_dict(
      model,
      model_state_dict={model.submodule: new_submodel_state_dict},
      options=StateDictOptions(strict=False),
  )
  

• FullyShardedDataParallel (FSDP) Deprecate FSDP.state_dict_type and redirect users to distributed_state_dict (#127794) In 2.3 and before, users can do:

  model = AnyModel(device=torch.device("cuda"))
  fsdp_model = FSDP(model)
  # Users can do both ways below
  get_model_state_dict(model)
  with FSDP.state_dict_type(fsdp_model, StateDictType.FULL_STATE_DICT):
      fsdp_model.state_dict()
  

  But from 2.4 forward, if users call state_dict or set state_dict with the FSDP.state_dict_type, users will see warnings. And the recommended solution now is to use get_model_state_dict and set_model_state_dict directly:

  model = AnyModel(device=torch.device("cuda"))
  fsdp_model = FSDP(model)
  
  get_model_state_dict(model)
  # Deprecating warnings thrown for the below way of calling API
  with FSDP.state_dict_type(fsdp_model, StateDictType.FULL_STATE_DICT):
      fsdp_model.state_dict()
  

Profiler

• Remove FlameGraph usage steps from export_stacks docstring (#123102) The export_stacks API will continue to work as before, however we’ve removed the docstring to use FrameGraph. PyTorch doesn’t own FrameGraph, and cannot guarantee that it functions properly.

Quantization

• Remove deprecated torch._aminmax operator (#125995). torch._aminmax -> torch.aminmax instead

Export

• Start deprecation of capture_pre_autograd_graph (#125848, #126403)

XPU

• Refactor autocast C++ APIs to be device-agnostic(#124359) at::autocast::get_autocast_gpu_dtype() -> at::autocast::get_autocast_dtype(at::kCUDA) at::autocast::get_autocast_cpu_dtype() -> at::autocast::get_autocast_dtype(at::kCPU)
• Refactor autocast Python APIs(#124479) torch.get_autocast_gpu_dtype() -> torch.get_autocast_dtype(“cuda”), torch.set_autocast_gpu_dtype(dtype) -> torch.set_autocast_dtype(“cuda”, dtype), torch.is_autocast_enabled() -> torch.is_autocast_enabled(“cuda”), torch.set_autocast_enabled(enabled) -> torch.set_autocast_enabled(”cuda”, enabled), torch.get_autocast_cpu_dtype() -> torch.get_autocast_dtype(“cpu”)
• Make torch.amp.autocast more generic (#125103) torch.cuda.amp.autocast(args…) -> torch.amp.autocast(“cuda”,args…), torch.cpu.amp.autocast(args…) -> torch.amp.autocast(“cpu”, args…),
• Deprecate device-specific GradScaler autocast API(#126527) torch.cuda.amp.GradScaler(args…) -> torch.amp.GradScaler(“cuda”, args…), torch.cuda.amp.GradScaler(args…) -> torch.amp.GradScaler(“cpu”, args…),
• Generalize custom_fwd&custom_bwd to be device-agnostic (#126531) torch.cuda.amp.custom_fwd(args…) -> torch.amp.custom_fwd(args…, device_type=’cuda’),

ONNX

• Remove more caffe2 files (#126628)

New Features

Python frontend

• Add
  • support for unsigned int sizes for torch.unique (#123643)
  • torch.OutOfMemoryError to signify out of memory error from any device (#121702)
  • new device-agnostic API for autocast in torch.amp.* (#124938)
  • new device-agnostic API for Stream/Event in torch.{Stream,Event} (#125757)
  • channels last support to max, average and adaptive pooling functions (#116305)
  • torch.serialization.add_safe_globals that allows users to allowlist classes for weights_only load (#124331, #124330, #127808)
  • pickling support for torch.Generator (#126271)
  • torch.utils.module_tracker to track position within torch.nn.Module hierarchy (#125352)

Composability

• Add
  • OpOverload.redispatch; use it in new custom ops API (#124089)
  • mutated_args field to custom_op (#123129)
  • new Python Custom Operators API
  • register_autograd to register backward formulas for custom ops (#123110)
  • torch.library.opcheck (#124496), torch.library.register_autograd (#124071), torch.library.register_kernel (#124299)
• Blanket ban kwarg-only Tensors (#124805)
• Change register_autograd to reflect ordering of setup_context and backward (#124403)
• Ensure torch.library doctests runs under xdoctest (#123282)
• Fix torch.library.register_fake's module reporting (#125037)
• New Custom Ops Documentation landing page (#127400)
• Refresh OpOverloadPacket if a new OpOverload gets added (#126863, #128000)
• Rename
  • impl_abstract to register_fake, part 1/2 (#123937)
  • register_impl to register_kernel (#124200)
• Schema inference now includes default values (#123453)
• Stop requiring a pystub for register_fake by default (#124064)
• Support TensorList inputs/outputs (#123615)
• Update the functionalization error message (#123261)
• add ability to provide manual schema (#124180)
• fix schema inference for kwarg-only args (#124637)
• mutated_args -> mutates_args (#123437)
• register_autograd supports non-tensor kwargonly-args (#124806)
• set some tags when constructing the op (#124414)
• setup_context fills in default values (#124852)
• torch.library.register_fake accepts more types (#124066)
• use new python custom ops API on prims ops (#124665)

Optim

• Enable torch.compile support for LRScheduler with Tensor LRs (#123751, #123752, #123753, #127190)

nn frontend

• Add RMSNorm module (#121364)

linalg

• Implement svd_lowrank and pca_lowrank for complex numbers (#125580)
• Extend preferred_backend on ROCm backend.
• Add cuBLASLt gemm implementation (#122106)

Distributed

c10d

• Implemented IntraNodeComm primitives for allgather_matmul (#118038)
• Add first differentiable collective all_to_all_single_grad (#123599)
• Add P2P versions of send/recv_object_list operations (#124379)
• Add a new Collectives API for doing distributed collectives operations in the Elastic store with more performant and debuggable primitives (#126695)

FullyShardedDataParallel v2 (FSDP2)

• FSDP2 is a new fully sharded data parallel implementation that uses DTensor-based dim-0 per-parameter sharding for improved flexibility (e.g. mixed-dtype all-gather, no constraints on requires_grad) without significant cost to performance. See the document for more details and a comparison with FSDP1 (#122888, #122907, #123142, #123362, #123491, #123857, #119302, #122908, #123953, #120952, #123988, #124293, #124318, #124319, #120256, #124513, #124955, #125191, #125269, #125394, #126070, #126267, #126305, #126166, #127585, #127776, #127832, #128138, #128117, #128242)

Pipelining

• PyTorch Distributed pipeline parallelism APIs were upstreamed from the PiPPy project and are available as a prototype release in PyTorch 2.4. The package is under torch.distributed.pipelining and consists of two parts: a splitting frontend and a distributed runtime. The splitting frontend takes your model code as-is, splits it up into “model partitions”, and captures the data-flow relationship. The distributed runtime executes the pipeline stages on different devices in parallel, handling things like micro-batch splitting, scheduling, communication, and gradient propagation. For more information please check out the documentation and tutorial (#126322, #124776, #125273, #125729, #125975, #126123, #126419, #126539, #126582, #126732, #126653, #127418, #127084, #127673, #127332, #127946, #128157, #128163, #127796, #128201, #128228, #128240, #128236, #128273, #128279, #128276, #128278, #127066)

Profiler

• Add profiler support for PrivateUse1 (#124818)

Dynamo

• torch.compile is compatible with Python 3.12.
• Guarding on nn modules attributes (#125202) - TorchDynamo guards on nn module attributes. This was a frequently raised issue in the past (examples (#111785, #120248, #120958, #117758, #124357, #124717, #124817)). This increases TorchDynamo soundness with minimal perf impact.
• Hardened the recently introduced tracing rules infrastructure. This allows torch.compile users to easily control TorchDynamo tracing of PyTorch internal code.
• Extended torch.compile support for RAdam and Adamax optimizer. Compiler optimizers now demonstrate SOTA performance.
• Experimental feature - We introduced a new experimental flag torch._dynamo.config.inline_inbuilt_nn_modules to enable torch.compile to reuse compiled artifacts on repeated blocks in the models. This gives another point in the tradeoff space of compilation time and performance speedup. By moving torch.compile from full model to a repeated block (e.g. moving torch.compile from full LLM to a repeated Transformer block), we can now achieve faster compilation time with some performance dip compared to full model. We plan to make this flag default to True in the 2.5 release.

Export

• Introduce ShapesCollection, a dynamic shapes builder API (#124898)

Inductor

• Add higher order associative scan operator (#119430)

jit

• Add aten::sort.any op for sorting lists of arbitrary elements (#123982)

MPS

• Conform torch.mps to device module interface (#124676)

XPU

• Inductor Intel GPU backend (#121895)
• a new autocast API torch.amp.is_autocast_available(#124938)
• attributes to xpu device prop (#121898)
• XPU implementation for PyTorch ATen operators (#120891)
• generic stream/event on XPU backend (#125751)
• gpu trace on XPU (#121795)
• Switch to torch.float16 on XPU AMP mode (#127741)

ONNX

• quantized layer norm op to opset 17 (#127640)
• symbolic_opset19.py and symbolic_opset20.py to support opset 19/20, extend opset 18 support (#118828)
• Support for Some Bitwise Ops in Onnx Exporter (#126229)
• Allow ONNX models without parameters (#121904)
• Integrate onnxscript optimizer (#123379)

Vulkan

• quantized transposed 2D convolutions (#120151, #122547)
• the quantized ReLU operator (#123004)

Improvements

Python frontend

• bfloat16 support for torch.binary_cross_entropy on CPU (#123823)
• MAP_SHARED option for torch.load when mmap=True (#124889)
• default value when printing function signature (#127059)
• all variants of upsampling functions to be done in high precision in autocast (#121324)

Composability

• FakeTensors, meta tensors and python decompositions are used to perform shape propagation when tracing out a graph in torch.compile. There were much coverage improvements this release:
  • New metas / fake tensor rules:
    • aten._embedding_bag_dense_backward, aten._embedding_bag_per_sample_weights_backward (#125785), aten.randint.out, aten.rand.out (#122375), aten.unique2 (#124306), aten.histc (#124548), aten.channel_shuffle (#123033), aten._masked_scale (#127389), aten.addcdiv.ScalarList, aten.addcmul.ScalarList (#123486)
• New decomps:
  • Aten.resize_as (#122317), several out= variants of ops with existing decomps (#122979, #115437)

Autograd frontend

• nn.functional.batch_norm: add forward AD rule for miopen backend (#125069)
• nn.functional.scaled_dot_product_attention: add backward rule for cuDNN backend (#122510)

Release Engineering

• Add CI support for aarch64 linux. The CI is triggered when the ciflow/linux-aarch64 label is added. (#120931, #121284, #125255, #121136, #124781, #125599)
• Add experimental CUDA pip wheels for ARM architectures supporting the NVIDIA Hopper architecture as nightly binaries and a prototype for the PyTorch 2.4.0 release. (#126174, #127514)
• Add support for CUDA 12.4 in CI/CD (#121684, #121956, #127825, #125944, #128250)
• Add support for numpy 2.0.0rc1 in CI and CD (#123286, #122157)
• Enable support for torch.compile and triton with Python 3.12 CI/CD (#127547, #123307, #126218)
• Intel GPU enablement in CI (#122254, #123920, #125655)
• Migrated CI/CD jobs to macOS 14 (#127582, #127853, #125801)
• ROCM: upgrade CI/CD to 6.1 (#124811, #118216, #124300, #125646)
• CUDNN version 9.1.0.70 for CUDA 11.8, 12.1, 12.4 builds (#123475)
• NCCL submodule v2.20.5 (#121635)
• submodule oneDNN v3.4.2 (#126137)
• Wrapped deprecated function/class with typing_extensions.deprecated (#127689)

nn frontend

• Add swap_tensors path to nn parametrizations (#124130)
• Relax use_count constraints for swap_tensors when AccumulateGrad node holds a reference (#127313)
• Increase numel limit to 2^63 for replicatepad1d (#122199)
• Use int64_t indexing for Upsample2d backwards (#123682)
• Remove warning from LazyModuleMixin constructor (#123968)

Optim

• Radam and Nadam support the flag for "maximize" (#126765, #127214)
• Include scheduler_on_plateau in optim.h (#121722)

Foreach

• Allow foreach ops to run for any backend, not just CPU (#127412)

cuda

• Update CUDA out of memory message with private pool info (#124673)
• Add autocast rule for torch.vdot (#125697)
• Fix type hint for cuda.get_device_name() and cuda. get_device_capability() (#126743)

Quantization

• X86 Inductor backend
  • Enable linear and linear-unary post-op gelu quant recipe for X86InductorQuantizer (#114853)
  • Add Quantization recipe filter per operator type for X86InductorQuantizer (#122775)
  • Add Matmul recipe into X86InductorQuantizer (#122776)
  • Improve performance of qconv by reducing integration overhead (#123240)
• PT2E quantization flow
  • Add support for conv transpose + bn + {relu} weights fusion in PTQ and QAT (#122046, #123652)
  • Simplify fake_quant_per_channel (#123186)
  • Support fp8 quantization (#123161)
  • Propagate get_attr meta through known ops only (#124415)
  • Fix issue of lowering nn.linear ops with kwargs (#126331)

Distributed

c10d

• TORCH_NCCL_HIGH_PRIORITY option for ProcessGroupNCCL (#122830)
• __repr__ to P2POp class (#126538)
• commCreateFromRanks to c10d (#127421, #127982)
• dist.get_node_local_rank helper (#123992)
• an option to enable TCPStore libuv backed for c10d rendezvous (#124684)
• Captured dtype in Flight Recorder (#126581)
• Enable ncclCommDevIdxMap unconditionally (#122049)
• Extended the flight recorder dump from timeout to any exception (#123023)
• Make TCPStore server use libuv by default (#127957)
• Make get_node_local_rank() accept fallback_rank (#126737)
• Make abort communicators in destroy_process_group call on default and code cleanup (#124334)
• Mapped float8 types to uint8 for allgather (#126556)
• Optionally avoided rethrowing CUDA Errors in NCCL Watchdog (#126587)
• Wrapped TCPStore check in a try/catch (#127030)
• ProcessGroupWrapper support custom backend (#124447)
• ncclComm is not aborted before checking exception (#124466)

DeviceMesh

• Add a private init backend option (#124780)
• Initialized mesh tensor with CPU context (#124767)
• Add DeviceMesh.from_group() (#124787)
• Make _validate_tp_mesh_dim support 3D (#125763)
• Supported N groups in from_group (#126258)
• Make sure device mesh can be imported from torch.distributed (#126119)

Distributed quantization

• Used BFloat16 in distributed quantization when supported by NCCL (#125113)

DistributedDataParallel (DDP)

• Add a mode to avoid clone() in DDPSink (#122927)

Distributed Checkpointing (DCP)

• Add type_check param to copy state dict utils (#127417)
• Add strict option to DefaultPlanner (#123869)
• Always created requests for non-tensor objects (#125334)
• Always flattened mapping even if no tensors present (#125335)
• Correctly handle _extra_state (#125336)
• Implement broadcast_from_rank0 option for model/optim state_dict (#125338, #125339)
• Introduced async staging extension points (#122965)
• Make distributed state_dict support torch.distributed is not initialized case (#127385)
• Make param name consistent with overridden function (#124770)
• Remove the support of Dict[nn.Module, Dict[str, Any]] state_dict (#127070)
• Supported flattening the optimizer state_dict when saving and unflattening when loading (#127071)
• Unified the API signatures of set_model_state_dict and set_optimizer_state_dict (#127384)

DTensor

• backward support for scaled_dot_product_attention (flash-attention) (#122541)
• more foreach ops (#123214)
• op support for view_as_complex and view_as_real (#122569)
• op support for memory efficient attention (#122996)
• support for fused_adam and fused_adamw when lr is a tensor (#126750)
• ASGD foreach optimizer with associated unit tests (#121942)
• the handle of DTensor.device_mesh.device_type in dynamo (#118803)
• the support of placement kwargs for DTensor.to_local() in dynamo (#119947)
• scatter op with simple replication (#126713)
• distributed topk operator (#126711)
• Make Partial placement public (#127338, #127420)
• ensure expected input spec have correct tensor meta (#122949)
• ensure meta tensor random op does not alternate rng state (#125693)
• Move early return check into redistribute autograd function (#121653)
• Move some modules to private namespace (#127339)
• Standardized multi mesh-dim strategy with utils (#126712)
• 2D clip_grad_norm_ (#121945)
• simple replicate strategy for SVD (#127004)
• Turned on foreach implementation for (1) clip_grad_norm_ for DTensor by default (#126423), (2) optimizer for DTensor by default (#123394)

FullyShardedDataParallel (FSDP)

• device in pin_memory argument (#119878)
• private _unshard API (#124304)
• privateuse1 in FSDP's sharded grad scaler (#126971)
• Avoided CPU sync in clip_grad_norm_ (#122001)
• Marked pre_backward_hook unserializable (#125464)
• Skipped FSDP hooks base on dynamo config (#123021)
• Used generic device handle instead of cuda (#121620)

ShardedTensor

• Supported non-contiguous rank validation in sharded tensor (#123230)

TorchElastic

• debug info logging interface for expired timers (#123883)
• health check server hook in torch elastic (#122750, #123504)
• option for sharing TCPStore created by rendezvous handlers (#125743)
• support for binding to TCP in WorkerServer (#127986)
• Applied "distributed debug handlers" (#127805)
• Cleared timer for already terminated process (#122324)
• Skipped expired timer logging for empty expired timers (#125039)

Tensor Parallel

• wildcard support for Tensor Parallel parallelize_plan (#122968)
• kwargs support to prepare_module_input (#124114)

Profiler

Profiler torch.profiler:

• metrics for performance timing and other statistics collection (#123412)
• Kineto traces will export ns granularity for finer timestamps (#122425, #123650)
• Unified the device (CUDA, XPU, PrivateUse1) in profiler’s post processing (#123247)
• Improve profiler post processing by iterating frontend function events rather than all function events (#124596)
• Report strides in json traces (#125851)
• Register COLLECTIVE_COMM profiler activity type when available (#121461)
• Support third-party devices emit a range for each autograd operator (#125822)

Memory Snapshot torch.cuda.memory._dump_snapshot:

• Improve the description of blocks with missing frames in the Memory Visualizer (#124784)
• Add recordAnnotations to capture record_function annotations (#124179)

Profiler record_function:

• For with_effects, skip over profiler.record_function_exit (#121829)
• support for RecordFunctionFast to take inputs (#123208)
• support for kwargs in RecordFunctionFast (#123600)
• Collecting autograd sequence numbers on PythonTLSSnapshot dispatch keys for Nested Tensor (#123304)

Export

• a printer to the unflattened module (#124315)
• disable_forced_specializations flag (#124949, #126925)
• export support for auto_functionalize (#121990, #122177, #122246)
• readable placeholder names to ExportedProgram nodes (#123587, #123590, #124765)
• set_grad_enabled higher order operator (#123391, #125066, #121736)
• stack_trace for non-strict export (#121034)
• torch_fn, a more consistent metadata across strict and non-strict export (#122693)
• torchbind tracing support (#122619, #123370, #122622, #125490)
• Allow static constraints in dynamic_shapes (#121860)
• Ignore logging.Logger.* calls during dynamo export (#123402)
• Make metadata serialization more strict (#124411)
• Populate ShapeEnv's var_to_val during deserialization (#121759)
• Prototype TorchScript 2 ExportedProgram Converter (#126920, #127466)
• Provide refine function for automatically accepting dynamic shapes suggested fixes (#127436)
• Save/load example inputs in the ExportedProgram (#122618)
• Suggest constant dim values in dynamic shapes fixes (#125458)
• Support map in pre-dispatch functionalization (#121444)
• We introduced the concept of “effect tokens”, which is how we allow side-effectful operators in torch.compile/export (#121552, #122357)

Fx

• shape inference tool (#120097)
• device_ordinal to Subgraph in splitter_base (#125616)
• exclusion function to minimizer base (#124504)
• missing forbidden mutation methods in immutable collections (#125468)
• option to turn on return_tuple in _SplitterBase (#123868)
• prefix option to CapabilityBasedPartitioner (#126382)
• Create block traverse mode in minimizer for graph aware debugging (#125613)
• Implement Graph Transform Observer (#127427)
• Option to include stride and device annotation in gm.print_readable() (#123690)
• Register create_node_hook (#126671)

Dynamo

• We performed a careful audit and fixed all known memory leaks in TorchDynamo.
• We hardened torch.compile + __torch_function__ support by onboarding Scaled Dot Product Attention (SDPA) and TensorDict.

Inductor

• 0 initialization to Triton masked loads (#127311)
• HalideCodeCache (#126416)
• clone if output is a view from constant (#123200)
• config to allow buffer mutation (#126584)
• decompose_mem_bound_mm to the customization pre and post grad passes (#123376)
• inductor support (#123709)
• kernel_code logging artifact (#126631)
• lowering for avg_pool{1, 3}d (#116085), cummax, cummin (#120429)
• missing files to torch_key (#128230)
• mode to MemoryDep to track atomic accumulates (#123223)
• pybind for tensor_converter util functions (#121744)
• qlinear_pointwise.binary op for X86Inductor backend (#123144)
• support for multiple flexattention calls in a single compile (#125516)
• tensor_constantX to pass constant buffer update's check (#122562, #122690)
• the quant lift up pass in convert phase (#122777)
• a decomposition for select_scatter (#124426)
• Allow multiple cudagraph recordings per compiled graph (#126822)
• Automatic detection for buffer mutation and binary linking (#126706)
• Change
  • OverridesData to take callables instead of strings (#123397)
  • aot_compile callsites (#122225)
• Clean up for removing 2 decompose patterns (#123422)
• Codegen runtime asserts in Inductor (#124874)
• Customize pre grad and post grad patterns (#121915)
• Disallow fusions of foreach and reductions (#127048)
• Enable
  • lowering of qlinear-binary(-unary) fusion for X86Inductor (#122593)
  • mmaped weights when CUDA is used (#124346)
  • meta internal AOTInductor compilation on ROCM (#124123)
• Enhance RecordFunctionFast input args and use input args in triton_heuristics.py (#123459)
• Filter non input symexprs from codecache guards (#128052)
• Get PT2 Cutlass backend working under fbcode (#125688)
• Hipifying aoti code_wrapper (#124241)
• Improve group batch fusion with same parent/users fusion enablement (#127648)
• Inductor respects strides for custom ops by default (#126986)
• Initial implementation of Inductor FX Graph Remote Cache (#124669)
• Make
  • torch._inductor.dependencies.Dep a proper class (#124407)
  • c10/util ostream function implementations to their headers (#123847)
  • some cudagraphs checks into C++ (#122251)
• Pass triton kernel info to record function (#123871)
• Read the patterns from the config instead of hard-code passes (#125136)
• Remove
  • API that allows for extra deferred runtime asserts during lowering (#124864)
  • assertion for cat target_func (#125540)
• Serialize large weights (#123002)
• Specialize on unguarded alignment of example inputs (#123319)
• Split cat customization (#123045)
• Support
  • CUDA_INC_PATH env variable when compiling extensions (#126808)
  • custom op in JIT with cpp wrapper (#122554)
  • pytrees as associative_scan input (#122137)
  • use_runtime_constant_folding for CPU (#122563)
• Try to reuse old symbol name rather than new symbol name when renaming (#124782)
• Update the cpp_wrapper entry function signature (#121745)
• Use source code hash instead of torch version (#126092)
• Various improvements to error handling during autotuning (#126847)
• batch pointwise op + unbind stack pass in post grad (#126959)
• config target platform (#126306)
• disable comprehensive padding in fbcode (#124191)
• enable software pipelining on AMD devices (#125858)
• epilogue support for gemm template (#126019)
• make mask_rcnn inference work in max-autotune mode (#123008)
• pt2 dper passes: run shape prop before each pass (#122451)
• remove 2 decompose patterns (#123371)
• switch assume_aligned_inputs to False (#124336)
• unified the vectorized conversion with at::vec::convert for all data types (#119979)

jit

• Shape function fix for _batch_norm_with_update (#122430)
• Attach target function to OSError when source can't be found (#125248)
• Support getattr/hasattr on NamedTuple (#121863)

ONNX

• Allow fake models to run with ONNXProgram.call (#122230)
• Fix ONNX export with print (#123368)
• Improve torch.onnx.export runtime from O(n^2) to O(n) (#123025, #123027, #123063, #124909, #123028, #123028, #123029, #123026, #124912)
• Make ONNXProgram.model_proto and disk file the same (#122196)
• Skip optimizer when it fails (#127349)
• Update decomposition table to core ATen ops (#127353)
• beartype to emit warning instead of error by default (#123205)

MPS

• Add naive quantized int4_mm, int8_mm and .gputrace capture hooks (#125163)
• Better error-check for linear op (#124952)
• Enable
  • index_select for complex types (#122590)
  • torch.mm and other ops for complex dtypes (#127241)
• Implemented isin_Tensor_Tensor_out for MPS backend (#124896)
• Improve F.adaptive_avg_pool2d error messages on MPS backend (#124143)
• Native non-zero op implementation (#125355)

XPU

• Generalize host allocator to be device-agnostic(#123079)
• Make macro with AMP more generic(#124050)
• Refactor
  • CUDA’s AMP autocast policy to be generic(#124051)
  • gpu trace to be device-agnostic(#121794)
• Support generic Stream/Event on CUDA/HIP backend(#125757)

Bug fixes

Python frontend fixes

• DtoH sync in torch.index_put_ (#125952)
• torch.load map_location for wrapper subclass and device being serialized through numpy (#126728)
• memory leak in torch.dtype.to_complex() (#125154)
• nn.Parameter constructor type hint (#125106)
• parameter name in torch.can_cast to from_ (#126030)
• support of paths with space in torch.utils.cpp_extensions (#122974)
• Support numpy array in Tensor.eq (#122249)

Composability fixes

• FakeTensors, meta tensors and python decompositions are used to perform shape propagation when tracing out a graph in torch.compile. There were a number of bug fixes improvements this release:
  • FakeTensor fixes:
    • Handle symbolic size access in FakeTensor (#124760)
    • Avoid cuda init in FakeTensorMode (#124413)
    • Do not run CUDA lazy init if it is triggered with fake mode on (#122636)
    • Refactor faketensor ops that produce unbacked symints to memoize (#125623)
  • Meta device fixes:
    • fix meta tensor set_() incorrectly modifying nbytes of the storage (#123880)
    • Fix aten._weight_int4pack_mm meta registration for float16 inputs (#124136)
  • Fixes to python decompositions:
    • aten.upsample_bicubic2d: support for uint8 (#120411)
    • aten.upsample_nearest* ops: properly registered decomp to dispatch keys (#122782), (#122783)
    • _refs.masked_fill: support privateuse1 device when value.device.type is cpu (#124835)
    • _refs._reshape_view_helper: specialization shortcut for converting n-d to 1-d and 1-d to 2-d views (#127641)
    • Fix decomp for torch.tensor(...) constructor with nested python lists(#125639)
    • Aten.rrellu_: fix decomp when default values are missing (#126978)
• AOTDispatcher is the component of the torch.compile stack that functionalizes and normalizes the graph, and adds support for compiling the backward during training. There were several bugfixes and improvements to AOTDispatcher:
  • Fix torch.compile used with triton kernels under inference_mode (#124489)
  • Fix incorrect graph when functionalizing aten.expand followed by mutation (#122114)
  • Properly keep input mutations in the graph when they are under torch.no_grad, even if there are outstanding aliases (#122433)
  • Replay original views from the user code instead of falling back to as_strided in a few cases, which can improve performance of the backward pass in cases where torch.compile captures small graphs with outputs that alias graph inputs (#121007)
• For __torch_dispatch__-based tensor subclasses, support custom layout overrides under torch dispatch mode (#125379)

cuda fixes

• cuda array for empty arrays (#121458)
• a perf regression in kernel launcher for the foreach_* family of ops (#123566)
• CUDA out of memory error message formatting (#123984)
• CUblasLt compilation on windows (#125792)

Autograd frontend fixes

• torch.utils.checkpoint: Use pytrees to improve determination of what RNG state to stash (#121462)
• Fix error message of autograd (#123154)

Release Engineering fixes

• Fix mypy issues in fake_tensor.py (#124428)
• Fix running of: lintrunner --all-files --take FLAKE8 (#124771)
• Fix libc and libstdcxx installation on conda environments (#121556)
• Release engineering tooling and CI fixes. Workflows, Trymerge, Bot Labeler, Mergebot (#125042, #121762, #121920, #124965, #122155, #123301, #121733, #127567, #128080)

nn frontend fixes

• access to unitialized memory in VSX vector functions for quantized values (#122399)
• swap_tensors path in nn.Module._apply for modules that inherit from RNNBase (RNN, GRU, LSTM) (#122800)
• ctc_loss zero/negative length corner cases (#123193)
• _LazyConvXdMixin.initialize_parameters and add related tests (#123756)
• load_state_dict with unexpected key whose prefix matches a valid key (#124385)
• requires_grad propagation in nn.utils.parametrize (#124888)
• nan with large bfloat16 values for FlashAttention backend of nn.functional.scaled_dot_product_attention
• issue in affine_grid_backward when grad_grid is non-contiguous (#124370)
• Add error checks for invalid inputs on thnn_conv2d (#121906)(#122135)

Optim fixes fixes

• Wrong ASGD implementation (#125440, #126375)
• loading optimizer options from archive (#125215)

linalg fixes

• svd_lowrank(..., M) in the presence of broadcasting (#122681)
• linalg.vector_norm when used with autocast(cuda) (#125175)

CPP fixes

• Handle all types c10::isSigned (#125637)
• crash for AVX512 int4 matrix multiplication if weights are unaligned (#124128)
• loading custom C++ extension within DataParallel-ized model (#125404)

Distributed fixes

c10d

• coalescedCollective op Flight Recording (#120430)
• group_name/group_desc set up in eager initialization (#127053)
• bug in _update_process_group API (#128262)
• bug in update_process_group DDP API (#128092)
• excepthook crash on exit after destroy_process_group (#126739)
• various errors in TCPStoreLibUvBackend.cpp (#127230)
• work handle for coalescing manager (#122849)
• Add check gloo availability when doing _ProcessGroupWrapper check (#124233)
• Add initialize lastEnqueuedSeq_ and lastCompletedSeq_ in ProcessGroupNCCL (#121980)
• Ensured gil is not released when calling to PyBytes (#128212)
• Guarded gpu context during abort (#127363)
• Make monitorThread sleep when we try to dump flight recorder (#123788)
• Only included NCCL related header file with macro USE_C10D_NCCL (#127501)
• Prevented wait_tensor() calls on graph inputs from getting DCEd for AsyncCollectiveTensor (#125677)

DeviceMesh

• hash and eq not match (#123572)
• device type issue in _get_device_handle (#124390)
• Enable cache and reuse of sliced result to prevent funky behaviors and NCCL deadlock at large scale (#122975)
• Make dtype of mesh tensor from init_device_mesh() consistent with directly calling DeviceMesh() (#123677)

DistributedDataParallel (DDP)

• DDP no_sync when find_unused_parameters is True (#124193)

Distributed Checkpointing (DCP)

• to remove non_persistent buffer in distributed state dict (#125337)
• set_optimizer_state_dict() changes the parameters with some optimizers (#125708)
• various bugs for broadcast_from_rank0 (#127635)
• Remove the check of FSDP has root (#121544)
• Kept params in torch.distributed.checkpoint.state_dict.set_optimizer_state_dict (#127644)

FullyShardedDataParallel (FSDP)

• FSDP 2D state_dict to use run_check=False (#123802)
• HSDP: sharding placement (#123778), validation error msg (#123019)
• summon_full_params on submodule (#123290)

TorchElastic

• Make torch.multiprocessing.ProcessContext.join() wait for all child procs to exit before return (#125969)

Profiler fixes

• an asynchronous trace bug where end timestamp overflows and events are years in the future (#124080)
• torch.profiler Schedule Function (Function Event only) to accumulate events (#125510)
• Add a sanity test to the unit testing (#124773)
• Add missing field device_resource_id in profiler events (#121480)
• Cleaned up deprecated use_cuda by default (#126180)
• Do not emit a warning when using CPU profiler only (#125654)
• Handle more cases of symbolic sizes/strides detection (#123696)
• Reduced warning msg in torch.profiler when using AMD (#124469)
• Release gil in prepareProfiler (#121949)
• Remove a redundant *1000 to timestamp since we already have ns precision (#124374)
• Split up profiler test file (#124856)

Dynamo fixes

• 'Could not infer dtype of SymBool' on torch.tensor call (#125656)
• 'get_attr' call in dynamo 'run_node' (#127696)
• 'get_real_value' on placeholder nodes (#127698)
• assume_constant_result for UnspecializedNNModuleVariable methods (#127695)
• guard_size_oblivious on non-symbolic expression (#123743)
• tvm backend interface (#126529)
• Add support for tensor's is_complex method (#124927)
• Allow asserts to fail (#126661)
• Forward OptimizedModule.setattr to the wrapped module (#122098)
• Initial exception handling support in dynamo (#126923)
• Keep track of ViewMeta with symbolic inputs (#125876)
• Support macOS and Linux/aarch64 platforms (#128124)

Export fixes

• GraphModuleDeserializer handling of signature (#122342)
• bug in get_update_constraint (#125194)
• conv decomp when decomposing to core-aten (#123283)
• mode not on stack error for while loop (#122323)
• runtime assertions to add call_function (#125878)
• to_copy to be inserted in the exported graph (#125628)
• unflattening with duplicate tensors (#125192)
• up nn_module_stack for nodes occurred around tracepoint ops (#124457)
• leaky fake tensor on attribute assignment, support buffer assignment (#122337)
• Allow Dim(1,2) for export dynamic shapes (v2 after revert) (#121910)
• Allow modules to be created in the forward (#125725)
• Correctly serialize empty list based on argument type (#123748)
• Forward fix failures for torch.export switch to predispatch (#126081)
• Handle param aliasing (#127471, #125509, #125758)
• Make error name private (#126715)
• More strictly respect scope when removing inputs in unflattener (#127607)
• Skip nn_module_stack verifier for non-fx.GraphModule modules (#122210)

Fx fixes

• fx graph triton import bug (#122041)
• graph partitioner and make runtime assertion work with submodules in export (#125793)
• infinite recursion in API BC test (#125706)
• mem size mismatch from split/chunk in const folding (#125199)
• triton import time cycles (#122059)
• Don't intersect when clamping for size oblivious (#123675)
• Don't use Proxy torch function in the sym size calls (#121981)
• FakeTensorProp assert consistency of sizes when metadata previously existed (#124059)
• Keep set_() input mutations in the AOTDispatcher graph, ban other cases (#122981)
• Make
  • check_is_size clamp to sys.maxsize - 1, so sys.maxsize comparison returns False (#122372)
  • torch._check understand Eq commutativity (#125629)
• Preserve
  • node.meta when fusing subgraph (#125261)
  • partitioner order (#122111)
  • unbacked SymInt on SymNode (#120816)
• Remove
  • duplicated nodes in dfs_iter_find_cycle (#125585)
  • incorrect check (#123616)
• Skip index_put_ in dce (#122683)

Inductor fixes

• AFOC QPS Regression (#122944)
• C++ compilation error for tensor array in abi_compatible mode
• FakeTensorUpdater logic for updating fake tensors (#116168)
• a bool value codegen issue when calling custom ops (#127398)
• a bug when mutated buffer meets .to (#127671)
• a codegen issue when .item() is used for kernel arg (#126575)
• a dynamic shape problem when lowering diagonal (#121881)
• an internal test regression (#123481)
• another out-of-bounds access (#122580)
• cat backwards wrapping on symints (#121527)
• compilation_latency regression caused by #127060 (#127326)
• constant propagation pass (#114471)
• cuda compilation under fbcode remote execution (#126408)
• cummax and cummin lowering for empty case (#126461)
• cutlass path in inductor (#125463)
• edge case in JIT vs. AOT fusion after finalizing MultiTemplateBuffer (#126622)
• includes to system Python (#125285)
• issue with randint + symbolic shapes (#122428)
• issues in pre_grad passes (#123181)
• mask propagation in the presence of where (#125574)
• memory planning compile error (#123867)
• missing unbacked def for unbacked in input expr (#127770)
• nextafter in inductor CPP codegen (#126876)
• ops.scan for non-commutative operators (#126633)
• out-of-bounds read/write in cvt_int64_to_[fp32|int32] (#122511)
• scheduler typehints (#127769)
• test with inlining flag (#128200)
• to #126656 (#127050)
• triton codegen main do_bench_gpu import error (#126213)
• unbacked symbol in stride when using item() (#122298)
• unsupported type of output=s1 (#126797)
• ScatterFallback codegen (#124580)
• a constant tensor device move issue (#128265)
• an assertion for node debug str (#127021)
• grid z bug for large grid (#127448)
• invalid call to aoti_torch_tensor_copy_ (#126668)
• linear_add_bias path (#127597)
• loop ordering test (#127807)
• miss isa bool check (#128274)
• post_grad pattern (#127457)
• redis-related env vars in remote_cache.py (#127583)
• Add missing acosh op to vec256_float_neon.h (#122513)
• Back out
  • "Added a check in register_lowering to avoid decomposed ops (#117632)" (#122709)
  • "Precompile triton templates (#121998)" (#123305)
• Backport https://github.com/openai/triton/pull/3433 (#122470)
• Correctly calculate the numel with symint in DDP fusion (#124422)
• Disable stack allocation when there is a fallback op (#122367)
• Do not forward parent's value range to CSE variable for variables created within codegen (#123099)
• Do not propogate (#124769)
• Don't clamp slices generated from cat kernel (#124139)
• Enable B019 - flags memory leaks through LRU cache on method (#127686)
• FX graph cache: Fix bug handling constants (#121925)
• Fall back to eager mode when viewing with differing bitwidths (#120998, #121786)
• Implement masked_load for integral types (#122608)
• Improve unbacked SymInt input support in Inductor (#124739)
• Inductor: fix Conv output stride for dynamic shapes (#121400)
• Remove symbol exports in C shim for Windows (#125472)
• Revert "Inductor respects strides for custom ops by default (#126986)" (#127923)
• Use pexpr, not texpr in Triton launch codegen (#128038)
• turn off triton memcache for amd devices (#122560)
• typing scheduler.py [1/2]: Bug fix (#126610)
• use two pass reduction for deterministic reduction order (#115620)
• Forward fixes
  • for D56289438 (#124882)
  • for templates + views (#127446)

ONNX fixes

• Fix list dtype finding bug in dispatcher (#122327)
• Rename ort to maia in dynamo's ort backend (#124967)
• Cast checkpoint weights to match model parameter's dtype (#122100)
• Reduce excessive warning to info (#122442)
• Prevent dup initializers when ONNXProgram.save is called many times (#122435)

MPS fixes

• FFT descriptor fields to resolve precision issue (#125328)
• FFT implementation bug dropping negative frequency components (#123274)
• GELU, LeakyRELU and MISH on non-contiguous tensors (#123049)
• abs for complex types (#125662)
• copies larger than 4GB (#124635)
• crash with binary_cross_entropy is invoked for half dtypes (#124258)
• for MPS regression in scalar creation (#123234)
• for addcdiv contiguous problem (#124442)
• naive matmul for BFloat16 (#121731)
• nextafter for negative values (#125029)
• overflow in cumsum when dtype is bool (#125318)
• strided ELU op correctness issue (#125692) and mse_loss correctness issue (#125696)
• Fwd-fix for clamp regression (#122148)
• Remove in place views fixing various crashes (#124895)

XPU fixes

• record issue on XPUGuardImpl (#123523)

Performance

Python frontend

• Use sleef on macOS Apple silicon by default (#126509)

cuda

• Speed up torch.softmax kernel (#122970)

nn frontend

• Parallelize upsampling ops across the batch/channel dimension (#127082)

Optim

• Add fast fused kernels for Adam, AdamW, SGD, and Adagrad on CPU (#123074, #123629, #124905)

linalg

• Improvements
  • the CPU performance of linalg.vector_norm when reducing over a dimension of length 1 (#122143)
  • performance of FP16 gemv on ARM (#126297, #126745, #126746, #126877, #127033) and BF16 gemm fallback on ARM (#126592)
  • autotuning through TunableOp on ROCm (#124362)

Foreach

• Allow int vals to go down the fastpath for _foreach_max (#127303)
• _foreach_copy now supports different source/dest dtypes on the fast path (#127186)

Distributed

C10d

• Disable compute of collective duration by default (#122138)

DTensor

• Used str for reduce_op instead of c10d enum (#125172)
• Make early return for _split_tensor (#125810)
• Make directly return local_tensor under no_grad (#128145)

Distributed Checkpointing (DCP)

• Improve the performance of distributed state_dict (#125501)

TorchElastic

• Changed monitor_interval for torchelastic default value to 0.1 sec (#124692)
• Add timing events to different stages of rendezvous (#125636)

jit

• Fix exponential memory usage when TorchScript types share the same name (#121874), (#121928)

Fx

• Add side table to FX Graph for O(1) op/target query (#121565)
• Apply guard knowledge to all simplifications (#123342)
• Do not calculate hint in advice_is_size (#124472)
• Enable FX graph and symbolic shape caching (#121697, #125258, #123724, #124610)
• Flatten/Unflatten micro optimization in proxy_tensor.py (#121993)
• Minor compile time optimization in has_free_symbols (#122144)
• Skip assert in check_is_size (#124209)
• Teach ShapeEnv that a <= b => a < b + 1 (#123436)
• Use sympy xreplace instead of subs (#124208)
• _find not update unchanged replacements (#124274)
• eval_static: guards, unbacked compute once (#124217)

Inductor

• Speedup convert<float>(Vectorized<half>::loadu(ptr, 8)) on ARM (#125889)
• Add more mm kernel choices (#125000)
• Add NEON ISA support on
  • arm64 Macs (#122217)
  • aarch64 (#123584)

MPS

• Improvements to perf of int4pack_mm (#125983, #127135, #125704)
• Making copy_cast, softmax and cat_out unranked (#123191)

XPU

• Intel GPU
  • Convolution&Deconvolution aten operators(#117529)
  • Matmul aten operators(addmm, badbmm, etc.)(#117202)
• Support xpu host allocator (#123080)
• oneDNN
  • Conv primitive integration (#117512)
  • Matmul primitive integration (#117112)
  • library compilation for Intel GPU support (#117098)

Documentation

Python frontend

• Add doc for
  • torch.distributions.utils.clamp_probs (#128136)
  • the legacy constructor for Tensor (#122625)
  • torch.Size.numel (#124186)
  • torch.utils.benchmark.utils.compare.Compare (#125009)
  • torch.utils.collect_env.get_env_info (#128021)
• Clarify Security Policy (#120531)
• Fixes doc
  • example of torch.masked_scatter (#123664)
  • for torch.load map_location (#125473)
• Improve doc for
  • torch.set_default_dtype (#121730)
  • torch.load weights_only argument (#127575)
• Update doc for
  • functions in torch.multinomial (#125495)
  • functions in torch.random (#125265)
  • torch.dot (#125908)

Composability

• Add extended debugging options for troubleshooting torch.compile issues (#122028)

cuda

• Add doc for torch.cuda.nccl.version (#128022)
• Add documentation for nvtx.range (#121699)

Autograd frontend

• torch.autograd.Function: update docs for separate context and forward functions (#121955)
• torch.utils.checkpoint: Improve error message when use_reentrant=True is used with .grad() (#125155)
• Improve the clarity of the torch.Tensor.backward doc (#127201)
• Fix typing for torch.autograd.Function with ctx-less forward (#122167)

Release Engineering

• Fix torch and torch.compile links (#121823, #121824)
• Add
  • fuzzer instructions to pt2 bug template (#123156)
  • better instructions for pytorchbot merge command on cancel (#124947)
  • instructions on how to run doc coverage locally (#123688)

nn frontend

• Fixes
  • KLDiv example (#126857)
  • torch.nn.TripletMarginLoss allowing margin less or equal to 0 (#121978)
  • example and typo in nn.ChannelShuffle and nn.PReLU docs (#123959)
  • redundant tensor in nn.MaxUnpool2d example (#127850)
  • wording in nn.Linear docstring (#127240)
• Improvements
  • NLLLoss documentation (#127346)
  • documentation of torch.nn.utils.rnn (#123559)
  • return value documentation for nn.Module.load_state_dict (#123637)
  • the example description for torch.nn.utils.rnn.pad_sequence (#123183)
• Update the is_causal explanation in the nn.functional.scaled_dot_product_attention doc (#127209)
• Warn SDPA users about dropout behavior (#126294)

Optim

• Document complex optimizer semantic behavior (#121667)
• Add missing parameter doc of Adagrad (#125886)

linalg

• Improve docs on the sorting of eig/eigvals (#127492)

Distributed

c10d

• Add
  • a doc page for NCCL ENVs (#128235)
  • migration notes for --local-rank option style change for torchrun for PyTorch 2.0 onwards (#109480)
• Documents
  • 'tag' limitation for nccl send/recv (#125278)
  • destroy_process_group usage (#122358)
• Fixes
  • example in torch.distributed.new_subgroups docstring (#123492)
  • the document of distributed.new_group() (#122703)

Distributed Checkpointing (DCP)

• Corrected typos in assert (#122633)

DTensor

• Add comment on replicate -> partial for _NormPartial (#121976)
• Updated public API docs for DTensor (#127340)

FullyShardedDataParallel (FSDP)

• Remove excessive warnings and rewrite FSDP docstrings (#123281)
• Fix docs for inter/intra node PG helpers (#126288)
• Updated docstring to include device_mesh arg (#126589)

Profiler

• Updated PT2+Profiler docs (#122272)

Export

• Fix documentation for register_fake_class (#126422)

Fx

• Document for add_var_to_val (#121850)

Dynamo

• Add a Dynamo deepdive to documentation (#122305)
• Update compile doc to suggest Module.compile (#123951)
• Fixes
  • links rendering when surrounding code in Dynamo deepdive (#123427)
  • the link to torch.compiler_custom_backends (#125865)
  • typos in torch._dynamo.config.py (#126150)
  • NumPy + backward example (#126872)

Inductor

• Fix aoti doc to avoid cannot bind non-const lvalue reference error (#121672)
• documentation for pattern_matcher.py (#127459)

ONNX

• Fix pytorch version for onnx in doc (#124182)
• Add docstring to masked_fill, expand, select, unsqueeze, cat fns (#128055)
• Documenting torch.onnx.operator.shape_as_tensor (#128051)
• Init sigmoid comments (#127983) (edited)

XPU

• PyTorch 2.4 XPU Getting Started (#127872)
• Update Intel GPU Support on README (#126001)
• Tensor (#126383 #127280)
• Stream (#121398)
• AMP (#127276 #127278)
• torch.compile with XPU support (#127879)

Developers

Composability

• cpu_fallback for aten::triu_indices on custom device crash (#121306)
• API to check whether running in torch_dispatch mode (#122339)
• clarify c10::Dispatcher kernel static asserts (#124519)

Release Engineering

• TD (target determination) reorders tests in CI based on heuristics and removes tests it believes to be irrelevant to the changes in the PR. (#121835, #121836, #122279, #122615, #122901, #124082, #122976, #125931)
• torchbench on-demand test workflow (#122624).
• BE: Ruff lint improvements (#124743, #124570)
• ability to save TORCH_COMPILE_DEBUG logs for CI failures (#124408)
• freezing option for cpu inductor accuracy test in inductor CI (#124715)

Optim

• Modify device check in capturable optimizer to support more devices (#124919)
• Improve typing and error messages in LRScheduler (#125556, #127943, #121633, #125161)
• Only initialize state if needed in SGD (#123757)
• Exempt torch.compile from more checks in Adamax (#123498)
• Merged the pyi files into py files of optimizer (#125153, #125452)
• Tighten fallback conditions for compiled optimizer (#125825)

Distributed

c10d

• Updated error message for sparse all-reduce (#121644)
• Add
  • generic scuba logging capability into c10d (#121859)
  • log the target of Flight Recorder dump (#122345)
  • the source rank in the logs when detecting the timeout (#122850)
  • more fields for periodic logging (#123860)
  • pg_name and pg_desc to logger (#126409)
  • Work's numel to logger for debugging purposes (#127468)
• Allow user to pass process group description for ProcessGroupNCCL (#123472)
• Print the duration of the broadcast of ncclunique_id (#123963)
• Pass and recorded process_group_name when creating ProcessGroupNCCL (#123117)
• Pass pg name and desc to NCCL communicator (#124149)
• Make only PG0 should dump when monitoring thread timed out (#125356)
• split seq_id to collective_seq_id and p2p_seq_id (#125727)
• Print certain logs only on the head rank of each node (#125432)
• Make warn env vars only once during program (#127046)

DTensor

• Add some initial c10d ops to CommDebugMode (#125475)
• Remove unused failed_reason (#126710)
• Add all_reduce_coalesced tracing to CommDebugMode (#127025)

Distributed Checkpointing (DCP)

• additional logging for improved observability in DCP (#121352)

FullyShardedDataParallel (FSDP)

• Remove unnecessary warnings (#126365)
• warnings on wrapping ModuleList/ModuleDict (#124764)

Miscellaneous

• Remove dist_ prefix from TORCH_LOGS shortcuts (#126499)
• Make torch.distributed.breakpoint() to work under Python/Meta contexts (#118645)

TorchElastic

• Make log directory creation idempotent (#126496)

Fx

• Suggest TORCHDYNAMO_EXTENDED_DEBUG_ envvars when appropriate (#122473)

Inductor

• aoti_torch_item as a util function (#126352)
• model_type and global_rank for the scuba log for the dashboard Optimus pattern frequency monitor (#123398)
• Change the log for the group batch fusion (#122245)
• Do not use importlib.load_module (#122542)
• Enable FX graph caching on another round of inductor tests (#121994)
• Improves
  • exception typing. Remove NOQAs (#125535)
  • generate_extern_kernel_out's signature (#123351)
  • logging (#122932)
  • the optimus scuba log (#122361)
• Misc refactors (#126945)
• Only print bw result for the first time we benchmark a kernel (#123568)
• Refactor
  • MultiOutput. codegen_list_tuple_access to use subclass type checks (#121662)
  • indexing() into triton.py
  • part of IterationRangesEntry into triton.py (#126944)
  • some fallback op util functions (#126182)
  • is_legacy_abi_kernel and abi_compatible_kernel (#121523)
• Renamed mutationlayout/aliasedlayout (#122474)
• Unify val_to_arg_str and val_to_cpp_arg_str (#126916)
• Update
  • DTYPE_TO_CPP mapping (#126915)
  • opinfo tests (flattened diff) (#124657)
  • tensor_converter util functions (#121743)
  • triton pin (#121268)
• Use C++17 helper templates (#122607)
• delete inductor config.trace.compile_profile (#127143)
• log pt2 config dict to signpost from inductor post grad (#124593)
• refactor
  • code to use define_kernel and call_kernel similar to CUDA (#123704)
  • device dispatch inside do_bench (#125736)

MPS

• Reorganize logics and naming in copy.mm (#123310)
• Pointer to the non-zero limit ticket#124244
• Introduce MetalShaderLibrary class (#125550)
• Include MPSGraphVenturaOps.h for complex types on macOS12 (#127859)
• Define _compute_tolerances (#121754)

XPU

• Support general device runtime Interface for Intel GPU (#121883)
• Enable triton installation for Intel GPU (#122254)
• Reuse inductor test for Intel GPU (#122866, #124147)
• Update Intel triton for Pytorch 2.4 release (#128615)
• Support reduction split for Intel GPU (#129337)
• call empty_cache for dynamo tests (#126377)
• Support xpu autocast policy (#124052)

Security

Python frontend

• warning for weights_only (#129239, #129396, #129509) (see Deprecations section)

Release Engineering

• Vulnerability related updates of packages used in CI (#124614, #124675, #124976, #124983, #125698, #126805, #126989)

This release is meant to fix the following issues (regressions / silent correctness):

Torch.compile:

• Remove runtime dependency on JAX/XLA, when importing torch.__dynamo (https://github.com/pytorch/pytorch/pull/124634)
• Hide Plan failed with a cudnnException warning (https://github.com/pytorch/pytorch/pull/125790)
• Fix CUDA memory leak (https://github.com/pytorch/pytorch/pull/124238) (https://github.com/pytorch/pytorch/pull/120756)

Distributed:

• Fix format_utils executable, which was causing it to run as a no-op (https://github.com/pytorch/pytorch/pull/123407)
• Fix regression with device_mesh in 2.3.0 during initialization causing memory spikes (https://github.com/pytorch/pytorch/pull/124780)
• Fix crash of FSDP + DTensor with ShardingStrategy.SHARD_GRAD_OP (https://github.com/pytorch/pytorch/pull/123617)
• Fix failure with distributed checkpointing + FSDP if at least 1 forward/backward pass has not been run. (https://github.com/pytorch/pytorch/pull/121544) (https://github.com/pytorch/pytorch/pull/127069)
• Fix error with distributed checkpointing + FSDP, and with use_orig_params = False and activation checkpointing (https://github.com/pytorch/pytorch/pull/124698) (https://github.com/pytorch/pytorch/pull/126935)
• Fix set_model_state_dict errors on compiled module with non-persistent buffer with distributed checkpointing (https://github.com/pytorch/pytorch/pull/125336) (https://github.com/pytorch/pytorch/pull/125337)

MPS:

• Fix data corruption when coping large (>4GiB) tensors (https://github.com/pytorch/pytorch/pull/124635)
• Fix Tensor.abs() for complex (https://github.com/pytorch/pytorch/pull/125662)

Packaging:

• Fix UTF-8 encoding on Windows .pyi files (https://github.com/pytorch/pytorch/pull/124932)
• Fix import torch failure when wheel is installed for a single user on Windows(https://github.com/pytorch/pytorch/pull/125684)
• Fix compatibility with torchdata 0.7.1 (https://github.com/pytorch/pytorch/pull/122616)
• Fix aarch64 docker publishing to https://ghcr.io (https://github.com/pytorch/pytorch/pull/125617)
• Fix performance regression an aarch64 linux (https://github.com/pytorch/builder/pull/1803)

Other:

• Fix DeepSpeed transformer extension build on ROCm (https://github.com/pytorch/pytorch/pull/121030)
• Fix kernel crash on tensor.dtype.to_complex() after ~100 calls in ipython kernel (https://github.com/pytorch/pytorch/pull/125154)

Release tracker https://github.com/pytorch/pytorch/issues/125425 contains all relevant pull requests related to this release as well as links to related issues.

PyTorch 2.3 Release notes

• Highlights
• Backwards Incompatible Changes
• Deprecations
• New Features
• Improvements
• Bug fixes
• Performance
• Documentation

Highlights

We are excited to announce the release of PyTorch® 2.3! PyTorch 2.3 offers support for user-defined Triton kernels in torch.compile, allowing for users to migrate their own Triton kernels from eager without experiencing performance complications or graph breaks. As well, Tensor Parallelism improves the experience for training Large Language Models using native PyTorch functions, which has been validated on training runs for 100B parameter models.

This release is composed of 3393 commits and 426 contributors since PyTorch 2.2. We want to sincerely thank our dedicated community for your contributions. As always, we encourage you to try these out and report any issues as we improve 2.3. More information about how to get started with the PyTorch 2-series can be found at our Getting Started page.

<table> <tr> <td> <strong>Stable</strong>

</td> <td> <strong>Beta</strong>

</td> <td> <strong>Prototype</strong>

</td> <td> <strong>Performance Improvements</strong>

</td> </tr> <tr> <td> </td> <td>User-defined Triton kernels in torch.compile </td> <td>torch.export adds new API to specify dynamic_shapes </td> <td>Weight-Only-Quantization introduced into Inductor CPU backend </td> </tr> <tr> <td> </td> <td>Tensor parallelism within PyTorch Distributed </td> <td>Asynchronous checkpoint generation </td> <td> </td> </tr> <tr> <td> </td> <td>Support for semi-structured sparsity </td> <td> </td> <td> </td> </tr> </table>

*To see a full list of public feature submissions click here.

Tracked Regressions

torch.compile on MacOS is considered unstable for 2.3 as there are known cases where it will hang (#124497)

torch.compile imports many unrelated packages when it is invoked (#123954)

This can cause significant first-time slowdown and instability when these packages are not fully compatible with PyTorch within a single process.

torch.compile is not supported on Python 3.12 (#120233)

PyTorch support for Python 3.12 in general is considered experimental. Please use Python version between 3.8 and 3.11 instead. This is an existing issue since PyTorch 2.2.

Backwards Incompatible Changes

Change default torch_function behavior to be disabled when torch_dispatch is defined (#120632)

Defining a subclass with a torch_dispatch entry will now automatically set torch_function to be disabled. This aligns better with all the use cases we’ve observed for subclasses. The main change of behavior is that the result of the torch_dispatch handler will not go through the default torch_function handler anymore, wrapping it into the current subclass. This allows in particular for your subclass to return a plain Tensor or another subclass from any op.

The original behavior can be recovered by adding the following to your Tensor subclass:

@classmethod
def __torch_function__(cls, func, types, args=(), kwargs=None):
      return super().__torch_function__(func, types, args, kwargs)

ProcessGroupNCCL removes multi-device-per-thread support from C++ level (#119099, #118674)

• Python level support was removed in 2.2.
• To simplify ProcessGroupNCCL’s code, we remove support for multiple cuda devices per thread. To our knowledge, this is not an active use case, but it adds a large burden to our codebase. If you are relying on this, there is no workaround other than rewriting your pytorch program to use one device per process or one device per thread (multi-threads per process is still supported).

Removes no_dist and coordinator_rank from public DCP API's (#121317)

As part of an overall effort to simplify our public facing API's for Distributed Checkpointing, we've decided to deprecate usage of the coordinator_rank and no_dist parameters under torch.distributed.checkpoint. In our opinion, these parameters can lead to confusion around the intended effect during API usage, and have limited value to begin with. One concrete example is here, https://github.com/pytorch/pytorch/issues/118337, where there is ambiguity in which Process Group is referenced by the coordinator rank (additional context: https://github.com/pytorch/pytorch/issues/118337). In the case of the no_dist parameter, we consider this an implementation detail which should be hidden from the user. Starting in this release, no_dist is inferred from the initialized state of the process group, assuming the intention is to use collectives if a process group is initialized, and assuming the opposite in the case it is not.

<table> <tr> <td> <strong>2.2</strong>

</td> <td> <strong>2.3</strong>

</td> </tr> <tr> <td>

# Version 2.2.2
import torch.distributed.checkpoint as dcp

dcp.save(
	state_dict={"model": model.state_dict()},
       checkpoint_id="path_to_model_checkpoint"
       no_dist=True,
       coordinator_rank=0
)
# ...
dcp.load(
	state_dict={"model": model.state_dict()},
       checkpoint_id="path_to_model_checkpoint"
       no_dist=True,
       coordinator_rank=0
)

</td> <td>

# Version 2.2.3
# no dist is assumed from pg state, and rank 0 is always coordinator.
import torch.distributed.checkpoint as dcp

dcp.save(
	state_dict={"model": model.state_dict()},
       checkpoint_id="path_to_model_checkpoint"
) 
# ...
dcp.load(
	state_dict={"model": model.state_dict()},
       checkpoint_id="path_to_model_checkpoint"
)

</td> </tr> </table>

Remove deprecated tp_mesh_dim arg (#121432)

Starting from PyTorch 2.3, parallelize_module API only accepts a DeviceMesh (the tp_mesh_dim argument has been removed). If having a N-D DeviceMesh for multi-dimensional parallelism, you can use mesh_nd["tp"] to obtain a 1-D DeviceMesh for tensor parallelism.

torch.export

• Users must pass in an nn.Module to torch.export.export. The reason is that we have several invariants the ExportedProgram that are ambiguous if the top-level object being traced is a function, such as how we guarantee that every call_function node has an nn_module_stack populated, and we offer ways to access the state_dict/parameters/buffers of the exported program. We'd like torch.export to offer strong invariants—the value proposition of export is that you can trade flexibility for stronger guarantees about your model. (#117528)
• Removed constraints in favor of dynamic_shapes (#117573, #117917, #117916, #120981, #120979)
• ExportedProgram is no longer a callable. Instead users will need to use .module() to call the ExportedProgram. This is to prevent users from treating ExportedPrograms as torch.nn.Modules as we do not plan to support all features that torch.nn.Modules have, like hooks. Instead users can create a proper torch.nn.Module through exported_program.module() and use that as a callable. (#120019, #118425, #119105)
• Remove equality_constraints from ExportedProgram as it is not used or useful anymore. Dimensions with equal constraints will now have the same symbol. (#116979)
• Remove torch._export.export in favor of torch.export.export (#119095)
• Remove CallSpec (#117671)

Enable fold_quantize by default in PT2 Export Quantization (#118701, #118605, #119425, #117797)

Previously, the PT2 Export Quantization flow did not generate quantized weight by default, but instead used fp32 weight in the quantized model in this pattern: fp32 weight -> q -> dq -> linear. Setting fold_quantize=True produces a graph with quantized weights in the quantized model in this pattern by default after convert_pt2e, and users will see a reduction in the model size: int8 weight -> dq -> linear.

<table> <tr> <td> <strong>2.2</strong>

</td> <td> <strong>2.3</strong>

</td> </tr> <tr> <td>

folded_model = convert_pt2e(model, fold_quantize=True)
non_folded_model = convert_pt2e(model)

</td> <td>

folded_model = convert_pt2e(model)
non_folded_model = convert_pt2e(model, fold_quantize=False)

</td> </tr> </table>

Remove deprecated torch.jit.quantized APIs (#118406)

All functions and classes under torch.jit.quantized will now raise an error if called/instantiated. This API has long been deprecated in favor of torch.ao.nn.quantized.

<table> <tr> <td> <strong>2.2</strong>

</td> <td> <strong>2.3</strong>

</td> </tr> <tr> <td>

# torch.jit.quantized APIs

torch.jit.quantized.quantize_rnn_cell_modules

torch.jit.quantized.quantize_rnn_modules
torch.jit.quantized.quantize_linear_modules

torch.jit.quantized.QuantizedLinear
torch.jit.QuantizedLinearFP16

torch.jit.quantized.QuantizedGRU
torch.jit.quantized.QuantizedGRUCell
torch.jit.quantized.QuantizedLSTM
torch.jit.quantized.QuantizedLSTMCell

</td> <td>

# Corresponding torch.ao.quantization APIs

torch.ao.nn.quantized.dynamic.RNNCell

torch.ao.quantization.quantize_dynamic APIs

torch.ao.nn.quantized.dynamic.Linear

torch.ao.nn.quantized.dynamic.GRU
torch.ao.nn.quantized.dynamic.GRUCell
torch.ao.nn.quantized.dynamic.LSTM

</td> </tr> </table>

Remove deprecated fbgemm operators (#112153)

TorchScript models that were exported with the deprecated torch.jit.quantized API will no longer be loadable, as the required internal operators have been removed. Please re-export your models using the newer torch.ao.quantization API instead.

Other

• Make List::get() const match List::operator[]() const (#117568)
• Delete C10_IS_TRIVIALLY_COPYABLE (#120120)
• Fix synchronization behavior for copies with type change (#121341)

Deprecations

torch.autograd.Function: Using the torch.autograd.function.traceable decorator and getting/setting torch.autograd.Function's is_traceable is now deprecated (#121413)

These decorators were previously marked for internal use only. They will be removed in version 2.4.

torch.utils.checkpoint: not passing use_reentrant explicitly to activation checkpoint and checkpoint_sequential is deprecated (#116710)

torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.

(Note that this was already deprecated in a previous release. In this version, we improve the deprecation message.)

Deprecated <code>torch.backends.cuda.sdp_kernel</code> and replace with <code>torch.nn.attention.sdpa_kernel</code> (#114689)

This PR deprecated torch.backends.cuda.sdp_kernel, users can now use torch.nn.attention.sdpa_kernel instead. The old code will raise the following warning: FutureWarning: torch.backends.cuda.sdp_kernel() is deprecated. In the future, this context manager will be removed. Please see, torch.nn.attention.sdpa_kernel() for the new context manager, with updated signature.

<table> <tr> <td> <strong>2.2</strong>

</td> <td> <strong>2.3</strong>

</td> </tr> <tr> <td>

import torch
from torch.backends.cuda import sdp_kernel

with sdp_kernel(enable_math=False, enable_flash=False, enable_mem_efficient=True):
    torch.nn.functional.scaled_dot_product_attention(...)

</td> <td>

import torch
from torch.nn.attention import sdpa_kernel, SDPBackend

with sdpa_kernel(backends=[SDPBackend.EFFICIENT_ATTENTION]):
    torch.nn.functional.scaled_dot_product_attention(...)

</td> </tr> </table>

Distributed API

• [c10d] Deprecate Work.result() (#117565)
• Deprecate torch.distributed.pipeline in favor of the PiPPy library (#121464)
• Add deprecation msg for NO_SHARD (#119553)
• Add composable API fully_shard deprecation warning (#120929)
• [DTensor] Change distribute_module input/output_fn to accept module, deprecating the input/output_fn that does not accept a module (#120895)

Releng

• Removal of macOS x86 binaries build jobs following their deprecation in 2.2 (#116726)
• CircleCI removed (#115701)

New Features

Autograd API

• Add basic autograd TORCH_LOGS support. (#115438)
• Autograd attaches logging hooks only in debug level (#116522)
• Update torch.autograd.graph logging to not print out grad_output (#116523)
• Added "any" mode to register_multi_grad_hook (#117984)

CUDA

• Add bfloat16 CUDA support for smoothl1loss (#116933), RNN (#116927), gamma unary functions (#116929), binomial distribution (#116932), and multinomial (#116951)
• Add float16 support to CUDA logaddexp2 (#116948)
• Add bfloat16 + fp16 support to fractional_max_pool for CUDA and CPU (#116950)
• Make torch.cuda.has_magma a build time check (#116299)

Distributed API

• C10d:
  • Install an excepthook which annotates exceptions with rank information when distributed is initialized (#118190)
  • Flight recorder for debugging failed collectives: (#119837, #116905, #114817, #118044, #115139, #118046, #119249, #118047 , #118142, #115771, #120063, #120331, #120544, #120262, #120724, #120270, #120893, #120502)
  • explicitly abort communicators in destroy_process_group call (#119250)
  • ProcessGroupGloo::allgather_into_tensor_coalesced (#118910)
  • ProcessGroupGloo::reduce_scatter_tensor_coalesced (#118911)
  • Create a python c10d API _set_pg_timeout to set timeout (#115453)
  • Introduce 3 low-latency, intra-node allreduce algorithms for small messages to PyTorch (#114001) (#116125)
  • Add complex support for P2P (#121240)
• Distributed Checkpointing (DCP):
  • Adds Checkpointer Wrapper for DCP [3/N] (#114603)
  • Adds async save, makes checkpointer private (#116293)
  • Makes async_save public (#121325)
  • Adds storage reader and planner classes for online loading/sharding of models in torch.save format (#119816)
  • Adds support for meta tensor loading for DCP.load_state_dict() (#113319)
  • [state_dict] Implement pin_memory and shared_memory copy for _offload_state_dict_to_cpu (#120378)
  • Add tests to demonstrate DCP checkpoint conversion (#117773)
  • Adds utility for converting dcp to torch save format (#119814), converting torch save to dcp (#119815)
  • Add distributed checkpoint support for custom device (#120201)
  • Add a profiler function for benchmarking save and load (#116007)
  • Enables load balancing duplicates in DCP (#116469)
  • Enable filesystem/fsspec auto detection (#118888)
• DTensor:
  • Add op support for aten.gather.default (#118513)
  • Add op support for nll_loss_backward (#119256)
  • Implement scaled dot product attention (flash-attention, forward only) (#120298)
  • Add async_op option to redistribute and some refactor (#121477)
  • [TP] add support for loss parallel (#119877)
  • [TP] Introduce Sequence Parallel Style for Laynorm/RMSNorm/Dropout (#121295)
  • allow OpStrategy to represent ops whose return type is a tuple (#115682)
  • Add op support for nll_loss_forward (#118917)
  • Supported foreach=False for clip_grad_norm_ (#120238)
  • Supported foreach=True for clip_grad_norm_ (#120910)
  • Add layer norm backward support (#115683)
  • Add Silu operator support (#118702)
  • Enable adadelta foreach optimizer (#115564)
  • Enable radam foreach optimizer (#115566)
  • Enable Adamax foreach optimizer (#119850)
  • Make add_.Tensor/div_.Scalar to be linear pointwise instead (#121294)
  • [DeviceMesh] Allow 1d slice from 1d mesh (#118895)
  • Add a implicit replication flag (#115297)
  • Adds tool to visualize sharding (#114307)
  • Initialized RNG tracker if needed (#121328)
  • get CommsDebugMode to work with DTensor (#118769)
  • [DTensor][XLA] support XLA backend in distribute_module API (#121355)
• DDP
  • Use compiled_autograd to trace DDP backward allreduce (#110662)
• Functional Collectives
  • Support broadcast in native funcol (#119229)
  • Allow using native c10d_functional via _functional_collectives (#113057)
  • Implements permute_tensor in functional collectives (#115078)
  • Support tracing native functional collective via python APIs (#119103)
  • Directly import DeviceMesh to avoid circular dependency (#115649)
  • Port all_to_all_single to native c10d_functional (#113438)
  • Make native c10d_functional ops work with AOTInductor (#113735)
  • fix an issue where mutation on views fails in inductor (#118333)
  • Run device mesh tests with native funcol enabled (#118437)
  • Change the .clone() in native funcol's all_reduce to use at::MemoryFormat::Contiguous (#120042 )
  • Make tests using CommDebugMode work for both legacy and native funcol (#120070)
  • Temporarily support ranks + tag as pg identifier in native funcol (#120226)
  • don't import torchdynamo when running torchdeploy (#120900)
  • Preliminary DeviceMesh + native c10d functional integration (#118423)
  • Change native funcol inductor tests to use fake pg (#119104)
  • Prepare test_inductor_collectives.py for native funcol migration (#120025)
  • Prepare test_dtensor.py for native funcol migration (#120043)
  • Disable GroupRegistry's thread isolation by default (#121457)

FX

• Experimental non-strict mode (#114658)
• Add _assert_scalar and teach Inductor to codegen it (#114148)
• Add symbol_guard_limit_before_specialize (#119347)
• Add _assert_scalar and teach Inductor to codegen it (#114148)

torch.compile

Dynamo

• Mutable variable trackers - TorchDynamo variable trackers are now mutable. Therefore, we can just mutate the variable tracker during symbolic tracing, instead of creating a new variable tracker. This improves compilation time for models with frequent mutations.
• (beta feature) Improved automatic deletion of compilation units - TorchDynamo caches the compilation units internally on every compilation event. Earlier, this cache was not cleared automatically when the original nn module went out of scope, holding on to the GPU memory. Therefore, users needed to call torch._dynamo.reset() to clear the cache. Now, we automatically clear the cache as soon as the nn module goes out of scope.
• (beta feature) torch.compile works with User defined triton kernels - Allows for PyTorch code that contains a triton kernel to be executed natively using torch.compile. This will allow users to migrate code containing triton kernels from eager PyTorch to using torch.compile without running into performance complications or graph breaks. A good entry point to how it works is on https://github.com/pytorch/pytorch/blob/main/torch/_higher_order_ops/triton_kernel_wrap.py
• (beta feature) Improved tracing rules infrastructure - Added a central infrastructure to control whether to inline or skip TorchDynamo tracing.
• Create a sentinel file for each dynamo test skips (Part 1) (#120500)
• Windows Dynamo Error Removal CI Check (#115969)
• Add TORCHDYNAMO_EXTENDED_DEBUG_GUARD_ADDED (#118750)
• feat: Add min, max ranges to mark_dynamic API (#119737)
• Add support for dynamic shapes in round (#115259)
• Add support for torch.cond in vmap (#114523)
• Added dynamic shapes support for math trigo ops: sin(h), cos(h), tan(h) ... (#114866)
• Add dynamo support for operator.abs (#117442)
• Add support for operator.truth (#117463)
• Add support for compiling SDPAParams (#117207)
• Add more dynamo call_methods and getattr support or Placement (#117733)
• Add size(), get_coordinate() support for DeviceMesh in dynamo (#117710)
• Initial torchbind support in PT2 (#117697)
• Add compilable foreach RAdam support (#117912)
• [HigherOrderOp] support while_loop in dynamo (#116913)
• Add Support for CausalBias to torch compile (#116071)
• support dict.clear() (#119197)
• Add support for labels to ttir analysis (#119836)
• Add jacrev support in torch.compile (#121146)
• Introduce size oblivious guards (#118579)

Inductor

• Some basic support for uint{16,32,64} codegen in CPU inductor (#116810)
• Enable for MacOS (#118076)
• Add torch.cond support to JIT Inductor (#119759) and AOT Inductor (#121120)
• Double buffering for Weights (#114446)
• Add Runtime Constant-folding for AOTInductor (#118765)
• Intel GPU backend Upstream: Generalize device-bias code in code generation (#116020), Register and add Intel GPU Inductor backend (#116330)

torch.export

• Introduced pre-dispatch IR through torch.export._trace._export(..., predispatch=True). This returns a graph that contains ATen operators at a higher level (aten.linear.default will not become decomposed) and are also safe for training (#117278)
• Support higher order op functionalization in predispatch IR (#115314)
• Ignore autograd ops for predispatch export (#116527)
• Introduced non-strict export (#118607, #118609, #119297, #119446, #119602)
• Transformed global state mutating operators torch._C.set_grad_enabled into a higher order operator (#119732, #119736, #119810, #119913, #119915)
• Added while_loop support (#116823)
• A very preliminary torchbind object tracing support was added. (#116985, #117978, #117979, #118158, #118684)

Linalg

• Add HIPSolver backend to linalg.eigh (#115177)
• Add MKLDNN backend to matmul for bfloat32 (#116015)
• Support torch.mm with conjugate transposed inputs (#117238)

MPS

• Add complex_out to MPS backend (#110851)
• Add Conv3D support for MPS (#114183)
• Enable select/[broad]cast ops for complex dtypes (#115727)
• Add torch.fft. support (#119670)
• Add support for 64-bit index operations (#116942)

torch.nn API

• Added swap_tensors path to nn.Module._apply (#117167)
• Integrated swap_tensors into nn.Module.load_state_dict ((#117913)
• Added assign argument to torch.Tensor.module_load (#121158)
• Added an attention bias subclass for a lower right causal masking (#114823)
• Added config to disable TransformerEncoder/MultiHeadAttention fastpath (#112212)

Profiler

• Enable profiler initialization to enable on-demand profiling for CPU only mode (#118320)
• Add execution_trace_observer as an optional argument to Profiler API (#119912)

Python API

• Add new torch.utils.swap_tensors function that can be used to exchange Tensor while preserving references from other objects (#111747)
• Add unsigned integer dtypes with full interop with third party (#116594, #116805, #116806, #116807, #116808, #116803, #116804)
• Add {Untyped,Typed}Storage.resizable() to check if a Storage can be resized (#119286)
• add Half support for flash attention (#119247)

Sparse

• Add bfloat16 support to torch.sparse.addmm for CPU (#115535)
• Add tune_bsr_dense_addmm as an API to find optimal triton kernel parameters for bsr_dense_addmm (#115499)
• Add values backward support for sparse CSR, CSC, BSR, and BSC tensors (#115586)
• Add batched sparse CSR/CSC/BSR/BSC to sparse COO conversion support (#116206)
• Add meta device support to sparse compressed tensors (#120498)
• Add sparse compressed meta tensor support (#120707)
• Add sparse compressed fake tensor support (#120920)
• [semi-structured] Enable fp32 support, separate sparse and dense constraints (#115550)
• Support add(sparse_compressed, dense) (#115432)
• Support csc layout for add sparse/dense. (#115433)
• Add in out_dtype support (i8i8->bf16, i32) for cusparselt (#119296)

Vulkan

• Added Vulkan support for 1D for the following ops: Convolution (#117780, #118660, #118833, #118834, #118835) Linear (#118690)

XPU

• Intel GPU Runtime Upstreaming for Device (#116019, #116833, #116850, #116869, #117611, #117619), Event (#117734), Device Allocator (#118091), Guard (#118523), and Generator (#118528, #118613)

Other

• Enable compiled Adam in the benchmarks (#116093)
• [ROCm] TunableOp (#114894)
• [pytree] Add access api (#117771)

Improvements

Autograd API

• Out-variant of ops for which autograd also does not have a formula now produce an improved error message when the out= argument is a leaf tensor (#121089)
• Support custom autograd Function forward AD return non-Tensor in forward (#118234)
• In-place foreach to returns the input list as-is instead of None (#121405)

Composability

• FakeTensors and meta tensors are used to perform shape propagating when tracing out a graph in torch.compile. There were a number of op coverage improvements this release:
  • New metas
    • _foreach_norm (#119927)
    • _upsample_bicubic2d_aa (#117347)
  • Fixes to metas
    • _efficient_attention_forward for jagged inputs (#118657)
    • _flash_attention_forward() (#119812)
    • efficient_attention_forward fix for NT inputs (#120594)
• We have python “reference” decompositions for many aten operators. These are used during the tracing step of torch.compile in a few ways: sometimes they are used to directly decompose operators in the captured graph. Other times, they are used as an alternative to a shape-propagation rule for an operator. There were several improvements to operator coverage in this release
  • New decomps
    • reflection_pad{1, 2, 3}d (#115100)
    • replication_pad (#115113)
    • torch.block_diag (#115096)
    • torch.take (#114813)
    • rsub (#118288)
    • roll (#119857)
    • frexp (#119217)
    • pad_sequence (#116285)
    • linalg.cross (#119809)
  • Fixes to some decomps:
    • SDPA decomp: actually use attn_mask (#117579)
    • aten.diag_embed (#120549)
    • Isin (#120821)
    • linalg norm (#120993)
    • Im2col: Remove opmath cast (#121363)
    • Index_copy: fix 0-dim Index (#117065)
    • Rrelu_with_noise: add default parameters (#117141)
    • _refs.linalg.svd: don’t check is_conj() (#117972)
    • Weight norm interface decomp: fix missing default dim param (#118762)
    • SDPA decomposition: add tests for different dtypes (#119239)
    • _to_copy fix (#119868)
    • torch.Tensor decomp fix to support sequences of tensors (#120872)
    • Avoid mkldnn ops appearing in graph in some cases (#115448)
  • Decomps relevant to the Core ATen opset defined here.
    • Add decomp pixel_shuffle/unshuffle (#118239), (#118921), (#120092)
    • Modify SDPA decomposition to decompose _scaled_dot_product_flash_attention_for_cpu (#117097)

Dynamic shapes:

• Improved unbacked SymInt support
  • Support symbolic min/max on unbacked SymInt (#118953)
  • Rewrite maybe_reduce more carefully for unbacked SymInt (#119562)
  • Size oblivious test for slice optimization (#119625)
  • If data dependent, check if guard_size_oblivious would fix problem and report if so (#121011)
  • Prevent unbacked symbol reallocation by forcing unification for unbacked symbol def sites (#114368)
• Improved dynamic shapes support
  • Always accept 0-d scalar tensors as int, even if index fails (#117451)
  • SymIntify prod_backward (#120776)
  • Add is_integer to SymFloat (#114703)
  • Dedupe symbolic shapes in tracing (#116158)
  • [Dynamic] Fix dynamic shape size inspection bug (#120341)
• Docs / debugging tools
  • Add basic reference documentation for symbolic_shapes.py (#118997)
  • Rename unbacked SymInt prefix to u (#117859)
  • Augment create_symbol with user/infra backtrace fragment (#118215)

CPP API

• Add TensorIteratorConfig::add_const_input to avoid COW materialize (#118053)
• Reserve sizes in c10::VaryingShape::concrete_sizes(), c10::TensorType::computeStrideProps() (#119189)

CUDA

• Faster gc_count update for CUDACachingAllocator (and avoid nullptr de… (#117064)
• Reduce register usage of fused adam(w) (#118361)
• Improve CUDACachingAllocator lock contention (#118550)
• Back scalar value to pinned memory for .item() (#119202)
• Avoid COW materialization for TensorInfo with const type (#119502)
• [CUDA graphs] Pool argument for make_graphed_callables (#121475)
• [CUDA Caching Allocator] Export sync-stream-and-free-HBM counter in memory_stats for performance debugging (#120050)

Distributed API

• c10d:
  • ProcessGroup/NCCL logging improvements: (#115801, #116059, #116060, #116520, #117291, #117868, #118335, #113238, #118455, #118582, #118924, #116489 )
  • NCCL ProcessGroup watchdog improvements: (#115403, #115577, #116702, #116267, #116717, #116545, #117312, #117093, #117682, #117297, #118016, #115770, #116661, #118344, #117699, #117168, #117738, #121132
  • Use TCPStore to record NCCL timeout and dump debug info (#115226)
  • Extend NCCL communicator splitting to more use cases (#114916)
  • Let all_reduce_coalesced accept one tensor as well (#115650)
  • Add stream info during nccl comm abort call (#116076)
  • Pass group global rank information to NCCL PG (#114736)
  • Store PG global rank information in tracing logs (#115730)
  • Only open NCCL dump pipe file once per process (#115798)
  • Dynamo + collectives: allreduce remap (#115950), all_gather_into_tensor remap (#117224)
  • Expose check method to Python for store via pybind (#116144)
  • Refactor all_reduce variants as private methods (#120855)
  • To make ProcessGroupNCCL to use globalStore for coordination (#117075)
  • Allow nonblocking wrap of ncclCommInitRankConfig (#118256)
  • Do not print NCCL_DEBUG before NCCL init (#117328)
  • Update the work progress of PG periodically (#120438)
  • Add NCCL work sequence number to work info (#120596)
  • [UCC] Retain CUDA context in progress_loop (#121446)
  • Change watchdog log from "NCCL" to "Process group" (#118121)
  • [IntraNodeComm] accept P2P buffer size as constructor argument (#120856)
• FSDP:
  • [torch.compile] FSDP changes (#115497)
  • Remove unused flat_param_part_view (#117082)
  • Replace acc_grad hooking with register_post_accumulate_grad_hook on flat_param (#112184)
  • Fix deprecation warning on typed storage (#116714)
  • Pass DTensor shape/stride during tensor unflatten in 2D (#117340)
  • Cloned unsharded tensor slice in optim state dict load (#117261)
  • Drop all gather stats to debug not warning (#117669)
  • Idempotent reshard (#117997)
  • Removed .detach in clip_grad_norm_ (#120612)
  • Vlean up unwanted _fsdp_wrapped_module FQNs (#120600)
• Distributed Checkpointing (DCP):
  • Automatically set no_dist if distributed is unavailable (#119813)
  • Let distributed_state_dict filter out the compiler prefix (#119830)
  • Only wait on AsyncCollectiveTensor after DTensor-based state dict loading (#119716)
  • Improve the readability of filesystem and fsspec filesystem (#116246)
  • Uses Serial Loader for DCP.save when more then one thread is used. (#118114)
  • Skip log line if no tensors were dedupped (DCP) (#119742)
  • Removes Checkpoint Wrapped Prefix from state dict fqns (#118119)
  • [DCP] Replaced storage() with untyped_storage() (#121538)
  • Let _offload_state_dict_to_cpu to return the companion_obj if it exist. (#121273)
  • Asserts CPU backend for async_save (#120241)
  • Allow users to save and load without creating storage reader and writer (#117772)
  • Passes process group to _all_gather_keys in dcp.load (#118301)
• DTensor:
  • [DeviceMesh] Ensure mesh tensor is a cpu tensor (#120046)
  • Standardize tuple strategy handling for foreach ops (#120695)
  • Add mesh_dim_names to DeviceMesh repr if it exists (#115579)
  • Remove assert to allow tensor sharding dimension < Shard(x).ndim (#115114)
  • Refactor sharding cost model to count for latency (#119897)
  • Make tensor_flatten more compatible for dynamo getattr (#118209)
  • DTensor: use memory_format in the hash for all aten ops that use that arg (e.g. aten.clone) (#118667)
  • [DeviceMesh] Removed print of self._dim_group_infos (#118527)
  • Account for empty list when turning to OpStrategy (#115298)
  • [debug] have visualize_sharding correctly print for sub-mesh DTensor (#121216)
  • switch softmax backward ops to OpStrategy (#119255)
  • Make DTensor from_local backward partial() to replicate() pass through (#115967)
  • Refactor partial redistribution logic (#113334)
  • Refactor redistribute and fix uneven sharding redistribution (#115525)
  • Switch softmax forward ops to OpStrategy (#117723)
  • Simplify outputs wrapping handling (#120297)
  • Relaxed to_local requires_grad warning (#118186)
  • Make local_shard_size_on_dim be staticmethod (#118078)
• TorchElastic:
  • Support for overprovisioning in C10 based rendezvous (#117066)
  • [rendezvous] Add option to enable libuv for TCPStore based rendezvous backend (#118944)
  • Create root log directory by default (#121257)
  • Refactoring to support non-default logging strategy (#120691)
  • [Logging] Pluggable logsspecs using python entrypoints and option to specify one by name. (#120942)
  • Refactor SubprocessHandler to separate module for easier subclass (#120373)

torch.compile

Dynamo

• Fewer graph breaks for dicts - ConstDictVariableTracker now supports many more types of keys, earlier it was just string and ints.
• Improved TorchDynamo reliability by fixing many bugs, increasing the pass rate of TorchDynamo wrapped PyTorch tests to 90%.
• Improve TORCHDYNAMO_EXTENDED_DEBUG for GuardOnDataDependentSymNode (#119412)
• Make torch._dynamo.mark_static work inside graph (#118962)
• Expand dynamic dims support for traceable subclasses (#114311)
• Extend auto_functionalized to support ops that return Tensors (#115135)
• Improve support for dynamic shapes str.format and _assert (#115203)
• make __lookup_backend return None when cache misses (#114766)
• Make subclass type instances constants (like UserDefinedClasses) (#115323)
• [HigherOrderOp] make MapHigherOrder use should_flatten_output=True (#115204)
• Move the shape env symint cache to a symbol cache, better routing for subclass fakification [re-pr 115227] (#115396)
• Remove always restore (#115317)
• Remove replace_all and make VTs mutable (#113725)
• Support torch function user objects (#111765)
• Check tensor subclass when using torch.compile + SAC (#115960)
• Ensure wrapping subclasses with as_subclass is supported (#116091)
• Add CALL_FINALLY opcode (#116159)
• Add a wrapper to transform a NumPy function into a PyTorch function (#114610)
• [HigherOrderOp] set set_subgraph_inputs to flatten_manual for map (#115853)
• Graphbreak when creating a map with unsupported keys (#116460)
• Specialize SymNodeVariable when used as module index (#114377)
• Impl. call_hasattr for BaseUserFunctionVariable (#116049)
• [HigherOrderOp] change signature of map_impl (#117161)
• Error if compiled nondeterministic backward called in deterministic mode (#114780)
• Add hasattr support for TupleVariable (#117694)
• Break on unsupported keys for dicts / elements for sets (#117630)
• Implement set in terms of dict (#110524)
• Add DictView variable tracker (#108420)
• add common methods to DistributedVariable (#117590)
• make ConstantSource propagate through built-in ops for TensorVariable (#117704)
• Add compilable and capturable foreach adamax with tests (#117835)
• Enhance torch.vmap support from inside torch.compile (#116050)
• Install module globals per output_graph (#117998)
• Remove optimizer.step patching for profiler hook (#115772)
• add username in debug path (#117820)
• avoid graph break on tensor.element_size() (#118229)
• avoid graph break on torch.backends.cuda.matmul.allow_tf32 (#118236)
• move torch._C._get_cublas_allow_tf32 to constant_fold_functions (#118342)
• inline torch.jit._unwrap_optional (#118434)
• support inference_mode with no arguments (#118427)
• constant fold torch.cuda.get_device_properties to avoid graph break (#118422)
• Faster empty LIST_LENGTH guard (#118542)
• Expose dynamic_shapes api at multiple levels (#118695)
• graph break on isinstance calls if we don't know the type (#118778)
• Print the malformed guard when there's a guard error. (#117982)
• Use SourcelesBuilder in BuiltinVariable (#118098)
• [optim] Place guards on the args before assuming they exist (#117983)
• Make variables in dict LazyTrackers (not lazily guarded yet) and avoid using DICT_KEYS guard (#117625)
• Make dict guards amenable to the CSE pass (#118194)
• Add Typing variable to possible dict keys (#118003)
• Don't assume all subclasses of BaseUserFunctionVariable have a fn attribute (#118208)
• Add functools.partial and UserDefinedFunction to dict keys (#118199)
• [optim] Use the actual sources from the parameters when tracing "params" in an optimizer (#118535)
• Optimize dict keys guard when all the keys are constant (#118855)
• bypass graph break due to masking if inference mode (#119056)
• decrease logging level for graph break in higher order op. (#119079)
• Add torch.backends.mha.get_fastpath_enabled to FUNC_INLINELIST (#118979)
• support comparing stream with constant (#119199)
• Functools partial reconstruct (#118583)
• Print the value of constants in str (#119276)
• inlining into iter of user defined object (#119243)
• Support kwargs for lazy module (#119445)
• In dynamo tracing for index() use None as the default indicator for end and not -1 (#119151)
• Capture untyped_storage().resize_() (#119647)
• Respect autograd.Function + multiple save_for_backward calls (#117667)
• Support attribute access on tensor subclasses without sources (#117666)
• [functional_collectives] Add all_to_all_single, all_gather_list, reduce_scatter_list to dynamo remapping (#119683)
• [Optimus] Log the optimus graph transformation to the scuba (#119745)
• Update tracing rules for new cudnn functions (#120268)
• [guards-cpp-refactor] WEAKREF_ALIVE guard (#120344), DictGuardManager (#120359)
• derived dim (#118729)
• Let torch dynamo inline torch.func.grad (#118407)
• Teach dynamo about vjp (#119405)
• Support module backwards hooks (#120685)
• DICT_CONTAINS guard (#120673)
• KeyValueDictGuardManager (#121147)
• Re-dispatch torch.Tensor.new into torch.Tensor.new_empty method. (#121075)
• guard on grads being None in compiled optimizers (#121291)
• Use type check for also is_not (#113859)
• Relax missing symbols runtime assert (#121339)
• Add operator length hint support (#121495)
• Improve Dynamo support for torch function and class methods in general (#121365)
• Switch cudagraph backend to cudagraph trees (#121019)
• Support _unsafe_set_version_counter (#121086)

Inductor

• [custom ops] Add tag to custom ops to preserve stride orders in inductor (#117298)
• Fix argument unused during compilation warning (#118077)
• Add a decomposition for isin() (#115390)
• Add no weight change version of fuse_parallel_linear (#115791)
• Allow user to explicitly specify Device to run on (#117413)
• Add torch._export.aot_load (#117610)
• Support .item() in the ABI-compatible mode (#117989)
• Add _scaled_dot_product_efficient_attention to C shim (#118169)
• Support scalar to tensor in the ABI-compatible mode (#118024)
• Replicate split_cat from torch IR to predispatch IR" (#118590)
• Change the cpp wrapper codegen for sdpa (#120592)
• Store OpOverload in ir.ExternKernel (#120629)
• Use torchgen to generate C shim functions (#120513)
• Update cpp wrapper codegen to use v2 C shim (#120714)
• Reuse generated kernels between constant graph and main graph (#121564)
• Update AOTI runner util (#116971)
• Remove caching for compiled model.so (#117087)
• Retrieve original FQNs for weights (#116157)
• Enable Dequant Promotion when Linear input dimension size exceeds 2 (#113912)
• Enable QLinear weight prepack when input dimension size exceeds 2 (#113928)
• Make some improvements to FX graph caching (#117888)
• Fuse parallel linear based on pre grad aten IR (#114776)
• Fuse pointwise operators in the post grad (#114778)
• Enable mkldnn op weight pre-packing on aarch64 (#115037)
• Add sub and div pointwise ops to the post grad fusion (#115389)
• [optimus] enable smart fusion (#115471)
• Parameterize ir.Scan on combine_fn (#109132)
• Add input numel assert for minimal arrayref interface (#113577)
• Remove ArrayRefTensor::dtype (#113578)
• Added non-integer expr support for floordiv in triton codegen (#115751)
• Updated upsample_bilinear2d decomposition (#104182)
• Consolidate usage of fp8 linears for inference models (#115808)
• Add lowerings for reflection_pad{1, 3}d_backward (#115645)
• Support Predispatch functionalization (#113728)
• Support sym exprs in lowering constant promotion (#116196)
• Add Support For Symbolic Shapes in Register_replacement, SDPA Pattern Matching (#115441)
• Handle more edge cases in slice and slice_scatter (#117377)
• Add statically_known_true utility for SymBool (#117359)
• Allow explicit shutdown of the compile-worker pools (#117664)
• Add runtime numeric check for pt2 Optimus in the pre grad pass (#115142)
• Allow reinplacing before meta-only users (#117121)
• Allow reinplacing functionalized scatter ops (#116899)
• Handle cum{sum,prod} on zero-dim tensors (#117990)
• Use an op counter to decide when to realize a kernel (#117030)
• Remove follow_imports = skip from sympy (#118469)
• Complete decomposition for aten.round (#118635)
• Never reuse accumulated gradients' buffers (#119334)
• Vectorization support for int32/int64 (#119001)
• Update the compile options for CppPythonBindingsCodeCache (#119415)
• Add CUDAEvent recording for constant folding to show up. (#119216)
• Decompose torch.ops.higher_order.auto_functionalized in Inductor (#118673)
• Support storage resizing (#119749)
• Use torch.cuda.clock_rate instead of triton.testing.nvsmi (#118662)
• Handle aliases correctly in foreach (#119508)
• Add Runtime Constant-Folding function of AOTInductor for AOTInductorModels used internally. (#119823)
• Simplify indexing when doing ModularIndexing + index propagation. (#119863)
• Add split cat pattern to remove cat nodes (#115004)
• Decompose memory bound mm (#120047)
• [cond] make sure subgraphs in cond are decomposed according to current decomp table (#120366)
• Use a dtype property in torch inductor nodes (#119227)
• Add unbind node normalization (#120253)
• Reinplace auto_functionalized (#120829)
• Change the split cat log to debug (#120823)
• Add decompostition for mm in backward (#120933)
• Do not use warm_pool() if TorchTnT is used (#121047)
• Triage the remaining fallbacks (#121312)
• Enable ABI-compatible mode for cpp-wrapper JIT (#121309)
• Change assertion throw to error message for const_run_impl call. (#121396)
• Split predispatch pass into multiple passes (#121592)
• Port remove_split_ops to PT2 pre-grad passes (#121674)
• Replace lld with the default ld linker (#115478)
• Emit static constexpr int array vars when possible (#112174)
• Avoid aoti_torch_data_ptr calls for constants at inference time (#112405)
• Use static_cast, not dynamic_cast (#112798)
• Add minimal arrayref interface (#112800)
• Add updaing constant buffer to active buffer. (#116001)
• Add aoti_torch_view_dtype in C shim (#118705)
• Support _embedding_bag in C shim (#118706)
• Skip launching kernels with zero grid in AOTInductor when using backed symints (#118654)
• Support copy_, _fft_c2c and view_as_real in C shim (#119125)
• Migrate fuse_split_linear_add from dper_pass to AOTI based on predispatch IR (#118983)
• Add C-shim for index_put (#116667)
• Port fuse_parallel_linear (without changing weights) to PT2 pre-grad (#121617)
• [OAT] move matmul precision out of system info (#115242)
• [OAT] toggle for forcing matmul precision matching (#115326)
• Remove hashing of tensor data for constants (#115356)
• [Triton] Replace triton.runtime.jit.get_cuda_stream with torch.cuda.c… (#115397)
• De-duplicate triton helper functions (#115546)
• Don't print disable_cudagraphs_reason when cudagraphs is disabled (#115489)
• Implement a deduplist data structure for name to user tracking (#115609)
• SDPA extend backward realized tensor alignment checking to forward realized tensors (#116069)
• Serve multistream graph captures from correct pool (#116199)
• Add input shape check for quantized conv binary lowering (#115247)
• Preserve strides of custom Triton kernel args (#116219)
• Add ABI shim function for torch.scatter_reduce (#116700)
• Use max sm clock when calculating device tflops (#116754)
• Replace recursive stable_topological_sort() with iterative. (#116761)
• Remove the float16 restriction for cpu cpp_wrapper (#116205)
• Sort unbacked symbols before iterating on them (#116421)
• Decompose bmm if batch2's last dim size is 1 and coordinate_descent_tuning is enabled (#116582)
• Control the cpp_wrapper mode with an env variable (#116615)
• Add shape checks to ExpandView (#113839)
• Add remaining user check for qconv binary fusion (#115809)
• add predispatch_pass to hold pass functions to be run when config.is_predispatch is true (#116788)
• [ROCm] Add opt-in option for inductor's layout optimisation on ROCm (#116329)
• Disable pointwise_cat on CPU (#116313)
• Don't access cluster_dims for too old version of triton (#117192)
• Check nan/inf for graph inputs (#117189)
• Iterative percolate tags (#117306)
• Update JIT Inductor cpp wrapper entry function signature (#119280)
• Make auto_functionalized HOP fallback in inductor (#117084)
• Realize non-ReinterpretView Views in custom Triton kernel args (#117468)
• Add torch.complex128 and torch.complex32 to DTYPE_TO_ATEN dictionary. (#117929)
• correctly retrieve the "shared" attribute from a Triton binary (#120666)
• Add lowering for adaptive_max_pool2d (#120254)
• Track constant's original_fqn mapping (#120524)
• enable fp8 cast for inductor CPU (#117737)
• Express y grid > 2^16 in terms of z grid (#121554)
• Use codegen reference for buffer to string (#117838)
• move disable_cudagraph_reason disabling after codecache is accessed (#117823)
• Add new pattern matchers for SDPA (#113004)
• sympy.Symbol is a subclass of sympy.Expr (#117857)
• For View.create(x, sizes) call realize_input() instead of realize() when handling unbacked symints (#117013)
• [ac][pattern matcher] Do not percolate tags beyond the inputs of matched portion (#118034)
• Add lowering to special.bessel_j0 (2nd try) (#118565)
• Dont fuse write into read if indexing differs (#118210)
• Handle special values correctly in ir.Scan codegen (#118788)
• Limit reductions into pointwise cat fusion (#118452)
• Update pointwise concat heuristics (#118453)
• Add lowering to special.bessel_j1 (#118992)
• Support ProxyExecutor argument codegen for sympy.Expr (#119166)
• Implementing missing magic methods on IR values. (#118933)
• Support sympy.expr in user-defined Triton kernel grid fn (#119165)
• Add lowering to special.modified_bessel_i0 (#118993)
• Add split scan kernel (#117992)
• Add lowerings to special functions (#119187)
• Use list comprehension to initialize unused_views. (#119618)
• Rewrite group_batch_fusion.find_independent_subset_greedy() to be iterative. (#118324)
• Recursivly unwrap_storage_for_input when convert_to_reinterpret_view fails (#119867)
• Replace generators with map. (#119818)
• Reorder if check to avoid more expensive check. (#119817)
• [scheduler] Use set for origin (#119861)
• Allow padding mm/bmm/addmm in the presence of dynamic dims (#120073)
• Enhance next_power_of_2 function (#120153)
• Always allow 64 bit in next_power_of_2 (#120164)
• Use two pass reduction for deterministic reduction order (#115620)
• Remove redundant to_dtype in Fused Schedular Nodes (#118365)
• Remove dependency of triton during inductor codegen (#120193)
• Pass device_str for async_compile.triton function (#120202)
• Colorization improvements for bandwidth profiler (#120343)
• Disable masked load for non-fp data types (#120558)
• Apply fx passes recursively to nested subgraphs (#120665)
• Make triton_meta be part of user defined triton kernel cache (#120809)
• Emit grid wrapper inlined with the user defined triton kernel (#120824)
• Add lowering for fraction_max_pool2d (#120460)
• Fix accuracy failure for a few models under freezing (#121054)
• Add a decomposition for torch.put, 2. (#120179)
• Skip foreach kernel for benchmark fusion (#121168)
• Move JK check to on-demand (#121182)
• Correctly read the cache key for remote cache (#121151)
• Make configs hash part of remote cache key (#121152)
• Fuse nodes with sizes (s0_s1_...,) and (s0, s1, s2, ...) (#120077)
• Use indices for constants in triton_meta (#121427)
• Skip welford combine on first reduciton loop iteration (#121488)
• Changes to support newer triton pin (#121267)

torch.export

• Added effect token to export (#121424)
• Preserve constant fqn (#120664)
• Require pytree serialized_type_name (#120636)
• Added 'is_lifted_tensor_constant' and 'get_lifted_tensor_constant' utils (#120546)
• Use forward hooks to capture module signatures. (#120468)
• Allow str inputs in non-strict tracing (#120536)
• Support output types that are non tensors (#120804)
• Allow None as the meta value for tensor output. (#116664)
• Make spec comparison indifferent to fx collections (#118718)
• Support non-tensor tuple hoo outputs (#119402)
• Make control flow operators respect global decomp table (#120412)
• Make balance_gradient preserved in export (#120332)
• Ensure optional fields in the schema always have default value. (#121163)

FX

• Ignore ill-formed solution of reduce_inequalities (#117310)
• Add an option to not retrace when doing op fusion (#118120)
• [minimizer] Defined traverse (#118889)
• [pytree] Properly register immutable collections (#120036)
• Skip less replacements (#119570)
• Refine value ranges on inequalities (#120800)
• Remove dead get_shape_groups (#120813)
• Simplify guards using info from previous guards (#121463)
• Inspect get_attr nodes for _decline_if_input_dtype (#118760)
• Optimize recursive_add_node in fx splitter (#117969)
• Slightly faster FX graph iterator (#121611)
• More strong typed codegen for partial specialized code on boolean (#117201)
• Add torch.fx.interpreter to uninteresting_files (#117460)
• Report function name in stack trace annotations (#117459)
• Cache dfs path in propose_partitions and re-use that later when trying to find cycles in the graph (#115943)
• [pytree] update treespec num_children access (#116370)
• [pytree] Allow tree_map_only to support predicate function as filter (#119974)
• Register torch.return_types in torch.fx._pytree (#120027)
• [pytree] Add key path api (#116786)
• [pytree] Reuse flatten_fn in flatten_with_keys_fn to ensure consistency (#117656)

JIT

• Release the GIL in serialization when it is safe to do so (#120818)
• Improve support for boolean inputs to operators in TorchScript (#113835)

NestedTensors

• Support ragged_idx != 1 on aten::is_same_size, aten::_to_copy (#118442)
• view: basic support for ragged_idx != 1 and _unsafe_view (#118317)
• Support nested tensor in check_trace (#121039)
• Add is_nested_int() (#119975)
• Rename singleton int to nested int (#119661)

Linalg

• Avoid COW materialization in at::parallel_for/parallel_reduce (#120455), TensorAccessors with const type (#119501), and input materialization in more forward ops (#121070)
• [executorch] Run llama in xplat (#118831)

MPS

• Add MacOS 14 runtime check (#115512)
• Add support for MPSDataTypeComplexFloat[16|32] (#115513)
• Fix sum and prod for complex types (#115554)
• Add support for MPSDataTypeComplexFloat[16|32] (#115513)
• Fix sum and prod for complex types (#115554)
• Enable torch.rand[n] for complex types (#115514)
• Implement aten::upsample_linear1d on mps (#115031)
• Speedup addmm (#116548)
• Enable bfloat16 support on MacOS 14 (#119641)
• Add naive std_mean implementation (#119777)
• Implement aten::upsample_linear1d on mps (#115031)
• Use dispatch with rethrow for indexing (#116903)
• Add function to materialize COW storages (#117053)
• Make addmm support empty matmul (#117223)

torch.nn API

• Add python and C++ support for LPPool3d (#114199)
• Add compatibility with channels_last_3d for conv3d (#114790)
• Add Half support for interpolate operators on CPU (#105648)
• Add nn.Module.to_empty() suggestion in the error message (#119353)
• Explicitly set nn.Module.set_extra_state return type to None (#120161)
• Updated nn.Module._apply to not gate on should_use_set_data when swap_tensors is set (#120659)
• Add Half support for AvgPool2d on CPU (#109578)
• Fix AdaptiveAvgPool1D to account for shmem limit for certain input sizes (#115231)
• Add back initial Flash Attention support on ROCM (#115981)
• Add 64-bit indexing for CUDA avg_pool_backward (#114193)
• Add Half support for layer_norm on CPU (#99590)
• Add Half support for flash attention on CPU (#118368)

ONNX

• Introduce decomposition skips using custom operator (#117314)
• Apply Modularizarion to ExportedProgram during ONNX Export (#119498)
• Disable opmath type promotion for div (#119112)
• Prevent instance_norm decomp for export (#120866)
• Allow ONNXProgram.save to use torch.load(..., mmap=True) for large models (#117295)
• Enable llama attention with dynamic shapes for onnxrt backend (#117009)
• Add bfloat16 support for scaled_dot_product_attention (#117878)
• Require Module to be passed to export (#117528)
• Improve support to mmap for ONNXProgram.save (#117863)
• Use environment variable ONNXRT_DUMP_PATH to dump onnx models created by onnxrt backend (#117551)
• Remove monkey-patch for torch.utils._rebuild_tensor (#120446)
• Enable custom ONNX model transforms in onnxrt dynamo backend (#120854)
• Add Float8 support to onnx exporter (#121281)
• Add support to save safetensors checkpoint directly into onnx (#121001)
• Update submodule onnx==1.16.0 (#123125)

Optimizer

• Allow torch.float64 scalars (in addition to torch.float32) for forloop + foreach implementations (#115841)
• Add beta1 support to CyclicLR momentum (#113548)
• Add guardrails preventing complex params in SparseAdam and add complex support for L-BFGS (#118161, #118184)
• Add capturable API for the forloop/single tensor (foreach=False) implementation in Adamax, RAdam, and ASGD (#121183, #121260, #121264)

Profiler

• Clean up first line of element text for readability (#120245)
• Add Total memory used after allocation in Trace View (#120339)
• Track context for SEGMENT_FREE and SEGMENT_UNMAP (#118055)
• Add CUDAAllocatorConfig details into snapshot metadata (#119404)
• Log process group config information in GPU trace’s distributedInfo field (#119443)
• Support GPU annotations for auto-trace jobs similar on-demand support (#114638)
• Record nccl version in distributed info (#121044)
• Add a function to allow adding preset user-defined metadata to traces (#121487)

Python API

• Enable eye on CPU for bfloat16 dtype (#116616)

Quantization

PT2 Export Quantization Flow:

• Relax constraints on dtype and qscheme to allow for customizations (#116287)
• Skip conv-bn folding when there are no batchnorm ops (#116440)
• Add move_exported_model_to_train (#113492)
• Allow users to override train/eval behavior (#119091)
• Add convert callback to Observer module (#115001)
• Fix _disallow_eval_train error message (#119694)
• Add model_is_exported util function (#119726)
• Relax model_is_exported input (#120720)
• Add the operator of decomposed fake quant per channel (#121297)
• Add error check for input_edge annotation in Quantizer (#121536)
• Call sub-quantizers' transform_for_annotation in ComposableQuantizer (#121548)

XNNPACKQuantizer:

• XNNPACKQuantizer skip inserting observers for non-float Tensors (#114999)
• Add support for linear_relu in XNNPACKQuantizer (#117052)
• Support custom qmin/qmax for activation and weight for XNNPACKQuantizer (#117305)
• Fix module name filter for underscores (#119344)

X86 CPU Inductor Backend:

• Enable QLinear input with multi dims in x86 CPU inductor backend (#113733)
• Add int8 linear op gelu for x86 CPU Inductor backend (#114852)
• Add dynamic quantization config for x86 inductor backend (#115337)
• Enable QConv2d with hardswish post op (#117487)
• Add Hardswish Conv2d Unary Annotation (#117488)

DTypes:

• Add uint1 to uint7 dtypes (#117208)
• Add float8 types to dtypes table (#117375)
• Enable cat for cuda torch.bits types (#115044)

Others:

• Skip privateuse1's checkZeroPoints (#114117)
• Support lowering for operator.matmul in fx graph mode quantization (#113954)
• Add quantized gelu (#119935)
• Update Quantizable LSTM to support QAT (#121448)

Releng

• Addition of linux cpu test for 3.12 (#117853)
• Bazel CUDA tests timeout increased to 480s (#120443)
• Update torchbench commit pin, add sam_fast benchmark (#121420)
• Trigger a mergability check on ghstack prs (#115944)
• Use matrix generate script for docker release workflows (#115949)
• [CI] Addition of initial inductor cpu smoketest for performance (#116456)
• [CI] Addition of python test skip logic for XPU (#117621)
• [ROCm] upgrade CI to 6.0 (#119495)
• Improved Dynamo testing convenience (#116173)
• Improved the error message when a PR lacks the necessary approvals (#116161)
• [CI] Addition of initial ci build test for XPU (#116100)
• Addition of CPU inductor merge rule (#116679)

ROCm

• Add hipblaslt support (#114329)
• Initial Flash Attention support on ROCM (#114309)
• Make ATen-cpu cuda/rocm agnostic (#121082)
• Autocast RNN Support (#121539)
• Add Flash Attention support on ROCM (#121561)
• Initial ir.Scan/aten.cumsum lowering support on ROCm (#119369)

Other

• Multiprocessing api to use sigkill if sigterm doesn't kill the process (#115219)
• Explicitly error out if CuDNN older than 8.5 (#118235)
• Set maximum supported version of Python as 3.12 (#119743)
• Update cslt to 0.5.2.1 (#115988)
• Update cutlass from 3.3.0 to 3.4.1 (#120434)
• Preserve metadata for MutableMapping and MutableSequence in pin_memory and collate_fn (#120553)
• Add inf norm support for _foreach_norm (#118441)
• Expose aggressive_recomputation as an inductor config (#118943)
• Disables denormal floating numbers on ARM CPU (#115184)
• Enable TORCH_TRACE by default in all Tupperware like environments (#120915)
• Upgrade submodule oneDNN to v3.3.6 (#122164)
• Pin protobuf to 3.20.2 on macOS (#121918)
• Make PyTorch compilable against upcoming Numpy-2.0 (#121880)
• Upgrade submodule pybind to 2.12.0 (#122899)

Bug Fixes

Autograd API

• Fast gradcheck bug no longer ignores specified eps argument when recomputing in slow mode to produce the error message (#115634)
• Properly handle retains_grad hook on the base when view of it is mutated (#117552)

Composability

• Fixed hash issue in fx_graph_cse graph pass (#119567)
• Better support for fakeifying torch_dispatch tensor subclasses that are views (#118405)
• Bugfix for where output of a compiled graph is a subclass, but is a view of a graph input (#118191)

CPP API

• Add the bound check for flatten with out_dim (#120894)
• torch check the division by zero in batch_norm_update_stats (#120882)
• Fixed crash when calling pad_packed_tensor when packed with cuda tensors and ensure_sorted=false due to indexing with tensors on different devices (#115028)
• [Caffe2] Fix bug in str on wide types (#117531)
• Try creating a bf16 tensor as a last resort of is_bf16_supported(). (#115924)
• Fix admm over empty tensors and broadcastable input (#118619)

Distributed API

• C10d:
  • [c10d] Fix Store check condition in NCCL PG watchdog (#115475)
  • [c10d] Fix compilation of NCCL_EXP path (#119805)
  • Don't add NCCL backend by default without CUDA (#119149)
  • [IntraNodeComm] fix a hybridCubeMeshAllReduceKernel breakage caused by a recent refactor (#121575)
  • Add partial read test for libuv backend and fix an error which only happens when partially reading a buffer (#116141)
  • Fix get_rank under a non-default group. (#120481)
  • Fix a bug where nn.functional._AllGather.backward produces wrong gradients (#120582)
  • Fix default world_size when running on 1 or 0 GPU (#119372)
  • Fix distributed debug w/ non-equal split (#115483)
  • [AMD] Fix build for intra_node_comm (#116291)
  • Fix timeout dump path write path overlap when there are multiple PGs (#116218)
  • Handle unwaited work objects on process termination (#119881)
  • Guarantee init cuda before attaching hooks (#120052)
  • Remove backend_id from pg_info (#120038)
  • Fix logic for default group=None in _set_pg_timeout (#120686)
  • Make _set_pg_timeout work with DeviceMesh PG (#120850)
  • Fix false positive ‘errors’ due to ‘reason’ string (#120863)
  • Fix the hang issue in store.check(TIMEOUT_DUMP) (#116297)
• FSDP:
  • Sharded grad scaler: copy found_inf after waiting on async reduce_all (#115710)
  • Fix FSDP + TP state dict in param unflattening (#115105)
  • Fixed device_mesh and auto wrap (#119064)
  • Added warning about unsupported double backwards (#120926)
• Distributed Checkpointing (DCP):
  • Call os.sync if os.fsync does not work for fsspec (#119287)
  • Fixes expected behavior when no_dist=True in state_dict_loader.load (#115660)
  • Fix no shard state dict loading (#120367)
• DTensor:
  • Force re-compute sharding when normalized_shape differs in fwd layer norm (#115250)
  • Fix is_tensor_shardable to correctly handle Replicate placement (#117726)
  • Fix unnecessary redistribute in new_factory_strategy (#118037)
  • Make input contiguous for DTensor reduce scatter to fix the incorrect numerical values (#115847)
  • DTensor + dynamo: fix is_shard/replicate always inlining to False (#118668)
  • to_local backward grad placement passthrough (#121474)
  • nn.Module: use swap_tensors for Tensor subclasses (#122755)
  • Fix swap_tensors path in _apply for modules that inherit from RNNBase (RNN, GRU, LSTM) (#122800)
• DeviceMesh
  • Fix fsdp device mesh depenency issue (#121061)
  • Cache and reuse sliced result (#122975)
• DDP
  • Pass inductor strides forward in ddp optimizer (#120523)
  • Ignore gradient sync if the gradient is not defined (#120419)
  • DistributedDataParallel._post_forward, fix return (#114678)
  • Lazily compile submodules - to propagate real tensor strides to backend compiler (#114154)
  • Fix wrong behavior of is_alias_of and c10d::reducer on MTIA (#115553)
• TorchElastic:
  • Correctly detect SystemExit with code == 0 when using –run-path #119697
• Misc
  • Fix ChunkShardingSpec metadata offsets for empty shards (#121002)

torch.compile

Dynamo

• Fix F632 bug in dynamo (if statement is always false) (#116867)
• Properly trace into mark_static (#120232)
• Handle guard_size_oblivious in user code (#120379)
• Do not attempt to make nditer spawned arrays writable (#120868)
• Fix autograd.Function x enum input x torch.compile (#115206)
• Fix handling of one_hot (#116338)
• Fix functools.reduce() function with None as initial (#116398)
• Fix sum() function with start argument (#116389)
• Fix torch function kwarg dispatch (#117083)
• Fix several bugs related to unbacked SymInt codegen in inductor (#117862)
• Fix Auto Functionalize to handle specified default values (#118331)
• Fix __name__ on a reconstructed NestedUserFunctionVariable (#118768)
• [HigherOrderOp] fix stack trace to report user stack (#118826)
• Fix dupe deprecated warning in dynamo export (#120896)
• Fix support for nn.Parameter constructor (part 1) (#120163)
• Fix gradient refcounts in pybind and compiled autograd (#118817)

Inductor

• Fix torch.bernoulli decomposition return type (#115699)
• Fix angle decomposition return type (#115700)
• Allow sympy expressions to participate in type promotion (#115676)
• Do variance calculation in opmath type (#115181)
• Properly unwrap_storage tensors sent to DynamicScalar (#117444)
• Catch some missing unbacked symbol dependencies (#117650)
• Don't try to directly compare symbols, it won't work (#117674)
• Changed return type of randint64_cpu to int64_t to prevent codegen is… (#117443)
• Realize inputs to DynamicScalar before unwrapping storage (#118125)
• Prevent DCE'ing unbacked SymInt for view outputs (#119552)
• Exclude operators that produce unbacked symbols (#120917)
• Fix guards for code objects (#120909)
• Fix a bug in batch linear fusion in the post grad (#115061) (#115131)
• Add missing include to model.h (#118075)
• Fix a bug in the torch._export.aot_load API (#118039)
• Fix a None as index codegen issue (#118187)
• Forward fix #117989 (#118291)
• Fix a strict-aliasing warning (#120628)
• Fix broadcast_tensors with unbacked symints when translation validation is off (#118066)
• Fixed issue with true div on integer input with dyn shapes (#115920)
• Fix QConv Binary Inplace Layout Issue (#115613)
• [Optimus] Fix batch layernorm numerical issue (#117404)
• Fix a bug in merge_splits (#117707)
• [Optimus] Fix a bug in gradients computation for runtime numeric check (#118105)
• Fix key error in pre_grad fx_passes_numeric_check (#118325)
• [Runtime numeric check] Fix compatibility issue (#118578)
• Fix a RAIIAtenTensorHandle premature deallocation bug (#118963)
• Fix FallbackKernel behavior on mutable ops (#118649)
• Fix compile error on scan with no mask (#119555)
• Fix a bug in merge_splits (#119956)
• Fix lint after #105590 (#120461)
• Fix "example_value" absent for stack nodes (#120655)
• Fix missing "example_value" for nodes introduced by group batch fusion (#120974)
• Forward fix lint after 121202 (#121425)
• Fix cudagraph check message (#115664)
• Fix constant folding and extern kernel mutation tracking bugs (#115908)
• Fix cpp_wrapper inputs mismatch (#116197), codegen for ir.ComplexView (#116481), cumsum codegen (#116171)
• Fix Conv Binary Inplace Fusion issue (#115153
• Fixed issue in upsample_nearestnd lowering with scales (#117538)
• Fix inductor pattern match error for qlinear with bmm (#117633)
• Fix cpp backend relu codegen with inf input (#117622)
• Fix CPP wrapper codegen for ExternKernel args (#117931)
• Fix sympy_subs to preserve integer and non-negative properties. (#118150)
• Fix Argmax codegen with Nan input (#118358)
• Fix constant folding bug with sym size tensor (#118411)
• Fix a typo in should_pad_bench (#118598)
• Fix codegen bug with Native Triton kernels with ReinterpretView args (#118569)
• Fix an internal test issue (#118903)
• Fix a cpp kernel missing arg type issue (#119021)
• Fix Inductor CSE Across Separate Reductions (#119410)
• Fix bandwidth extimation for StarDep (#120266)
• Fix bug around out of order constexprs in inductor (#120287)
• Fix compiler check (#120492)
• Fix q/dq per channel lowering with 64-bit qparams (#120984)
• Fix the layout problem for nll_loss2d_backward (#121173)
• Fix for Wait kernel lowering in inductor not accepting MultiOutputs from non-collective calls (#121428)
• Correct index propagation for % (#119864)
• Fix a missing declaration for the result of item() (#115175)
• Make sure bitcast input and target type have the same bitwidth (#115619)
• Avoid inplace for ComplexView (#115166)
• SDPA extend backward realized tensor alignment checking to forward realized tensors (#116069)
• Ignore SIGINT in codecache workers (#116380)
• Add more alias and mutation check for other input of Conv Binary Inplace fusion (#117330)
• Fail Conv Binary Inplace check when act and accum are same tensor (#117331)
• [CPU] Disable floating-point contraction when compiling (#116318)
• Use wait stream instead of synchronize() in cudagraph warmup (#117578)
• Place .lrodata later in the binary (#117575)
• Correctly generate grid info for benchmark_kernel (#118202)
• Do not reuse buffers across scopes in mem planning (#120777)
• Fix profiler (#119959)
• Wrap remote cache creation with a try-catch (#121340)
• Inductor cpp wrapper: fix dtype of ShapeAsConstantBuffer (#122297)

torch.export

• Handle transposition pattern seen in SDPA with unbacked SymInts (#121005)
• Fix bug removing node from wrong graph (#121574)
• Fix graph signature for primitive outputs (#118818)
• Fixed bug with user input mutations (#118942)
• Prevent specialization on backends (#118683)
• Fixed nn_module_stack in retracing (#121423)
• Fixed accidental specialization with faketensor input checks (#121460)
• Fixed name collision on constant name (#121145)
• Don't error if nn_module_stack doesn't contain a class (#119753)
• Fixed tuple return with symints (#119829)
• Fixed canonicalization for input mutations (#119533)
• Fixed getting meta["val"] (#117313)
• Add pass to remove auto functionalized hop (#122246)
• Fix auto_functionalize (#121990)
• Hack skip index_put_ in DCE (#122683)
• Various fixes to .module() (#118272)
• Do not rewrite state dict when unlifting (#118611)

FX

• Fix pass_manager type annotation (#119499)
• Suggested fixes for congruences (#121418)
• Fix: set codegen in _SplitterBase partitioner (#120361)
• Fixed FxGraphDrawer compat constructor (#119767)
• [sigmoid] Fix for FX tracing unflattened modules (#115708)
• Fix for subgraph rewriter (#119052)
• Fix F821 error in torch/fx/experimental (#116587)
• Support printing storage while FakeTensorMode is enabled (#118780)
• Don't guard if there are unbacked SymInts (#119312)
• Avoid performing replacements when it would unrefine ranges (#117356)
• Fix none type comparison (#116399)

JIT

• Fix RuntimeError: NYI: Named tensors are not supported with the tracer errors when using torch.jit.trace (#118393)
• Fix LLVM18 build (#115652, #117086)
• Fix handling of broadcasted inputs in Linear-BN Fusion (#119264)

Linalg

• Fix mm accuracy in ROCm for some inputs (#116537)
• Update matmul heuristics in the presence of gradients (#117067) (#118617)
• Workaround a cusolver bug on CUDA < 12.1 in triangular_solve (#117636)

MPS

• Fix addmm (#116547)
• Fix SegFault when torch.all/any dispatched to mps or other backends (#116457)
• Increase metal language support to 2.3 (#117472)
• Fix torch.mm correctness for large matrices (#117549)
• Fix lintear for 5D tensors (#117837)
• Fix use_metal_mm condition (#118830)
• Add support for complex scalars (#119318)
• Use dyspatch_sync_with_rethrow in searchsorted (#119646)
• Fix cfloat->chalf conversion on MacOS13 (#119681)
• Enable conj and conj_physical (#119669)
• Fix out resize logic in torch.where (#121476)
• Fix CrossEntropyLoss for float16 (#116597)
• Do not crash if Metal function can not be found (#116938)
• Fix float32 error on mps, in linalg.matrix_rank and linalg.pinv (#114771)
• Fix placeholder tensor is empty for relu in mps (#118965)
• Fix boundary checks in generateKernelOffsets (#116915)
• Fix torch.clamp in MPS to handle NaN correctly (#121381)
• Fwd-fix for clamp regression (#122148)
• Fix naive matmul for BFloat16 (#121731)
• Fix for MPS regression in #122016 and #123178 (#123234)

torch.nn API

• Fixed numpy warning when importing torch without numpy installed (#115867)
• Fixed edge case for size 1 channels dim in AdaptiveMaxPool (#116482)
• Fixed module pre bw hooks when input doesn't require grad but gradients are changed by the user (#116454)
• Fixed TransformerEncoderLayer for bias=False (#116760)
• Fixed error checking for LSTM with wrong input shape (#115542)
• Removed an incorrect type specification from AdaptiveMaxPool1d (#118162)
• Fixed an illegal memory access in cross entropy loss when using an index that is not a valid class (#117561)
• Fixed pool padding assertion to account for dilation (#118897)
• Fixed flash_attn_bw impl to match meta implementation when k and v have different strides (#119500)
• Fixed nonlinearity arg issue in RNN (#120234)
• Fixed requires_grad preservation for nn.Module.load_state_dict(assign=True) (#121157)
• Fixed last_dim stride check for singleton dimensions (#117001)
• Fixed gradients on cuda for interpolate::trilinear on non-contiguous grad output (#117373)
• Added Half support for masked_softmax on CPU (#117028)
• Fixed an issue where nn.Linear would cause an internal int underflow (#119221)
• Fixed segfault in torch.native_channel_shuffle when input is empty (#121199)

Nested Tensors

• Proper view support for jagged layout NestedTensor (#113279)

ONNX

• Add decomposition for upsample_linear{1d, 3d} (#114774)
• Fix upsample_bilinear2d decomp skip with output shape (#118823)
• Fix ONNXRT torch.compile backend running with OrtValueVector (#116124)
• Fix output mismatch issue of repeat_interleave when dim is None (#116689)
• Update initializer path for ONNXProgram.save due to onnx.checker limitation (#117294)
• Set proper fqn in lift constant tensor pass (#115222)
• Fix type promotion pass (#118246)
• Perform implicit casting of constants for the onnx::where operator (#118733) (#120619)
• Fix onnxrt backends with inputs on mix devices (#121159)
• Fix breaking changes for ONNX Runtime Training (#122000)
• beartype to emit warning instead of error by default (#123205)

Optimizer

• Rectify capturable testing and fix load_state_dict bugs with capturable! (#118326)
• Use torch.no_grad decorator for clip_grad_norm APIs vs local detaches (#120638)
• ReduceLROnPlateau allow get_last_lr to not error (#119556)

Profiler

• Stop clearing history when changing context (#120436)
• Fix conversion of max memory allocated and reserved from GB to GiB (#120172)
• Fix the missing device string in _memory_profiler (#119751)
• Log process group id instead of backend id in GPU traces (#120475)
• Add kineto init delay when used in daemon mode (#120276)
• Fix recorded profiler step number (#121127)
• [ET] Fix deadlock in ExecutionTraceObserver (#119242) (#119398)

Python API

• Fix index range checks when index is the minimum int value (#116062)
• Fix slots handling in torch.utils.swap_tensor (#116128)
• Fix NaN bug in torch.signal.windows.kaiser (#116470)
• Fix handling of empty inputs in torch.fft.fftn (#117368)
• Fix and/or ops on torch.uint8 tensors only return 0x00 or 0x01 (#117827)
• Fix inf handling in torch.nn.functional.scaled_dot_product_attention (#119577)
• Fix serialization of torch.complex32 dtype (#120388)
• Fix torch.gradient check for spacing arg list length (#115686)
• Fix type hints on torch.nn.attention.sdpa_kernel (#119140)
• Fix dimension checks in torch.distributions.MixtureSameFamily (#118947)

Quantization

• Make HistogramObserver handle torch.inf and closeby values (#103467)
• Fix equal_quantized_cpu for QUInt4x2 and QUInt2x4 (#116307)
• Fix XNNPACKQuantizer set_module_type issue (#115252)
• Fix a segfault when calling topk on a quantized scalar tensor. (#116337)
• Fix a segfault issue when passing an empty kernel to quantized_max_pool1d (#116342)
• Fix batchnorm folding in pt2e quantization (#118720)
• Update PerChannelMinMaxObserver default _load_from_state_dict (#118659)

Releng

• Fix for sparse windows on CPU with MKL (#102604)
• Fix for ExecuTorch pinned commit update failure (#117518)

Sparse

• Fix a crash in sparse compressed tensor invariants check when nnz == 0 (#115825)
• Fix sparse compressed tensor invariants checks when nnz==0 (#115826)
• Fix segfault when trying to permute empty tensor (#116335)

Other

• Remove compute capability 3.5 for CUDA 12 (#114930)
• VSX: Fix overflow in complex division (#116972)
• VSX: Fix vectorized abs function for complex tensors (#116859)
• Add complex support to parametrizations.spectral_norm (#121452)
• Fix crash in SymInt unary minus (#116160)
• Fix for out of bounds read in mobile interpreter INTERFACE_CALL opcode handler (#110301)
• Fix for out of bounds registers_ access in mobile TorchScript interpreter (#110300)

Performance

Composability

• In torch.compile, avoid allocating extra buffers unnecessarily in cases where the compiled function returns a mutated input directly (#120514)
• Min-cut partitioner always saves tensors that are returned as-is in backward (#114970)

CUDA

• Speed up triu_tril_kernel (#115013)

Inductor

• Add an autotune cache for inductor generated kernels (#120963)
• Add ArrayRefTensor (#112115)
• Replace cached thread_locals with stack allocation in AOTI (#112116)
• Autotune with matrix_instr_nonkdim for AMDGPU (#120742)
• Enable lowering of dynamic qlinear for X86Inductor (#120605)
• [NFC][Autotune] Use device_prop.regsPerMultiprocessor instead of hardcoded reg number. (#115094)
• [Autotune] Enable register pressure handling logic for H100. (#115295)
• Support vectorization for index_expr that depends on tiling itervar or with indirect indexing (#114545)
• Load as scalar for the index invariant in the vector range (#116387)
• Inductor qlinear int8_fp32 with bmm (#116599)
• Inductor qlinear int8_bf16 with bmm (#116604)
• Enable fp16 mkldnn fusion/prepack in inductor (#117206)
• Improve vector contiguous checks for FloorDiv and ModularIndexing (#117221)
• Load as scalar for the index invariant in the vector range (#116387)
• Use sleef implementation for CPP backend acosh codegen (#118350)
• Don't skip register-spilling configs in custom Triton kernel auto-tuning (#119634)
• be more consrevative until regression is debugged (#119583)
• Check alignment of ReinterpretView args of custom Triton kernels (#119649)
• Apply simplify_index_in_vec_range in select_tiling_indices to enable more contiguous vec load (#117260)
• Apply simplify_index_in_vec_range to vector store and vector transpose (#117263)
• Multi-kernel support (#103469)
• Enable the Inductor Lowering of QConv2d post op hardswish (#117489)
• Change CppWrapperCodeCa

Note truncated.

This release is meant to fix the following issues (regressions / silent correctness):

• Properly raise an error when trying to use inductor backend on non-supported platforms such as Windows (https://github.com/pytorch/pytorch/pull/115969)
• Fix mkldnn performance issue on Windows platform (https://github.com/pytorch/pytorch/pull/121618)
• Fix RuntimeError: cannot create std::vector larger than max_size() in torch.nn.functional.conv1d on non-contiguous cpu inputs by patching OneDNN (https://github.com/pytorch/builder/pull/1742) (https://github.com/pytorch/builder/pull/1744)
• Add support for torch.distributed.fsdp.StateDictType.FULL_STATE_DICT for when using torch.distributed.fsdp.FullyShardedDataParallel with the device_mesh argument (https://github.com/pytorch/pytorch/pull/120837)
• Fix make triton command on release branch for users building the release branch from source (https://github.com/pytorch/pytorch/pull/121169)
• Ensure gcc>=9.0 for build from source and cpp_extensions (https://github.com/pytorch/pytorch/pull/120126)
• Fix cxx11-abi build in release branch (https://github.com/pytorch/builder/pull/1709)
• Fix building from source on Windows source MSVC 14.38 - VS 2022 (https://github.com/pytorch/pytorch/pull/122120)

Release tracker https://github.com/pytorch/pytorch/issues/120999 contains all relevant pull requests related to this release as well as links to related issues.

This release is meant to fix the following issues (regressions / silent correctness):

• Fix missing OpenMP support on Apple Silicon binaries (https://github.com/pytorch/builder/pull/1697)
• Fix crash when mixing lazy and non-lazy tensors in one operation (https://github.com/pytorch/pytorch/pull/117653)
• Fix PyTorch performance regression on Linux aarch64 (https://github.com/pytorch/builder/pull/1696)
• Fix silent correctness in DTensor _to_copy operation (https://github.com/pytorch/pytorch/pull/116426)
• Fix properly assigning param.grad_fn for next forward (https://github.com/pytorch/pytorch/pull/116792)
• Ensure gradient clear out pending AsyncCollectiveTensor in FSDP Extension (https://github.com/pytorch/pytorch/pull/116122)
• Fix processing unflatten tensor on compute stream in FSDP Extension (https://github.com/pytorch/pytorch/pull/116559)
• Fix FSDP AssertionError on tensor subclass when setting sync_module_states=True (https://github.com/pytorch/pytorch/pull/117336)
• Fix DCP state_dict cannot correctly find FQN when the leaf module is wrapped by FSDP (https://github.com/pytorch/pytorch/pull/115592)
• Fix OOM when when returning a AsyncCollectiveTensor by forcing _gather_state_dict() to be synchronous with respect to the mian stream. (https://github.com/pytorch/pytorch/pull/118197) (https://github.com/pytorch/pytorch/pull/119716)
• Fix Windows runtime torch.distributed.DistNetworkError: [WinError 32] The process cannot access the file because it is being used by another process (https://github.com/pytorch/pytorch/pull/118860)
• Update supported python versions in package description (https://github.com/pytorch/pytorch/pull/119743)
• Fix SIGILL crash during import torch on CPUs that do not support SSE4.1 (https://github.com/pytorch/pytorch/issues/116623)
• Fix DCP RuntimeError in get_state_dict and set_state_dict (https://github.com/pytorch/pytorch/pull/119573)
• Fixes for HSDP + TP integration with device_mesh (https://github.com/pytorch/pytorch/pull/112435) (https://github.com/pytorch/pytorch/pull/118620) (https://github.com/pytorch/pytorch/pull/119064) (https://github.com/pytorch/pytorch/pull/118638) (https://github.com/pytorch/pytorch/pull/119481)
• Fix numerical error with mixedmm on NVIDIA V100 (https://github.com/pytorch/pytorch/pull/118591)
• Fix RuntimeError when using SymInt input invariant when splitting graphs (https://github.com/pytorch/pytorch/pull/117406)
• Fix compile DTensor.from_local in trace_rule_look up (https://github.com/pytorch/pytorch/pull/119659)
• Improve torch.compile integration with CUDA-11.8 binaries (https://github.com/pytorch/pytorch/pull/119750)

Release tracker https://github.com/pytorch/pytorch/issues/119295 contains all relevant pull requests related to this release as well as links to related issues.

PyTorch 2.2 Release Notes

• Highlights
• Backwards Incompatible Changes
• Deprecations
• New Features
• Improvements
• Bug fixes
• Performance
• Documentation

Highlights

We are excited to announce the release of PyTorch® 2.2! PyTorch 2.2 offers ~2x performance improvements to scaled_dot_product_attention via FlashAttention-v2 integration, as well as AOTInductor, a new ahead-of-time compilation and deployment tool built for non-python server-side deployments.

This release also includes improved torch.compile support for Optimizers, a number of new inductor optimizations, and a new logging mechanism called TORCH_LOGS.

Please note that we are deprecating macOS x86 support, and PyTorch 2.2.x will be the last version that supports macOS x64.

Along with 2.2, we are also releasing a series of updates to the PyTorch domain libraries. More details can be found in the library updates blog.

This release is composed of 3,628 commits and 521 contributors since PyTorch 2.1. We want to sincerely thank our dedicated community for your contributions. As always, we encourage you to try these out and report any issues as we improve 2.2. More information about how to get started with the PyTorch 2-series can be found at our Getting Started page.

Summary:

• scaled_dot_product_attention (SDPA) now supports FlashAttention-2, yielding around 2x speedups compared to previous versions.
• PyTorch 2.2 introduces a new ahead-of-time extension of TorchInductor called AOTInductor, designed to compile and deploy PyTorch programs for non-python server-side.
• torch.distributed supports a new abstraction for initializing and representing ProcessGroups called device_mesh.
• PyTorch 2.2 ships a standardized, configurable logging mechanism called TORCH_LOGS.
• A number of torch.compile improvements are included in PyTorch 2.2, including improved support for compiling Optimizers and improved TorchInductor fusion and layout optimizations.
• Please note that we are deprecating macOS x86 support, and PyTorch 2.2.x will be the last version that supports macOS x64.
• torch.ao.quantization now offers a prototype torch.export based flow

<table> <tr> <td> <strong>Stable</strong> </td> <td><strong>Beta</strong> </td> <td><strong>Prototype</strong> </td> <td><strong>Performance Improvements</strong> </td> </tr> <tr> <td> </td> <td>FlashAttentionV2 backend for scaled dot product attention </td> <td>PT 2 Quantization </td> <td>Inductor optimizations </td> </tr> <tr> <td> </td> <td> AOTInductor </td> <td> Scaled dot product attention support for jagged layout NestedTensors </td> <td>aarch64-linux optimizations (AWS Graviton) </td> </tr> <tr> <td> </td> <td> TORCH_LOGS </td> <td> </td> <td> </td> </tr> <tr> <td> </td> <td>torch.distributed.device_mesh </td> <td> </td> <td> </td> </tr> </tr> <tr> <td> </td> <td>torch.compile + Optimizers </td> <td> </td> <td> </td> </tr> </table>

*To see a full list of public 2.2 - 1.12 feature submissions click here.

Tracked Regressions

Performance reduction when using NVLSTree algorithm in NCCL 2.19.3 (#117748)

We have noticed a performance regression introduced to all-reduce in NCCL 2.19.3. Please use version 2.19.1 instead.

Poor numeric stability of loss when training with FSDP + DTensor (#117471)

We observe the loss will flatline randomly while training with FSDP + DTensor in some instances.

Backwards Incompatible Changes

Building PyTorch from source now requires GCC 9.4 or newer (#112858)

GCC 9.4 is the oldest version fully compatible with C++17, which the PyTorch codebase has migrated to from C++14.

Updated flash attention kernel in scaled_dot_product_attention to use Flash Attention v2 (#105602)

Previously, the v1 Flash Attention kernel had a Windows implementation. So if a user on Windows had explicitly forced the flash attention kernel to be run by using sdp_kernel context manager with only flash attention enabled, it would work. In 2.2, if the sdp_kernel context manager must be used, use the memory efficient or math kernel if on Windows.

with torch.backends.cuda.sdp_kernel(enable_flash=True, enable_math=False, enable_mem_efficient=False):
  torch.nn.functional.scaled_dot_product_attention(q,k,v)

</td> <td>

# Don't force flash attention to be used if using sdp_kernel on Windows
with torch.backends.cuda.sdp_kernel(enable_flash=False, enable_math=True, enable_mem_efficient=True):
  torch.nn.functional.scaled_dot_product_attention(q,k,v)

</td> </tr> </table>

Rewrote DTensor (Tensor Parallel) APIs to improve UX (#114732)

In PyTorch 2.1 or before, users can use ParallelStyles like PairwiseParallel and specify input/output layout with functions like make_input_replicate_1d or make_output_replicate_1d. And we have default values for _prepare_input and _prepare_output. The UX of Tensor Parallel was like:

from torch.distributed.tensor.parallel.style import (
    ColwiseParallel,
    make_input_replicate_1d,
    make_input_reshard_replicate,
    make_input_shard_1d,
    make_input_shard_1d_last_dim,
    make_sharded_output_tensor,
    make_output_replicate_1d,
    make_output_reshard_tensor,
    make_output_shard_1d,
    make_output_tensor,
    PairwiseParallel,
    parallelize_module,
)
from torch.distributed.tensor import DeviceMesh

module = DummyModule()
device_mesh = DeviceMesh("cuda", list(range(self.world_size)))
parallelize_module(module, device_mesh, PairwiseParallel(_prepare_input=make_input_replicate_1d))
...

Starting from PyTorch 2.2, we simplified parallel styles to only contain ColwiseParallel and RowwiseParallel because other ParallelStyle can consist of these two. We also deleted the input/output functions, and started using input_layouts and output_layouts as kwargs instead to specify the sharding layout of both input/output tensors. Finally, added PrepareModuleInput/PrepareModuleOutput style, and no default arguments for layouts in these two styles and users need to specify them to think about the sharding layouts.

from torch.distributed.tensor.parallel.style import (
    ColwiseParallel,
    PrepareModuleInput,
    RowwiseParallel,
    parallelize_module,
)
from torch.distributed._tensor import init_device_mesh

module = SimpleMLPModule()
device_mesh = init_device_mesh("cuda", (self.world_size,)))
parallelize_module(
   module,
   device_mesh,
   {
      "fqn": PrepareModuleInput(
                input_layouts=Shard(0),
                desired_input_layouts=Replicate()
             ),
      "fqn.net1": ColwiseParallel(),
      "fqn.net2": RowwiseParallel(output_layouts=Shard(0)),
   }
)
...

UntypedStorage.resize_ now uses the original device instead of the current device context (#113386)

Before this PR, UntypedStorage.resize_ would move data to the current CUDA device index (given by torch.cuda.current_device()). Now, UntypedStorage.resize_() keeps the data on the same device index that it was on before, regardless of the current device index.

<table> <tr> <th>2.1</th> <th>2.2</th> </tr> <tr> <td>

>>> import torch
>>> with torch.cuda.device('cuda:0'):
...:     a = torch.zeros(0, device='cuda:1')
...:     print(a.device)
...:     a = a.untyped_storage().resize_(0)
...:     print(a.device)
cuda:1
cuda:0

</td> <td>

>>> import torch
>>> with torch.cuda.device('cuda:0'):
...:     a = torch.zeros(0, device='cuda:1')
...:     print(a.device)
...:     a = a.untyped_storage().resize_(0)
...:     print(a.device)
cuda:1
cuda:1

</td> </tr> </table>

Wrapping a function with set_grad_enabled will consume its global mutation (#113359)

This bc-breaking change fixes some unexpected behavior when set_grad_enabled is used as a decorator.

<table> <tr> <th>2.1</th> <th>2.2</th> </tr> <tr> <td>

>>> import torch
>>> @torch.set_grad_enabled(False)  # unexpectedly, this mutates the grad mode!
    def inner_func(x):
        return x.sin()

>>> torch.is_grad_enabled()
True

</td> <td>

>>> import torch
>>> @torch.set_grad_enabled(False)  # unexpectedly, this mutates the grad mode!
    def inner_func(x):
        return x.sin()

>>> torch.is_grad_enabled()
False

</td> </tr> </table>

Deprecated verbose parameter in LRscheduler constructors (#111302)

As part of our decision to move towards a consolidated logging system, we are deprecating the verbose flag in LRScheduler.

If you would like to print the learning rate during execution, please use get_last_lr()

<table> <tr> <th>2.1</th> <th>2.2</th> </tr> <tr> <td>

optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)
scheduler = ReduceLROnPlateau(optimizer, 'min', verbose=True)
for epoch in range(10):
    train(...)
    val_loss = validate(...)
    # Note that step should be called after validate()
    scheduler.step(val_loss)

</td> <td>

optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)
scheduler = ReduceLROnPlateau(optimizer, 'min')
for epoch in range(10):
    train(...)
    val_loss = validate(...)
    # Note that step should be called after validate()
    scheduler.step(val_loss)
	print(f"Epoch {epoch} has concluded with lr of {scheduler.get_last_lr()}")

</td> </tr> </table>

Removed deprecated c10d multi-gpu-per-thread APIs (#114156)

In PyTorch 2.1 or before, users can use our multi-gpu c10d collective APIs such as all_reduce_multigpu: <table> <tr> <th>2.1</th> <th>2.2</th> </tr> <tr> <td>

import torch.distributed as dist


dist.broadcast_multigpu
dist.all_reduce_multigpu
dist.reduce_multigpu
dist.all_gather_multigpu
dist.reduce_scatter_multigpu
...

</td> <td>

optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)
scheduler = ReduceLROnPlateau(optimizer, 'min')
for epoch in range(10):
    train(...)
    val_loss = validate(...)
    # Note that step should be called after validate()
    scheduler.step(val_loss)
	print(f"Epoch {epoch} has concluded with lr of {scheduler.get_last_lr()}")

</td> </tr> </table>

In PyTorch 2.2, these APIs are removed because PyTorch Distributed's preferred programming model is one device per thread, as exemplified by the APIs in its document. The multi-GPU functions (which stand for multiple GPUs per CPU thread) have been deprecated since PyTorch 1.13.

Rename torch.onnx.ExportOutput* to ONNXProgram* (#112263)

The torch.onnx.dynamo_export’s output was renamed from torch.onnx.ExportOutput to torch.onnx.ONNXProgram to better align with torch.export.export API terminology which returns a torch.export.ExportedProgram. With this change, any ambiguity that could arise with either API is eliminated.

<table> <tr> <th>2.1</th> <th>2.2</th> </tr> <tr> <td>

export_output: torch.onnx.ExportOutput = torch.onnx.dynamo(...)

</td> <td>

onnx_program: torch.onnx.ONNXProgram = torch.onnx.dynamo(...)

</td> </tr> </table>

Fix functional::smooth_l1_loss signatures to not override beta (#109798)

Previously, there were two possible options to pass in beta to smooth_l1_loss, either as a SmoothL1LossFuncOption parameter or a function parameter.

Before, the beta specified as a function parameter would override the other beta if it was set, which was unexpected behavior. Now, we throw an error when beta is passed in both cases.

Deprecations

Autograd API

Deprecate not passing use_reentrant kwarg to torch.utils.checkpoint.checkpoint_sequential explicitly (#114158)

The use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants. Note that not passing use_reentrant kwarg to torch.utils.checkpoint.checkpoint has been previously deprecated in a previous release.

<table> <tr> <th>2.1</th> <th>2.2</th> </tr> <tr> <td>

a = torch.randn(3, requires_grad=True)
modules_list = [
    torch.nn.Linear(3, 3),
    torch.nn.Linear(3, 3),
    torch.nn.Linear(3, 3)
]

# This would produce a warning in 2.2
checkpoint_sequential(modules_list, 3, a)

</td> <td>

# Recommended
checkpoint_sequential(modules_list, 3, a, use_reentrant=False)

# To preserve existing behavior
checkpoint_sequential(modules_list, 3, a, use_reentrant=True)

</td> </tr> </table>

Deprecate "fallthrough" as autograd fallback default (#113166)

Custom operators that do not have a kernel registered to the Autograd keys (e.g. AutogradCPU and AutogradCUDA) will now produce a warning when used with autograd. If your custom operator previously returned floating-point or complex Tensors that do not require grad, they will now require grad as long as grad mode is enabled and the inputs require grad. For users who would like the old behavior, register torch::CppFunction::makeFallthrough() to your Autograd key, as shown here.

The below example uses the torch library API, but if you are writing an operator in a cpp extension, please read this doc for more information.

import torch
import numpy as np

# Define the operator
torch.library.define("mylibrary::sin", "(Tensor x) -> Tensor")

# Add implementations for the cpu device
@torch.library.impl("mylibrary::sin", "cpu")
def f(x):
    return torch.from_numpy(np.sin(x.detach().numpy()))
x = torch.randn(3, requires_grad=True)
y = torch.ops.mylibrary.sin(x)
y.sum().backward()

<table> <tr> <th>2.1</th> <th>2.2</th> </tr> <tr> <td>

RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

</td> <td>

UserWarning: mylibrary::sin: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd.

</td> </tr> </table>

Linalg

Deprecate torch.cross default behavior (#108760)

Calling torch.cross without specifying the dim arg is now deprecated. This behavior will be changed to match that of torch.linalg.cross in a future release.

Jit

NVFuser functionality has been removed from TorchScript (#110124, #111447, #110881)

Neural Network Compiler (NNC) has replaced NVFuser as the default GPU fuser for TorchScript in PyTorch 2.1, which also added a deprecation warning for NVFuser. The TorchScript functionality for NVFuser has now been fully removed and is no longer supported.

Optimizer

SparseAdam constructor will no longer accept raw Tensor type for params (#114425)

SparseAdam is now consistent with the rest of our optimizers and only accepts containers instead of individual Tensors/Parameters/param groups.

<table> <tr> <th>2.1</th> <th>2.2</th> </tr> <tr> <td>

import torch
param = torch.rand(16, 32)
optimizer = torch.optim.SparseAdam(param)

</td> <td>

optimizer = torch.optim.SparseAdam([param])

</td> </tr> </table>

New Features

torch.compile

Dynamo

• Fully enabled compiled optimizers (#115906)
• Cudagraphs support for compiled optimizers (#107504)
• Experimental support for TorchDynamo tracing with DTensor (#108329)
• Experimental support for torch.compile, activation checkpointing and FSDP (#103953)
• Dynamo variable trackers are mutable (#113725) - Improves Dynamo compilation time
• Reduce cache size limit to 8 - Quickly fallback to eager for non-compile friendly functions (#108526)

Inductor

• CUTLASS Backend with epilogue fusion support (#107802, #107847, #107901, #107931, #108015, #109338, #110890, #112762)
• FP8 support (#109168, #112527, #114974, #111122, #112297)
• Support user defined triton kernel (#111434, #111627, #111939, #111956, #111970, #112228, #112290, #112523, #112752, #113090, #114002, #114475)
• Support registering out of tree customized pre/post grad pattern matchers (#108615, #113823)

torch.export

• Introduce dynamic_shapes API in favor of constraints (#108448, #112298, #110101, #110638, #110276)
• Add torch.export.register_dataclass API (#109152)
• Expose torch.ops.higher_order.map (#111404)
• Change export to return full ATen IR (not Core ATen) and add a run_decomposition() function to allow users to pass in a decomposition table (or by default it will decompose to the core ATen decomposition table) (#111030, 8be26111f93, #110236, #114714)

Build

• Add Hopper (CUDA arch 9.0a) support (#110587)

Python API

• Add torch.unravel_index (#110580)
• Add multi-dim reductions for `torch.{any,all} (#110310)
• Add file name and size to the serialization metadata logging (#113077)
• Add torch.distributions.InverseGamma distribution and fix sign bug in torch.distributions.PowerTransform (#104501)
• Add torch.utils.deterministic.fill_uninitialized_memory flag (#111377)

Profiler

• Show shapes for lists of tensors in chrome traces (#109751)
• Add src/dst information to NCCL send/recv (#111811)
• Populate in/out split size information for NCCL all_to_all from CPU to CUDA kernel (#112308)

Quantization

• Add CUTLASS-based support for mixed dtypes matrix multiplication (#110981)

Sparse API

• Add torch.compile support and padding for semi-structured sparsity (#111049, #110583)
• Add CSR tensor with non-contiguous values support to CuSparseSpMatCsrDescriptor (#111742)
• Add scatter_mm and bsr_scatter_mm operations. (#110396, #111796)
• Add is_sparse as a property of MaskedTensor (#110725)

NestedTensor API

• Add unary out-of-place sin / cos support (#107891)
• Add binary out-of-place ge.Scalar / eq.Scalar support (#107892)
• Add binary op support for (B, C, *, *) NT with (C, 1, 1) dense (#107890)
• Add support for cat with dim=0 (#108361)
• Add support for matmul of (B, *, C, D) NT with dense (D, E) (#108370)
• Add support for narrow() on dim=0 (#108362)
• Add support for cat with dim > 0 when representable as jagged (#108428)
• Add public API for constructing NT with jagged layout from tensor list (#111078)

Misc

• Python 3.10 Union operator | support for JIT (#109293)
• Allow specifiying inputs as GradientEdge in autograd APIs (#110867, dev-discuss)
• Use CapturedTraceback symbolizer for C++ exceptions from Python library (#113207)
• Add sparse tensor support to dataloader (#112842)
• Add 0dim Tensor overload for _foreach_div (#113688)
• Add global_step parameter to SummaryWriter.add_hparams (#109572)

Fx

• Add a matcher that supports name to node mapping (#110743)
• Add splitting by tags feature (#109332)
• Allow tracing calls with Python Enum values. (#109507)
• Add function to port FX minified graph to HLO via StableHLO (#109084)

ONNX

• Add symbolic shape support for torch.onnx.dynamo_export(#112179)
• Add optional torch.export.ExportGraphSignature to ONNXProgram (#113477)
• Add ONNXProgram.__call__ API to run model with ONNX Runtime (#113495)
• Add decomposition support for dynamo_export + ExportedProgram (#112444)
• Add user input mutation support for dynamo_export + ExportedProgram (#114596)
• Add mutated buffer support for dynamo_export + ExportedProgram (#112272)
• Add FakeTensor support for dynamo_export + ExportedProgram (#114407)

CPU

• Add support for torch.cpu.set_device() and torch.cpu.current_device() (#110716, #110987)

MPS

• Pixel shuffle unshuffle support (#99306)
• Add lgamma, digamma, and polygamma implementations (#106292)
• Add support for aten::nextafter (#109685)
• Adding weight_norm_interface support for mps (#108008)
• Add searchsorted op (#112829)
• Add bucketize op (#112830)

Vulkan

• Add Vulkan support for several ATen operators:
  • aten::randlike (#108086)
  • aten::randn_like and aten::normal (#109075)
  • aten::bmm (#109360)
  • aten::baddbmm (#109360)
  • aten::floor_divide (#110785, #112190)
  • aten::mean.dim (#111609)
  • aten::var.dim (#111965)
  • aten::log and aten::log_softmax (#112828)
  • aten::layer_norm (#112322, #114701)
  • aten::native_layer_norm (#113573)
• Partial implementation of 1D convolution (only supports stride=1, padding=0, dilation=1 for now) (#112880)
  • Add support for 0-size tensors (i.e. a tensor with a size of 0, for example with sizes {2, 1, 0}) (#111512)
  • Add support for 0-dim tensors (i.e. a tensor with sizes of {}) (#111680)

Improvements

torch.compile

Dynamo

• Dispatch numpy.take_along_axis to torch.take_along_dim (#108880)
• Force specialization on INT_LIST (#111216)
• Add custom treespec fqn field (#112428)
• Better error handling for cond (#108817)
• Support 'BaseOutput' and subclasses from 'diffusers' in dynamo (#111978)
• Add infinite generators itertools.{count, repeat, cycle} (#110967)
• Add support for dict.fromkeys() / OrderedDict.fromkeys() / defaultdict.fromkeys() (#115010)
• Add support for dict.update(seq2) / OrderedDict.update(seq2) / defaultdict.update(seq2) (#115011)
• Add support for dict.copy() / OrderedDict.copy() / defaultdict.copy() (#115012)
• Force synced KJT to trace unbacked SymInt (#108960)

Inductor

• max-autotune improvements
  • max-autotune in multi-processes with multi-GPUs (#109126, #109127, #109500)
  • Allow customizing benchmark input generation (#108242)
• Make codegen stateless (#107320, #107617)
• Add or improve lowering rules for prims.div, reflection_pad2d, full, sub, _local_scalar_dense, index, reflection_pad2d, index_put, unfold (#102809, #110988, #108166, #108518, #109893, #111015, #111212, #113204, #113259)
• Add or improve decomposition rules for grid_sampler_2d, full, torch.ops.quantized.embedding_bag_byte_unpack, amax/amin, native_dropout, bmm, mm, complex dtype addition, upsample_nearest_exactNd (#104710, #108443, #109398, #110311, #115040, #109836, #110740, #113749)
• Add reinplacing pass for scatters + incremental fake tensor updating (#106192)
• Add meta-registration for _sparse_semi_structured_linear, _cslt_sparse_mm (#114477, #114685 )
• Provide fallback values for unbacked symint (#109893, #110520)
• Decompose addmm on cpu for a few special cases (e.g. dot product or small matrix vector multiplication) (#110010, #110456)
• Don't tune beyond 32 warps (which is a CUDA limit) for the coordinate descent tuner (#108997)
• Avoid special characters in cache_dir path (#110945)
• Add DeviceInterface abstraction to make inductor code more device agnostic (#109486 )
• Allow matmul to have flexiable layout when we are not autotuning (#110726)
• Allow backend compiler skipping a frame for transient errors (#111153)
• Handle item() on boolean tensor (#114157)
• Replace rand[n].generator with inductor prim if generator=None (#115051)
• Support channel last for XPU convolution in inductor layout optimization path (#111018)
• Early work to improve the static memory planning algorithm (#111402)
• Added support for symbolic shapes in FX graph cache (#111421)
• Added config to specify the shape attribute for the generated svg graphs (#114811)
• Quantization
  • Enable quantization dynamic batch size support (#108550)
  • Enable QConv2d Unary & Binary int8-mixed-bf16 Lowering (#112550, #112551)
  • Enable QLinear int8-mixed-bf16 Lowering (#112486)
  • Enable the lowering of quantized reshape (#114443)
  • Enable the Inductor Lowering of QConv2d post op hardtanh (#114580)
• Improve the CPU backend
  • Support S390X (#111367, #112723)
  • Support MkldnnRnnLayer (#107858)
  • Add GIL release and acquire (#111888)
  • Vectorize support for truediv (#112234)
  • Support QConv (#112373)
  • Vectorize embedding lookup (#114062)
  • Avoid redundant lowp type cast for direct load/store (#115006)
• Improve the Fx pattern matching passes
  • Generalize pointless_cumsum_replacement pattern (#108373)
  • Improve mem efficiency of constant folding (#108421)
  • Make sure unfuse_addmm and addmm patterns don't overlap (#110235)
  • Improve reinplace_scatters pass (#112801)
  • Make pattern-matcher failure diagnostics lazy and add an error message if format string is too long (#112923)
• Foreach kernel compilation time improvement
  • Skip searching getitem in group batch fusion pass reduces optimizer compilation time by 60s (#112088)
  • Re-inplace foreach when safe and allow aliasing during lowering (#112440)
• AOTInductor
  • ABI-compatible mode support
    • Add a C shim layer for libtorch (#109391, #109834)
    • Support _scaled_dot_product_flash_attention fallback (#110085)
    • Add AOTI ABI shim function for repeat_interleave.Tensor (#110745)
    • Add size, stride, storage_offset to RAIIAtenTensorHandle (#110764)
    • Add AOTI ABI shim function for torch.nonzero (#110766)
    • Enable floor_div indexing to work under ABI-compat mode (#113276)
    • Add ABI shim function for torch.scatter (#114027)
    • Support ReinterpretView in ABI mode (#114169)
    • Support at::convolution for AOTInductor (#114961)
  • ProxyExecutor for custom ops support
    • ProxyExecutor skips serializing missing args with default value (#111425)
    • Support List[Tensor] return type (#110182)
    • Proxy Executor for Extern Fallback kernels (#108350)
    • Switch ProxyExecutor to use AtenTensorHandle (#109748)
    • ProxyExecutor supports custom op with tuple output (#110140)
    • ProxyExecutor supports Tuple of Tensor and List[Tensor] in returns (#110187)
    • ProxyExecutor support ReinterpretView inputs (#110451)
    • ProxyExecutor support Dynamic Shape (#110526)
    • Allow using ProxyExecutor for ATen fallbacks (#112976)
    • Use ProxyExecutor for aten op if c-shim is missing (#113918)
  • CPU performance improvement
    • Generate reused thread_locals when tensors probably have static shape (#110892)
    • Cache dtypes and device types at DSO load (#111820)
    • Emit CACHED_TORCH_TYPE only as needed (#113997)
  • UX improvement and refactoring
    • Use array of constants (#111815)
    • Write weight files only if they do not exist yet (#111379)
    • Enforce no_grad for 'run' entry points (#111613)
    • Improve validation for C++ wrapper codegen (#111102)
    • Avoid generating redundant kernel loading code (#110510)
    • Group AOTInductor configs under aot_inductor class (#108369)
    • Include constants in the generated .so file (#108473)
    • Do not hardcode directory with .cubin files (#109151)
    • Add is_cpu for AOTInductorModelContainer (#109287)
    • Pass TorchIR to AOTInductor (#110020)
    • A lightweight model runner (#110158)
    • Remove CUDA dependency for cpp backend (#110409)
    • Delay the fallback kernel naming decision to the codegen time (#113660)
    • Move constant loading logic from Container to Model (#112197)
    • Allow specifying a .so name in the aot_inductor.output_path config (#112651)
    • Improve the two-pass wrapper codegen (#114067)

torch.export

• Address constant tensors in ExportedPrograms (#113689, #108592)
• Remove replaced symbols from range_constraints (#110644)
• Copy graph module before calling PassManager (#108321)
• Made aot_export_module uses dynamo's fake_mode (#114009, #114381)
• Core ATen Opset
  • Registered additional ATen operators as core (#110882)
  • De-registered full_like and empty_like as core (#110924)
  • Added div.Tensor_mode, div.Scalar_mode, and copy as core operators (#109812)

Composability

• FakeTensors and meta tensors are used to perform shape propagating when tracing out a graph in torch.compile. There were a number of op coverage improvements this release:
  • masked_scatter (#108802), _segment_reduce (#109359), foreach ops (#112281), linear_backward (#114359), scaled_mm (#112609)
• We have python “reference” decompositions for many aten operators. These are used during the tracing step of torch.compile. In a few ways: sometimes they are used to directly decompose operators in the captured graph. Other times, they are used as an alternative to a shape-propagation rule for an operator. There were several improvements to operator coverage in this release
  • linalg.vecdot (#108188)
  • aten.dot/vdot (#108194)
  • aten.tensor_split.tensor_indices_or_sections (#107251)
  • view_as_complex (#108005)
  • aten.take_along_dim (#108185)
  • scaled_dot_product_attention (#108180, #108608, #108371, #113102)
  • unsafe_split{,_with_sizes} (#109668)
  • _weight_norm_interface (#112193)
• We also have an opset known as “Core ATen IR” as defined here. Several ops were either added to core ATen, or had decompositions for them added, that decompose into other core ATen operators:
• decompositions:
  • std.correction (#108733)
  • unsafe_split.Tensor (#108544)
  • clamp_min (#108717)
  • clamp_max (#108718)
  • _unsafe_view (#108713)
  • baddbmm (#108534)
  • floor_divide (#110046)
  • aten.all (#110093)
  • split.Tensor + unbind (#110323)
  • aten.sum + aten.squeeze (#110645)
  • std + std_mean (#109667)
  • Scaled dot product attention (#117097)
  • New core aten ops:
    • trunc (#109319, #109902)
    • glu (#110043)

Python API

• Add a UserWarning when using torch.{std,var,std_mean,std_var} with dof<=0 (#109824)
• Add torch.half support for torch.multinomial on CPU (#104178)
• Add support for serializing torch.float8_* dtypes (#114662)
• Add different out dtypes support to torch.addc{mul,div} (#112682)

torch.nn API

• Add __all__ for torch.nn.utils (#111026)
• Add Half support for AdaptiveAvgPool2d and AdaptiveMaxPool2d on CPU (#102079)
• Add Half support for GroupNorm on CPU (#100234)
• Add Half support for torch.nn.functional.{softmax/log_softmax} on CPU (#103315)
• Add BFloat16 support to torch.nn.functional.grid_sample (#112331)
• Add BFloat16 support for nn.utils.parametrizations.weight_norm (#114785)

Linalg API

• Add fp16 support for gemm on CPU (#99498)
• Add quantized int4 support for mm (#111403)

Optimizer API

• Allow torch.float64 scalars for forloop + foreach implementations (#115841, #111008)
• Add NAdam support for complex dtypes, with has_complex shortcut (#110634)
• Set default learning rate (lr) value of SGD to 1e-3 (#114467)
• Add capturable ASGD impl (#107857)

torch.func

• Add vmap support for various in-place operations (#110692, #113513)
• Add vmap support for torch.unsafe_chunk (#110862)
• Add vmap support for Tensor.index_add_ (#112276)
• Add vmap support for torch.linspace and torch.logspace (#105451)
• Add vmap support for torch.linalg.eigh (#110640)
• Add dynamic shapes support for vmap over torch.squeeze and alias (#107577)
• Add dynamic shapes support for vmap over torch.is_same_size and torch.split_with_sizes (#111491)

Misc

• Add torch.utils.checkpoint.set_checkpoint_debug_enabled (#110728)
• StackDataset batched sampling (#110694)
• Add option to flop counter formula registration to get raw values (#110591)

Quantization

• Bits Types
  • Enable cat for bits types (e.g.torch.bits8) in cuda (#115044)
  • Enable copy/clone/reshape/contiguous operations for bits types (#113508)
• PyTorch 2 Export Quantization:
  • Add reference representation for dynamic quantized linear (#108073)
  • Use input_qspec_map for weight quantization of linear (#107105)
  • Make annotation util functions return annotated nodes (#107106)
  • Add dequantize operator duplication pass (#107900)
  • Add metadata porting for nodes added by quantization (#107107)
  • Move to BFS instead of DFS to check for connectedness (#108572)
  • Support int16 quantization (#108453)
  • Support cat (#108382), conv1d (#109830) and mul (#110428) in XNNPACKQuantizer
  • Enable constant folding for quantize ops (#109343)
  • Add util function to convert scalars to attrs (#110427)
  • Support cudnn_batch_norm (#109908) and miopen_batch_norm (#110653) in QAT fusion
  • Preserve source_fn_stack after QAT fusion (#110899, #111515)
  • Cleanup observer insertion logic (#111828) (#112453)
  • Fix QAT conv-bn bias using DerivedQSpec (#112159)
  • Refactor QAT q-dq patterns (#112279)
  • Add "quantization_tag" as metadata to fx.Proxy (#108764)
  • Inductor cpp wrapper: support QLinear (#112378)
  • Enable QAT Quantization flow in X86InductorQuantizer (#111280)
  • Add ConvBNAdd(ReLU) Annotation (#111281), adaptive_avg_pool2d and flatten (#114442), Hardtanh and ReLU6 for conv2d (#114579) in X86InductorQuantizer
  • Enable oneDNN QConv (#112010) and QLinear (#112126) FP32/BF16 output
  • Enable oneDNN QConv2d with hardtanh post op (#114578)
  • Remove the output Annotation of Conv/Linear in x86InductorQuantizer (#112140)
  • Enable quantize_per_tensor/quantize_per_channel to accept bfloat16 input (#112225)
  • Support quantized conv bias in QAT fusion (#112528)
  • Fix custom dtype per channel weight in QAT (#112612)
  • Support allow_implicit_sharing flag (#112929)
  • Add transform_for_annotation method in Quantizer (#113115)
  • Remove add/relu from conv-bn QAT pattern (#113006)
  • Rewrite QAT annotations using SubgraphMatcherWithNameNodeMap (#113709)
  • Support conv1d-bn QAT fusion (#113714)
  • XNNPACKQuantizer skip quantization for input and output to workaround histogram observer problem (#113405)
  • Add support for QAT dynamic quantization for linear in XNNPACKQuantizer (#113288)
• Support Subclasses of FloatFunctional in torch.ao.quantization.prepare (#109646)
• Enable pickling model prepared with QAT QConfig (#109288)
• Suppress empty translation unit warning in QNNPACK (#111475)
• new_qtensor support privateuseone allocator. (#111464)
• Add support for float8_e4m3fnuz and float8_e5m2fnuz (#107586)
• Overload vec::dequantize to eliminate rounding error for quantized sigmoid (#114098)

NestedTensor API

• Pickle support for NT (#110219)
• Multiprocessing support for NT (#110292)
• pin_memory support for NT (#110404)
• Implement split_with_sizes backward for NT (#110647)
• Multiprocessing support for NT (#110292)
• Support for as_nested_tensor() with jagged layout + fixed nested_tensor() semantics (#112304)
• Do not generate zero-numel NT by default in helper and improve to_padded_tensor msg (#113162)
• Backward support for broadcasting binary ops (#112519)
• Implement narrow from a regular tensor to jagged tensor (#112770)

Distributed

• c10d
  • Make TCPStore more robust to one-time interruptions. (#108425)
  • Add functional collective all_to_all_single and support it in Inductor (#110195)
  • Set ProcessGroupNCCL default timeout to 10 min (#110947)
  • Add an explicit _shutdown method to ProcessGroupNCCL (#111392)
  • Enable coalescing manager in DETAIL debug mode (#111878)
  • Avoid recording stream for all-gather, reduce-scatter, broadcast and scatter (#111431, #112896)
  • Relax tensor contiguity requirement for P2P ops (#114982)
  • Add .boxed() to c10d::ProcessGroup and c10d::Work's pybind (#111997)
  • Make init_process_group timeout kwarg override pg_options (#112611, #113094)
  • Use allocator trace callbacks for ProcessGroupNCCL register (#112850)
  • Make FakeProcessGroup traceable (#113314)
  • Add Bfloat16 scalar support to gloo backend (#113557)
  • Opportunistically use ncclCommSplit when creating new NCCL groups (#114385)
  • Add API _set_group_name and group_name to track pg names in C++. (#108813)
• Distributed Checkpointing (DCP):
  • Stateful Checkpointing for Distributed (#113867)
• DistributedDataParallel (DDP):
  • Add an API to DDP for dynamically updating the underlying process group. (#113580, #114194)
• DTensor
  • Supported convolution ops (#113123)
  • Add DTensor constructor: randn (#108285)
  • Add grad placements kwarg to to_local API (#110629)
  • Support aten.where and enabled implicit scalar promotion (#110584)
  • Support lt/gt op (#110585)
  • Enable DTensor TP in the inference mode (#110751)
  • Refactor Parallel Style and TP API to improve UX (#111160, #111166, #111176, #111346, #111353, #111625, #111521)
  • Enable embedding sharding in TP API (#111177)
  • Enable adagrad foreach support (#114151)
  • Enable RMSprop optimizer foreach support (#114152)
  • Introduce full_tensor API to DTensor (#112224, #113322)
  • Enable foreach operators for adam optimizer (#112108)
  • Don’t use make_fx for strategy propagation (#108262)
  • Add assert of shard dim to be less than tensor ndim (#112404)
  • Add rand_like, randn_like, randint_like ops to shard propagation (#112576)
  • Enable min, max and prod sharding propagation rules (#112403)
  • Add support for layer norm op in DTensor (#113105, #113244)
  • Add foreach_zero_ support (#113897)
  • Make _Partial, Replicate frozen dataclasses (#113919)
  • Make replicate -> partial DTensor do division instead (#110898)
  • Use new placements for neg dim in redistribute (#113924)
  • Use new placements for neg dim in from_local (#114134)
  • Use new placements for neg dim in distribute_tensor (#113930)
  • Ensure grad_placements was tuple (#113925)
  • Support Xla backend in distribute_tensor API (#110275)
• FullyShardedDataParallel (FSDP):
  • Not materialized ignored modules for FSDP (#108032)
  • Not moved ignored params / buffers to device (#108033)
  • Make checkpoint_wrapper default to NO_REENTRANT (#108435)
  • Make ModuleWrapPolicy callable (#109117)
  • Enable cpu_offload config for optimizer state_dict (#108434)
  • Enable FSDP on CPU when GPU is still available (#112145, #112144)
  • Add cpu_only and ranks_only support for _gather_state_dict (#112836)
  • Implement cpu_offload and full_state_dict for get_state_dict (#112837)
• TorchElastic:
  • Ensure grandchild processes are restarted correctly (#113231)
  • Avoid terminating parent process if exit code from child isn't valid (#111961)

CPU

• Use cpuinfo to determine c10::ThreadPool thread number (#107010)
• Add Half support for cummax, cummin, cumprod, logcumsumexp, and prod on CPU (#112132)
• Add Half support for poisson and use float for Half cumulative distribution on CPU (#112124)
• Remove memory efficient attention checks (#112375)
• Enable THP for buffer sizes >=2MB (5a4f1363409)

CUDA

• Add lazy initialization for p2p access function (#108589)
• Add support of CudaHostRegister (#108488)
• Create per thread task pool for mapping memory space (#111545)
• Add AMP support to linalg.vecdot. (#108165)
• bfloat16 support in erfinv (#111257)
• Add Bfloat16 support to CrossKernel.cu (#108941)
• Add bf16 support to replicate padding (#112099)
• Preserve operations order between vectorized and non-vectorized in ln grad input (#111488)

Fx

• Add mechanism for make_fx to not error on data-dependent-ops (#114129)
• Preserve non-forward method during torch package serialization (#114702)
• Add Graph input option for replace_pattern (#112409)
• Allow preserving non-forward methods during deepcopy (#114849)
• Replace node.meta source_fn with source_fn_stack (#108595)
• Fix tree spec matching behavior (#109679)
• Assert that output must be the last node of the FX graph (#114973)
• Misc improvements to visualization + utility (#114984)
• Add stylistic improvements for fx.split_module (#113373)

Jit

• Skip builtins while enumerating class methods (#91805)
• Support lovelace for NVRTC (#87611)
• Add expanded symbolic shape support (movedim) (#91696)

MPS

• Add complex support for fill, mul, add, sub (#111885, #111937, #108395, #108394)
• Add support for sgn to MPS backend (#110829)
• Add support for new activation functions (Mish, Softshrink) (#109786) (#110814)
• Generalize toAccumulateType() (#108248)

ONNX

• torch->onnx export support: quantized::linear_relu (#109755)
• Add Half for aten2, logaddexp, logaddexp2, hypot, and nextafter on CPU (#112138)
• Support None in fx.args as torchlib inputs (#108708)
• Support attn_mask fp16 type (#110306)
• A better way to safe guard 2GB model serialization (#111984)
• Fix scalar type promotion between fp16 tensor and fp32 scalar (#113404)
• Add 'aten::rsub' type promotion (#113697)
• Relax unsupported node analysis on complex dtype (#113785)

ROCm

• enable hipSparse const descriptors for version >= 2.4.0 (#110317)

Vulkan

• Improve binary operators to be able to handle the other argument being a 0-dim tensor (#109035)
• Improve binary operators to automatically convert the other argument to float in order to handle mismatched input dtype (#114145)
• Improve aten::addmm and aten::linear to be able to broadcast the bias argument (#108199)
• Improve aten::layernorm to be able to handle 2D inputs (#110796)
• Improve aten::slice to be able to return 0-size output (#112879)

Bug fixes

Autograd API

• Fix in-place custom autograd Functions to not fail when grad returned from backward is undefined (#108353)
• Update custom Function preserve torch function when inputs returned as-is (#109825)
• Do not error when printing view created in no-grad modified in-place in no-grad (#113716)
• Fix torch.autograd.gradcheck when fast_mode=True and default device is set (#114560)
• Fix torch.prod double backward when input tensor contains more than one zero (#113969)

Cpp API

• Check results dtype in index_out (#108167)
• Add the appropriate check on div_value to the cpp frontend (#114671)
• Add input check at the beginning for C++ API interpolate (#108506)
• Fix the coredump described by #106702 (#108002)
• Fix torch.nn.GRUCell segfaulting (#108340)
• Add checks to num_layers for RNN, LSTM, GRU (#108853)
• torch::nn::AdaptiveLogSoftmaxWithLoss: check length of cutoffs (#106777)

Foreach API

• Fix 0-size handling for real (#109402)

Linalg API

• Fallback to GEMM if mkldnn_matmul fails on aarch64 (#115936)
• Preserve input's NaN values to prevent undefined behavior for matrix_exp function (#111539)

NestedTensor API

• Fix torch.load(..., weights_only=True) for NT (#112516)

Optimizer API

• ReduceLROnPlateau now subclasses LRScheduler (#113659)
• Fix adagrad sparse handling due to incorrect early exit (#110454)
• Solving pickle error when saving CyclicLR state_dict (#110931)

Python API

• Fix type checking of lazy submodule import (#109683)
• Fix unhandled exceptions in torch.{finfo,iinfo} calls (#109743)
• Fix torch.{size|stride}(dim=None)` (#111991)
• Fix legacy typed storage warning line pointer (#113601)
• Fix cpu detection error handling (#113771)

Sparse API

• Fix semi-structured sparse shape mismatch bug (#110420)

torch.compile

Dynamo

• Add torch.distributed get_rank and get_world_size to constant_fold_functions (#109029)
• Implement traceable torch.tensor when you have SymInt/SymFloat inputs (#109515)
• Avoid throwing exception in ClosingTHPObjectPtr (#109758)
• Fix inductor CI (by updating graph break count) (#110160)
• Error if you try to run Dynamo compiled function under torch.jit.trace (#111321)
• Adjust _list_with_default to also work with SymInt input (#113073)
• Avoid eager imports of classes with custom VariableTrackers (#112319)
• Uniformly use SourcelessBuilder to handle user defined types (#113390)
• Register SymInt-aware meta function for mm out, symintify resize (#113202)
• use sourceless builder for builtin getattr (#113340)
• use sourceless builder for builtin getattr (#113340)
• Don't toggle torch logger to NOTSET if it is not set; always use pre-existing (#113842)
• Fix dict.get with no default (#115048)
• Improve support for list subclasses (#115052)

Inductor

• Properly handle unbacked symint in various scenarios (#109603, #109609, #111803)
• Using floating point 0 rather than integer 0 as default value for tl.load (#113047)
• Avoid passing a None 'generator' argument to aten.rand which does not accept a generator argument (#112240)
• Avoid recursion error because of accumulating too much computation in a pointwise IRNode (#112320)
• Fix 0-sized views of tensors in cudagraphs (#109055)
• Explicitly use the result's dtype for 'other' values in a masked load to avoid unexpected type promotion (#109325)
• Work around triton issue that loading from int1 pointer returns int8 (#110388)
• Avoid max-autotune benchmarking messing up the random number generator (RNG) state (#111381)
• Fix an out of shared memory issue by avoiding a single invalid triton config causes fatal problem (#112916)
• Make TORCH_COMPILE_DEBUG=1 work again (#112917)
• Fix inductor <> ddp_optimizer issue (#108081)
• Fix visualize_overlap for Inductor comm reordering (#113066)
• Fix cat decomp that the first tensor was returned if it is empty and there is only one non-empty tensor (#113514)
• Correctly codegen math.inf in Inductor (#114159)
• Do not promote int to float for torch.mm (#115043)
• Dont pad broadcasting bias dimension in pad mm (#115098)
• Bug fix for the CPU backend
  • Fix argmax with >1 reduction dims (#113168)
  • Fix add/sub with uint8 dtype to avoid unexpected type promotion (#113253)
  • Fix non-contiguous reduction store (#113261)
• Bug fix for the Fx pattern matching passes
  • Fix a bug in the merge getitem cat pattern (#113822)
  • Fix shape mismatch in SDPA pattern matcher (#115038)
• AOTInductor
  • make AOTInductor work with pip installed torch (#108319)
  • Fixing a redefining symbol bug (#110041)
  • Make freezing work with AOTInductor (#110055)
  • Make a free function in AOTInductor header file inline to avoid redefining symbol error (#110445)
  • Fix a weight loading issue when the weight size can be 0 (#114280)
  • Handle empty input args (#114682)

torch.export

• Fix how pass base uses constant fake tensors (#111140)

torch.func API

• Fix vmap support for `torch.real, torch.imag (#110508)
• Fix vmap support for torch.isfinite, torch.isreal, and torch.log_sigmoid (#110896)
• Fix vmap support for torch.movedim, torch.tensor_split, Tensor.to, to.* (#110999)
• Fix vmap support for torch.flatten, torch.linalg.*, torch.linear, torch.log_softmax, torch.logdet, torch.special.* (#110985)

torch.nn API

• Fix precision issues for torch.nn.LayerNorm on CPU (#108089)
• Madforward outputs of type collections.namedtuple are preserved instead of being changed to tuple when there are backward hooks on nn.Module (#112433)
• Fixug in mem_eff kernel with attention mask and MQA (bc244ee2cdc)
• Fix allowed dtypes for CUDA devices less than SM80 for memory_efficient_attention (#116026)
• Enfced that both input tensors to nn.CosineEmbeddingLoss have the same size (#112782)
• Fix type hints for nn.Module.to (#108767)
• Fix torch.nn.utils.rnn.pad_sequence type hint to allow sequences to be an iterable (#108765)
• Fix num_batches_tracked of nn.BatchNorm{*}D in load_state_dict to not be reset to 0 if the state_dict does not contain num_batches_tracked (#110850)
• Fix convert_sync_batchnorm to return SyncBatchNorm layer with same training flag as BatchNorm layer being converted (#111998)
• Fix 64-bit indexing support for cross-entropy CUDA kernels (#112096)

Build

• Fix finding Intel MKL, LAPACK, cuDNN and cuSPARSELt on Windows (#108040)
• Fix ppc64le clang compilation errors (#106446)
• Compile FBGEMM with ASAN (#111266)

Composability

• FakeTensors and meta tensors are used to perform shape propagating when tracing out a graph in torch.compile. There were a number of op coverage improvements this release:
  • Bugfixes to several aten operator meta implementations.
    • index_select.out (#111364)
    • meta_randperm (#109721)
    • qlinear_pointwise (#112390)
  • Other meta bugfixes: (#108988, #113634, #108989, #111705, #113635)
• There were several bugfixes to our python decompositions and reference implementations of our aten operators this release:
  • Operator specific
    • Several FFT operators (#108360, #109083)
    • threshold_backward (#110689)
    • aten.add with int32 + scalar input (#113965)
    • aten.add non-contiguous out handling (#111758)
    • aten.normal with strided layout input (#111205, #112467)
    • aten.baddbmm (#109714)
    • index_add (#108826)
    • aten.split_with_sizes (#113984)
  • General bugfixes
    • fix infinite loop with primtorch and .to(meta) (#109632)
    • Fix registering jit decompositions for jvp for out wrapped decomps (#109367)
    • Fix python decomps for OpOverloadPackets and add tests (#107707)
• fix issue with lift_fresh_copy when using export + compile (#108243)
• Removed spurious warnings from calling torch.overrides.get_overridable_functions (#109890)

CPU

• Fix NULL dereference in binary CPU ops (e57f089704a)
• Fix cpuinfo related crash on ppc64 (#110708)

CUDA

• Release GIL in torch.cuda ops wherever possible. (#109159)
• Skipped CUDA Flags if C++ Extension Name includes "arch" Substring (#111211)
• Don't set CUDA_HOME when not compiled with CUDA support (#106310)

Distributed

• C10d
  • Fix gloo cuda sparse_allreduce dispatch (#111485)
  • Add timeout for master store if clients do not join (#111805)
  • Add cuda to MPI backend capabilities (#109614)
  • Fix send()/recv() to make them respect the timeout same as non-p2p collectives (#109611)
  • Change default NCCL_ASYNC_ERROR_HANDLING to 3:SkipCleanUp to avoid calling ncclCommAbort which in some cases hangs (#110723)
  • Distributed Checkpointing (DCP):
  • Fix torch.cpu has no attribute current_device in checkpoint/optimizer.py (#110299)
• DTensor:
  • Fix DTensor.from_local() returns DTensor with wrong size for uneven sharded tensor (#110781)
  • Make DTensor handle negative dim correctly and fixed TP regression (#111750)
  • Fix pointwise op strategy linearity (#112107)
  • Fix empty shape init for DTensor constructors (#115091)
• FullyShardedDataParallel:
  • Fix non-Node 0 unable receive parameters from Node 0 for HSDP (#108331)
  • Add device to _shard_utils.py to explicitly use the correct device from fsdp_state (#109631)
  • Propagate requires_grad attribute to unsharded params (#109892)
  • Fix logics for fsdp exec order pre fwd record (#110138)
  • Move local optimizer state to FSDP compute_device (#110929)
  • Fix the FSDP to not reshard parameters twice (#110948)
  • Fix FSDP to reset prefetch flag upon reshard (#111354)
  • Fix FSDP when SHARD_GRAD_OP and forward_prefetch is turned on (#110139)
  • Fix FSDP summon_full_params(..., with_grads=True) when grad precision is not fp32 (#112746)
  • Fix fsdp state_dict to use run_check=False (#114995)
  • Fix pylance issues for torch.distributed.fsdp (#109922)
• RPC:
  • Fix assertion on vector length during message parsing (#108414)

Fx

• Fix functorch.compile.minifier error of “'Node' object is not iterable” (#103011)
  • Skip mode issue in minimizer (#109399)
  • Skip the Tensor node in __annotations__ (#109853)
  • Fixed dict size change during iteration error (#111267)
  • Made sure fx code is valid in python (#113345)
  • Updated symbolic_trace’s nn_module_stack format (#114422)
  • Fixed missing meta for proxy.node (#114659)
• Correctly restore pybind11 error_already_set (#93238)
• Remove proxy tensor's check for data dependent output (#93265)
• Make ShapeEnv deepcopy-able (#93403)
• Fix SubgraphMatcher for case of no anchor found (#86421)
• Fix for partitioner with symbolic shapes (#86425)
• Fix getitem in partitioner and make metadata storage more consistent (#87012)
• Fix magic method try reverse protocol (#88030)
• Fix FakeTensorProp on Module with Parameters or Buffers (#88700)
• Fix PassManager to not use a class variable mutable list (#89108)
• Prevent tracing when we track_tensor_tree (#89139)
• Make all make_fx invocations isolated (opaque to higher make_fx invocations) by default (#93290)
• Fix matching args in PatternMatcher (#94375)
• Allow FakeTensorProp to run on graphs traced with some None inputs (#94569)
• Copy codegen in legalize_graph (#90023)
• Fix proxy unwrapping for cond() (#91907)

Jit

• Fix optimize_for_inference to support modules that don't have a forward method (#110013)
• Fix errors found by fuzzing and sanitizers (#108417, #108413, #110303, #110441)
• Fix deprecated python usage for python 3.12 in TorchScript (#113981)
• Support newer versions of LLVM (#110200, #113455)

Lazy

• Fix error when inferring shape in AdaptiveAvgPool3d (#109822)

Mps

• Fix and refactor unary/binary ops with non-zero offset or non-contiguous output (#97085)
• Fix memory leak in copy_from_mps_ (#114197)
• Fix crash if nonzero is called concurrently (#108996)
• Fix nll_loss with default ignore_index (#109574)
• Fix sort with empty tensor. (#109584)

ONNX

• Fix module attribute retrieval in ONNX export (#109759)
• Add dynamic input support for MaxPool (#113318)
• Fix op-level debug for complex dtype (#114885)
• Fix indexing for meshgrid op (#109350)
• Fix torch.diagonal for torch.onnx.export when dim1<0 or dim2<0 (#111130)
• Fix scope name when parent scope is empty for torch.onnx.export (#112654)
• Cast ‘scale’ back to float16 after _attention_scale. (#112554)
• Fix aten::layer_norm for ONNX opset 17 (#114058)
• Add support for negative dim in _index_fill_reshape_helper (#114050)
• Disable opmath type promotion (#113780)

Profiler

• Fix missing dependency in torch.utils.tensorboard (#115598) (#115598)
• Fix torch.utils.benchmark API while use privateuse1. (#108548)
• Ignore some properties when symbolic size/strides exist (#112458)
• Fix description to use nelems rather than size (#114735)
• Use PyCFunction_Check to check both PyCMethod_Type and PyC… (#110002)
• Disable CUPTI Teardown when using CUDA Graphs (#112507)
• Fix the Chrome trace loading issue with all_to_all input split length > 30 (#113392)

Quantization

• Make mutation test work with quantized tensors (#108935)
• Support transitive sharing for SharedQuantizationSpec (#111172)

Releng

• Fix focus builds of macOS apps on apple silicon. (#96966) (#107816)
• Use jemalloc for CUDA builds (#116900) (#116900)

Visualization

• Fix TensorBoard summary writer encoding for torch.bfloat16 tensors (#108351)

Vulkan

• Fix for a bug in aten::sum.dim_IntList where providing negative dims in the opt_dim argument and setting keepdim=false results in wrong inputs (#111586)

Performance

Autograd API

• Avoid saving input for torch.mean backward (#109935)

Cpp API

• Add ScalarTensor or 0dim overload for _foreach_add (#111079)
• Vectorize torch.exp2 on CPU and add complex support (#92115)
• Add various performance fixes to c++ STL usage (#94034)

Linalg API

• Speedup torch.matrix_exp performance (#110848)
• Improve speedup of cholesky_solve_backward using output_mask (#112981)

NestedTensor API

• Reduce overhead in split and chunk for NestedTensor (#108213)

Optimizer API

• Use for loop with shortcut in Optimizers to speedup inductor against list comprehensions (e.g. complex conversion) (#110613, #112722)
• Speed up dynamo tracing of optimizer by shortcutting is_sparse iteration in foreach SGD (#110648)

Sparse API

• Add NVIDIA A100 optimized meta parameters to bsr_dense_mm (#111760)
• Improve triton bsr_dense_mm performance on column-major ordered inputs with float32 dtype (#108512)
• Add bsr_dense_addmm triton kernel (#114595)
• Use more performant bsr_scatter_mm within bsr_dense_mm when blocksize is 16. (#111489)

torch.compile API

Inductor

• Support convolution layout optimization for models with SDPA (#112045)
• Scaling down XBLOCK/RBLOCK to increase occupancy for kernels exposing large parallelism and having register pressue (#109275, #109839, #113039, #114284)
• Horizontal fusion for concat (#111437)
• Avoid an extra memory copy for views on ExternKernelAlloc (#108635)
• More memory and performance efficient implementation for Conv+BatchNorm block in eval mode (#109398, #109722)
• Pointwise fuse cat with pointwise inputs or outputs and <= 4 inputs (#111233)
• Add a way to force fusion of int_mm with mul (#111413)
• Add a heuristic to multi-layer reduction to increase the chance that the first splitted reduction can have compatible shape to fuse with a preceding reduction (#111781)
• benchmark fusion: either use this to skip slow fusions or analyze patterns from slow fusions (#112450)
• optimize sympy expression where div denominator is -1 (#112878)
• Use different conv layout optimization heuristics for inference (#114600)
• Add or improve Fx pattern matching passes
  • pre grad batch relu fusion (#111146)
  • new split cat pattern detection (#110923)
  • Add split-stack-tahn-unbind pattern detection (#111854)
  • Remove split nodes with split section size one (#112922)
  • Normalize nodes created by users (#113179)
  • post_grad batched linear fusion (#112504)
  • More SDPA patterns (#109156, #110001)
  • Horizontally fusing two matmuls in freezing phase (#111232)
  • A bunch of pattern matcher + indexing fixes (#112476)
• CPU Backend
  • Fallback scatter_add to eager on CPU to avoid bad perf (#108220)
  • Make OneDNN matmul inputs contiguous to avoid degraded performance (#108560)

torch.func API

• Add vmap batching rule for: bitwise operators (#91971), nansum & nanmean (#91372), all & any (#91966), torch.linalg.vander (#91749), slogdet (#86815), torch.index_fill (#91364), narrow_copy (#88130), view_copy (#88150), greater_equal.Scaler (#91324)

CPU

• S390x complex division (#108516)
• Add Half support for CPU autocast on eager mode (#112484)
• Add scalar conversion using avx instructions for half (#102140)

CUDA

• Release the allocator lock on the slow path (#108367)
• Faster gc_count update for CUDACachingAllocator (#108071)
• baddmm should fall back to addmm for batch=1 (#114992, #114992)
• Speed-up casts to FP8 (#110251)
• int4 mm kernel enhancement (#111460)
• vectorized implementation for layer_norm_grad_input_kernel (#111021)

Distributed

• c10d:
  • Push TCPStore scalability further by staggering client connection and increasing the backlog to 16k. (#109217)
  • Make the minimum wait time in _store_based_barrier to be adaptative based on the number of ranks. (#109218)
• DTensor:
  • Fix and improve the sharding cache behavior (#109306, #109428)
  • Switch DTensor and Functional Collective to use optree (#110670)
  • Skip move to device when device_type match (#110774)
  • Skip pytree when not necessary (#110132)
  • Introduce cost model for sharding (#109145)
  • Group dispatch unwrapping to a method (#113846)
  • Cache hash for DTensorSpec (#113915)
  • Compute and recompute DTensorSpec hash lazily (#114322, #114379)
  • Reduce to one isinstance call in is_shard (#114140)
• FullyShardedDataParallel (FSDP):
  • Fuse allgather for optim_state_dict when use_orig_params is True (#108298)
  • Make the new optimizer allgather fusion work with fine-tuning models (#110540)
  • Skip the parameter in optim state dict if the parameter does not belong to the current FSDP instance (#112804)

Fx

• Use deque instead of list for BFS (#91139)
• Refactor the dfs cyclic search from recursive to iterative approach (#91042)

Vulkan

• Improve matrix multiplication shader performance by up to 50% through new packing schemes and algorithmic improvements (#112918, #113627, #113883, #113943)

Documentation

Autograd API

• Improve docstring issues in various places (#113266)

Dataloader API

• Add and update docstrings for torch.utils.data (#112244, #112817, #112765)

Linalg API

• Fix typo in example of torch.linalg.solve_triangular (#112361)
• Remove duplicate sentences in description of torch.linalg.eig (#108230)
• Fix bug in matrix_power documentation (#108585)

Optimizer API

• Update documentation for PolynomialLR (#110151)
• Clarify maximize option in optimizer.py (#112724)
• Fix docstring errors inside torch/cuda/ and torch/optim/ (#112964)

Python API

• Fix docstring issues in torch.utils (#113335)
• Add docstring to Timer.adaptive_autorange (#111612)
• Fix a typo in torch.cholesky_inverse documentation (#110364)
• Document torch.from_file and UntypedStorage.from_file properly (#111688)
• Clarify difference between Tensor.share_memory_ and torch.from_file (#111856)
• Improve torch.unique docs (#113424)
• Fix torch.lgamma docs (#108719)
• Update torch.take_along_dim docs to include dim=None case (#109120)
• Fix torch.searchsorted docs (#109364)
• Clarify torch.multinomial usage (#112892)

torch.compile API

Inductor

• Add a tutorial for AOTInductor (#112457)
• Add document for cudagraph_mark_step_begin API (#111722)

torch.export API

• Add torch.cond doc (#108691)
• Add ir spec (#110394)
• Update docs to say that export returns full ATen IR (#111161)

torch.func API

• Fix per-sample-grads notebook (#107988)

torch.nn API

• Add examples for nn.CosineEmbeddingLoss(#108215)
• Fix attn_bias in code block in scaled_dot_product_attention documentation (#109086)
• Add documentation for torch.nn.utils.parametrizations.weight_norm (#113783)
• Improve type annotation for device parameters when a device ordinal is allowed (#113647)
• Update scaled_dot_product_attention documentation to point to flash-attn-v2 (#114124)
• Fix extending torch native API docs (#114863)

Build

• Fix doc preview page url at CONTRIBUTING.md (#108580)
• Fix typo in cpp/installing when wheel is used (#111143)

Composability

• Fix ScalarTensor repr in Extending PyTorch example (#86330)
• Fix incorrect wrapping of function decorator (#94446)
• Add all to torch.{autograd, fx, cuda} submodules (#85343)

CUDA

• Show CUDAExtension example commands as code (#112764)
• Rewrite docs to describe CUDACachingAllocator semantics (#113282)

Distributed

• c10d:
  • Fix TCPStore doc for arg wait_for_workers (#111807)
  • Fix the warning messages when avoidRecordStreams_ so that correct name of Environment variable is shown in the warning message (#108759)
  • Fix an incorrect indent in documentation for dist.send (#108273)
  • Fix warnings and descriptions about distributed exceptions in the logging section of PyTorch Distributed document (#110157)
  • Fix batch_isend_irecv example incorrect usage (#110408)
  • Clarify the behavior of apply_optimizer_in_backward in the document (#110903)
  • Correct docstring errors for torch.distributed files (#112735, #113511, #113523, #112693, #113241, #113216)
  • Print NCCL_SUFFIX in NCCL version log at PG init (#112560)
  • Distributed Checkpointing (DCP):
  • Fix the comment no_dist for in load_state_dict (save -> load) (#112217)
  • Improve DDP checkpoint documentation (#106985)
• DistributedDataParallel (DDP):
  • Fix import in DDP notes (#111833)
  • Add errors when using _dynamo.optimize_ddp=True and _inductor.keep_output_stride=False together inside DDP (#108235)
• DTensor:
  • Improve TP documentation (#115880, #115974)
  • FullyShardedDataParallel (FSDP):
  • Fix docstring of FSDP.optim_state_dict_to_load to reflect right ctors (#108383)
  • Fix docstring for FSDP.set_state_dict_type to contain missing Args (#103864)
  • Remove "on CPU" in the comment of FSDP initialization doc (#113753)
• TorchElastic:
  • Fix a typo in rendezvous/registry.py (#111352)
• RPC:
  • Fix torch.distributed.rpc example incorrect usage (#112367)
• Activation checkpointing
  • Clean up comments in activation checkpoint (#86622)
• Distributed (c10d)
  • Improve documentation for various functions (#87018, #94543, #91116,#89905, #86438 )
• DistributedDataParallel
  • Improve Documentation (#86221, #91832)
• RPC
  • Fix non-existing parameters in docstrings in benchmarks (#91115)
• Tensor parallelism and DTensor:
  • Add more clarifications and fix errors in tensor parallelism docs (#94786)
  • Update 2D parallelism API naming and docs (#94771)
• FullyShardedDataParallel
  • Add docs to explain the running the forward pass of of submodules in FSDP (#86343)
  • Clarify warnings to mention collectives (#87478)
  • Remove HSDP Zero-2 from doc (#90503)
  • Improve the comments for FSDP (#92359)
• Distributed Checkpoint
  • Enable documentation for Distributed Checkpoint. (#92813)
• Torch Elastic
  • Fix a minor typo in documentation (#90667)
  • Fix torch.distributed.run init connect timeout by comparing host with the current IP list (#90221)

Mps

• Resolve docstring errors (#113311)

ONNX

• Update exporter issue report instructions for quantized models (#113494)
• Fix sample code in onnx_dynamo.rst (#114770)

Profiler

• Improve the docstring for export_memory_timeline (#110949)
• Improve torch/csrc/profiler/README.md - stubs, RecordFunction, Autograd interaction (#108470)

Quantization

• Add pt2 export quantization to main doc (#110260)
• Add x86 inductor quantization docs (#112648)
• Use \odot everywhere instead of mixing \odot and * for the Hadamard product (#111763)
• Add documentation for prepare_pt2e, prepare_qat_pt2e and convert_pt2e (#110097)
• Docstyle fix for some quantization code (#112992)
• Updating docs for embedding_bag support for fx and eager (#107623)
• fix docstring errors in quantized modules (#112695)

Security

Releng

• Use secure setup-ssh action from test-infra (#111922)
• Automate passing Conda PyTorchBot Test Token for release (#111821)
• Migrate MacOS wheel binary builds to ephemeral M1 runners (#110432)

This release is meant to fix the following issues (regressions / silent correctness):

• Fix crashes for float16 empty tensors (https://github.com/pytorch/pytorch/pull/115183)
• Fix MPS memory corruption when working with tensor slices (https://github.com/pytorch/pytorch/pull/114838)
• Fix crashes during Conv backward pass on MPS devices (https://github.com/pytorch/pytorch/pull/113398)
• Partially fix nn.Linear behavior on AArch64 platform (https://github.com/pytorch/pytorch/pull/110150)
• Fix cosine_similarity for tensors of different sizes (https://github.com/pytorch/pytorch/pull/109363)
• Package missing headers needed for extension development (https://github.com/pytorch/pytorch/pull/113055)
• Improve error handling of torch.set_num_threads (https://github.com/pytorch/pytorch/pull/113684)
• Fix profiling traces generation (https://github.com/pytorch/pytorch/pull/113763)

The Cherry pick tracker https://github.com/pytorch/pytorch/issues/113962 contains all relevant pull requests related to this release as well as links to related issues.

This release is meant to fix the following issues (regressions / silent correctness):

• Remove spurious warning in comparison ops (#112170)
• Fix segfault in foreach_* operations when input list length does not match (#112349)
• Fix cuda driver API to load the appropriate .so file (#112996)
• Fix missing CUDA initialization when calling FFT operations (#110326)
• Ignore beartype==0.16.0 within the onnx package as it is incompatible (#111861)
• Fix the behavior of torch.new_zeros in onnx due to TorchScript behavior change (#111694)
• Remove unnecessary slow code in torch.distributed.checkpoint.optimizer.load_sharded_optimizer_state_dict (#111687)
• Add planner argument to torch.distributed.checkpoint.optimizer.load_sharded_optimizer_state_dict (#111393)
• Continue if param not exist in sharded load in torch.distributed.FSDP (#109116)
• Fix handling of non-contiguous bias_mask in torch.nn.functional.scaled_dot_product_attention (#112673)
• Fix the meta device implementation for nn.functional.scaled_dot_product_attention (#110893)
• Fix copy from mps to cpu device when storage_offset is non-zero (#109557)
• Fix segfault in torch.sparse.mm for non-contiguous inputs (#111742)
• Fix circular import between Dynamo and einops (#110575)
• Verify flatbuffer module fields are initialized for mobile deserialization (#109794)

The https://github.com/pytorch/pytorch/issues/110961 contains all relevant pull requests related to this release as well as links to related issues.