NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
PyPI, #397 most downloaded on PyPI
Tensors and Dynamic neural networks in Python with strong GPU acceleration
Latest stable
2.15.0
Last release 23 days ago
28 Oct 2026
Ships fairly regularly
a new release about every 2 months
Nearly every release is documented
notes for 52 of 52 stable releases
Nothing withdrawn
no release was ever pulled
8 years old
52 releases · first in 2018
Drag across the ruler to compare, or tap a release to open it.
This upgrade is doing the same BC-breaking changes as the DLPack release. Objects in torch.utils.dlpack have been updated to reflect these changes, su…
<table> <tr> <td><strong>Unstable (API-Unstable)</strong></td> </tr> <tr> <td>Updates to the stable libtorch ABI for third-party C++/CUDA extensions</td> </tr> <tr> <td>Symmetric memory that enables easy programming of multi-GPU kernels</td> </tr> <tr> <td>The ability to arbitrarily toggle error or resume on graph breaks in torch.compile</td> </tr> <tr> <td>Expanded wheel variant support to include ROCm, XPU and CUDA 13</td> </tr> <tr> <td>FlexAttention enablement on Intel GPUs</td> </tr> <tr> <td>Flash decoding optimization based on FlexAttention on X86 CPU</td> </tr> <tr> <td>ARM Platform improvements and optimizations</td> </tr> <tr> <td>Enablement of Linux aarch64 binary wheel builds across all supported CUDA versions</td> </tr> </table>
For more details about these highlighted features, you can look at the release blogpost. Below are the full release notes for this release.
The minimum version of Python required for PyTorch 2.9.0 is 3.10. We also have 3.14 and 3.14t available as preview with this release.
This is a reminder that outputs of PyTorch custom operators (that are registered using the torch.library or TORCH_LIBRARY APIs) are not allowed to return Tensors that share storage with input tensors. The violation of this condition leads to undefined behavior: sometimes the result will be correct, sometimes it will be garbage.
After #163227, custom operators that violated this condition that previously returned correct results under torch.compile may now return silently incorrect results under torch.compile. Because this is changing the behavior of undefined behavior, we do not consider this to be a bug, but we are still documenting it in this section as a "potentially unexpected behavior change".
This is one of the conditions checked for by torch.library.opcheck and is mentioned in The Custom Operators Manual
Outputs of PyTorch custom operators are not allowed to return Tensors that share storage with input tensors
For example, the following two custom operators are not valid custom operators:
@torch.library.custom_op("mylib::foo", mutates_args=())
def foo(x: torch.Tensor) -> torch.Tensor:
# the result of `foo` must not directly be an input to foo.
return x
@torch.library.custom_op("mylib::bar", mutates_args=())
def bar(x: torch.Tensor) -> torch.Tensor:
# the result of bar must not be a view of an input of bar
return x.view(-1)
The easiest workaround is to add an extra .clone() to the outputs:
@torch.library.custom_op("mylib::foo", mutates_args=())
def foo(x: torch.Tensor) -> torch.Tensor:
return x.clone()
@torch.library.custom_op("mylib::bar", mutates_args=())
def bar(x: torch.Tensor) -> torch.Tensor:
return x.view(-1).clone()
A common way to get into this situation is for a user to want to create a custom operator that sometimes mutates the input in-place and sometimes returns a new Tensor, like in the following example.
@torch.library.custom_op("mylib::baz", mutates_args=["x"])
def baz(x: torch.Tensor) -> torch.Tensor:
if inplace:
x.sin_()
return x
else:
return x.sin()
This dynamism is not supported and leads to undefined behavior. The workaround is to split the custom operator into two custom operators, one that always mutates the input in-place, and another that always returns a new Tensor.
@torch.library.custom_op("mylib::baz_outplace", mutates_args=())
def baz_outplace(x: torch.Tensor) -> torch.Tensor:
return x.sin()
@torch.library.custom_op("mylib::baz_inplace", mutates_args=["x"])
def baz_inplace(x: torch.Tensor) -> torch.Tensor:
x.sin_()
def baz(x):
if inplace:
baz_inplace(x)
return x
else:
return baz_outplace(x)
PyTorch MPS is only supported on MacOS-14 or later. If you need to use MPS on MacOS Ventura, please avoid updating to Python-3.9 or above
This upgrade is doing the same BC-breaking changes as the DLPack release. Objects in torch.utils.dlpack have been updated to reflect these changes, such as DLDeviceType.
See the PR for details on the exact changes and how to update your code.
torch.cat (#158249)torch.cat now raises ValueError, IndexError or TypeError where appropriate instead of the generic RuntimeError. If you code was catching these errors, you can update to catch the new error type.
dynamo=True for ONNX exporter (#159646, #162726)Previously torch.onnx.export(...) used the legacy TorchScript exporter if no arguments were provied. The ONNX exporter now uses the newer torch.export.export pipeline by default (dynamo=True). This change improves graph fidelity and future-proofs exports, but may surface graph capture errors that were previously masked or handled differently.
Previously in torch 2.8.0:
# API calls the legacy exporter with dynamo=False
torch.onnx.export(...)
Now in torch 2.9.0:
# To preserve the original behavior
torch.onnx.export(..., dynamo=False)
# Export onnx model through torch.export.export
torch.onnx.export(...)
Recommendation: first try the new default; only fall back if you hit blocking issues and report them upstream. Long term solution: fix the root cause instead of relying on fallback or TorchScript exporter.
To enable runtime asserts, use export(..., prefer_deferred_runtime_asserts_over_guards=True). Also kills the allow_complex_guards_as_runtime_asserts flag, merging it into the former option.
Additionally, exported_program.module() will generate a call to a _guards_fn submodule that will run additional checks on inputs. Users who do not want this behavior can either remove this call in the graph, or do exported_program.module(check_guards=False) to avoid the generation.
Opset 20 enables newer operator definitions. If your tooling or downstream runtime only supports opset 18, pin it explicitly. For the latest ONNX operators, you can experiment with opset 23.
Previously in torch 2.8.0:
# opset_version=18
torch.onnx.export(...)
Now in torch 2.9.0:
# To preserve the original behavior
torch.onnx.export(..., opset_version=18)
# New: opset_version=20
torch.onnx.export(...)
# Use the latest supported opset: opset_version=23
torch.onnx.export(..., opset_version=23)
draft_export in exporter API (#161454, #162225)Remove implicit draft tracing from the default exporter path, achieving clearer behaviour and faster failures.
The expensive torch.export.draft_export diagnostic path is no longer auto-invoked (which could take hours on large models). You can still opt in for deep diagnostics:
Previously in torch 2.8.0:
# If both torch.export.export(..., strict=False) and
# torch.export.export(..., strict=True) fail to capture
# the model graph, torch.export.draft_export(...) will be triggered,
# and uses real tensor to trace/export the model.
#
# Inside export_to_onnx.py:
# ... torch.onnx.export(..., dynamo=True)
python export_to_onnx.py
Now in torch 2.9.0:
# To trigger torch.export.draft_export once
# torch.export.export strict=False/True both
# fail:
TORCH_ONNX_ENABLE_DRAFT_EXPORT=True python export_to_onnx.py
torch.onnx.dynamo_export and the onnxrt torch compile backend (#158130, #158258)torch.onnx.dynamo_export is removed. Please use torch.onnx.export instead.
The experimental ONNX Runtime compile backend (torch.compile(backend="onnxrt")) is no longer supported.
torch.onnx.enable_fake_mode (#161222)The dynamo=True mode uses FakeTensors by default which is memory efficient.
Deprecated members in torch.onnx.verification are removed. Previously private torch.onnx.symbolic_opsets* functions will no longer be accessible. Consider making a copy of the source code if you need to access any private functions for compatibility with the TorchScript based exporter.
torch.onnx.symbolic_caffe2 (#157102)Support for caffe2 in the ONNX exporter has ended and is removed.
/d2implyavx512upperregs flag that slows build (#159431)Re-introduced AVX512 optimizations for Windows VS2022 builds, may cause issues with specific versions of VS2022, see #145702
ScalarType to shim conversion and stable::Tensor.scalar_type (#160557)Before, user extensions could only in abstract pass around obfuscated dtypes appearing as int32_ts. Now, users can confidently use torch::headeronly::ScalarType in their extensions for major scalar types. This PR enables ABI stability by adding a translation layer through the shim, so that even if the ScalarType enum values change in the future, user extensions need not fear.
This change adds ScalarType support for user extensions and is only narrowly BC breaking for unpopular dtypes: quint*s, qint*s, Bits*, dummy_uint*s, dummy_int*s, Float8_e8m0fnu, and Float4_e2m1fn_x2 in the use case where an extension retrieves a Tensor dtype of the above and passes it into aoti_torch_call_dispatcher.
pin_memory_device param in torch.utils.data.DataLoader (#158323)We move enabling pin_memory back inside BaseDataLoaderIter. This is required for StatefulDataloader which leveraged BaseDataLoaderIter direclty rather than the Dataloader class init
torch.export.export_for_training API in favor of equivalent torch.export.export API (#158203)torch.export.export_for_training exists because we couldn't migrate internal usages of export to the final IR. Now that we have completed the migration, we deprecated and deleted this API.
__torch_function__ handler to be triggered by elements within a list (#160256)torch.hash_tensor reduction function (#154149)is_fx_symbolic_tracing flag (#161385)Introduces an API for annotating dynamic integer inputs & attributes for torch.compile, by wrapping plain ints with DynamicInt().
DynamicInt objects also work in eager mode, acting as their underlying values when passed as scalar inputs.
a = DynamicInt(4)
y = a + 2 # DynamicInt(6)
z = torch.ones(a) # torch.ones(4)
fn = torch.compile(torch.ones)
fn(a) # compiled fn takes a dynamic integer input
fn(2) # returns torch.ones(2) without recompiling
torch/csrc/stable/ops.h: amax, narrow, new_empty + new_zeros dtype variant, pad, (#159328, #158974, #159508, #161597, #160214)torch::stable::Tensor() default constructor, is_cpu, and get_device_index(#159507, #160212, #160143)torch::stable::accelerator with support for DeviceGuard and Stream (#159679, #160453)torch/headeronly: c10 Macros, STD_TORCH_CHECK, ScalarTypes (like BFloat16 and Half, #158035, #158365, #157912, #158377, #159302, #159414, #159412, #159415, #159411, #159911)TORCH_ERROR_CODE_CHECK in headeronly from AOTI (#159604)wheel from build requirements (#158027)TORCH_STABLE_ONLY is defined in TensorBase.h (#161658)torch/csrc/stable (#158160)zero_() and empty_like(t) to torch/csrc/stable/ops.h (#158866)Add support for CUDA 13.0 in CI/CD builds. Enable CUDA compression mode for binary size reduction for CUDA 13.0 builds (#160956, #161073, #161257, #161663, #161316, #160201, #160770, #161013, #161916, #162268, #162322, #162383, #161833)
Enable CUDA 12.6, 12.8 and 13.0 support for Linux ARM64 CD builds (#162364, #160720, #159481)
Add support for Python 3.14 in CI/CD builds (#156889, #157559, #159261, #159869, #160593, #160788, #161255, #159725)
Enable NVSHMEM integration (#151261, #153010, #154538, #155506, #156685, #158938, #161321, #160778, #159907, #160465)
cudnn_batch_norm_out kernel to replace the autogen approach (#123020)avg_pool3d, max_unpool1d/2d/3d, max_pool3d, max_pool3d bwd pass, and avg_pool3d bwd pass for MPS (#158877,#159789, #156467, #157498, #159089)FlexAttention on Intel GPU (#143553)torch.load under FakeTensorMode by reducing random reads (#157931)torch.utils.benchmark.utils.timer accelerator agnostic (#157131)register_buffer with Tensor-like objects (#159455)NLLLoss (#161412)requires_grad=True to a scalar (#160389)SequentialLR deprecation warning about invoking step(epoch) (#149392)torch.nn.Upsample mode="trilinear" backward (#154239)ProcessGroupNCCL (#156748)ProcessGroupNCCL (#156790)ProcessGroupGloo (#158128)TCPStore (#159165)all_gather (#149913)unsafe_get_ptr for dist.ProcessGroupNCCL.NCCLConfig (#161136)send/recv_object_list (#160342)ProcessGroupGloo (#156633)work.isStarted (#160398)device_mesh argument constraint in local_map (#157049)DTensor slice (#157953)histc op (#158298)propagate_tensor_meta function that skips cache if _are_we_tracing (#161334)local_map as a decorator (#161353)init_device_mesh (#159371)all_gather and reduce_scatter comms (#155189)set_allocate_memory_from_process_group if used together with custom comm hooks (#157487)reduceOpSum when world size is 1 (#157529)allgather when world size is 1 (#160135)post_reduce_stream.record_event() on hsdp+cpuoffload (#160481)parallelize_module API to support more cases (#157182)torchrun (#149334)eval() API to schedule (#157795)OVERLAP_F_B computation type (#158978)DualPipeV schedule (#159591)is_impure() (#151524, #157981)self.module_stack in ModuleStackTracer (#159956)node_name_match to subgraph rewriter (#157574)lists (e.g. #153969)sets (e.g. #153150)dicts (e.g. #154794)iter (e.g. #156371)itertools (e.g. #159693)collections (e.g. #159365)collections.NamedTuple (#159367)dataclasses.dataclass (#159529)TorchDispatchMode to ignore torch.compile internals (#161648){non-inf, > int32_max} upper bound is provided (#159433)None & ellipsis slicing/select in non-strict (#157821)triton_kernel_wrapper_functional HOP (#161314)while_loop HOP subgraphs (#158467)aot_export_joint_with_descriptors and aot_compile_joint_with_descriptors (#158715)prepare_aot_module_simplified for use in next PR (#158319)aot_export_joint_with_descriptors (#159814)aten.add.Scalar (#161332)aten.expand_copy decomp (#161688)aten.linalg_vector_norm (#155111)aten.complex (#160894)HistogramObserver (#156457)bias=None for fbgemm_linear_fp16_weight CPU op (#158535)wrapped_fbgemm_linear_fp16_weight for Sigmoid (#160451)log_softmax() support (#159662)vector.reserve() consistently for non-inplace foreach operations (#161128)has_integral_tensor() (#161042)torch.tensor warning in ONNX symbolic_opset10 export (#158835)AllocatorConfig to be device-agnostic via new AcceleratorAllocatorConfig (#149601, #150312)Scalar::isUnsigned() method (#159877)ModelRunner from nativert as public (#159989)torch.binomial enforcing float inputs (#157658)Dependencies.cmake (#159702)libtorch without NVSHMEM (#160910)repeat_interleave kernel (#157996)kernelLaunchCheck to print help string (#158896)Wunused-but-set-variable (#159276)libc++ (#161101)shifted_chebyshev_polynomial_[tuvw], igamma/igammac,grid_sampler_3d, native_dropout/native_dropout_backward (#157488, #161927, #160541, #162108)index_put to complex types (#160159)addmm to integral types (#160270)kthvalue (#161817)logcumsumexp metal kernel (#156858)dlpack integration (#158888)avg_pool2d to use Metal kernel when ceil_mode=True (#161011)composable_kernel (CK) backend user interface to improve user experience (#152951)rocSOLVER for Cholesky inversion. (#157154)torch.backends.miopen.immediate to toggle MIOpen Immediate Mode instead of relying on deterministic=True and benchmark=False (#158951)reshape_ or unexpectedly change memory formats (#161687)device_id to Intel GPU properties to distinguish iGPUs with identical names (#156481)torch.utils.cpp_extension.load_inline to override gencode (#156850)max_width computation in Tensor printing (#126859)pin_memory error message on CPU-only systems (#159994)F.embedding DTensor-aware (#162117)torch.autograd.Function memory leak due to torch.utils.checkpiont early stopping (#161171)torch.autograd.graph.GradientEdge for torch.autograd.Function (#160098)setGroupName and setGroupDesc in group_split and merge_remote_group (#159429)batch_isend_irecv with 2D tensor views by making P2P tensors dense (#163719)allgather/reducescatter inputs (#163712)DDPOptimizer and donated buffers (#160745)OpSchema equality check (#161231)grouped_mm strategy for invalid stride cases (#158245)F.one_hot in DTensor (#162307)ShardingPropagation cache if compiling (#156868)pin_memory (#157147)NO_SHARD correctly by flattening tensors before copying (#154369)fsdp_pre_all_gather (#160817)set_reduce_scatter_divide_factor errors and MixedPrecisionPolicy (#155964)no_grad() (#159293)eval() (#159475)import torch if compiled without TensorPipe (#159461)torch.distributed.elastic.multiprocessing.start_processes() (#160396)split_module with symint (#160093)getattr_recursive with ModuleList (#161204)torch.Tensor (#162224)torch.compiler.reset() (#156527)FlexAttention (#163677)return_lse warning message in FlexAttention (#163578)FlexAttention head broadcast (#163426)constant_pad_nd (#159878)MutationOutput Buffer (#162020)NaN behavior (#159308)dtype consistency (#160851)FallbackKernel alias function to avoid incorrect aliasing for custom ops (#163227)load_constants (#161887)gen_aoti_c_shim (#159904)aoti_torch_as_strided (#162118)wait_tensor returned tensor (#159502)all_reduce (#159818)from_node provenance in unlift pass (#157943)NaN serialization (#155359)nn_module_stack for assert_tensor_metadata nodes (#159625)move_to_device_pass (#159992, #160528, #162301)dynamo.disables in unflattening (#161306).contiguous() when saving weights to raw bytes to preserve original storage size of tensor (#163587)NaN in fp8 output of CPU qlinear and qconv ops (#160957)choose_qparams_optimized (#161966)chunk_size should always be int64_t for Foreach functors (#156872)rotary_embedding_23 implementation (#162865)None as output (#160200)dynamo=True (#161056)index_put_ usage (#161263)TORCH_CUDA_ARCH_LIST to only log when running on non-distributed or on rank 0 (#162764)torch.utils.cpp_extension parser for clang version 20.1.7+libcxx (#157666)MakeTensor::computeStorageSize() calculation (#158690)AllocatorConfig (#159629)BUILD_BUNDLEPTXAS=1 to allow compile on newer GPUs(#163988)torch.backends.cuda.matmul.fp32_precision (#161102)cudaErrorNotSupported (#162412)__syncthreads in MultiMarginLoss backward (#158994)kernel size != 1 for cuDNN 9.8+ (#163581)MKLDNN path (#162168)tensor dim > INT_MAX (#158824)exponential_ for MPS (#159386)ConvTranspose3d (#160345)conv3d autocast bias dtype mismatch (#160423)torch.var on scalar (#160889)index_add for complex + int64, int64 input + zerodim index (#160926, #161511)constant_pad_nd_mps bug when pad is empty (#161149)index_select for scalar_types (#161206)index_copy for scalars and index_copy for strided indices (#161267, #161333)NaNs if SDPA is called with all values masked from query (#157727)scaled_dot_product_attention using MPS (#163598)fillBuffer into 4Gb slices to avoid regression on MacOS 26 (#164108)hip:0 device error (#161221)cpp_extension.py on Windows (#159790)logsumexp needs scaling back to natural base (#156903)memcpy_and_sync inline function (#158165)cpp_extension compatibility with intel-deep-learning-essentials-2025.2 (#161012)ErrorReport::CallStack thread-safe (#160386)RemoveProfileNodesAndSpecializeTypes handling for Tensor? that is resolved to None (#161538)addmm to improve Newton–Schulz orthogonalization in Muon (#161379)AveragedModel.update_parameters() (#157705)dict tag optimization for faster guard evaluation (#159183)GEMM template (#159127, #161148)fmod.Scalar and scale_gradient (#160654, #160454)torch.full for 1-byte types (#158874)argmax/argmin (#159524)max_pool3d (#157875)max_pool3d impl (#157874)max_pool2d to Metal for stride != 1 (#157876)hipblaslt is used by default on gfx908 for ROCm >= 6.3 (#159092)tensor.item() (#158486)fbgemm_gpu genai sources for grouped GEMM support (#160676)torch.lobpcg, torch.clone, torch.matmul, torch.max, torch.gather, torch.Tensor.scatter_, torch.empty_like, torch.randint, torch.mul, torch.min, torch.max. torch.sort, torch.full_like, torch.histogramdd, torch.hamming_window (#156139, #157007, #161424, #156153, #157929, #157920, #158050, #158731, #160312, #161539, #162051, #158275, #152682)torch.set_float32_matmul_precision docs (#158191)torch.nn.utils.clip_grads_with_norm_ to reflect clamping behavior (#158200)torch.gradient (#159130)torch.segment_reduce docs (#154352)torch.is_floating_point and torch.is_complex docs (#161951)padding for avg_poolnd (#159142)CrossEntropyLoss docs with example of incorrect target specification (#155649)scaled_dot_product_attention example (#161613)torch.optim.adam.Adam, properly (#158483, #158669, #160194)zero_grad description (#161239)torch.inference_mode docs and error message (#161164)device_id (#159389)distributed_c10d.py (#162158)TupleStrategy (#158132)redistribute_costs (#158495)torch/ (torch/fx/, #156604)torch/fx/experimental/migrate_gradual_types/util.py (#157236)symbolic_multi_out (#160702)onnx.md to simplify deprecated entities (#159312)fallback=False by default (#162622, #162726)waitcounter for watchdog and heartbeat monitoring thread (#157480)torch.distributed.breakpoint set a long timeout (#158481)check_rng_sync util (#160283)FlightRecorder support for ProcessGroupXCCL (#158568)early_stop kwarg to torch.utils.checkpoint (#160781)register_foward_pre_hook not supported on ScriptModule error (#156904)__eq__ function to NodeSource (#158170)__hash__ function to NodeSource (#158322)tlparse error (#158469)expanded_def option for FX printing, render descriptor, update tests (#158708)co_lnotab in favor of co_linetable (#159227)CompiledFxGraph (#159311)allow_tf32 in tl.dot(..., allow_tf32=...), use tl.dot(..., input_precision=...) (#160711)jit_post_compile_hook within Inductor Triton Kernel compile path (#161443)torch.backends.cuda.matmul.allow_tf32 in Inductor cache key (#159480)guard_size_oblivious on data dependent errors (#160510)torch.utils.data samplers benchmark script (#156974)torch.utils.data.Dataloader benchmark script (#159432)setup.py develop with pip install -e for development builds (#155998, #156027, #156710) (#156709)Your coding agent can read these notes before it upgrades. Set up the MCP server →
This is a BC-breaking change for the build system interface. Downstream projects that previously got NVTX3 through cmake/public/cuda.cmake (i.e.. call…
<table> <tr> <td><strong>Unstable</strong></td> </tr> <tr> <td>torch::stable::Tensor</td> </tr> <tr> <td>High-performance quantized LLM inference on Intel CPUs with native PyTorch</td> </tr> <tr> <td>Experimental Wheel Variant Support</td> </tr> <tr> <td>Inductor CUTLASS backend support</td> </tr> <tr> <td>Inductor Graph Partition for CUDAGraph</td> </tr> <tr> <td>Control Flow Operator Library</td> </tr> <tr> <td>HuggingFace SafeTensors support in PyTorch Distributed Checkpointing</td> </tr> <tr> <td>SYCL support in PyTorch CPP Extension API</td> </tr> <tr> <td>A16W4 on XPU Device</td> </tr> <tr> <td>Hierarchical compilation with torch.compile</td> </tr> <tr> <td>Intel GPU distributed backend (XCCL) support</td> </tr> </table>
For more details about these highlighted features, you can look at the release blogpost. Below are the full release notes for this release.
Due to a bug introduced in CUDA 12.9.1, we are unable to complete full Windows wheel builds with this
version, as compilation of torch.segment_reduce() crashes the build. Thus, we provide a wheel
without torch.segment_reduce() included in order to sidestep the issue. If you need support
for torch.segment_reduce(), please utilize a different version.
Due to binary size limitations, support for sm50 - sm60 architectures with CUDA 12.8 and 12.9 has been dropped for the 2.8.0 release. If you need support for these architectures, please utilize CUDA 12.6 instead.
NotImplementedError instead of RuntimeError (#155470)Please update exception handling logic to reflect this.
In 2.7.0
try:
torch.nn.Hardshrink()(torch.randint(0, 5, (10,)))
except RuntimeError:
...
In 2.8.0
try:
torch.nn.Hardshrink()(torch.randint(0, 5, (10,)))
except NotImplementedError:
...
autograd.Function (#153094)In 2.8.0, if a custom autograd.Function mutates a view of a leaf requiring grad,
it now properly raises an error. Previously, it would silently leak memory.
class Func(torch.autograd.Function):
@staticmethod
def forward(ctx, inp):
inp.add_(1)
ctx.mark_dirty(inp)
return inp
@staticmethod
def backward(ctx, gO):
pass
a = torch.tensor([1.0, 2.0], requires_grad=True)
b = a.view_as(a)
Func.apply(b)
Output:
Version 2.7.0
Runs without error, but leaks memory
Version 2.8.0
RuntimeError: a view of a leaf Variable that requires grad is being used in an in-place operation
tensordot when called with a requires_grad=True tensor (#150270)Please avoid passing an out tensor with requires_grad=True as gradients cannot be
computed for this tensor.
In 2.7.0
a = torch.empty((4, 2), requires_grad=True)
b = torch.empty((2, 4), requires_grad=True)
c = torch.empty((2, 2), requires_grad=True)
# does not error, but gradients for c cannot be computed
torch.tensordot(a, b, dims=([1], [0]), out=c)
In 2.8.0
a = torch.empty((4, 2), requires_grad=True)
b = torch.empty((2, 4), requires_grad=True)
c = torch.empty((2, 2), requires_grad=True)
torch.tensordot(a, b, dims=([1], [0]), out=c)
# RuntimeError: tensordot(): the 'out' tensor was specified and requires gradients, and
# its shape does not match the expected result. Either remove the 'out' argument, ensure
# it does not require gradients, or make sure its shape matches the expected output.
mark_dynamic applied now correctly errors (#152661)Prior to 2.8, it was possible for a guard on a symbolic shape to be incorrectly
omitted if the symbolic shape evaluation was previously tested with guards
suppressed (this often happens within the compiler itself). This has been fixed
in 2.8 and usually will just silently "do the right thing" and add the correct
guard. However, if the new guard causes a tensor marked with mark_dynamic to become
specialized, this can result in an error. One workaround is to use
maybe_mark_dynamic instead of mark_dynamic.
See the discussion in issue #157921 for more context.
Version 2.7.0
import torch
embed = torch.randn(2, 8192)
x = torch.zeros(8192)
torch._dynamo.mark_dynamic(x, 0)
@torch.compile
def f(embedding_indices, x):
added_tokens_mask = torch.where(x > 10000, 1, 0)
ei = torch.narrow(embedding_indices, 1, 0, x.size(0))
return ei.clone()
f(embed, x)
Version 2.8.0
import torch
embed = torch.randn(2, 8192)
x = torch.zeros(8192)
torch._dynamo.maybe_mark_dynamic(x, 0)
@torch.compile
def f(embedding_indices, x):
added_tokens_mask = torch.where(x > 10000, 1, 0)
ei = torch.narrow(embedding_indices, 1, 0, x.size(0))
return ei.clone()
f(embed, x)
torch.compile have been renamed or removedenable_cpp_framelocals_guard_eval has changed to no longer have any effect (#151008).rocm.n_max_profiling_configs is deprecated (#152341).
Instead, use ck-tile based configs rocm.ck_max_profiling_configs and
rocm.ck_tile_max_profiling_configs.autotune_fallback_to_aten is deprecated (#154331).
Inductor will no longer silently fall back to ATen. Please add "ATEN" to
max_autotune_gemm_backends for the old behavior.use_mixed_mm and mixed_mm_choice are deprecated (#152071). Inductor now supports prologue fusion, so there is no need for
special cases now.descriptive_names = False is deprecated (#151481). Please use one of the other available
options: "torch", "original_aten", or "inductor_node".custom_op_default_layout_constraint has moved from inductor config to functorch config (#148104). Please reference it via
torch._functorch.config.custom_op_default_layout_constraint instead of
torch._inductor.config.custom_op_default_layout_constraint.emit_current_arch_binary is deprecated (#155768).aot_inductor.embed_cubin has been renamed to aot_inductor.embed_kernel_binary (#154412).aot_inductor.compile_wrapper_with_O0 has been renamed to compile_wrapper_opt_level (#148714).HigherOrderOperators (e.g. cond), which will explicitly error out if alias/mutation among inputs and outputs is unsupported (#148953, #146658).For affected HigherOrderOperators, add .clone() to aliased outputs to address this.
Version 2.7.0
import torch
@torch.compile(backend="eager")
def fn(x):
return torch.cond(x.sum() > 0, lambda x: x, lambda x: x + 1, [x])
fn(torch.ones(3))
Version 2.8.0
import torch
@torch.compile(backend="eager")
def fn(x):
return torch.cond(x.sum() > 0, lambda x: x.clone(), lambda x: x + 1, [x])
fn(torch.ones(3))
guard_or_x and definitely_x have been consolidated (#152463)We removed definitely_true / definitely_false and associated APIs, replacing them with
guard_or_true / guard_or_false, which offer similar functionality and can be used to
achieve the same effect. Please migrate to the latter.
Version 2.7.0
from torch.fx.experimental.symbolic_shapes import definitely_false, definitely_true
...
if definitely_true(x):
...
if definitely_false(y):
...
Version 2.8.0
from torch.fx.experimental.symbolic_shapes import guard_or_false, guard_or_true
...
if guard_or_false(x):
...
# alternatively: if guard_or_false(torch.sym_not(y))
if not guard_or_true(y):
...
torch.export.export_for_inference has been removed in favor of torch.export.export_for_training().run_decompositions() (#149078)Version 2.7.0
import torch
...
exported_program = torch.export.export_for_inference(mod, args, kwargs)
Version 2.8.0
import torch
...
exported_program = torch.export.export_for_training(
mod, args, kwargs
).run_decompositions(decomp_table=decomp_table)
strict=False in torch.export.export and export_for_training (#148790, #150941)This differs from the previous release default of strict=True. To revert to the old default
behavior, please explicitly pass strict=True.
Version 2.7.0
import torch
# default behavior is strict=True
torch.export.export(...)
torch.export.export_for_training(...)
Version 2.8.0
import torch
# strict=True must be explicitly passed to get the old behavior
torch.export.export(..., strict=True)
torch.export.export_for_training(..., strict=True)
torch.onnx.export is now 18 (#156023)When dynamo=False, the default ONNX opset version has been updated from 17 to 18. Users can set opset_version to explicitly select an opset version.
Version 2.7
# opset_version=17
torch.onnx.export(...)
Version 2.8
# To preserve the original behavior
torch.onnx.export(..., opset_version=17)
# New: opset_version=18
torch.onnx.export(...)
JitTraceConvertStrategy has been removed (#152556)Support for JIT traced and scripted modules in the ONNX exporter when dynamo=True has been removed. You are encouraged to export an nn.Module directly, or create an ExportedProgram using torch.export before exporting to ONNX.
onnxscript>=0.3.1 is required for the dynamo=True option (#157017)You must upgrade onnxscript to version 0.3.1 or higher for it to be compatible with PyTorch 2.8.
torch/types.h include from Dispatcher.h (#149557)This can cause build errors in C++ code that implicitly relies on this include (e.g. very old versions of torchvision).
Note that Dispatcher.h does not belong as an include from torch/types.h and was only present as a
short-term hack to appease torchvision. If you run into torchvision build errors, please
update to a more recent version of torchvision to resolve this.
DLPack to 1.0 (#145000)As part of the upgrade, some of the DLDeviceType enum values have been renamed. Please switch
to the new names.
Version 2.7.0
from torch.utils.dlpack import DLDeviceType
d1 = DLDeviceType.kDLGPU
d2 = DLDeviceType.kDLCPUPinned
...
Version 2.8.0
from torch.utils.dlpack import DLDeviceType
d1 = DLDeviceType.kDLCUDA # formerly kDLGPU
d2 = DLDeviceType.kDLCUDAHost # formerly kDLCPUPinned
...
cmake/public/cuda.cmake to cmake/Dependencies.cmake (#151583)This is a BC-breaking change for the build system interface. Downstream projects that previously got NVTX3 through cmake/public/cuda.cmake
(i.e.. calling find_package(TORCH REQUIRED)) will now need to explicitly configure NVTX3 support in the library itself (i.e. use USE_SYSTEM_NVTX=1).
The change is to fix the broken behavior where downstream projects couldn't find NVTX3 anyway due to the PROJECT_SOURCE_DIR mismatch.
Version 2.7.0:
-DUSE_SYSTEM_NVTX would be able to find NVTX3 and torch::nvtx3 via PyTorch's cmake/public/cuda.cmake logic.-DUSE_SYSTEM_NVTX would encounter build errors with CUDA 12.8 or above.Version 2.8.0:
-DUSE_SYSTEM_NVTX will not be able to find NVTX3 or torch::nvtx3 via PyTorch's cmake/public/cuda.cmake. The downstream project now needs to explicitly find NVTX3 and torch::nvtx3 by implementing the same logic in PyTorch's cmake/Dependences.cmake.-DUSE_SYSTEM_NVTX will proceed building without NVTX unless another part of the build process re-enables NVTX.PyTorch 2.8 is the last release that will support GPU acceleration on MacOS Ventura. In the next release (2.9), MacOS Sonoma (released in Sept. 2023) or above will be required to use the MPS backend.
torch.ao.quantization is deprecated and will be removed in 2.10 (#153892)To migrate:
torch.ao.quantization.quantize, torch.ao.quantization.quantize_dynamic)
torchao eager mode quantize_.torchao PT2E quantization.torch.ao.quantization.quantize_fx.prepare_fx, torch.ao.quantization.quantize_fx.convert_fx): use torchao PT2E quantization (torchao.quantization.quantize_pt2e.prepare_pt2e, torchao.quantization.quantize_pt2e.convert_pt2e).Note that PT2E quantization has been migrated to torchao (https://github.com/pytorch/ao/tree/main/torchao/quantization/pt2e). See https://github.com/pytorch/ao/issues/2259 and https://docs.pytorch.org/ao/main/quick_start.html#pytorch-2-export-quantization for more details.
dynamo=False (current default) option for torch.onnx.export is deprecated (#152478, #155580)The default will be dynamo=True starting from PyTorch 2.9. You are encouraged to migrate to use the dynamo=True option in torch.onnx.export. This flag makes torch.export.export the default export path, replacing TorchScript.
To maintain the old behavior, set dynamo=False explicitly. You are encouraged to also experiment with the fallback=True option that will make the exporter fall back to the dynamo=False path if there are errors.
nested_compile_region (#156449)guard_filter_fn (#150936)dont_skip_tracing decorator to skip over most Dynamo skipfiles rules (#150586)draft-export, an export variant designed to consistently produce a graph and generate a debugging report of issues encountered during tracing (#152637, #153219, #149465, #153627, #154190, #155744, #150876, #150948, #151051, #151065, #150809, #151797)TorchBind objects (#150196, #154265)aot_inductor.model_name_for_generated_files for specifying model name (#154129)MPSInductor: torch.compile for Apple GPUs (#150121, #149342, #151449, #151754, #149687, #149180, #149221, #153598, #152788, #153787, #152214, #151152, #155891, #154578, #151272, #151288, #153997, #151871, #153362, #156566, #150661, #153582)Added new strategy draft_export (#147529, docs) to provide debugging information upon data-dependent / constraint errors when obtaining an ExportedProgram with torch.onnx.export
Added support for symbolic operators in the dynamo=True export path (#148905, #149678, #150038, docs). Two operators torch.onnx.ops.symbolic and torch.onnx.ops.symbolic_multi_out are defined to allow you to create symbolic ONNX operators directly in your PyTorch models. You can use them in a forward method:
def forward(self, x: torch.Tensor) -> torch.Tensor:
# Optionally use is_in_onnx_export to control the behavior during onnx export
if torch.onnx.is_in_onnx_export():
# Create a symbolic ONNX operator with the name "CustomOp" in the "custom_domain" domain.
# The output tensor will have the specified dtype and shape
return torch.onnx.ops.symbolic(
"custom_domain::CustomOp",
(x,),
dict(attr_key="attr_value"),
dtype=x.dtype,
shape=x.shape,
version=1,
)
else:
return x
torch.float4_e2m1fn_x2 dtype (#148791)TORCH_CUDA_ARCH_LIST (#152715, #155314)bicubic mode for torch::nn::functional::grid_sample (#150817)no_implicit_headers mode for load_inline() on custom CUDA extensions (#149480)TCPStore with clone and queuing features (#150966, #151045, #150969, #151485)getDefaultBackend more fault tolerant without relying on exceptions (#149152)masterListenFd in TCPStoreLibUvBackend (#150215)TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK (#150682)global_rank when group_rank is used (#151373)ProcessGroupNCCL via an unsafe API (#152496)needs_contiguous_strides tag in functional collective (#153399, #153523)split_group to work with non-nccl backends (#152175)new_subgroups() by using new_subgroups_by_enumeration() (#153843)ProcessGroupNCCL (#153990)c10::Half for gloo (#153862)get_process_group_ranks() to accept group=None (#154902)init_process_group support index-only device id (#156214)ProcessGroup (#151723)reduce_scatter and ReduceOp::AVG in ProcessGroupGloo (#149781, #149869)ProcessGroupNCCL (#152706)ibverbs backend in gloo and enabled gloo CUDA when used with a backend that supports GPUDirect (#153015, #153425, #153406)use_python_reducer to C++ reducer (#152735)
DistributedStateDict (DSD)write_size in planner write items (#149699)StridedShard support uneven sharding (#150490)torch.cumsum (#151071)DTensor redistribute fwd/bwd datatype conversion to enable SimpleFSDP mixed precision training (#150740)torch.distributed.tensor.debug.visualize_sharding (#152027)PrivateUse1 backend in FSDP collectives and device type to pre forward hook (#147260, #149487)set_reshard_after_forward (#149103)reshard_after_forward=True for root model and kept root unsharded when not specifying reshard_after_forward (#154704, #155319)all_reduce_event only if it's not CPU device (#150316)get_pipeline_order() for Gpipe and 1F1B (#155935)ShardedTensor and recalculated metadata from all_gather (#152583)ParallelStyle PrepareModuleInputOutput (#150372)__torch_function__, and namedtuple subclasses (#153150, #149792, #153982)reason field to torch.compiler.disable (#150341)lru_cache warnings for functions in the top-level torch namespace (#157718)aot_inductor.custom_ops_to_c_shims and aot_inductor.custom_op_libs: allow for specifying custom op C shim (#153968)max_fusion_buffer_group_pairwise_attempts: limits fusions to specified node distance (#154688)cuda.cutlass_enabled_ops: controls CUTLASS operation selection (#155770)triton.cudagraph_capture_sizes: allows specifying certain shapes for which to capture CUDAGraphs; skips CUDAGraphs for other shapes (#156551)use_static_cuda_launcher: enables launching compiled triton statically to improve cold start times (#148890)assume_unaligned_fallback_output: allows inductor to track unaligned outputs (#150777)cuda.cutlass_tma_only: controls whether or not to only use TMA-compatible kernels in CUTLASS (#152815)static_launch_user_defined_triton_kernels: enables statically launching user defined triton kernels (#153725)precompilation_timeout_seconds: controls the timeout on precompilation (#153788)disable_decompose_k: disables new DecomposeK GEMM Kernels (#154421)min_num_split: sets the minimum number of splits in a split reduction (#155941)max_autotune_flex_search_space: allows specifying the size of the search space for flex attention autotuning (#156307)LOG_AUTOTUNE_RESULTS for autotune log (#156254)min, max, math.pow) (#151348)pytree.register_dataclass (#147752)jit.scripted functions in export (#155180)num_runners to AOTIModelPackageLoader (#149364)== (#150611)normalize_function (#143689)graph_code_verbose_log artifact for FX passes (#153775)fx.passes.split_module to normalize input names (#157793)cross (#154999)torch.special operations as well as index_copy, hardshrink, rsub, col2im, and isin (#149174, #149203 #149123, #149368, #149378, #149563, #149687, #149705, #149783, #149407/#149680, #150279, #151754, #153786, #154326, #155304, #156263, #155382, #154010, #149816, #152282, #156090, #150060, #151600, #155002, #154671)index_put with half precision floats (#151869)ConvTranspose3D with FP32 and complex (#154696)log1p and sigmoid with int64 (#151791)weight_norm on CPU (#148878)dynamo=True (#149901, #154596)Attention-23 and RotaryEmbedding-23 as native PyTorch ops (#156431, #156367, #154745)torch.scan (#154513)group_norm support from opset 21 (#152138)asdict method to VerificationInfo class (#151024)dynamic_shapes behavior to use torch.export.dim.DYNAMIC (#153065)sym_float, sym_not, sym_min, sym_max (#153200, #152111, #152196)TensorLR variant for fused Adagrad on CPU (#153078)lr_lambda type check in MultiplicativeLR (#151973)torch.AcceleratorError (#152023)Size.__radd__() (#152554)get_default_device() to also respect torch.device context manager (#148621)mul / add / add_relu and batch_norm2d), qconv1d-relu fusion, and lowering pass (#151112, #152411, #152811, #150751, #149708)torch.fused_moving_avg_obs_fake_quant on CUDA (#153699)cpp_extension (#152432)mm/bmm/addmm (#153262)PrivateUse1 extension (#149374)torch.Tensor.scatter_add_ (#150543), torch.matrix_exp (#155202)embed_cubin and multi_arch_kernel_binary options in AOTI for Intel GPU (#154514, #153924)UserDefineClass (#155787)CMake-4.x (#150203)gcc-12+ (#150847)/permissive- flag (#149035)torch.norm for scalar input (#144073)log_softmax reduced-precision fp kernel (#156379)torch.backends.cuda.matmul.allow_fp16_accumulation crash when using cuBLASLt (#153083)AsyncMM on Blackwell (#153519)torch.cuda.MemPool for multithreaded use-cases (#153356)sum() on a default-constructed gamma / beta in layer_norm (#156600)empty_cache under mempool context (#158180)all_to_all (#149485)group input argument in new_subgroups() (#152765, #153798)broadcast_object util function (#155912)DDPOptimizer issue on static tensor index (#155746)local_map with multi-threading (#149070)new_local_tensor in redistribute be None case (#152303)TensorPipe (#154382)gather when a local tensor on certain ranks has zero elements (#150914)dict(mapping_proxy), and the FlexAttention HOP (#157754, #157515, #157519)lru_cache method (#158689, #157308)TORCH_LOGS argument is passed (#151678)aten.is_nonzero (#149637), torch.bincount() (#152497), aten.div (#150874) slicing (#150104), and attn_mask (#158618), aten.to (#153972), scalar tensor construction (#154661)dynamic_shapes spec for kwargs (#148772, #149528, #150103)functools.partial (#153408), and higher order ops (#149295)None inputs (#150515), math module (#154643), call_torchbind (#155647), and enums (#154821)update_constant_buffer issue (#149243)model_package_loader (#152334)AOTIModel if they don't exist (#152692)ConstantFolding (#153152)dot and gemv (#152676)torch.lobpcg to compute same largest eigenvalue as scipy and np.linalg.eig (#152789)ReducedPrecisionGemV (#150949)2**32+ element inputs, binary ops with inputs with different dtypes, ops with complex scalar inputs, cholesky decomp, floor_divide type promotion, index_kernel with large inputs, lerp with complex inputs, logit with half/bfloat16 inputs, SDPA memory leak, torch.special.entr, tri[ul], matrix inversion with N>1024, and where with non-contiguous cond (#152479, #155183, #149233, #151176, #151282, #158239, #152371, #149974, #158237, #146754, #158867, #155184, #152204)load_state_dict behavior for nn.LazyLinear (#147599)onnx_program callable (#151121)lr_scheduler unexpectedly calls step() when init argument last_epoch > -1 (#149312)CosineAnnealingWarmRestarts resetting T_cur (#151289)MixtureSameFamily distribution (#151317)Wishart or Uniform distribution modifies constraints on the first (#154361)torch::utils::tensor_to_numpy symbol (#154178)torch.[con]cat[enate] to avoid crashing on empty inputs (#155460)torch.tensor and torch.ops.aten.scalar_tensor behavior (#158655)ScaledGEMM (#149677)ScaledGEMM (#152403)torch.is_vulkan_available() on Mac (#155595)offset > 0 (#154495)torch.xpu.is_bf16_supported to correctly report presence of Intel GPU (#152317)ELU(0) with the cheaper definition (#155765)cat and index_select (#150233, #152380, #151715)SubsetRandomSampler by iterating over list instead of tensor (#149126)cpp.use_small_dequant_buffer to use a small dequant buffer for WOQ int4 GEMM (#156395)torch.dot with float16/bfloat16 (#152799)LayerNorm, mm / bmm, sum / prod reductions, arithmetic ops,
binary kernels, SDPA, linear, and cumsum / cumprod (#152010, #150541, #150566, #147644, #149730, #152781, #152210, #157494)torch.tensordot when contracting to a scalar (#145936)softmax, NLLLoss, in-place sum, max pooling backward / reductions on NHWC
inputs, max pooling, multi-dimensional reductions, and non-vectorized elementwise kernels (#149076, #149779, #149548, #151230, #152267, #154522, #154619, #155806, #153184)HipSparseLT to further accelerate semi-structured (e.g. 2:4) sparsity (#150578)addmm, baddmm to reduce oneDNN integration overhead on Intel GPU (#153051)ctx.save_for_backward is important in note about extending autograd (#153005)torch.autograd.graph.saved_tensors_hooks to avoid refcycle (#153049)torch.amin and torch.amax (#155071)NCCLConfig with QOS variable (#151821)get_default_backend_for_device (#158236)ignored_params docstring and added unit tests (#149074)Dims and ExportGraphSignature (#156262, #156244)torch.linalg.norm()'s ord argument of +2 & -2 (#155148)nn.RNN, nn.functional loss functions, interpolate saturate cast behavior, ConvTranspose2d stride / output_size arguments, and register_full_backward_hook (#155123, #153620, #148436, #151304, #150819, #150609, #151785)nn.Sequential and nn.LazyModuleMixin (#147304, #150596)nn.modules.padding and AvgPoolND (#155618, #152680)LRSchedulers (#149189)CosineAnnealingLR to accurately reflect its recursive learning rate schedule (#152936)Adafactor documentation (#145209)load_state_dict hint doc about invoke order work with lr_scheduler (#149942)torch.Library's kind have no default value to be consistent with the code (#149390)requires_grad=True in tensor.to() (#150913)cdist param description (#151178)Example: and not Example:: in docs (#153978)as_strided() docs (#149146)keepdim param optional description (#151197)torch.trapezoid docs (#151190)out_dtype arg for torch GEMM operations (#151704)torch.min(), torch.max(), torch.all(), and torch.any() (#152658)torch.triu_indices, torch.tril_indices dtype description (#150749)torch.equal description (#149618)get_default_qat_qconfig in prepare_qat_fx docs (#155100)nccl_version and thread name/id, for flight record in PGNCCL (#150356, #150513, #151048, #152648, #155142, #155754)new_subgroups() for Non-Divisible World Sizes (#154124)get_backend() with more details (#141796)FlatParamHandle (#151336)rpc_init to CPython (#154325)torch.distributed.run option to provide destination for event logging (#155268)TracingContext (#149294)detect_attr_assignment (#151824)AOTInductor runtime API for Intel GPU (#153929)stable::Tensor is_contiguous API (#156228)lr_scheduler.py (#151219)step() with default value (#153367)setup-python from for Mac tests (#155698)This release is meant to fix the following issues (regressions / silent correctness):
This release is meant to fix the following issues (regressions / silent correctness):
Fix Excessive cudagraph re-recording for HF LLM models (#152287) Fix torch.compile on some HuggingFace models (#151154) Fix crash due to Exception raised inside torch.autocast (#152503) Improve Error logging in torch.compile (#149831) Mark mutable custom operators as cacheable in torch.compile (#151194) Implement workaround for a graph break with older version einops (#153925) Fix an issue with tensor.view(dtype).copy_(...) (#151598)
Fix assertion error due to inductor permuting inputs to flex attention (#151959) Fix performance regression on nanogpt speedrun (#152641)
Fix extra CUDA context created by barrier (#149144) Fix an issue related to Distributed Fused Adam in Rocm/APEX when using nccl_ub feature (#150010) Add a workaround random hang in non-blocking API mode in NCCL 2.26 (#154055)
Fix MacOS compilation error with Clang 17 (#151316) Fix binary kernels produce incorrect results when one of the tensor arguments is from a wrapped scalar on MPS devices (#152997)
Improve PyTorch Wheel size due to introduction of addition of 128 bit vectorization (#148320) (#152396) Fix fmsub function definition (#152075) Fix Floating point exception in torch.mkldnn_max_pool2d (#151848) Fix abnormal inference output with XPU:1 device (#153067) Fix Illegal Instruction Caused by grid_sample on Windows (#152613) Fix ONNX decomposition does not preserve custom CompositeImplicitAutograd ops (#151826) Fix error with dynamic linking of libgomp library (#150084) Fix segfault in profiler with Python 3.13 (#153848)
Users should move to use the dynamo=True option on torch.onnx.export as torch.onnx.dynamo_export is now deprecated. Leverage the `dynamic_shapes` argu…
<table> <tr> <td><strong>Beta</strong> </td> <td><strong>Prototype</strong> </td> </tr> <tr> <td>Torch.Compile support for Torch Function Modes </td> <td>NVIDIA Blackwell Architecture Support </td> </tr> <tr> <td>Mega Cache </td> <td>PyTorch Native Context Parallel </td> </tr> <tr> <td> </td> <td>Enhancing Intel GPU Acceleration </td> </tr> <tr> <td> </td> <td>FlexAttention LLM <span style="text-decoration:underline;">first token processing</span> on X86 CPUs </td> </tr> <tr> <td> </td> <td>FlexAttention LLM <span style="text-decoration:underline;">throughput mode optimization</span> on X86 CPUs </td> </tr> <tr> <td> </td> <td>Foreach Map </td> </tr> <tr> <td> </td> <td>Flex Attention for Inference </td> </tr> <tr> <td> </td> <td>Prologue Fusion Support in Inductor </td> </tr> </table>
For more details about these highlighted features, you can look at the release blogpost. Below are the full release notes for this release.
Some users with 12.2 CUDA driver (535 version) report seeing "CUDA driver error: invalid argument" during NCCL or Symmetric Memory initialization. This issue is currently under investigation, see #150852. If you use PyTorch from source, a known workaround is to rebuild PyTorch with CUDA 12.2 toolkit. Otherwise, you can try upgrading the CUDA driver on your system.
py_limited_api=True is now built with -DPy_LIMITED_API (#145764)We formally began respecting the py_limited_api=True kwarg in 2.6 and stopped linking libtorch_python.so when the flag was specified, as libtorch_python.so does not guarantee using APIs from from the stable Python limited API. In 2.7, we go further by specifying the -DPy_LIMITED_API flag which will enforce that the extension is buildable with the limited API. As a result of this enforcement, custom extensions that set py_limited_api=True but do not abide by the limited API may fail to build. For an example, see #152243.
This is strictly better behavior as it is sketchy to claim CPython agnosticism without enforcing with the flag. If you run into this issue, please ensure that the extension you are building does not use any APIs which are outside of the Python limited API, e.g., pybind.
torch.Tensor.new_tensor() to be on the given Tensor's device by default (#144958)This function was always creating the new Tensor on the "cpu" device and will now use the same device as the current Tensor object. This behavior is now consistent with other .new_* methods.
With Migration to manylinux_2_28 (AlmaLinux 8 based), we can no longer support OS distros with glibc2_26. These include popular Amazon Linux 2 and CentOS 7. (#143423, #146200, #148028, #148135, #148195, #148129)
torch.onnx.dynamo_export now uses the ExportedProgram logic path (#137296)Users using the torch.onnx.dynamo_export API may see some ExportOptions become
unsupported due to an internal switch to use torch.onnx.export(..., dynamo=True): diagnostic_options, fake_context and onnx_registry are removed/ignored by ExportOptions. Only dynamic_shapes is retained.
Users should move to use the dynamo=True option on torch.onnx.export as
torch.onnx.dynamo_export is now deprecated. Leverage the dynamic_shapes argument in torch.onnx.export for specifying dynamic shapes on the model.
Version 2.6.0
torch.onnx.dynamo_export(model, *args, **kwargs)
Version 2.7.0
torch.onnx.export(model, args, kwargs=kwargs, dynamo=True)
LRScheduler.print_lr() along with the verbose kwarg to the LRScheduler constructor. (#147301)Both APIs have been deprecated since 2.2. Please use LRScheduler.get_last_lr() to access the learning rate instead.print_lr and verbose were confusing, not properly documented and were little used, as described in #99270, so we deprecated them in 2.2. Now, we complete the deprecation by removing them completely. To access and print the learning rate of a LRScheduler:
Version 2.6.0
optim = ...
lrsched = torch.optim.lr_scheduler.ReduceLROnPlateau(optim, verbose=True)
// lrsched will internally call print_lr() and print the learning rate
Version 2.7.0
optim = ...
lrsched = torch.optim.lr_scheduler.ReduceLROnPlateau(optim)
print(lrsched.get_last_lr())
Previously, the symbols in libtorch_python.so were exposed with default visibility. We have transitioned to being more intentional about what we expose as public symbols for our python API in C++. After #142214, public symbols will be marked explicitly while everything else will be hidden. Some extensions using private symbols will see linker failures with this change.
torch.export.export instead of capture_pre_autograd_graph to export the model for pytorch 2 export quantization (#139505)capture_pre_autograd_graph was a temporary API in torch.export. Since now we have a better longer term API: export available, we can deprecate it.
Version 2.6.0
from torch._export import capture_pre_autograd_graph
from torch.ao.quantization.quantize_pt2e import prepare_pt2e
from torch.ao.quantization.quantizer.xnnpack_quantizer import (
XNNPACKQuantizer,
get_symmetric_quantization_config,
)
quantizer = XNNPACKQuantizer().set_global(
get_symmetric_quantization_config()
)
m = capture_pre_autograd_graph(m, *example_inputs)
m = prepare_pt2e(m, quantizer)
Version 2.7.0
from torch.export import export
from torch.ao.quantization.quantize_pt2e import prepare_pt2e
# please get xnnpack quantizer from executorch (https://github.com/pytorch/executorch/)
from executorch.backends.xnnpack.quantizer.xnnpack_quantizer import (
XNNPACKQuantizer,
get_symmetric_quantization_config,
)
quantizer = XNNPACKQuantizer().set_global(
get_symmetric_quantization_config()
)
m = export(m, *example_inputs)
m = prepare_pt2e(m, quantizer)
torch.fx.passes.graph_transform_observer.GraphTransformObserver to enable Node Level provenance tracking (#144277)We now track a mapping between the nodes in the pre-grad and post-grad graph. See the issue for an example frontend to visualize the transformations. To update your GraphTransformObserver subclasses, instead of overriding on_node_creation and on_node_erase, there are new functions get_node_creation_hook, get_node_erase_hook, get_node_replace_hook and get_deepcopy_hook. These are registered on the GraphModule member of the GraphTransformObserver upon entry and exit of a with block
Version 2.6.0
class MyPrintObserver(GraphTransformObserver):
def on_node_creation(self, node: torch.fx.Node):
print(node)
Version 2.7.0
class MyPrintObserver(GraphTransformObserver):
def get_node_creation_hook(self):
def hook(node: torch.fx.Node):
print(node)
return hook
torch.ao.quantization.pt2e.graph_utils.get_control_flow_submodules is no longer public (#141612)We are planning to make all functions under torch.ao.quantization.pt2e.graph_utils private. This update marks get_control_flow_submodules as a private API. If you have to or want to continue using get_control_flow_submodules, please make a private call by using _get_control_flow_submodules.
Example: Version 2.6:
>>> from torch.ao.quantization.pt2e.graph_utils import get_control_flow_submodules
Version 2.7:
>>> from torch.ao.quantization.pt2e.graph_utils import get_control_flow_submodules
ImportError: cannot import name 'get_control_flow_submodules' from 'torch.ao.quantization.pt2e.graph_utils'
>>> from torch.ao.quantization.pt2e.graph_utils import _get_control_flow_submodules # Note: Use _get_control_flow_submodules for private access
torch.onnx.dynamo_export is deprecated (#146425, #146639, #146923)Users should use the dynamo=True option on torch.onnx.export.
Version 2.6.0
torch.onnx.dynamo_export(model, *args, **kwargs)
Version 2.7.0
torch.onnx.export(model, args, kwargs=kwargs, dynamo=True)
XNNPACKQuantizer is deprecated in PyTorch and moved to ExecuTorch, please use it from executorch.backends.xnnpack.quantizer.xnnpack_quantizer instead of torch.ao.quantization.quantizer.xnnpack_quantizer. (#144940)XNNPACKQuantizer is a quantizer for xnnpack that was added into pytorch/pytorch for initial development. However, as it is not related to our core quantization workflow, we have moved it to ExecuTorch instead. Please use it from executorch.backends.xnnpack.quantizer.xnnpack_quantizer instead of torch.ao.quantization.quantizer.xnnpack_quantizer.
Version 2.6.0
from torch._export import capture_pre_autograd_graph
from torch.ao.quantization.quantize_pt2e import prepare_pt2e
from torch.ao.quantization.quantizer.xnnpack_quantizer import (
XNNPACKQuantizer,
get_symmetric_quantization_config,
)
quantizer = XNNPACKQuantizer().set_global(
get_symmetric_quantization_config()
)
m = capture_pre_autograd_graph(m, *example_inputs)
m = prepare_pt2e(m, quantizer)
Version 2.7.0
# we also updated the export call
from torch.export import export
from torch.ao.quantization.quantize_pt2e import prepare_pt2e
# please get xnnpack quantizer from executorch (https://github.com/pytorch/executorch/)
from executorch.backends.xnnpack.quantizer.xnnpack_quantizer import (
XNNPACKQuantizer,
get_symmetric_quantization_config,
)
quantizer = XNNPACKQuantizer().set_global(
get_symmetric_quantization_config()
)
m = export(m, *example_inputs)
m = prepare_pt2e(m, quantizer)
torch.utils.serialization.config namespace for all serialization related configurations (#143324)torch.serialization.config.save.use_pinned_memory_for_d2h to speed up torch.save when passed gpu devices (#143342)torch.utils.serialization.config.load.calculate_storage_offsets to reduce random reads and significantly improve performance for storage with bad random access performance (#143880)__torch_function__ handler on dtype arguments, similar to subclass objects (#145085)torch.nn.functional.scaled_dot_product_attention over the sequence dimension. We implemented
Ring Attention (#131351) and an AllGather-based approach (#132820) where the all-gather is issued before the first local SDPA
and the subsequent local SDPAs will have to wait until the all-gather completes, and offered a user API (#142093) to select the desired approach. The implementation
currently supports three SDPA kernels: SDPBackend.FLASH_ATTENTION, SDPBackend.EFFICIENT_ATTENTION, and SDPBackend.CUDNN_ATTENTION (#148537). We also
verified that our Context Parallel implementation is compatible with other parallelisms and torch.compile.torch.compile (#145270)torch.cuda.gds APIs public (#147120)torch.compile on Windows Platform for XPU (#147637, #144316, #149511)torch.utils.cpp_extension APIs (#132945)contextlib.contextmanager in Dynamo (#136033)nonstrict_trace escape hatch to apply non-strict tracing to difficult-to-compile code (#146367)list subclasses (#146819)head_dim for FlexAttention (#133495).num_warps and num_stages (#139639).ConfigFuzzer: a new debugging tool designed to fuzz Torch compile configurations. Given a test function, it will identify combinations of configs that throw errors during compilation and execution (#139736) (#145565).TORCHINDUCTOR_PROLOGUE_FUSION enables this feature (#147008).TORCHINDUCTOR_CUTLASS_INSTANTIATION_LEVEL. Consult config.py for information (#146230).cuda.cutlass_max_profiling_swizzle_options (#146088).package_cpp_only is specified in AOTI (#143352).graph_partition functions. Set the graph_partition in inductor config to enable (#147038).experimentalConfig (#143659)torch.ops.aten._dyn_quant_matmul_4bit, while the weights, scaled and optional bias are packed in torch.ops.aten._dyn_quant_pack_4bit_weight. To use it on your model you can quantize it using the following example that leverages torchao:from torchao.dtypes import PlainLayout
from torchao.experimental.packed_linear_int8_dynamic_activation_intx_weight_layout import (
PackedLinearInt8DynamicActivationIntxWeightLayout,
)
from torchao.experimental.quant_api import (
int8_dynamic_activation_intx_weight,
)
from torchao.quantization.granularity import (
PerGroup,
PerRow,
)
from torchao.quantization.quant_api import quantize_
from torchao.quantization.quant_primitives import MappingType
my_model = Model()
quantize_(
my_model,
int8_dynamic_activation_intx_weight(
weight_dtype=torch.int4,
granularity=PerGroup(32), # PerRow() is also supported
has_weight_zeros=True, # Should be True
weight_mapping_type=MappingType.SYMMETRIC_NO_CLIPPING_ERR # MappingType.SYMMETRIC can also be used but increases error
layout=PackedLinearInt8DynamicActivationIntxWeightLayout(target="aten"),
),
)
torch.onnx.verification.verify_onnx_program (#148396, #148706, #148730, #148707)A new verification API torch.onnx.verification.verify_onnx_program can now be used to verify numerical accuracy of the exported ONNX model. Users can use the compare_intermediates option to identify any operator that causes numerical discrepancies in intermediate tensors. It is possible to use a tool like model-explorer to visualize the verification results.
dynamic_shapes (#146321)torch.onnx.export(dynamo=True) now optimizes the output model by default (#146187)torch.addcmul (#143264)-DPy_LIMITED_API flag for py_limited_api=True cpp_extensions (#145764)torch.jit.load (#143403)torch.save configurable (#147788)with statement on torch.Stream (#140138)torch.autograd.graph.GradientEdge as torch.autograd.backward outputs #144744residuals of torch.linalg.lstsq #148526reflection_pad2d_backward (#136241)in_order is False (#142324)device argument. device and pin_memory_device are discouraged and will be deprecated in the future. (#131858)torch.cum{min,max}. (#143920)chunk() backward on batch dim (#144584)*_like factory functions for NJT (#144889)matmul with NJTs via backward support and composition with dense tensors (#144587, #146405)strict kwarg to nn.Module.set_submodule and fix bug for non dot-delineated strings (#143455)reflection_pad1d, reflection_pad2d and reflection_pad3d (#141670)isAcceleratorExcluded (#144959)abort and shutdown by adding both to Backend and ProcessGroup objects (#148798)new_group instead of split_group on non-CUDA device (#141469)call_guard in pybind object init of c10d (#143598)getDefaultBackend more fault tolerant (#148596)init_sync option to control collectives during initialization (#142824)reduce_dtype in lazy init (#143297)aten.amin/amax to linear_reduction_strategy (#143747)src_data_rank to distribute_tensor API (#143883)_scaled_mm (#143760)aten.view.dtype op support (#144404)shard_dim_alltoall to use alltoall_single (#148868)_shard_tensor to use src_data_rank=None (#144171)aten.minimum (#145816)src_data_rank kwarg in TP API (#144005)etcd_rendezvous publicly importable (#145396)generate_stage_to_rank_mapping utility (#146193)stage_index_to_group_rank from schedule (#146217)brgemm (#143384)sharedMemPerMultiprocessor device property to python (#143119)cudaDeviceProps to python (#143226)index >= 0 of cuda device (#140791)get_stream_from_external API for CUDA backend (#143799)angle, entr, spherical_bessel_j0,xlog1py, sinc,round.decimals, linalg.det, cholesky.ex, bilineard2d_aa,linalg.solve, zeta, cholesky, fused_rms_norm, lu_unpack, lu_factor_ex, slogdet and logdet (#143449, #147948, #146818, #147687, #146539, #147266, #146279, #146799, #145526, #146531, #146465, #145701, #145301, #146681, #144651, #145341, #146771, #147914)angle and atan2 for long type, torch.special.sinc to complex, torch.mm / torch.bmm to integral types (#149017, #146648, #145809, #147526)torch.accelerator.synchronize() on MPS (#143171)gamma, zeta, sinc, spherical_bessel_j0, entr (#145341, #146465, #146539, #147650, #148128)convolution_backward output layout between fake tensor and real output tensor (#146880)torch.xpu.get_device_properties API error message (#144379)nested_layer_norm support for XPU (#148593)is_big_gpu() check in Inductor (#143491)dict subclasses (#143548)trace_rules.py skipfiles (#145856)transformers ModelOutput) (#143567)buffer.copy_(int) (#141161)torch.inference_mode (#147925)topk (#147017)interpolate(antialias=True) backward (#141198)nonzer_static (#146006)_compute_symbolic_stride() (#138844)torch._check (#144471)backed_size_oblivious config (#148696)mark_unbacked strict mode (#147333, #147342)Several operator decomps received improvements/bugfixes:
torch._refs.tensor (#143461)torch._refs.mean (#147188)linspace (#147997)addmv (#143792)
New meta tensor implementations for a few pytorch operators:nonzero (#144727)silu, sigmoid, _softmax, embedding (#147862)
New fake tensor implementation for a few pytorch operators:unique_consecutive (#145649)
Several general FakeTensor improvementsUntypedStorage.from_buffer(buf) to return meta storage under FakeTensorMode (#146642)meta_tensor.to(device='cpu') under fake_mode (#146729)swizzle (#147223).INDUCTOR_CPP_ENABLE_FLOATING_POINT_CONTRACT_FLAG will be passed to ffp-contract (#143450).TORCHINDUCTOR_WORKER_START to one of "subprocess", "fork", or "spawn" (#144491).do_bench (#133058).emulate_precision_casts: TORCHINDUCTOR_EMULATE_PRECISION_CASTS (#145948).TORCHINDUCTOR_CUTLASS_ALLOWLIST and TORCHINDUCTOR_CUTLASS_DENYLIST (#148161).TORCHINDUCTOR_SCALAR_ASSERTS (#146462).layout_optimization and comprehensive_padding (#148450).AOT_INDUCTOR_COMPILE_WRAPPER_WITH_O0=1 (#144866).global_scratch arg, fix cpp_wrapper (#148051, #149973)._int_mm in AOTI (#144571).torch.ops.aten._assert_tensor_metadata.default for AOTI (#145028).aot_compile and aoti_compile_and_package (#148506)._weight_int4pack_mm_cpu_tensor (#149031)"+export" logging to de/serialization process (#145283)builtins.getattr with serializable higher-order-op for tensor subclasses (#145772)Dim.AUTO) for TorchScript -> Export Converter (#138273)ShapesCollection (#147534)expression_created logging in draft export (#146859)report as return output for draft export, attached as ep._report (#147558)builtin bitshift ops in verifier (#145802)aoti_call_delegate higher-order-op for eager-mode runnability (#145630)FlatArgsAdapter (#146107)upsample_bilinear2d.vec, nearest2d.vec from default decomposition table (#147153)keep_original_weights in _lower_to_native_backend (#141049)torch.float8_e8m0fnu dtype to PyTorch (#147466)dynamic_axes to dynamic_shapes with torch.export.Dim.AUTO (#143158)torch.cdist into onnx and support 'compute_mode' (#144213)LegacyDynamoStrategy (#145442)torch.onnx call site (#147165)dynamic_shapes renaming (#147407)run_decomposition() (#148617)torch export to get dynamic_shapes for JIT convert strategy (#148627)torch.export.Dim.AUTO in dynamo_export (#144356)verify=True (#148619)torch.lerp type promotion (#141117)torch.Tensor when both slots and python gc are used (#143203)torch.bfloat16 support for __cuda_array_interface__. (#143042)torch.Tensor incorrect. (#145530)torch.load under FakeTensorMode to create FakeTensor with correct devices (for plain Tensors) (#147786)torch.acos, torch.asin, torch.atan, torch.exp, torch.sigmoid, torch.div, for torch.complex datatypes on CPU (#134838, #140358, #140391, #140375, #144749)torch.autograd.graph.allow_mutation_on_saved_tensors for inplace foreach ops #145520hardswish backward (#143899)batchnorm backward on CPU (#147353)torch.compile + ddp + non-reentrant AC pack hook firing count (#144271)eigh (#146456)min / max backward() for non-ragged reductions (#144583)frexp() to handle both outputs (#144585)fill.Scalar for contiguous inputs (#144586)#pragma diagnostic pop in VecLib (#148354)CudaEventCache for dangling references (#144496)all-reduce input contiguous in distributed.nn.all_reduce (#144267)Alltoallv specialization for PyTorch generic all_to_all (#145045)torch.device (#146290)dist.init_process_group on windows (#148266)isend and irecv (#148462)strict=False case for DDP (#143038)buffer_dtype without root parameters (#143989)torch.distributed._functional_collectives.AsyncCollectiveTensor for aten.to. (#134661)_scaled_dot_product_flash_attention sharding (#148125)all-reduce (#148761)asinh codegen (#142360)Conv/Linear + broadcast add fusion (#141759)PYTORCH_NO_CUDA_MEMORY_CACHING has effect only when value is 1 (#145905)complex128 scan (#143401)Int64 indexing fix for UpSampleNearest3D (#144865)avg_pool2d backward for SM 10.0 to prevent runtime crash (#145669)f8f8bf16 rowwise scaled matmul to SM 9.0 (precedes #148421 adding of kernel) (#145728)Upsample2D (#141923)_preload_cuda_deps (#149808)sm100, sm120 (#150640)gather_out in MPS backend (#135543)torch.add(x,y, alpha=2) crash (#143949)nllnd_loss_backward crash with different dtypes (#144170)_scaled_dot_product_attention_math_mps (#146623)cholesky_ex for empty inputs (#147159)torch.chalf (#148285)unary_kernel_strided logic (#148512)lstm_mps causing leaked memory (#145503)c10::metal::log_gamma correctness on M4 (#145740)enable_gqa crash on MPS (#149147)tril op not handling infs correctly (#149866)min/max reductions over large dims (#149004)masked/where for inf values (#144500)addmm op (#144519)torch.layer_norm invalid configuration when input is large tensor (#144007)isnan integer overload errors on MicroSoft STL (#146605)torch.utils._content_store to accelerate XPU tensor hashing for tensor serialization (#147785)OffsetBasedRNGTracker to unbreak torch.distributed (#148360)torch.backends.mkldnn.flags() CM should not warn (#150358)step() for iterative on-demand tracking behind environment variable (#144494)torch.profiler (#144237)torch._functorch import (#149683)torch.compile calls was ignored (#145131).nanj in cpp wrapper CPU (#144064).fractional_max_pool lowering in Inductor (#144395).associative_scan (#143048).torch.polygamma(n) when n == 0 (#144058).avg_pool that was causing 0 rounding (#144059).avg_pool with uint to match eager (#144313).torch.logit decomposition (#145576).benchmark_harness isn't generated, but is called in some cases (#145532).SVE256 features were run on SVE128 systems (#146207).mm_template (#146293).cpp_wrapper (#145527).AOTIModelPackageLoaderPybind::boxed_run (#146100).None and equal_to_1 arguments issue in Triton kernel generated by AOTI (#148102)AOTIModelPackageLoader() constructor defaults (#149082)get_source_partitions when weights are tied (#142446)and_ operator (#145506)unbacked_bindings (#144894)unbacked_bindings keys (#145777)unbacked_bindings in deserialization (#145882)nn_module_stack (#145901).requires_grad field (#146351)math.trunc ops for serialization (#146715)math.inf and NaN as strings (#146490)lazy_trace_handler bug in draft export logging (#146106)._modules corner case for nn_module_stack metadata in strict-mode (#142823)all_reduce, all_gather, all_gather_into_tensor, all_to_all_single, reduce_scatter_tensor) in non-strict mode (#147133, #147417)stack_trace field optional in insert_custom_op_guards pass (#146438)ScriptModules and ScriptObjects for TorchBind (#147399)export_tracepoint (#148709)transpose_ (#149057)rename_dynamic_shapes_with_model_inputs (#146002)dynamic_shapes string cases (#148025)clones throughout codebase (#148159)ALLOC_BUFFER_SIZE from 4000 to 4096 to be a power of 2 for TCPStore (#145759)ExpandableSegments object and the GIL in WorkNCCL destructor (#148805)fmadds (#144486)sort (#142391)prop_kind to forward_inference when grad is not needed for mkldnn_convolution_pointwise (#142855)add and max (#144065)conv src zp mask (#149473)PYTORCH_NO_CUDA_MEMORY_CACHING has effect only when value is 1 (#145905)complex128 scan (#143401)Int64 indexing fix for UpSampleNearest3D (#144865)avg_pool2d backward for SM 10.0 to prevent runtime crash (#145669)f8f8bf16 rowwise scaled matmul to SM 9.0 (precedes #148421 adding of kernel) (#145728)Upsample2D (#141923)masked_fill_scalar as shader (#147369)bilineard2d as shader (#145581)_load_dwordx4 ISA for BFloat16 and Half (#141397)persistent_tma (#142101)._weight_int4pack_mm_for_cpu (#146756).SmoothQuant int8 linear pattern (#142036).max_pool2d_with_indices as a reduction for large window sizes (#147876).Graph.nodes.__iter__ (#144631)map_aggregate(immutable_dict) (#147691)input in torch.addbmm() (#146664)torch.cat type promotion documentation (#141339)torch.topk indices stability when duplicate values (#143736)torch.diagonal documentation (#144214)torch.{min,max} documentation (#146725)torch.max optional args dim, keepdim description (#147177)torch.bucketize documentaion (#148400)torch.autograd.gradcheck #144287CrossEntropyLoss doc (#145444)set_requires_gradient_sync and no_sync (#148715)wait() (#143305)from_group API and add a 2D test (#146364)__create_chunk_list__ in the doc (#144100)set_optimizer_state_dict (#148918)record (#146968)suppress_errors on compiler error (#146553)emulate_precision_casts (#145579).inductor.config.descriptive_names = False is no longer a suggested option (#145523).dims() and suggested fixes parser: #142510torch.export.load(): #141490__add__ and __mul__ hints to torch.Size (#144322)fully_shard so that the return value can be chained with typing enabled (#147489)add_, addcmul, arange, baddbmm, bmm, clamp, div, div_, gelu, index_add, logical_and, mul_, sub_, topk, where} to operator benchmark (#145625)torch._dynamo.optimize() with torch.compile() (#142451)TORCH_LOGs under the name "autotuning" (#147222).set by OrderedSet: only use OrderedSet in the Inductor codebase (#138466).GPU_TYPE (#143634).qlinear (#143903).TORCH_COMPILE_DEBUG to TORCH_LOGS. For example, TORCH_LOGS="+ir_pre_fusion" (#147248).deepcopy for AOTICompiledModel (#145423)fx.node.map_arg() and .map_aggregate() generic (#146248)This is a backward compatibility-breaking change, please see this forum post for more details.
We are excited to announce the release of PyTorch® 2.6 (release notes)! This release features multiple improvements for PT2: torch.compile can now be used with Python 3.13; new performance-related knob torch.compiler.set_stance; several AOTInductor enhancements. Besides the PT2 improvements, another highlight is FP16 support on X86 CPUs.
NOTE: Starting with this release we are not going to publish on Conda, please see [Announcement] Deprecating PyTorch’s official Anaconda channel for the details.
For this release the experimental Linux binaries shipped with CUDA 12.6.3 (as well as Linux Aarch64, Linux ROCm 6.2.4, and Linux XPU binaries) are built with CXX11_ABI=1 and are using the Manylinux 2.28 build platform. If you build PyTorch extensions with custom C++ or CUDA extensions, please update these builds to use CXX_ABI=1 as well and report any issues you are seeing. For the next PyTorch 2.7 release we plan to switch all Linux builds to Manylinux 2.28 and CXX11_ABI=1, please see [RFC] PyTorch next wheel build platform: manylinux-2.28 for the details and discussion.
Also in this release as an important security improvement measure we have changed the default value for weights_only parameter of torch.load. This is a backward compatibility-breaking change, please see this forum post for more details.
This release is composed of 3892 commits from 520 contributors since PyTorch 2.5. We want to sincerely thank our dedicated community for your contributions. As always, we encourage you to try these out and report any issues as we improve PyTorch. More information about how to get started with the PyTorch 2-series can be found at our Getting Started page.
<table> <tr> <td>Beta </td> <td>Prototype </td> </tr> <tr> <td>torch.compiler.set_stance </td> <td>Improved PyTorch user experience on Intel GPUs </td> </tr> <tr> <td>torch.library.triton_op </td> <td>FlexAttention support on X86 CPU for LLMs </td> </tr> <tr> <td>torch.compile support for Python 3.13 </td> <td>Dim.AUTO </td> </tr> <tr> <td>New packaging APIs for AOTInductor </td> <td>CUTLASS and CK GEMM/CONV Backends for AOTInductor </td> </tr> <tr> <td>AOTInductor: minifier </td> <td> </td> </tr> <tr> <td>AOTInductor: ABI-compatible mode code generation </td> <td> </td> </tr> <tr> <td>FP16 support for X86 CPUs </td> <td> </td> </tr> </table>
*To see a full list of public feature submissions click here.
This feature enables the user to specify different behaviors (“stances”) that torch.compile can take between different invocations of compiled functions. One of the stances, for example, is
“eager_on_recompile”, that instructs PyTorch to code eagerly when a recompile is necessary, reusing cached compiled code when possible.
For more information please refer to the set_stance documentation and the Dynamic Compilation Control with torch.compiler.set_stance tutorial.
[Beta] torch.library.triton_op
torch.library.triton_op offers a standard way of creating custom operators that are backed by user-defined triton kernels.
When users turn user-defined triton kernels into custom operators, torch.library.triton_op allows torch.compile to peek into the implementation, enabling torch.compile to optimize the triton kernel inside it.
For more information please refer to the triton_op documentation and the Using User-Defined Triton Kernels with torch.compile tutorial.
[Beta] torch.compile support for Python 3.13
torch.compile previously only supported Python up to version 3.12. Users can now optimize models with torch.compile in Python 3.13.
[Beta] New packaging APIs for AOTInductor
A new package format, “PT2 archive”, has been introduced. This essentially contains a zipfile of all the files that need to be used by AOTInductor, and allows users to send everything needed to other environments. There is also functionality to package multiple models into one artifact, and to store additional metadata inside of the package.
For more details please see the updated torch.export AOTInductor Tutorial for Python runtime.
[Beta] AOTInductor: minifier
If a user encounters an error while using AOTInductor APIs, AOTInductor Minifier allows creation of a minimal nn.Module that reproduces the error.
For more information please see the AOTInductor Minifier documentation.
[Beta] AOTInductor: ABI-compatible mode code generation
AOTInductor-generated model code has dependency on Pytorch cpp libraries. As Pytorch evolves quickly, it’s important to make sure previously AOTInductor compiled models can continue to run on newer Pytorch versions, i.e. AOTInductor is backward compatible.
In order to guarantee application binary interface (ABI) backward compatibility, we have carefully defined a set of stable C interfaces in libtorch and make sure AOTInductor generates code that only refers to the specific set of APIs and nothing else in libtorch. We will keep the set of C APIs stable across Pytorch versions and thus provide backward compatibility guarantees for AOTInductor-compiled models.
[Beta] FP16 support for X86 CPUs (both eager and Inductor modes)
Float16 datatype is commonly used for reduced memory usage and faster computation in AI inference and training. CPUs like the recently launched Intel® Xeon® 6 with P-Cores support Float16 datatype with native accelerator AMX. Float16 support on X86 CPUs was introduced in PyTorch 2.5 as a prototype feature, and now it has been further improved for both eager mode and Torch.compile + Inductor mode, making it Beta level feature with both functionality and performance verified with a broad scope of workloads.
[Prototype] Improved PyTorch user experience on Intel GPUs
PyTorch user experience on Intel GPUs is further improved with simplified installation steps, Windows release binary distribution and expanded coverage of supported GPU models including the latest Intel® Arc™ B-Series discrete graphics. Application developers and researchers seeking to fine-tune, inference and develop with PyTorch models on Intel® Core™ Ultra AI PCs and Intel® Arc™ discrete graphics will now be able to directly install PyTorch with binary releases for Windows, Linux and Windows Subsystem for Linux 2.
For more information regarding Intel GPU support, please refer to Getting Started Guide.
[Prototype] FlexAttention support on X86 CPU for LLMs
FlexAttention was initially introduced in PyTorch 2.5 to provide optimized implementations for Attention variants with a flexible API. In PyTorch 2.6, X86 CPU support for FlexAttention was added through TorchInductor CPP backend. This new feature leverages and extends current CPP template abilities to support broad attention variants (e.x.: PageAttention, which is critical for LLMs inference) based on the existing FlexAttention API, and brings optimized performance on x86 CPUs. With this feature, it’s easy to use FlexAttention API to compose Attention solutions on CPU platforms and achieve good performance.
[Prototype] Dim.AUTO
Dim.AUTO allows usage of automatic dynamic shapes with torch.export. Users can export with Dim.AUTO and “discover” the dynamic behavior of their models, with min/max ranges, relations between dimensions, and static/dynamic behavior being automatically inferred.
This is a more user-friendly experience compared to the existing named-Dims approach for specifying dynamic shapes, which requires the user to fully understand the dynamic behavior of their models at export time. Dim.AUTO allows users to write generic code that isn’t model-dependent, increasing ease-of-use for exporting with dynamic shapes.
Please see torch.export tutorial for more information.
[Prototype] CUTLASS and CK GEMM/CONV Backends for AOTInductor
The CUTLASS and CK backend adds kernel choices for GEMM autotuning in Inductor. This is now also available in AOTInductor which can run in C++ runtime environments. A major improvement to the two backends is improved compile-time speed by eliminating redundant kernel binary compilations and dynamic shapes support.
There is a known regression (#144152) that torch.device(0) makes CUDA init fail in subprocess since PyTorch 2.5.0.
There was an attempt to fix the regressions, but it caused some complications and was reverted.
An easy workaround is to use torch.device('cuda') or torch.device('cuda:0') instead.
A regressions (https://github.com/pytorch/pytorch/issues/145220) was reported for PyTorch 2.6.0 with
compilation of the out= variant of the torch.all operator. This should be a rare use case, a workaround can be
rewriting the model code to avoid the out= variant.
We are closing the loop on the deprecation that started in 2.4 and flipped torch.load to use weights_only=True by default.
When this flag is set, instead of using the usual pickle module, torch.load uses a custom unpickler constrained to call only functions and classes needed for loading state dictionaries and basic types.
While this change is disruptive for users serializing more than basic types, we expect the increased security by default is a tradeoff that is worth it. Do note that, even though this default is safer, we still recommend only loading trusted checkpoints and rely on more constrained (and even safer) formats like safetensors for un-trusted checkpoints.
For full details, please refer to this dev-discuss post.
PyTorch will stop publishing Anaconda packages that depend on Anaconda’s default packages. We are directing users to utilize our official wheel packages from download.pytorch.org or PyPI, or switch to utilizing conda-forge (pytorch) packages if they would like to continue to use conda. For more details refer to this announcement
The PyTorch binaries shipped with CUDA 12.6.3 are built with CXX11_ABI=1 and are using the Manylinux 2.28 build platform. If you are building PyTorch extensions with custom C++ or CUDA extensions, please update these builds to use CXX_ABI=1 as well and report any issues you are seeing. For the next PyTorch 2.7 release we plan to switch all Linux builds to Manylinux 2.28 and CXX11_ABI=1, please see [RFC] PyTorch next wheel build platform: manylinux-2.28 for the details and discussion.
torch.onnx.export(..., dynamo=True) now creates ONNX models using IR version 10 (#141207)ONNX ir_version=10 is used to add support for UINT4, INT4 data types and include metadata in GraphProto and NodeProto. Make sure model consumers are able to accept IR version 10 ONNX models. You may read more about IRv10 on https://github.com/onnx/onnx/releases/tag/v1.16.0.
Some logging APIs, torch.onnx.ExportTypes, torch.onnx.export_to_pretty_string are removed. Users should remove usage of the APIs above.
torch.onnx.ONNXProgram has been reimplemented and improved (#136281)All ONNX "dynamo" APIs will return the new ONNXProgram class. Some notable methods available are save(), optimize(). It can also be directly applied on PyTorch tensors to leverage ONNX Runtime to verify the ONNX graph. Some legacy methods are no longer available.
The full release compatibility matrix matrix can be found in release.md
c10d::onCompletionHook (#142390)pg = dist.init_process_group()
def hook(work_info: torch._C._distributed_c10d.WorkInfo):
# do something
pg._register_on_completion_hook(hook)
# The hook will be triggered after the collective complete
pg.broadcast([tensor]).wait()
Instead of setting TORCHINDUCTOR_STACK_ALLOCATION, update your torch.compile call: torch.compile(options={"aot_inductor.allow_stack_allocation": True})(foo).
Tensor.dim_order() (#141632)ord argument for torch.linalg.{vector,matrix}_norm() (#137463)functional.affine_grid (#135494)device_id is given #138527)group_dst/group_src in c10d collectives (#140460, #139677, #140827, #140843, #140847)XCCL) in PyTorch distributed package (#141856)torch.compiler.set_stance to dynamically change torch.compile behavior without needing to re-apply torch.compile. (#137504)automatic_dynamic - automatically save and load automatic dynamic decisions to reuse on future runs (#139001)skip_guard_eval_unsafe compiler stance option for power users - skip guard checks when it is known to be safe to do so (#140251)torch.xpu.mem_get_info API: Introduces a new API to retrieve memory information for XPU devices. (#141230)elapsed_time method for XPU events: Introduces a method to measure elapsed time between XPU events. (#140865)torch.xpu.get_arch_list and torch.xpu.get_gencode_flags: Introduces new APIs to retrieve architecture lists and code generation flags for XPU. (#137773)torch.export.export_for_training() API to perform export that can run training. Note that this replaces the non-documented capture_pre_autograd_graph feature (#135374, #135918, #135549, #143224)torch._inductor.aoti_compile_and_package
torch._export.aot_compile), would return a path to a .so. However, this does not have a great user experience as actually there are other files that are used along with the .so, for example .cubin files and serialized extern kernels. So, we introduce a new package format, “PT2 archive”, which is what we intend to have AOTInductor return. This essentially contains a zipfile of all the files that need to be used by AOTInductor, and allows users to send to other environments. There is also functionality to package multiple models into one artifact, and to store additional metadata inside of the package.torch._inductor.aoti_compile_and_package, torch._indcutor.aoti_load_package, or running the loaded model of aoti_load_package on some inputs, you can use the AOTInductor Minifier to create a minimal nn.Module that reproduces the error. (#139351,#140999, #141159, #141156)export.export_for_inference and export.exported_program.core_aten_decompositions API. export_for_inference returns a functional, post-dispatch ATen IR. (#135912).torch.compile(options={"use_minimal_arrayref_interface": True})(foo) is now torch.compile(options={"aot_inductor.use_minimal_arrayref_interface": True})(foo) and torch.compile(options={"allow_stack_allocation": True})(foo) is now torch.compile(options={"aot_inductor.allow_stack_allocation": True})(foo).torch._utils.is_compiling to torch.compiler.is_compiling (#127690) Rewrite torch._utils.is_compiling() to torch.compiler.is_compiling().autotune_num_choices_displayed to control number of kernel options displayed (#138788)force_pointwise_cat concat support through inductor using pointwise kernels (#141966). This forces concat to be generated as a pointwise op with masked loads.annotate_training that adds Inductor annotations to NVTX. (#130429)triton_kernel_default_layout_constraint to tweak stride settings for user-defined Triton kernels, enhancing customization and flexibility (#135530).torch.compile(options={"triton_kernel_default_layout_constraint": "needs_fixed_stride_order"})(foo) (#135581).register_external_matmul for Matmul tuning candidates in Inductor (#130774).package_constants_in_so to exclude weights from .so files in AOTInductor (#141997).load_constants to the package API (#142246).torch.cond is supported (#137428)torch.cond is the recommended way to introduce control flows that can be converted to an ONNX model.
custom_translation_table to provide custom implementations for converting operators to ONNX (#135403)This is useful when you need to override an implementation or provide one that is not currently implemented. Refer to the tutorials for a more complete description of the operator registration mechanism.
# Define the translation using ONNX Script
from onnxscript import opset18 as op
def sym_not_onnx(input):
return op.Not(input)
torch.onnx.export(...
dynamo=True,
custom_translation_table = { # Then provide it here
torch.sym_not: sym_not_onnx,
})
ONNXProgram has a new optimize() method (#137667)Users can run optimize() to flatten nested structures in the ONNX graph, perform constant folding and remove redundancies in the ONNX model. Calling optimize() after exporting to ONNX is recommended.
onnx_program = torch.onnx.export(..., dynamo=True)
onnx_program.optimize() # Optimize the graph before saving is recommended
onnx_program.save(...)
torch.special.i1 (#137899)torch.save (#137735)torch.adaptive_pool3d on cpu (#136091)torch.masked_select (#141928)torch.{linalg.}norm complex half support (#133661)nn.utils.clip_grad_norm_ (#139662)weight argument to MSELoss, HuberLoss and L1Loss (#132049)torch.nn.Linear and torch.nn.Bilinear (#135596)ReduceLROnPlateau and Optimizer.add_param_group interaction by auto-updating min_lrs (#137637)SequentialLR to include ChainedScheduler (#133450)Operator decompositions, FakeTensors and meta tensors are used to trace out a graph in torch.compile and torch.export. They received several improvements:
aten.split_with_sizes (#135728)aten.max_unpool2d/aten.max_unpool3d (#133146)aten.dot (#138596)aten.layer_norm (#140557)aten.scaled_dot_product_attention (#135297)aten.matmul (#134568)aten._embedding_bag (#136774)aten.native_group_norm/aten.native_layer_norm (#137079)aten.to(..., non_blocking=True) (#136513)Aten.addmm (#138520)out= dtype checks for unary ops (#140288)aten.diagonal_copy (#136730)Dynamic shapes
We made many improvements and bugfixes to dynamic shapes in torch.compile
native_layer_norm_backward work with unbacked SymInts (#136798)masked_fill work with unbacked SymIntsl (#137060)torch.cat with large numbers of symbolic variables (#139653)canonicalize_bool_expr (#135621)sympy_generic_le (#135622)_maybe_evaluate_static (#135499)Add._from_args and Mul._from_args (#136143)_SympyT to enable more folding (#138235)new_unbacked_bindings if needed. (#140083)has_free_unbacked_symbols instead of bool(free_unbacked_symbols) (#140027)TORCHDYNAMO_EXTENDED_ADVICE (#137159) (#137196)get_implications (#139738)Custom operators
We improved the existing torch.library APIs and added new ones.
torch.library.triton_op API (#141880)torch.library.register_vmap to work with nested vmap (#137306)torch.library.custom_op APIs on torch.deploy (#139509)torch.library.custom_op overhead (#139513)torch.library.opcheck and register_autograd docs (#141883)cuStreamWriteValue32 (#136488)CudaDMAConnectivityDetector (#137530)world_size=0 to TCPStore (#137792)nccl_nonblocking_timeout (#138374)device_id provided (#140963)P2POp and batch_isend_irecv (#141054)CudaEventCache by default and add multi device support (#140975)PyProcessGroup (#141529)_reduce_scatter_base to c10d::ProcessGroupUCC (#138021)_set_unshard_async_op (#135523)fsdp_pre_all_gather (#136129)all-gather extensions (#137005)shard_placement_fn arg (#137496)set_unshard_in_backward(bool) (#137922)PipelineStage support meta initialization (#136243)rms_norm(), embedding_bag(), record_stream(), rad2deg(), embedding() backward, activation functions (#135872, #135888, #140736, #138627, #137099, #140290)torch.bmm(NST, NST) (#141380)torch.scatter_reduce (#135547)native_dropout_backward (#140140)torch.func.functional_call (#134643))X86InductorQuantizer (#136298)torch.uint16 in quantization and export (#136547)linear_dynamic_fp16 ops for OneDNN (#140376)linear_dynamic_fp16 for x86 (#141480)cuda_flip to unsigned types (#137781)upsample_bicubic2d as Metal op (#136123)scatter_reduce.two (#141948)torch.special.i1 op (#140196)unfold_backward on MPS (#135411)isposinf and isneginf (#136689)MetalShaderLibrary::getFunctionNames() (#141499)tri[lu]_indices (#137648)fmin/fmax/copysign and nextafter to bfloat16 (#136982)arange to bfloat16 (#136754)torch.linalg.cross for bfloat16 (#136984)nan_to_num for bfloat16 (#136986)c10::value_or_else (#138818)torch.Event for MPS (#142468)torch.float8e4m3fn dtype support to semi-structured sparse (#136397)num_threads before default (#136793, #137051)automatic_dynamic_shapes_mark_as - adds an option to cause automatic dynamic shapes to trigger unbacked SymInts rather than backed SymInts (#141415)torch.compile support for Python 3.13 (#139533)export preserves node.meta["custom"] field (#138266)neg and pos operator to serde/serialize (#138309, #143343)_inline_module util function to work with both args and kwargs (#136631)_replace_hook to a list of Callable (#142006)keep_original_order=True and no-op graph (#141340)fx.Graph (#139770)stride in TensorMetadata to always be a Tuple[int, ...] (#141106)from_node node meta to track source recursively (#142066)fuse_by_partitions to always return output as tuple (#142056)custom_op_default_layout_constraint in Inductor to optimize tensor layout (#135239).relu_nan_to_num option for handling NaNs in pre-grad passes in AOTInductor (#138545).env_name_default and env_name_force for better configuration management (#138956).from dynamic axes to shapes (#140488)torch.mean(..., out=) for fp16 and bf16 on CPU (#135174)torch.uint16, torch.uint32, torch.uint64 (#137184)torch.utils.module_tracker.ModuleTracker (#141960)nn.functional.softshrink returning 0 on NAN input (#138421)torch.nn.EmbeddingBag when per_sample_weights is differentiable but embedding weights are not (#142338)torch.compiling torch.scaled_dot_product_attention, in the case where the scale argument is a dynamic shape (#141728)torch.compiling torch.rrelu, in the case where it mutates any module buffers (#136008)destroy_process_group (#141510)ProcessGroup and Expose backend Options to reflect the correct code structure (#132931) (#135653)all-reduce (#137257)all-reduce (#137345)CUDAEventCache is enabled (#138040)alltoall (#141816)ncclCommSplit (#138781)IntraNodeComm::rendezvous() (#141200)_are_we_tracing() in dynamo for functional collectives (#142075)from_group when passing a tensor mesh (#137713)test_all_gather_extensions_monkey_patch (#136130).to(dtype) (#137593)stage_backward (#136507)_validate_schedule (#142211)torch.compile (#135819)sum(), unsqueeze(), to() on non-contiguous NJTs, where(), select(), chunk(), reductions (#131945, #141392, #137124, #141500, #139317, #141506, #141604)linear_backward() memory usage using a more efficient formula (#141163)nll_loss (#135434)torch.normal (#136863)channels_last_3d in nn.Conv3d (#141780)c10::Event unit test failure on XPU backend (#141800)F.logsigmoid (#141333)UnicodeDecodeError: 'utf-8' codec can't decode byte (#139062)ideep:lowp_kind to matmul_forward::compute on cache misses (#135058)torchao/quantization/utils.py to targeted device to avoid device mismatch issue (#135204)dilation in torch.quantized_max_pool3d() (#137845)prepare_pt2e graph with concat of the same node (#141651)bmm(COO, dense) illegal memory access for some shapes (#131977)sparse_csr layout (#134335)torch.package for 3.13 (#141409)_get_non_persistent_buffers for duplicate submodules (#136552)make_fx(#137240)export for constant outputs (#137547, #137993)export_for_training (#138292) AttrProxy._modules when module is overwritten as None (#139957)torch.Size (#137465)test_lazy_module_kwargs (#137705)is_export flag to FunctionalTensorMode in analysis pass (#138836)__init__.py to shape inference folder. (#135461)sympy.oo in bitwise_and/or value_ranges (#141522)abi_compatible (#136233)_dynamo.compiled_autograd.reset() (#137889)atomic_add_vec (#138744)clone_preserve_strides (#139224, #139458)torch.compile compiler error when march=aarch64+sve (#137795)Note truncated.
This release is meant to fix the following regressions:
This release is meant to fix the following regressions:
Besides the regression fixes, the release includes several documentation updates.
See release tracker https://github.com/pytorch/pytorch/issues/132400 for additional information.
The deprecated torch.onnx.SymbolicContext class has been removed. (Non-dynamo) custom symbolic functions can no longer take ctx: torch.onnx.SymbolicCo…
We are excited to announce the release of PyTorch® 2.5! This release features a new CuDNN backend for SDPA, enabling speedups by default for users of SDPA on H100s or newer GPUs. As well, regional compilation of torch.compile offers a way to reduce the cold start up time for torch.compile by allowing users to compile a repeated nn.Module (e.g. a transformer layer in LLM) without recompilations. Finally, TorchInductor CPP backend offers solid performance speedup with numerous enhancements like FP16 support, CPP wrapper, AOT-Inductor mode, and max-autotune mode. This release is composed of 4095 commits from 504 contributors since PyTorch 2.4. We want to sincerely thank our dedicated community for your contributions. As always, we encourage you to try these out and report any issues as we improve 2.5. More information about how to get started with the PyTorch 2-series can be found at our Getting Started page. As well, please check out our new ecosystem projects releases with TorchRec and TorchFix.
| Beta | Prototype |
|---|---|
| CuDNN backend for SDPA | FlexAttention |
| torch.compile regional compilation without recompilations | Compiled Autograd |
| TorchDynamo added support for exception handling & MutableMapping types | Flight Recorder |
| TorchInductor CPU backend optimization | Max-autotune Support on CPU with GEMM Template |
| TorchInductor on Windows | |
| FP16 support on CPU path for both eager mode and TorchInductor CPP backend | |
| Autoload Device Extension | |
| Enhanced Intel GPU support |
*To see a full list of public feature submissions click here.
The cuDNN "Fused Flash Attention" backend was landed for torch.nn.functional.scaled_dot_product_attention. On NVIDIA H100 GPUs this can provide up to 75% speed-up over FlashAttentionV2. This speedup is enabled by default for all users of SDPA on H100 or newer GPUs.
Regional compilation without recompilations, via torch._dynamo.config.inline_inbuilt_nn_modules which default to True in 2.5+. This option allows users to compile a repeated nn.Module (e.g. a transformer layer in LLM) without recompilations. Compared to compiling the full model, this option can result in smaller compilation latencies with 1%-5% performance degradation compared to full model compilation.
See the tutorial for more information.
This feature advances Inductor’s CPU backend optimization, including CPP backend code generation and FX fusions with customized CPU kernels. The Inductor CPU backend supports vectorization of common data types and all Inductor IR operations, along with the static and symbolic shapes. It is compatible with both Linux and Windows OS and supports the default Python wrapper, the CPP wrapper, and AOT-Inductor mode.
Additionally, it extends the max-autotune mode of the GEMM template (prototyped in 2.5), offering further performance gains. The backend supports various FX fusions, lowering to customized kernels such as oneDNN for Linear/Conv operations and SDPA. The Inductor CPU backend consistently achieves performance speedups across three benchmark suites—TorchBench, Hugging Face, and timms—outperforming eager mode in 97.5% of the 193 models tested.
We've introduced a flexible API that enables implementing various attention mechanisms such as Sliding Window, Causal Mask, and PrefixLM with just a few lines of idiomatic PyTorch code. This API leverages torch.compile to generate a fused FlashAttention kernel, which eliminates extra memory allocation and achieves performance comparable to handwritten implementations. Additionally, we automatically generate the backwards pass using PyTorch's autograd machinery. Furthermore, our API can take advantage of sparsity in the attention mask, resulting in significant improvements over standard attention implementations.
For more information and examples, please refer to the official blog post and Attention Gym.
Compiled Autograd is an extension to the PT2 stack allowing the capture of the entire backward pass. Unlike the backward graph traced by AOT dispatcher, Compiled Autograd tracing is deferred until backward execution time, which makes it impervious to forward pass graph breaks, and allows it to record backward hooks into the graph.
Please refer to the tutorial for more information.
Flight recorder is a new debugging tool that helps debug stuck jobs. The tool works by continuously capturing information about collectives as they run. Upon detecting a stuck job, the information can be used to quickly identify misbehaving ranks/machines along with code stack traces.
For more information please refer to the following tutorial.
Max-autotune mode for the Inductor CPU backend in torch.compile profiles multiple implementations of operations at compile time and selects the best-performing one. This is particularly beneficial for GEMM-related operations, using a C++ template-based GEMM implementation as an alternative to the ATen-based approach with oneDNN and MKL libraries. We support FP32, BF16, FP16, and INT8 with epilogue fusions for x86 CPUs. We’ve seen up to 7% geomean speedup on the dynamo benchmark suites and up to 20% boost in next-token latency for LLM inference.
For more information please refer to the tutorial.
Inductor CPU backend in torch.compile now works on Windows. We support MSVC (cl), clang (clang-cl) and Intel compiler (icx-cl) for Windows inductor currently.
See the tutorial for more details.
Float16 is a commonly used reduced floating point type for performance improvement in neural network inference/training. Since this release, float16 for both eager and TorchInductor is supported on the CPU path.
PyTorch now supports autoloading for out-of-tree device extensions, streamlining integration by eliminating the need for manual imports. This feature, enabled through the torch.backends entrypoint, simplifies usage by ensuring seamless extension loading, while allowing users to disable it via an environment variable if needed.
See the tutorial for more information.
Intel GPUs support enhancement is now available for both Intel® Data Center GPU Max Series and Intel® Client GPUs (Intel® Core™ Ultra processors with built-in Intel® Arc™ graphics and Intel® Arc™ Graphics for dGPU parts), which is to make it easier to accelerate your Machine Learning workflows on Intel GPUs in PyTorch 2.5 release. We also enabled the initial support of PyTorch on Windows for Intel® Client GPUs in this release.
These features are available through PyTorch preview and nightly binary PIP wheels. For more information regarding Intel GPU support, please refer to documentation.
[c10d] Remove Option for ProcessGroup and Expose backend Options to reflect the correct code structure (#132931)
# Users can pass in a basic option when creating an instance of ProcessGroup
base_pg_options = ProcessGroup.Options(backend=str(backend))
base_pg_options._timeout = timeout
pg: ProcessGroup = ProcessGroup(
store, rank, group_size, base_pg_options
)
# Users then need to create a backend option to create the comm backend (e.g., ProcessGroupNCCL)
pg_options = ProcessGroupNCCL.Options()
backend = ProcessGroupNCCL(
store, rank, group_size, pg_options
)
# No basic option is passed in when creating a instance of ProcessGroup
pg: ProcessGroup = ProcessGroup(store, rank, group_size)
pg._set_default_backend(Backend.backend_type_map[backend])
# Users then need to create a backend option to create the comm backend (e.g., ProcessGroupNCCL)
pg_options = ProcessGroupNCCL.Options()
backend = ProcessGroupNCCL(
store, rank, group_size, pg_options
)
dynamic_dim() (#134211)
dynamic_dim() method for specifying dynamic shapes in torch.export() has been removed. Please refer to the export tutorial for using Dims to specify dynamic shapes.[Torch] Support meta device in checkpoint (#132684)
Switch to internal benchmarking and update benchmarking path (#132827)
This change moves from using triton’s benchmarking utils to the internal inductor utils at torch._inductor.runtime.benchmarking. To update your benchmarking code:
# before
from torch._inductor.runtime.runtime_utils import do_bench_gpu
# ...
do_bench_gpu(kernel, rep=40, fast_flush=True)
# after
from torch._inductor.runtime.benchmarking import benchmarker
# ...
benchmarker.benchmark_gpu(kernel_call, rep=40, fast_flush=True)
Update fused kernels and call _safe_softmax from SDPA (#131863)
Before this PR, fully masked rows in the attn_mask passed to nn.functional.scaled_dot_product_attention would yield NANs in the output, after this PR, fully masked rows yield 0s.
Example:
2.4.0
B, S, D = 1, 1, 128
q = torch.randn(B, S, D, device='cuda')
k = torch.randn(B, S, D, device='cuda')
v = torch.randn(B, S, D, device='cuda')
attn_mask = torch.tensor([False], device='cuda')
F.scaled_dot_product_attention(q, k, v, attn_mask=attn_mask)
tensor([[[nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
nan, nan, nan, nan, nan, nan, nan, nan, nan]]], device='cuda:0')
2.5.0
B, S, D = 1, 1, 128
q = torch.randn(B, S, D, device='cuda')
k = torch.randn(B, S, D, device='cuda')
v = torch.randn(B, S, D, device='cuda')
attn_mask = torch.tensor([False], device='cuda')
F.scaled_dot_product_attention(q, k, v, attn_mask=attn_mask)
tensor([[[-0., -0., 0., -0., 0., 0., -0., -0., -0., -0., 0., -0., -0., -0., 0., -0., 0., 0., -0., -0., 0., -0., -0.,
0., 0., -0., 0., -0., -0., 0., -0., -0.]]], device='cuda:0')
GradScaler for respecting an already set grad_scale value (#123429)torch.onnx.export (except for the first three arguments) are now keyword-only (#131501)Options can be supplied by keywords only to allow for future addition and evolution of the torch.onnx.export API.
Example:
Version 2.4
torch.onnx.export(model, input, f, True, False)
Version 2.5:
torch.onnx.export(model, input, f, export_params=True, verbose=False)
torch.onnx._export has been removed (133824)torch.onnx._export is an internal API which is not meant for public consumption. Use the public torch.onnx.export instead.
Example:
Version 2.4
torch.onnx._export(...)
Version 2.5:
torch.onnx.export(...)
op_level_debug option from torch.onnx.ExportOptions has been removed (#134961)This option, designed to identify operator discrepancies, proved unreliable and has been removed. Instead, use torch.onnx.export(..., report=True, verify=True) option to validate exported models.
ONNXProgramSerializer class has been removed (#135261)The ONNX model in torch.onnx.ONNXProgram is now maintained and serialized by ONNX IR.
textproto, onnxtext, and json formats are supported by default when calling ONNXProgram.save() with a corresponding file extension.
SymbolicContext class has been removed (#132184)The deprecated torch.onnx.SymbolicContext class has been removed. (Non-dynamo) custom symbolic functions can no longer take ctx: torch.onnx.SymbolicContext as the first argument.
onnx_unpack_quantized_weights (#129021)is_caffe2_aten_fallback in torch.onnx.symbolic_helperCheckerError and InvalidExportOptionsError are removed. Users can always catch RuntimeError to handle torch.onnx export errors.
torch._dynamo.utils.CompileProfiler (#135133)None for specifying static dimensions in dynamic_shapes (#134877)<br>
The use of None at the dimension-level for specifying dynamic shapes is now deprecated, and a user warning will be raised, so please use Dim.STATIC in its place. Specifying None for an entire input, or an entire program, is still supported.search_autotune_cache (#133628)torch.onnx.export is deprecated (#131501)The ability to supply model keyword arguments as a final dictionary is deprecated. Users should use the kwargs parameter instead.
Deprecated:
torch.onnx.export(model, (arg1, arg2, {“kwarg1”: …}))
Future:
torch.onnx.export(model, (arg1, arg2), kwargs={“kwarg1”: …})
torch.onnx.OperatorExportTypes is deprecated (#131501)The ability to supply operator_export_type in torch.onnx.export() is deprecated. Exported ONNX graphs will always use the ONNX opset domain. Options ONNX_FALLTHROUGH, ONNX_ATEN and ONNX_ATEN_FALLBACK are no longer supported. The OperatorExportTypes class will be removed in a future release.
training option in torch.onnx.export is deprecatedSet the model training mode first before exporting instead.
Deprecated:
torch.onnx.export(model, inputs, path, training=torch.onnx.TrainingMode.EVAL)
Future:
model = model.eval()
torch.onnx.export(model, inputs, path)
torch.utils.checkpoint (#125795, #129262)all-gather and reduce-scatter (#130583)torch._dynamo.config.enable_compiler_collectives for syncing compilation across ranks (#130935)export_for_training [WIP/unstable] (#129092, #130062, #134677, #135549)FlexAttention API public (#130755)nn.Modules.set_submodule() like get_submodule (#127714)nn.Buffer like nn.Parameter (#125971)__torch_function__ handler to Tensor.get_device cpp (#132567)torch.serialization.skip_data context manager to create a metadata-only checkpoint (#134504)torch.serialization.safe_globals context manager to work with weights_only serialization (#127939)_numeric_debug_handle throguh deepcopy and re-export (#129287)numeric_debugger top level APIs (#130643)node.meta["custom"] field (#134040)dynamo=True option and new export logic (#132530, #133743, #134304, #134782, #135378, #135399, #135786, #136162, #135134, #134976, #135367, #135418, #135591, #135520)We introduce the dynamo=True option in torch.onnx.export(). This is recommended as a replacement for torch.onnx.dynamo_export starting in PyTorch 2.5.
Version 2.5:
onnx_program = torch.onnx.export(model, inputs, kwargs=kwargs, dynamo=True)
# Use the external_data option to save weights as external data
onnx_program.save(“model.onnx”, external_data=True)
# To save without initializers
onnx_program.save(“model.onnx”, include_initializers=False, keep_initializers_as_inputs=True)
torch.onnx.export(model, args, dynamo=True, report=True, verify=True) leverages torch.export and ONNX IR to convert captured ExportedPrograms to ONNX efficiently and robustly. This new process reduces memory consumption by half compared to dynamo_export in 2.4, while preserving rich tensor shape and stack trace information in the ONNX graph. You can leverage the report=True option to obtain a conversion report in markdown format to diagnose any conversion issues. Set verify=True to verify the ONNX model numerically with ONNX Runtime.
When using external_data=True to save model weights as external data to the .onnx file, weights larger than 1 MB are now aligned at 64 KB addresses. This allows runtimes to memory-map weights for better memory efficiency during inference.
[NOTE]
Thedynamo=Trueoption currently supports only ONNX opset 18. Future releases will expand support to newer opsets.
[NOTE]
Thedynamo=Trueoption requires the latest versions ofonnxscriptandonnxpackages.
torch.autograd.grad (#127766)torch.autograd.graph.increment_version accept List[Tensor] (#132652)torch.autograd.graph.Node.register_hook during Node execution are run (#134728)from __future__ import annotations (#128809)mutates_args=”unknown” for if you didn’t know the alias information of your custom op (#129614)register_flop_formula now actually does something for custom ops (#131777)Operator decompositions, FakeTensors and meta tensors are used to trace out a graph in torch.compile and torch.export. They received several improvements:
padding_side to pad_sequence with "left" and "right" options ("right" as default) (#131884)kwargs to composable AC API to enable full capabilities (#128516)ActivationWrapper an abstract class (#129808)somaxconn and enabled TCP_NODELAY to TCPStoreLibUvBackend (#128739)TCPStore (#129261)TCPStore (#129985)new_group eager when used with comm_split (#129284)split_group to create a process group (#130507)bfloat16 support for NAN check (#131131)ProcessGroupNCCL barrier all-reduce (#132701)ProcessGroupNCCL watchdog timeout (#133028)ncclCommAbort if comm is not initialized (#133630)DTensor to public namespace (#134203)slice_backward to use op strategy (#130287)from_local API with run_check (#130289)nn.init.orthogonal_ (#132104)scaled_dot_product_efficient_attention and scaled_dot_product_flash_attention more conservatively cached (#134146)DTensor for foreach ops so model doesn't have to be fully tp-ed when using 2D (#134551)DTensor (#132816)DTensor after torch.distributed.checkpoint.state_dict.set_optimizer_state_dict (#128004)state_dict (#128685)clip_grad_norm_ norm compute order deterministic (#134673)dataclass(frozen=True) (#135067)FSDPParamGroup record_functions (#128624)set_post_optim_event (#128975)contract to allow Sequence[nn.Module] (#127773) (#130947)List[nn.Module] as arg (#130949)fsdp.set_ op through lowering (#130786)set_reduce_scatter_divide_factor (#129286)_get_remote_device_str (#132120)FSDPParamGroup and FSDPParam (#132350)fsdp.set_ convey to functionalization that it mutates storage (#132322)FSDPParam.sharded_param in lazy_init (#132954)replicate and fully_shard (#133059)TCPStore by default when using c10d_rendezvous_backend (#128096)wait instead of get for store barrier (#130148)TCPStore backend type and rely on c10d to decide which backend to use (#134882)local_addr across all rendezvous implementations (#135262)mp.start_processes for forkserver (#134629)SequenceParallel and its documentation (#131346)dw_runner for PipelineStage (#128983)_Action from F1_1 to 1F1 (#129786)unshard/reshard pass (#129810)send/recv pass (#130378)zb1p schedule (#130210)get_schedule_class util (#132768)discard (#133317), intersection (#130672), remove (#132943))inspect.signature.bind (#132330) and inspect.signature.Parameter attribute access (#134636)random.Random objects (#133725)set on KeysView (#131389)dict conversion of objects derived from MutableMapping (#131367), constrained subclasses of dict and OrderedDict (#132558)__contains__ on __dict__ of user defined classes (#131378)abc.MutableMapping.get (#132363)frozenset (#134563)__new__ (#132977), object.__new__ (#133746)torch.cuda.device (#133385)id(Parameter) (#130100)str on UserDefinedObjectVariable (#130506)out-variant custom ops that return None (#129078)autograd.Function mark_non_differentiable (#134087), ctx.set_materialize_grads (#133978)ModelOutput class (#127780)mark_static can now be used on nn.Modules to make int attributes static (instead of automatic dynamic) (#134713)export_for_training (#128077, #129249, #134801)run_decompositions() (#130970, #131075)torch.amp.autocast_mode as a higher-order-op subgraph (#132677)ExportedProgram.validate() public-facing (#132777)“custom” metadata field (#131912)getitem deduplication pass (#133618, #134830)move_to_device_pass public-facing (#134263)while_loop higher-order-op public-facing (#128562)TORCHEXPORT_EXTENDED_DEBUG_CURRENT_LOC=1 flag for line-by-line debugging in non-strict mode (#134298)aten.full.default and aten.full_like.default (#130639)aten.to.dtype + mutation_op pattern for TSConverter (#132529)InterpreterModule to trace_rules (#132949)tracing_mode support for TorchBind (#129586)set_grad_enabled higher-order-op in dynamo to enable re-tracing (#134281)_foreach_sigmoid and _foreach_lgamma (#134253, #134344)get_isolated_graphmodule (#130886)meta["val"] that is a dict, for triton kernels and for the partitioner (#132466)max_acc_splits (#133041, #133724)fx.Graph.lint (#135069)python_code(verbose=True) (#132192)do_bench_using_profiling in TORCHINDUCTOR_PROFILE (#133523)TORCHINDUCTOR_CPP_GEMM_THREAD_FACTORS (#132730)nn.Module.mtia() (#131499)_init_group to allow better error checking in more cases (#131153)torch.serialization.set_default_mmap_options usable as a context manager (#134371)set_module_name_qconfig in X86InductorQuantizer (#126044)onnxruntime (#134662)torch.autograd.graph.register_multi_grad_hook (#132055)torch.autograd.graph.saved_tensors_hooks (#131700)torch.autograd.graph.saved_tensors_hooks to not detach when unpacking tensors that do not require grad (#127959)torch.utils.checkpoint (#128671)Fix interaction between torch.conj() tensor subclasses and scalar tensors (#131482)
Better complex number support for aliasing prim operators (#132699)
AOTDispatcher is a component in the torch.compile stack responsible for capturing a graph of normalized/functionalized ATen IR as well as capture the backward. A few bugfixes this release:
Dynamic Shapes
avg_pool_2d backward attempt 2 (#129818)commSplit bug by having every rank being called even though it is no-color (#128459)torch.distributed.barrier()'s streamSynchronize (#129908)ENABLE_INTRA_NODE_COMM=1 + multiple process groups leads to failure (#130269)all-reduce (#131244)split_group usage when there is a single rank (#131824)TCPStore (#131773) (#131913)DeviceMesh initialization (#133024)foreach_norm when ord is 2 (#130753)_MaskPartial when multiple embeddings coexist (#131264)unshard without lazy init (#129241)loss_parallel with BF16 logits (#130550).data mutations invisible to autograd (#131403)TorchDispatchMode is enabled (#131828)nn_module_stack is the same when inlining inbuilt nn.Modules (#128295)post_acc_grad_hooks (#134205) and ctx.saved_variables (#134286)OptimizedModule.training flag now mirrors the wrapped module. (#131546)map/zip and return map/zip instead of a tuple (#135074)list_backends (#132546)preserve_module_call_signature (#128260).requires_grad on FakeTensors in graph metadata (#128656)aot_export (#128844)source_fn_stack for training IR (#132033).requires_grad for unflattened parameters (#134353)aten::to for training IR (#134622)node.users when inlining higher-order-ops (#133144)linearize(grad(...)) call by moving DCE (#133364)linalg.lstsq (#130612)torch.[all|any] for 5+D tensors (#130542)toAccumulateType signature in Normalization.cpp (#134540)ldexp (#133519)scaled_dot_product_attention (#133314)scaled_dot_product_attention with float scale (#135594)nn.Module.__getattr__ (#129315)nn.Module._call_impl (#129285)torch.nn.Module.parameters (#129583)getitems (#130654)PolyfilledFunctionVariable.call_function (#134842)get_implications calls (#128410)Node.__update_args_kwargs (#135076)setattr in Node.__init__ (#135079)Note truncated.
This release is meant to fix the following issues (regressions / silent correctness):
This release is meant to fix the following issues (regressions / silent correctness):
Release tracker #132400 contains all relevant pull requests related to this release as well as links to related issues.
Vulnerability related updates of packages used in CI (#124614, #124675, #124976, #124983, #125698, #126805, #126989)
We are excited to announce the release of PyTorch® 2.4!
PyTorch 2.4 adds support for the latest version of Python (3.12) for torch.compile.
AOTInductor freezing gives developers running AOTInductor more performance based optimizations by allowing the
serialization of MKLDNN weights. As well, a new default TCPStore server backend utilizing libuv has been introduced
which should significantly reduce initialization times for users running large-scale jobs.
Finally, a new Python Custom Operator API makes it easier than before to integrate custom kernels
into PyTorch, especially for torch.compile.
This release is composed of 3661 commits and 475 contributors since PyTorch 2.3. We want to sincerely thank our dedicated community for your contributions. As always, we encourage you to try these out and report any issues as we improve 2.4. More information about how to get started with the PyTorch 2-series can be found at our Getting Started page.
<table> <tr> <td>Beta </td> <td>Prototype </td> <td>Performance Improvements </td> </tr> <tr> <td>Python 3.12 support for torch.compile </td> <td>FSDP2: DTensor-based per-parameter-sharding FSDP </td> <td>torch.compile optimizations for AWS Graviton (aarch64-linux) processors </td> </tr> <tr> <td>AOTInductor Freezing for CPU </td> <td>torch.distributed.pipelining, simplified pipeline parallelism </td> <td>BF16 symbolic shape optimization in TorchInductor </td> </tr> <tr> <td>New Higher-level Python Custom Operator API </td> <td>Intel GPU is available through source build </td> <td>Performance optimizations for GenAI projects utilizing CPU devices </td> </tr> <tr> <td>Switching TCPStore’s default server backend to libuv </td> <td> </td> <td> </td> </tr> <tr> <td> </td> <td> </td> <td> </td> </tr> </table>
*To see a full list of public feature submissions click here.
There is a reported issue (#131070) when using torch.compile if onnxruntime-training lib is
installed. The issue will be fixed (#131194) in v2.4.1. It can be solved locally by setting the environment variable
TORCHINDUCTOR_WORKER_START=fork before executing the script.
It was also reported (#130684) that the new version of triton uses cuda features that are not compatible with pre-cuda12 drivers.
In this case, the workaround is to set
TRITON_PTXAS_PATH manually as follows (adapt the code according to the local installation path):
TRITON_PTXAS_PATH=/usr/local/lib/python3.10/site-packages/torch/bin/ptxas python script.py
TreadPool size to number of physical cores (#125963)Changed the default number of threads used for intra-op parallelism from the number of logical cores to the number of physical cores. This should reduce core oversubscribing when running CPU workload and improve performance. Previous behavior can be recovered by using torch.set_num_threads to set the number of threads to the desired value.
torch.quasirandom.SobolEngine.draw default dtype handling (#126781)The default dtype value has been changed from torch.float32 to the current default dtype as given by
torch.get_default_dtype() to be consistent with other APIs.
torch._C._TensorBase directly (#125558)This is an internal subclass that a user used to be able to create an object that is almost a Tensor in Python and was advertised as such in some tutorials. This is not allowed anymore to improve consistency and all users should subclass torch.Tensor directly.
torch.compile will raise an error (#122502)The torch.compile flow involves functionalizing any mutations inside the region being compiled. Torch.as_strided is
an existing view op that can be used non-compositionally: meaning when you call x.as_strided(...), as_strided will only
consider the underlying storage size of x, and ignore its current size/stride/storage_offset when creating a new view.
This makes it difficult to safely functionalize mutations on views of as_strided that are created non-compositionally,
so we ban them rather than risking silent correctness issues under torch.compile.
An example of a non-compositional usage of as_strided followed by mutation that we will error on is below. You can avoid this issue by re-writing your usage of as_strided so that it is compositional (for example: either use a different set of view ops instead of as_strided, or call as_strided directly on the base tensor instead of an existing view of it).
@torch.compile
def foo(a):
e = a.diagonal()
# as_strided is being called on an existing view (e),
# making it non-compositional. mutations to f under torch.compile
# are not allowed, as we cannot easily functionalize them safely
f = e.as_strided((2,), (1,), 0)
f.add_(1.0)
return a
Previously, you could register a custom op through the operator registration APIs, but give it a schema that contained types unknown to the PyTorch Dispatcher. This behavior came from TorchScript, where “unknown” types were implicitly treated by the TorchScript interpreter as type variables. However, calling such a custom op through regular pytorch would result in an error later. As of 2.4, we will raise an error at registration time, when you first register the custom operator. You can get the old behavior by constructing the schema with allow_typevars=true.
TORCH_LIBRARY(my_ns, m) {
// this now raises an error at registration time: bar/baz are unknown types
m.def("my_ns::foo(bar t) -> baz");
// you can get back the old behavior with the below flag
m.def(torch::schema("my_ns::foo(bar t) -> baz", /*allow_typevars*/ true));
}
The torch.autograd.function.traceable(...) API, which sets the is_traceable class attribute on a torch.autograd.Function class was deprecated in 2.3 and is now being deleted. This API does not do anything and was only meant for internal purposes. The following raised an warning in 2.3, and now errors because the API has been deleted:
@torch.autograd.function.traceable
class Func(torch.autograd.Function):
...
SparseAdam weird allowance of raw Tensor input (#127081).Update get_group and add get_all_groups (#128097) In 2.3 and before, users can do:
mesh_2d = init_device_mesh(
"cuda", (2, 2), mesh_dim_names=("dp", "tp")
)
mesh_2d.get_group() # This will return all sub-pgs within the mesh
assert mesh_2d.get_group()[0] == mesh_2d.get_group(0)
assert mesh_2d.get_group()[1] == mesh_2d.get_group(1)
But from 2.4 forward, if users call get_group without passing in the dim, users will get a RuntimeError.
Instead, they should use get_all_groups:
mesh_2d = init_device_mesh(
"cuda", (2, 2), mesh_dim_names=("dp", "tp")
)
mesh_2d.get_group() # This will throw a RuntimeError
assert mesh_2d.get_all_groups()[0] == mesh_2d.get_group(0)
assert mesh_2d.get_all_groups()[1] == mesh_2d.get_group(1)
Retire torch.distributed.pipeline (#127354) In 2.3 and before, users can do:
import torch.distributed.pipeline # warning saying that this will be removed and users need to migrate to torch.distributed.pipelining
But from 2.4 forward, if users write the code above, users will get a ModuleNotFound error.
Instead, they should use torch.distributed.pipelining:
import torch.distributed.pipeline # -> ModuleNotFoundError
import torch.distributed.pipelining
Complete revamp of float/promotion sympy handling (#126905)
torch.load with default weights_only=False value (#129239, #129396, #129509).
A warning is now raised if the weights_only value is not specified during a call to torch.load, encouraging users to
adopt the safest practice when loading weights.torch.compile to provide a nice error
message when tracing custom ops into a graph that are not written in a PT2-friendly way (because, for example, they
try to directly access a tensor’s data pointer from a region of code being traced). More details on integrating custom
ops with torch.compile can be found here (#123292)torch.nn.hardtahn allowed min_val to be greater than max_val (#121627)model = AnyModel(device=torch.device("cuda"))
model_state_dict = get_model_state_dict(model)
set_model_state_dict(
model,
model_state_dict=new_model_state_dict,
options=StateDictOptions(strict=False),
)
# Below way of calling API is also legit
model_state_dict2 = get_model_state_dict(model, submodules={model.submodule})
set_model_state_dict(
model,
model_state_dict={model.submodule: new_submodel_state_dict},
options=StateDictOptions(strict=False),
)
But from 2.4 forward, if users call get_model_state_dict or set_model_state_dict with a submodule path or
state_dict, users will see a warning about the feature. To achieve the same functionality, users can manually
filter out the state_dict returned from get_state_dict API and preprocess the model_state_dict before
calling set_state_dict API:model = AnyModel(device=torch.device("cuda"))
model_state_dict = get_model_state_dict(model)
set_model_state_dict(
model,
model_state_dict=new_model_state_dict,
options=StateDictOptions(strict=False),
)
# Deprecating warnings thrown for the below way of calling API
model_state_dict2 = get_model_state_dict(model, submodules={model.submodule})
set_model_state_dict(
model,
model_state_dict={model.submodule: new_submodel_state_dict},
options=StateDictOptions(strict=False),
)
model = AnyModel(device=torch.device("cuda"))
fsdp_model = FSDP(model)
# Users can do both ways below
get_model_state_dict(model)
with FSDP.state_dict_type(fsdp_model, StateDictType.FULL_STATE_DICT):
fsdp_model.state_dict()
But from 2.4 forward, if users call state_dict or set state_dict with the FSDP.state_dict_type, users will see warnings. And the recommended solution now is to use get_model_state_dict and set_model_state_dict directly:model = AnyModel(device=torch.device("cuda"))
fsdp_model = FSDP(model)
get_model_state_dict(model)
# Deprecating warnings thrown for the below way of calling API
with FSDP.state_dict_type(fsdp_model, StateDictType.FULL_STATE_DICT):
fsdp_model.state_dict()
torch._aminmax operator (#125995).
torch._aminmax -> torch.aminmax insteadat::autocast::get_autocast_gpu_dtype() -> at::autocast::get_autocast_dtype(at::kCUDA)
at::autocast::get_autocast_cpu_dtype() -> at::autocast::get_autocast_dtype(at::kCPU)torch.get_autocast_gpu_dtype() -> torch.get_autocast_dtype(“cuda”),
torch.set_autocast_gpu_dtype(dtype) -> torch.set_autocast_dtype(“cuda”, dtype),
torch.is_autocast_enabled() -> torch.is_autocast_enabled(“cuda”),
torch.set_autocast_enabled(enabled) -> torch.set_autocast_enabled(”cuda”, enabled),
torch.get_autocast_cpu_dtype() -> torch.get_autocast_dtype(“cpu”)torch.cuda.amp.autocast(args…) -> torch.amp.autocast(“cuda”,args…),
torch.cpu.amp.autocast(args…) -> torch.amp.autocast(“cpu”, args…),torch.cuda.amp.GradScaler(args…) -> torch.amp.GradScaler(“cuda”, args…),
torch.cuda.amp.GradScaler(args…) -> torch.amp.GradScaler(“cpu”, args…),torch.cuda.amp.custom_fwd(args…) -> torch.amp.custom_fwd(args…, device_type=’cuda’),torch.compile support for LRScheduler with Tensor LRs (#123751, #123752, #123753, #127190)preferred_backend on ROCm backend.gemm implementation (#122106)allgather_matmul (#118038)all_to_all_single_grad (#123599)send/recv_object_list operations (#124379)PrivateUse1 (#124818)torch.compile is compatible with Python 3.12.torch.compile users to easily control TorchDynamo tracing of PyTorch internal code.torch.compile support for RAdam and Adamax optimizer. Compiler optimizers now demonstrate SOTA performance.torch.compile to reuse compiled
artifacts on repeated blocks in the models. This gives another point in the tradeoff space of compilation time and performance speedup.
By moving torch.compile from full model to a repeated block (e.g. moving torch.compile from full LLM to a repeated Transformer block),
we can now achieve faster compilation time with some performance dip compared to full model.
We plan to make this flag default to True in the 2.5 release.nn.functional.batch_norm: add forward AD rule for miopen backend (#125069)nn.functional.scaled_dot_product_attention: add backward rule for cuDNN backend (#122510)torch.compile and triton with Python 3.12 CI/CD (#127547, #123307, #126218)swap_tensors path to nn parametrizations (#124130)use_count constraints for swap_tensors when AccumulateGrad node holds a reference (#127313)int64_t indexing for Upsample2d backwards (#123682)LazyModuleMixin constructor (#123968)X86InductorQuantizer (#114853)X86InductorQuantizer (#122775)X86InductorQuantizer (#122776)qconv by reducing integration overhead (#123240)fake_quant_per_channel (#123186)TORCH_NCCL_HIGH_PRIORITY option for ProcessGroupNCCL (#122830)__repr__ to P2POp class (#126538)commCreateFromRanks to c10d (#127421, #127982)dist.get_node_local_rank helper (#123992)get_node_local_rank() accept fallback_rank (#126737)ProcessGroupWrapper support custom backend (#124447)DeviceMesh.from_group() (#124787)_validate_tp_mesh_dim support 3D (#125763)type_check param to copy state dict utils (#127417)DefaultPlanner (#123869)_extra_state (#125336)broadcast_from_rank0 option for model/optim state_dict (#125338, #125339)state_dict support torch.distributed is not initialized case (#127385)state_dict when saving and unflattening when loading (#127071)set_model_state_dict and set_optimizer_state_dict (#127384)scaled_dot_product_attention (flash-attention) (#122541)view_as_complex and view_as_real (#122569)fused_adam and fused_adamw when lr is a tensor (#126750)clip_grad_norm_ for DTensor by default (#126423), (2) optimizer for DTensor by default (#123394)pin_memory argument (#119878)clip_grad_norm_ (#122001)pre_backward_hook unserializable (#125464)parallelize_plan (#122968)prepare_module_input (#124114)torch.profiler:torch.cuda.memory._dump_snapshot:record_function:torch.compile + __torch_function__ support by onboarding Scaled Dot Product Attention (SDPA) and TensorDict.torch.load map_location for wrapper subclass and device being serialized through numpy (#126728)Aten.rrellu_: fix decomp when default values are missing (#126978)torch.compile stack that functionalizes and normalizes the graph, and adds
support for compiling the backward during training. There were several bugfixes and improvements to AOTDispatcher:
torch.compile used with triton kernels under inference_mode (#124489)torch.compile captures small graphs with outputs that alias graph inputs (#121007)__torch_dispatch__-based tensor subclasses, support custom layout overrides under torch dispatch mode (#125379)torch.utils.checkpoint: Use pytrees to improve determination of what RNG state to stash (#121462)swap_tensors path in nn.Module._apply for modules that inherit from RNNBase (RNN, GRU, LSTM) (#122800)ctc_loss zero/negative length corner cases (#123193)_LazyConvXdMixin.initialize_parameters and add related tests (#123756)load_state_dict with unexpected key whose prefix matches a valid key (#124385)requires_grad propagation in nn.utils.parametrize (#124888)nan with large bfloat16 values for FlashAttention backend of nn.functional.scaled_dot_product_attentionaffine_grid_backward when grad_grid is non-contiguous (#124370)thnn_conv2d (#121906)(#122135)svd_lowrank(..., M) in the presence of broadcasting (#122681)linalg.vector_norm when used with autocast(cuda) (#125175)coalescedCollective op Flight Recording (#120430)group_name/group_desc set up in eager initialization (#127053)_update_process_group API (#128262)TCPStoreLibUvBackend.cpp (#127230)_ProcessGroupWrapper check (#124233)USE_C10D_NCCL (#127501)wait_tensor() calls on graph inputs from getting DCEd for AsyncCollectiveTensor (#125677)_get_device_handle (#124390)init_device_mesh() consistent with directly calling DeviceMesh() (#123677)no_sync when find_unused_parameters is True (#124193)set_optimizer_state_dict() changes the parameters with some optimizers (#125708)broadcast_from_rank0 (#127635)torch.multiprocessing.ProcessContext.join() wait for all child procs to exit before return (#125969)linalg.vector_norm when reducing over a dimension of length 1 (#122143)gemv on ARM (#126297, #126745, #126746, #126877, #127033) and BF16 gemm fallback on ARM (#126592)TunableOp on ROCm (#124362)_foreach_copy now supports different source/dest dtypes on the fast path (#127186)_split_tensor (#125810) local_tensor under no_grad (#128145)monitor_interval for torchelastic default value to 0.1 sec (#124692)_find not update unchanged replacements (#124274)convert<float>(Vectorized<half>::loadu(ptr, 8)) on ARM (#125889)torch.compile issues (#122028)torch.autograd.Function: update docs for separate context and forward functions (#121955)torch.utils.checkpoint: Improve error message when use_reentrant=True is used with .grad() (#125155)torch.Tensor.backward doc (#127201)torch.autograd.Function with ctx-less forward (#122167)torch.compile links (#121823, #121824)KLDiv example (#126857)torch.nn.TripletMarginLoss allowing margin less or equal to 0 (#121978)nn.ChannelShuffle and nn.PReLU docs (#123959)nn.MaxUnpool2d example (#127850)nn.Linear docstring (#127240)NLLLoss documentation (#127346)torch.nn.utils.rnn (#123559)nn.Module.load_state_dict (#123637)torch.nn.utils.rnn.pad_sequence (#123183)is_causal explanation in the nn.functional.scaled_dot_product_attention doc (#127209)eig/eigvals (#127492)destroy_process_group usage (#122358)torch.distributed.new_subgroups docstring (#123492)distributed.new_group() (#122703)device_mesh arg (#126589)torch.compile with XPU support (#127879)torch.compile from more checks in Adamax (#123498)pg_name and pg_desc to logger (#126409)ncclunique_id (#123963)process_group_name when creating ProcessGroupNCCL (#123117)collective_seq_id and p2p_seq_id (#125727)CommDebugMode (#125475)all_reduce_coalesced tracing to CommDebugMode (#127025)ModuleList/ModuleDict (#124764)aoti_torch_item as a util function (#126352)importlib.load_module (#122542)IterationRangesEntry into triton.py (#126944)mutationlayout/aliasedlayout (#122474)DTYPE_TO_CPP mapping (#126915)opinfo tests (flattened diff) (#124657)config.trace.compile_profile (#127143)do_bench (#125736)This release is meant to fix the following issues (regressions / silent correctness):
This release is meant to fix the following issues (regressions / silent correctness):
torch.__dynamo (https://github.com/pytorch/pytorch/pull/124634)Plan failed with a cudnnException warning (https://github.com/pytorch/pytorch/pull/125790)format_utils executable, which was causing it to run as a no-op (https://github.com/pytorch/pytorch/pull/123407)device_mesh in 2.3.0 during initialization causing memory spikes (https://github.com/pytorch/pytorch/pull/124780)FSDP + DTensor with ShardingStrategy.SHARD_GRAD_OP (https://github.com/pytorch/pytorch/pull/123617)use_orig_params = False and activation checkpointing (https://github.com/pytorch/pytorch/pull/124698) (https://github.com/pytorch/pytorch/pull/126935)set_model_state_dict errors on compiled module with non-persistent buffer with distributed checkpointing (https://github.com/pytorch/pytorch/pull/125336) (https://github.com/pytorch/pytorch/pull/125337)Tensor.abs() for complex (https://github.com/pytorch/pytorch/pull/125662).pyi files (https://github.com/pytorch/pytorch/pull/124932)import torch failure when wheel is installed for a single user on Windows(https://github.com/pytorch/pytorch/pull/125684)tensor.dtype.to_complex() after ~100 calls in ipython kernel (https://github.com/pytorch/pytorch/pull/125154)Release tracker https://github.com/pytorch/pytorch/issues/125425 contains all relevant pull requests related to this release as well as links to related issues.
Fix breaking changes for ONNX Runtime Training
We are excited to announce the release of PyTorch® 2.3! PyTorch 2.3 offers support for user-defined Triton kernels in torch.compile, allowing for users to migrate their own Triton kernels from eager without experiencing performance complications or graph breaks. As well, Tensor Parallelism improves the experience for training Large Language Models using native PyTorch functions, which has been validated on training runs for 100B parameter models.
This release is composed of 3393 commits and 426 contributors since PyTorch 2.2. We want to sincerely thank our dedicated community for your contributions. As always, we encourage you to try these out and report any issues as we improve 2.3. More information about how to get started with the PyTorch 2-series can be found at our Getting Started page.
<table> <tr> <td> <strong>Stable</strong>
</td> <td> <strong>Beta</strong>
</td> <td> <strong>Prototype</strong>
</td> <td> <strong>Performance Improvements</strong>
</td> </tr> <tr> <td> </td> <td>User-defined Triton kernels in torch.compile </td> <td>torch.export adds new API to specify dynamic_shapes </td> <td>Weight-Only-Quantization introduced into Inductor CPU backend </td> </tr> <tr> <td> </td> <td>Tensor parallelism within PyTorch Distributed </td> <td>Asynchronous checkpoint generation </td> <td> </td> </tr> <tr> <td> </td> <td>Support for semi-structured sparsity </td> <td> </td> <td> </td> </tr> </table>
*To see a full list of public feature submissions click here.
This can cause significant first-time slowdown and instability when these packages are not fully compatible with PyTorch within a single process.
PyTorch support for Python 3.12 in general is considered experimental. Please use Python version between 3.8 and 3.11 instead. This is an existing issue since PyTorch 2.2.
Defining a subclass with a torch_dispatch entry will now automatically set torch_function to be disabled. This aligns better with all the use cases we’ve observed for subclasses. The main change of behavior is that the result of the torch_dispatch handler will not go through the default torch_function handler anymore, wrapping it into the current subclass. This allows in particular for your subclass to return a plain Tensor or another subclass from any op.
The original behavior can be recovered by adding the following to your Tensor subclass:
@classmethod
def __torch_function__(cls, func, types, args=(), kwargs=None):
return super().__torch_function__(func, types, args, kwargs)
no_dist and coordinator_rank from public DCP API's (#121317)As part of an overall effort to simplify our public facing API's for Distributed Checkpointing, we've decided to deprecate usage of the coordinator_rank and no_dist parameters under torch.distributed.checkpoint. In our opinion, these parameters can lead to confusion around the intended effect during API usage, and have limited value to begin with. One concrete example is here, https://github.com/pytorch/pytorch/issues/118337, where there is ambiguity in which Process Group is referenced by the coordinator rank (additional context: https://github.com/pytorch/pytorch/issues/118337). In the case of the no_dist parameter, we consider this an implementation detail which should be hidden from the user. Starting in this release, no_dist is inferred from the initialized state of the process group, assuming the intention is to use collectives if a process group is initialized, and assuming the opposite in the case it is not.
<table> <tr> <td> <strong>2.2</strong>
</td> <td> <strong>2.3</strong>
</td> </tr> <tr> <td>
# Version 2.2.2
import torch.distributed.checkpoint as dcp
dcp.save(
state_dict={"model": model.state_dict()},
checkpoint_id="path_to_model_checkpoint"
no_dist=True,
coordinator_rank=0
)
# ...
dcp.load(
state_dict={"model": model.state_dict()},
checkpoint_id="path_to_model_checkpoint"
no_dist=True,
coordinator_rank=0
)
</td> <td>
# Version 2.2.3
# no dist is assumed from pg state, and rank 0 is always coordinator.
import torch.distributed.checkpoint as dcp
dcp.save(
state_dict={"model": model.state_dict()},
checkpoint_id="path_to_model_checkpoint"
)
# ...
dcp.load(
state_dict={"model": model.state_dict()},
checkpoint_id="path_to_model_checkpoint"
)
</td> </tr> </table>
Starting from PyTorch 2.3, parallelize_module API only accepts a DeviceMesh (the tp_mesh_dim argument has been removed). If having a N-D DeviceMesh for multi-dimensional parallelism, you can use mesh_nd["tp"] to obtain a 1-D DeviceMesh for tensor parallelism.
Previously, the PT2 Export Quantization flow did not generate quantized weight by default, but instead used fp32 weight in the quantized model in this pattern: fp32 weight -> q -> dq -> linear. Setting fold_quantize=True produces a graph with quantized weights in the quantized model in this pattern by default after convert_pt2e, and users will see a reduction in the model size: int8 weight -> dq -> linear.
<table> <tr> <td> <strong>2.2</strong>
</td> <td> <strong>2.3</strong>
</td> </tr> <tr> <td>
folded_model = convert_pt2e(model, fold_quantize=True)
non_folded_model = convert_pt2e(model)
</td> <td>
folded_model = convert_pt2e(model)
non_folded_model = convert_pt2e(model, fold_quantize=False)
</td> </tr> </table>
All functions and classes under torch.jit.quantized will now raise an error if called/instantiated. This API has long been deprecated in favor of torch.ao.nn.quantized.
<table> <tr> <td> <strong>2.2</strong>
</td> <td> <strong>2.3</strong>
</td> </tr> <tr> <td>
# torch.jit.quantized APIs
torch.jit.quantized.quantize_rnn_cell_modules
torch.jit.quantized.quantize_rnn_modules
torch.jit.quantized.quantize_linear_modules
torch.jit.quantized.QuantizedLinear
torch.jit.QuantizedLinearFP16
torch.jit.quantized.QuantizedGRU
torch.jit.quantized.QuantizedGRUCell
torch.jit.quantized.QuantizedLSTM
torch.jit.quantized.QuantizedLSTMCell
</td> <td>
# Corresponding torch.ao.quantization APIs
torch.ao.nn.quantized.dynamic.RNNCell
torch.ao.quantization.quantize_dynamic APIs
torch.ao.nn.quantized.dynamic.Linear
torch.ao.nn.quantized.dynamic.GRU
torch.ao.nn.quantized.dynamic.GRUCell
torch.ao.nn.quantized.dynamic.LSTM
</td> </tr> </table>
TorchScript models that were exported with the deprecated torch.jit.quantized API will no longer be loadable, as the required internal operators have been removed. Please re-export your models using the newer torch.ao.quantization API instead.
List::get() const match List::operator[]() const (#117568)C10_IS_TRIVIALLY_COPYABLE (#120120)These decorators were previously marked for internal use only. They will be removed in version 2.4.
torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.
(Note that this was already deprecated in a previous release. In this version, we improve the deprecation message.)
This PR deprecated torch.backends.cuda.sdp_kernel, users can now use torch.nn.attention.sdpa_kernel instead. The old code will raise the following warning: FutureWarning: torch.backends.cuda.sdp_kernel() is deprecated. In the future, this context manager will be removed. Please see, torch.nn.attention.sdpa_kernel() for the new context manager, with updated signature.
<table> <tr> <td> <strong>2.2</strong>
</td> <td> <strong>2.3</strong>
</td> </tr> <tr> <td>
import torch
from torch.backends.cuda import sdp_kernel
with sdp_kernel(enable_math=False, enable_flash=False, enable_mem_efficient=True):
torch.nn.functional.scaled_dot_product_attention(...)
</td> <td>
import torch
from torch.nn.attention import sdpa_kernel, SDPBackend
with sdpa_kernel(backends=[SDPBackend.EFFICIENT_ATTENTION]):
torch.nn.functional.scaled_dot_product_attention(...)
</td> </tr> </table>
NO_SHARD (#119553)fully_shard deprecation warning (#120929)"any" mode to register_multi_grad_hook (#117984)foreach=False for clip_grad_norm_ (#120238)foreach=True for clip_grad_norm_ (#120910)permute_tensor in functional collectives (#115078)torch._dynamo.reset() to clear the cache. Now, we automatically clear the cache as soon as the nn module goes out of scope.operator.truth (#117463)torch.export._trace._export(..., predispatch=True). This returns a graph that contains ATen operators at a higher level (aten.linear.default will not become decomposed) and are also safe for training (#117278)torch.fft. support (#119670)swap_tensors path to nn.Module._apply (#117167)swap_tensors into nn.Module.load_state_dict ((#117913)assign argument to torch.Tensor.module_load (#121158)TransformerEncoder/MultiHeadAttention fastpath (#112212)add(sparse_compressed, dense) (#115432)Dynamic shapes:
TensorIteratorConfig::add_const_input to avoid COW materialize (#118053).detach in clip_grad_norm_ (#120612)no_dist if distributed is unavailable (#119813)storage() with untyped_storage() (#121538)_all_gather_keys in dcp.load (#118301)self._dim_group_infos (#118527)from_local backward partial() to replicate() pass through (#115967)to_local requires_grad warning (#118186)as_subclass is supported (#116091)torch.Tensor.new into torch.Tensor.new_empty method. (#121075)None in compiled optimizers (#121291)is_not (#113859)argument unused during compilation warning (#118077)num_children access (#116370)flatten_fn in flatten_with_keys_fn to ensure consistency (#117656)at::parallel_for/parallel_reduce (#120455), TensorAccessors with const type (#119501), and input materialization in more forward ops (#121070)MPSDataTypeComplexFloat[16|32] (#115513)sum and prod for complex types (#115554)MPSDataTypeComplexFloat[16|32] (#115513)sum and prod for complex types (#115554)torch.rand[n] for complex types (#115514)bfloat16 support on MacOS 14 (#119641)LPPool3d (#114199)nn.Module.to_empty() suggestion in the error message (#119353)nn.Module.set_extra_state return type to None (#120161)nn.Module._apply to not gate on should_use_set_data when swap_tensors is set (#120659)AdaptiveAvgPool1D to account for shmem limit for certain input sizes (#115231)avg_pool_backward (#114193)layer_norm on CPU (#99590)onnxrt dynamo backend (#120854)PT2 Export Quantization Flow:
move_exported_model_to_train (#113492)model_is_exported util function (#119726)model_is_exported input (#120720)XNNPACKQuantizer:
X86 CPU Inductor Backend:
DTypes:
Others:
MutableMapping and MutableSequence in pin_memory and collate_fn (#120553)fx_graph_cse graph pass (#119567)str on wide types (#117531)is_bf16_supported(). (#115924)get_rank under a non-default group. (#120481)device_mesh and auto wrap (#119064)no_dist=True in state_dict_loader.load (#115660)functools.reduce() function with None as initial (#116398)sum() function with start argument (#116389)__name__ on a reconstructed NestedUserFunctionVariable (#118768)model.h (#118075)RuntimeError: NYI: Named tensors are not supported with the tracer errors when using torch.jit.trace (#118393)torch.mm correctness for large matrices (#117549)use_metal_mm condition (#118830)dyspatch_sync_with_rethrow in searchsorted (#119646)conj and conj_physical (#119669)out resize logic in torch.where (#121476)AdaptiveMaxPool (#116482)TransformerEncoderLayer for bias=False (#116760)LSTM with wrong input shape (#115542)AdaptiveMaxPool1d (#118162)flash_attn_bw impl to match meta implementation when k and v have different strides (#119500)nonlinearity arg issue in RNN (#120234)requires_grad preservation for nn.Module.load_state_dict(assign=True) (#121157)PerChannelMinMaxObserver default _load_from_state_dict (#118659)Note truncated.
This release is meant to fix the following issues (regressions / silent correctness):
This release is meant to fix the following issues (regressions / silent correctness):
RuntimeError: cannot create std::vector larger than max_size() in torch.nn.functional.conv1d on non-contiguous cpu inputs by patching OneDNN (https://github.com/pytorch/builder/pull/1742) (https://github.com/pytorch/builder/pull/1744)torch.distributed.fsdp.StateDictType.FULL_STATE_DICT for when using torch.distributed.fsdp.FullyShardedDataParallel with the device_mesh argument (https://github.com/pytorch/pytorch/pull/120837)make triton command on release branch for users building the release branch from source (https://github.com/pytorch/pytorch/pull/121169)Release tracker https://github.com/pytorch/pytorch/issues/120999 contains all relevant pull requests related to this release as well as links to related issues.
This release is meant to fix the following issues (regressions / silent correctness):
This release is meant to fix the following issues (regressions / silent correctness):
_to_copy operation (https://github.com/pytorch/pytorch/pull/116426)param.grad_fn for next forward (https://github.com/pytorch/pytorch/pull/116792)AsyncCollectiveTensor in FSDP Extension (https://github.com/pytorch/pytorch/pull/116122)AssertionError on tensor subclass when setting sync_module_states=True (https://github.com/pytorch/pytorch/pull/117336)_gather_state_dict() to be synchronous with respect to the mian stream. (https://github.com/pytorch/pytorch/pull/118197) (https://github.com/pytorch/pytorch/pull/119716)torch.distributed.DistNetworkError: [WinError 32] The process cannot access the file because it is being used by another process (https://github.com/pytorch/pytorch/pull/118860)import torch on CPUs that do not support SSE4.1 (https://github.com/pytorch/pytorch/issues/116623)get_state_dict and set_state_dict (https://github.com/pytorch/pytorch/pull/119573)mixedmm on NVIDIA V100 (https://github.com/pytorch/pytorch/pull/118591)DTensor.from_local in trace_rule_look up (https://github.com/pytorch/pytorch/pull/119659)Release tracker https://github.com/pytorch/pytorch/issues/119295 contains all relevant pull requests related to this release as well as links to related issues.
This bc-breaking change fixes some unexpected behavior when set_grad_enabled is used as a decorator.
We are excited to announce the release of PyTorch® 2.2! PyTorch 2.2 offers ~2x performance improvements to scaled_dot_product_attention via FlashAttention-v2 integration, as well as AOTInductor, a new ahead-of-time compilation and deployment tool built for non-python server-side deployments.
This release also includes improved torch.compile support for Optimizers, a number of new inductor optimizations, and a new logging mechanism called TORCH_LOGS.
Please note that we are deprecating macOS x86 support, and PyTorch 2.2.x will be the last version that supports macOS x64.
Along with 2.2, we are also releasing a series of updates to the PyTorch domain libraries. More details can be found in the library updates blog.
This release is composed of 3,628 commits and 521 contributors since PyTorch 2.1. We want to sincerely thank our dedicated community for your contributions. As always, we encourage you to try these out and report any issues as we improve 2.2. More information about how to get started with the PyTorch 2-series can be found at our Getting Started page.
Summary:
scaled_dot_product_attention (SDPA) now supports FlashAttention-2, yielding around 2x speedups compared to previous versions.torch.distributed supports a new abstraction for initializing and representing ProcessGroups called device_mesh.torch.ao.quantization now offers a prototype torch.export based flow<table> <tr> <td> <strong>Stable</strong> </td> <td><strong>Beta</strong> </td> <td><strong>Prototype</strong> </td> <td><strong>Performance Improvements</strong> </td> </tr> <tr> <td> </td> <td>FlashAttentionV2 backend for scaled dot product attention </td> <td>PT 2 Quantization </td> <td>Inductor optimizations </td> </tr> <tr> <td> </td> <td> AOTInductor </td> <td> Scaled dot product attention support for jagged layout NestedTensors </td> <td>aarch64-linux optimizations (AWS Graviton) </td> </tr> <tr> <td> </td> <td> TORCH_LOGS </td> <td> </td> <td> </td> </tr> <tr> <td> </td> <td>torch.distributed.device_mesh </td> <td> </td> <td> </td> </tr> </tr> <tr> <td> </td> <td>torch.compile + Optimizers </td> <td> </td> <td> </td> </tr> </table>
*To see a full list of public 2.2 - 1.12 feature submissions click here.
We have noticed a performance regression introduced to all-reduce in NCCL 2.19.3. Please use version 2.19.1 instead.
We observe the loss will flatline randomly while training with FSDP + DTensor in some instances.
GCC 9.4 is the oldest version fully compatible with C++17, which the PyTorch codebase has migrated to from C++14.
scaled_dot_product_attention to use Flash Attention v2 (#105602)Previously, the v1 Flash Attention kernel had a Windows implementation. So if a user on Windows had explicitly forced the flash attention kernel to be run by using sdp_kernel context manager with only flash attention enabled, it would work. In 2.2, if the sdp_kernel context manager must be used, use the memory efficient or math kernel if on Windows.
with torch.backends.cuda.sdp_kernel(enable_flash=True, enable_math=False, enable_mem_efficient=False):
torch.nn.functional.scaled_dot_product_attention(q,k,v)
</td> <td>
# Don't force flash attention to be used if using sdp_kernel on Windows
with torch.backends.cuda.sdp_kernel(enable_flash=False, enable_math=True, enable_mem_efficient=True):
torch.nn.functional.scaled_dot_product_attention(q,k,v)
</td> </tr> </table>
In PyTorch 2.1 or before, users can use ParallelStyles like PairwiseParallel and specify input/output layout with functions like make_input_replicate_1d or make_output_replicate_1d. And we have default values for _prepare_input and _prepare_output. The UX of Tensor Parallel was like:
from torch.distributed.tensor.parallel.style import (
ColwiseParallel,
make_input_replicate_1d,
make_input_reshard_replicate,
make_input_shard_1d,
make_input_shard_1d_last_dim,
make_sharded_output_tensor,
make_output_replicate_1d,
make_output_reshard_tensor,
make_output_shard_1d,
make_output_tensor,
PairwiseParallel,
parallelize_module,
)
from torch.distributed.tensor import DeviceMesh
module = DummyModule()
device_mesh = DeviceMesh("cuda", list(range(self.world_size)))
parallelize_module(module, device_mesh, PairwiseParallel(_prepare_input=make_input_replicate_1d))
...
Starting from PyTorch 2.2, we simplified parallel styles to only contain ColwiseParallel and RowwiseParallel because other ParallelStyle can consist of these two. We also deleted the input/output functions, and started using input_layouts and output_layouts as kwargs instead to specify the sharding layout of both input/output tensors. Finally, added PrepareModuleInput/PrepareModuleOutput style, and no default arguments for layouts in these two styles and users need to specify them to think about the sharding layouts.
from torch.distributed.tensor.parallel.style import (
ColwiseParallel,
PrepareModuleInput,
RowwiseParallel,
parallelize_module,
)
from torch.distributed._tensor import init_device_mesh
module = SimpleMLPModule()
device_mesh = init_device_mesh("cuda", (self.world_size,)))
parallelize_module(
module,
device_mesh,
{
"fqn": PrepareModuleInput(
input_layouts=Shard(0),
desired_input_layouts=Replicate()
),
"fqn.net1": ColwiseParallel(),
"fqn.net2": RowwiseParallel(output_layouts=Shard(0)),
}
)
...
UntypedStorage.resize_ now uses the original device instead of the current device context (#113386)Before this PR, UntypedStorage.resize_ would move data to the current CUDA device index (given by torch.cuda.current_device()).
Now, UntypedStorage.resize_() keeps the data on the same device index that it was on before, regardless of the current device index.
<table> <tr> <th>2.1</th> <th>2.2</th> </tr> <tr> <td>
>>> import torch
>>> with torch.cuda.device('cuda:0'):
...: a = torch.zeros(0, device='cuda:1')
...: print(a.device)
...: a = a.untyped_storage().resize_(0)
...: print(a.device)
cuda:1
cuda:0
</td> <td>
>>> import torch
>>> with torch.cuda.device('cuda:0'):
...: a = torch.zeros(0, device='cuda:1')
...: print(a.device)
...: a = a.untyped_storage().resize_(0)
...: print(a.device)
cuda:1
cuda:1
</td> </tr> </table>
This bc-breaking change fixes some unexpected behavior when set_grad_enabled is used as a decorator.
<table> <tr> <th>2.1</th> <th>2.2</th> </tr> <tr> <td>
>>> import torch
>>> @torch.set_grad_enabled(False) # unexpectedly, this mutates the grad mode!
def inner_func(x):
return x.sin()
>>> torch.is_grad_enabled()
True
</td> <td>
>>> import torch
>>> @torch.set_grad_enabled(False) # unexpectedly, this mutates the grad mode!
def inner_func(x):
return x.sin()
>>> torch.is_grad_enabled()
False
</td> </tr> </table>
verbose parameter in LRscheduler constructors (#111302)As part of our decision to move towards a consolidated logging system, we are deprecating the verbose flag in LRScheduler.
If you would like to print the learning rate during execution, please use get_last_lr()
<table> <tr> <th>2.1</th> <th>2.2</th> </tr> <tr> <td>
optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)
scheduler = ReduceLROnPlateau(optimizer, 'min', verbose=True)
for epoch in range(10):
train(...)
val_loss = validate(...)
# Note that step should be called after validate()
scheduler.step(val_loss)
</td> <td>
optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)
scheduler = ReduceLROnPlateau(optimizer, 'min')
for epoch in range(10):
train(...)
val_loss = validate(...)
# Note that step should be called after validate()
scheduler.step(val_loss)
print(f"Epoch {epoch} has concluded with lr of {scheduler.get_last_lr()}")
</td> </tr> </table>
In PyTorch 2.1 or before, users can use our multi-gpu c10d collective APIs such as all_reduce_multigpu:
<table>
<tr>
<th>2.1</th>
<th>2.2</th>
</tr>
<tr>
<td>
import torch.distributed as dist
dist.broadcast_multigpu
dist.all_reduce_multigpu
dist.reduce_multigpu
dist.all_gather_multigpu
dist.reduce_scatter_multigpu
...
</td> <td>
optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)
scheduler = ReduceLROnPlateau(optimizer, 'min')
for epoch in range(10):
train(...)
val_loss = validate(...)
# Note that step should be called after validate()
scheduler.step(val_loss)
print(f"Epoch {epoch} has concluded with lr of {scheduler.get_last_lr()}")
</td> </tr> </table>
In PyTorch 2.2, these APIs are removed because PyTorch Distributed's preferred programming model is one device per thread, as exemplified by the APIs in its document. The multi-GPU functions (which stand for multiple GPUs per CPU thread) have been deprecated since PyTorch 1.13.
torch.onnx.ExportOutput* to ONNXProgram* (#112263)The torch.onnx.dynamo_export’s output was renamed from torch.onnx.ExportOutput to torch.onnx.ONNXProgram to better align with torch.export.export API terminology which returns a torch.export.ExportedProgram. With this change, any ambiguity that could arise with either API is eliminated.
<table> <tr> <th>2.1</th> <th>2.2</th> </tr> <tr> <td>
export_output: torch.onnx.ExportOutput = torch.onnx.dynamo(...)
</td> <td>
onnx_program: torch.onnx.ONNXProgram = torch.onnx.dynamo(...)
</td> </tr> </table>
functional::smooth_l1_loss signatures to not override beta (#109798)Previously, there were two possible options to pass in beta to smooth_l1_loss, either as a SmoothL1LossFuncOption parameter or a function parameter.
Before, the beta specified as a function parameter would override the other beta if it was set, which was unexpected behavior. Now, we throw an error when beta is passed in both cases.
use_reentrant kwarg to torch.utils.checkpoint.checkpoint_sequential explicitly (#114158)The use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.
Note that not passing use_reentrant kwarg to torch.utils.checkpoint.checkpoint has been previously deprecated in a previous release.
<table> <tr> <th>2.1</th> <th>2.2</th> </tr> <tr> <td>
a = torch.randn(3, requires_grad=True)
modules_list = [
torch.nn.Linear(3, 3),
torch.nn.Linear(3, 3),
torch.nn.Linear(3, 3)
]
# This would produce a warning in 2.2
checkpoint_sequential(modules_list, 3, a)
</td> <td>
# Recommended
checkpoint_sequential(modules_list, 3, a, use_reentrant=False)
# To preserve existing behavior
checkpoint_sequential(modules_list, 3, a, use_reentrant=True)
</td> </tr> </table>
"fallthrough" as autograd fallback default (#113166)Custom operators that do not have a kernel registered to the Autograd keys (e.g. AutogradCPU and AutogradCUDA) will now produce a warning when used with autograd.
If your custom operator previously returned floating-point or complex Tensors that do not require grad, they will now require grad as long as grad mode is enabled and the inputs require grad.
For users who would like the old behavior, register torch::CppFunction::makeFallthrough() to your Autograd key, as shown here.
The below example uses the torch library API, but if you are writing an operator in a cpp extension, please read this doc for more information.
import torch
import numpy as np
# Define the operator
torch.library.define("mylibrary::sin", "(Tensor x) -> Tensor")
# Add implementations for the cpu device
@torch.library.impl("mylibrary::sin", "cpu")
def f(x):
return torch.from_numpy(np.sin(x.detach().numpy()))
x = torch.randn(3, requires_grad=True)
y = torch.ops.mylibrary.sin(x)
y.sum().backward()
<table> <tr> <th>2.1</th> <th>2.2</th> </tr> <tr> <td>
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
</td> <td>
UserWarning: mylibrary::sin: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd.
</td> </tr> </table>
torch.cross default behavior (#108760)Calling torch.cross without specifying the dim arg is now deprecated. This behavior will be changed to match that of torch.linalg.cross in a future release.
NVFuser functionality has been removed from TorchScript (#110124, #111447, #110881)Neural Network Compiler (NNC) has replaced NVFuser as the default GPU fuser for TorchScript in PyTorch 2.1, which also added a deprecation warning for NVFuser. The TorchScript functionality for NVFuser has now been fully removed and is no longer supported.
SparseAdam constructor will no longer accept raw Tensor type for params (#114425)SparseAdam is now consistent with the rest of our optimizers and only accepts containers instead of individual Tensors/Parameters/param groups.
<table> <tr> <th>2.1</th> <th>2.2</th> </tr> <tr> <td>
import torch
param = torch.rand(16, 32)
optimizer = torch.optim.SparseAdam(param)
</td> <td>
optimizer = torch.optim.SparseAdam([param])
</td> </tr> </table>
torch.distributions.InverseGamma distribution and fix sign bug in torch.distributions.PowerTransform (#104501)src/dst information to NCCL send/recv (#111811)| support for JIT (#109293)ONNXProgram.__call__ API to run model with ONNX Runtime (#113495)torch.cpu.set_device() and torch.cpu.current_device() (#110716, #110987)aten::randlike (#108086)aten::randn_like and aten::normal (#109075)aten::bmm (#109360)aten::baddbmm (#109360)aten::floor_divide (#110785, #112190)aten::mean.dim (#111609)aten::var.dim (#111965)aten::log and aten::log_softmax (#112828)aten::layer_norm (#112322, #114701)aten::native_layer_norm (#113573)itertools.{count, repeat, cycle} (#110967)max-autotune improvements
rand[n].generator with inductor prim if generator=None (#115051)masked_scatter (#108802), _segment_reduce (#109359), foreach ops (#112281), linear_backward (#114359), scaled_mm (#112609)linalg.vecdot (#108188)aten.dot/vdot (#108194)aten.tensor_split.tensor_indices_or_sections (#107251)view_as_complex (#108005)aten.take_along_dim (#108185)scaled_dot_product_attention (#108180, #108608, #108371, #113102)unsafe_split{,_with_sizes} (#109668)_weight_norm_interface (#112193)std.correction (#108733)unsafe_split.Tensor (#108544)clamp_min (#108717)clamp_max (#108718)_unsafe_view (#108713)baddbmm (#108534)floor_divide (#110046)aten.all (#110093)split.Tensor + unbind (#110323)aten.sum + aten.squeeze (#110645)std + std_mean (#109667)trunc (#109319, #109902)glu (#110043)torch.half support for torch.multinomial on CPU (#104178)torch.float8_* dtypes (#114662)torch.addc{mul,div} (#112682)__all__ for torch.nn.utils (#111026)AdaptiveAvgPool2d and AdaptiveMaxPool2d on CPU (#102079)GroupNorm on CPU (#100234)torch.nn.functional.{softmax/log_softmax} on CPU (#103315)torch.nn.functional.grid_sample (#112331)nn.utils.parametrizations.weight_norm (#114785)NAdam support for complex dtypes, with has_complex shortcut (#110634)lr) value of SGD to 1e-3 (#114467)torch.unsafe_chunk (#110862)Tensor.index_add_ (#112276)torch.linspace and torch.logspace (#105451)torch.linalg.eigh (#110640)torch.squeeze and alias (#107577)torch.is_same_size and torch.split_with_sizes (#111491)torch.utils.checkpoint.set_checkpoint_debug_enabled (#110728)cat for bits types (e.g.torch.bits8) in cuda (#115044)copy/clone/reshape/contiguous operations for bits types (#113508)input_qspec_map for weight quantization of linear (#107105)int16 quantization (#108453)cat (#108382), conv1d (#109830) and mul (#110428) in XNNPACKQuantizercudnn_batch_norm (#109908) and miopen_batch_norm (#110653) in QAT fusionDerivedQSpec (#112159)fx.Proxy (#108764)X86InductorQuantizer (#111280)ConvBNAdd(ReLU) Annotation (#111281), adaptive_avg_pool2d and flatten (#114442), Hardtanh and ReLU6 for conv2d (#114579) in X86InductorQuantizeroneDNN QConv (#112010) and QLinear (#112126) FP32/BF16 outputoneDNN QConv2d with hardtanh post op (#114578)quantize_per_tensor/quantize_per_channel to accept bfloat16 input (#112225)allow_implicit_sharing flag (#112929)transform_for_annotation method in Quantizer (#113115)SubgraphMatcherWithNameNodeMap (#113709)XNNPACKQuantizer skip quantization for input and output to workaround histogram observer problem (#113405)XNNPACKQuantizer (#113288)FloatFunctional in torch.ao.quantization.prepare (#109646)QConfig (#109288)new_qtensor support privateuseone allocator. (#111464)float8_e4m3fnuz and float8_e5m2fnuz (#107586)all_to_all_single and support it in Inductor (#110195)ProcessGroupNCCL default timeout to 10 min (#110947)_shutdown method to ProcessGroupNCCL (#111392)DETAIL debug mode (#111878)c10d::ProcessGroup and c10d::Work's pybind (#111997)init_process_group timeout kwarg override pg_options (#112611, #113094)ProcessGroupNCCL register (#112850)FakeProcessGroup traceable (#113314)ncclCommSplit when creating new NCCL groups (#114385)_set_group_name and group_name to track pg names in C++. (#108813)randn (#108285)to_local API (#110629)aten.where and enabled implicit scalar promotion (#110584)full_tensor API to DTensor (#112224, #113322)rand_like, randn_like, randint_like ops to shard propagation (#112576)min, max and prod sharding propagation rules (#112403)_Partial, Replicate frozen dataclasses (#113919)redistribute (#113924)from_local (#114134)distribute_tensor (#113930)grad_placements was tuple (#113925)checkpoint_wrapper default to NO_REENTRANT (#108435)ModuleWrapPolicy callable (#109117)cpu_offload config for optimizer state_dict (#108434)cpu_only and ranks_only support for _gather_state_dict (#112836)cpu_offload and full_state_dict for get_state_dict (#112837)fx.split_module (#113373)torch.autograd.gradcheck when fast_mode=True and default device is set (#114560)torch.prod double backward when input tensor contains more than one zero (#113969)interpolate (#108506)num_layers for RNN, LSTM, GRU (#108853)torch::nn::AdaptiveLogSoftmaxWithLoss: check length of cutoffs (#106777)matrix_exp function (#111539)ReduceLROnPlateau now subclasses LRScheduler (#113659)adagrad sparse handling due to incorrect early exit (#110454)state_dict (#110931)torch.{finfo,iinfo} calls (#109743)torch.isfinite, torch.isreal, and torch.log_sigmoid (#110896)torch.movedim, torch.tensor_split, Tensor.to, to.* (#110999)torch.flatten, torch.linalg.*, torch.linear, torch.log_softmax, torch.logdet, torch.special.* (#110985)torch.nn.LayerNorm on CPU (#108089)collections.namedtuple are preserved instead of being changed to tuple when there are backward hooks on nn.Module (#112433)memory_efficient_attention (#116026)nn.CosineEmbeddingLoss have the same size (#112782)nn.Module.to (#108767)torch.nn.utils.rnn.pad_sequence type hint to allow sequences to be an iterable (#108765)num_batches_tracked of nn.BatchNorm{*}D in load_state_dict to not be reset to 0 if the state_dict does not contain num_batches_tracked (#110850)convert_sync_batchnorm to return SyncBatchNorm layer with same training flag as BatchNorm layer being converted (#111998)index_select.out (#111364)meta_randperm (#109721)qlinear_pointwise (#112390)threshold_backward (#110689)aten.add with int32 + scalar input (#113965)aten.add non-contiguous out handling (#111758)aten.normal with strided layout input (#111205, #112467)aten.baddbmm (#109714)index_add (#108826)aten.split_with_sizes (#113984)torch.overrides.get_overridable_functions (#109890)sparse_allreduce dispatch (#111485)cuda to MPI backend capabilities (#109614)send()/recv() to make them respect the timeout same as non-p2p collectives (#109611)NCCL_ASYNC_ERROR_HANDLING to 3:SkipCleanUp to avoid calling ncclCommAbort which in some cases hangs (#110723)torch.cpu has no attribute current_device in checkpoint/optimizer.py (#110299)DTensor.from_local() returns DTensor with wrong size for uneven sharded tensor (#110781)_shard_utils.py to explicitly use the correct device from fsdp_state (#109631)requires_grad attribute to unsharded params (#109892)compute_device (#110929)SHARD_GRAD_OP and forward_prefetch is turned on (#110139)summon_full_params(..., with_grads=True) when grad precision is not fp32 (#112746)functorch.compile.minifier error of “'Node' object is not iterable” (#103011)
__annotations__ (#109853)make_fx invocations isolated (opaque to higher make_fx invocations) by default (#93290)optimize_for_inference to support modules that don't have a forward method (#110013)AdaptiveAvgPool3d (#109822)torch.mean backward (#109935)torch.matrix_exp performance (#110848)cholesky_solve_backward using output_mask (#112981)Optimizers to speedup inductor against list comprehensions (e.g. complex conversion) (#110613, #112722)bitwise operators (#91971), nansum & nanmean (#91372), all & any (#91966), torch.linalg.vander (#91749), slogdet (#86815), torch.index_fill (#91364), narrow_copy (#88130), view_copy (#88150), greater_equal.Scaler (#91324)_store_based_barrier to be adaptative based on the number of ranks. (#109218)device_type match (#110774)DTensorSpec (#113915)DTensorSpec hash lazily (#114322, #114379)isinstance call in is_shard (#114140)optim_state_dict when use_orig_params is True (#108298)torch.utils.data (#112244, #112817, #112765)PolynomialLR (#110151)maximize option in optimizer.py (#112724)torch/cuda/ and torch/optim/ (#112964)Timer.adaptive_autorange (#111612)torch.cholesky_inverse documentation (#110364)torch.from_file and UntypedStorage.from_file properly (#111688)Tensor.share_memory_ and torch.from_file (#111856)torch.unique docs (#113424)torch.lgamma docs (#108719)torch.take_along_dim docs to include dim=None case (#109120)torch.searchsorted docs (#109364)torch.multinomial usage (#112892)torch.export APItorch.cond doc (#108691)nn.CosineEmbeddingLoss(#108215)attn_bias in code block in scaled_dot_product_attention documentation (#109086)torch.nn.utils.parametrizations.weight_norm (#113783)scaled_dot_product_attention documentation to point to flash-attn-v2 (#114124)wait_for_workers (#111807)avoidRecordStreams_ so that correct name of Environment variable is shown in the warning message (#108759)dist.send (#108273)batch_isend_irecv example incorrect usage (#110408)apply_optimizer_in_backward in the document (#110903)NCCL_SUFFIX in NCCL version log at PG init (#112560)no_dist for in load_state_dict (save -> load) (#112217)_dynamo.optimize_ddp=True and _inductor.keep_output_stride=False together inside DDP (#108235)FSDP.optim_state_dict_to_load to reflect right ctors (#108383)FSDP.set_state_dict_type to contain missing Args (#103864)rendezvous/registry.py (#111352)torch.distributed.rpc example incorrect usage (#112367)torch.distributed.run init connect timeout by comparing host with the current IP list (#90221)prepare_pt2e, prepare_qat_pt2e and convert_pt2e (#110097)This release is meant to fix the following issues (regressions / silent correctness):
This release is meant to fix the following issues (regressions / silent correctness):
torch.set_num_threads (https://github.com/pytorch/pytorch/pull/113684)The Cherry pick tracker https://github.com/pytorch/pytorch/issues/113962 contains all relevant pull requests related to this release as well as links to related issues.
This release is meant to fix the following issues (regressions / silent correctness):
This release is meant to fix the following issues (regressions / silent correctness):
torch.distributed.checkpoint.optimizer.load_sharded_optimizer_state_dict (#111687)planner argument to torch.distributed.checkpoint.optimizer.load_sharded_optimizer_state_dict (#111393)torch.distributed.FSDP (#109116)torch.nn.functional.scaled_dot_product_attention (#112673)nn.functional.scaled_dot_product_attention (#110893)torch.sparse.mm for non-contiguous inputs (#111742)The https://github.com/pytorch/pytorch/issues/110961 contains all relevant pull requests related to this release as well as links to related issues.
One column per quarter.