This release is meant to fix the following issues (regressions / silent correctness):

• RuntimeError by torch.nn.modules.activation.MultiheadAttention with bias=False and batch_first=True #88669
• Installation via pip on Amazon Linux 2, regression #88869
• Installation using poetry on Mac M1, failure #88049
• Missing masked tensor documentation #89734
• torch.jit.annotations.parse_type_line is not safe (command injection) #88868
• Use the Python frame safely in _pythonCallstack #88993
• Double-backward with full_backward_hook causes RuntimeError #88312
• Fix logical error in get_default_qat_qconfig #88876
• Fix cuda/cpu check on NoneType and unit test #88854 and #88970
• Onnx ATen Fallback for BUILD_CAFFE2=0 for ONNX-only ops #88504
• Onnx operator_export_type on the new registry #87735
• torchrun AttributeError caused by file_based_local_timer on Windows #85427

The release tracker should contain all relevant pull requests related to this release as well as links to related issues

Pytorch 1.13 Release Notes

• Highlights
• Backwards Incompatible Changes
• New Features
• Improvements
• Performance
• Documentation
• Developers

Highlights

We are excited to announce the release of PyTorch 1.13! This includes stable versions of BetterTransformer. We deprecated CUDA 10.2 and 11.3 and completed migration of CUDA 11.6 and 11.7. Beta includes improved support for Apple M1 chips and functorch, a library that offers composable vmap (vectorization) and autodiff transforms, being included in-tree with the PyTorch release. This release is composed of over 3,749 commits and 467 contributors since 1.12.1. We want to sincerely thank our dedicated community for your contributions.

Summary:

• The BetterTransformer feature set supports fastpath execution for common Transformer models during Inference out-of-the-box, without the need to modify the model. Additional improvements include accelerated add+matmul linear algebra kernels for sizes commonly used in Transformer models and Nested Tensors is now enabled by default.

• Timely deprecating older CUDA versions allows us to proceed with introducing the latest CUDA version as they are introduced by Nvidia®, and hence allows support for C++17 in PyTorch and new NVIDIA Open GPU Kernel Modules.

• Previously, functorch was released out-of-tree in a separate package. After installing PyTorch, a user will be able to import functorch and use functorch without needing to install another package.

• PyTorch is offering native builds for Apple® silicon machines that use Apple's new M1 chip as a beta feature, providing improved support across PyTorch's APIs.

You can check the blogpost that shows the new features here.

Backwards Incompatible changes

Python API

uint8 and all integer dtype masks are no longer allowed in Transformer (#87106)

Prior to 1.13, key_padding_mask could be set to uint8 or other integer dtypes in TransformerEncoder and MultiheadAttention, which might generate unexpected results. In this release, these dtypes are not allowed for the mask anymore. Please convert them to torch.bool before using.

1.12.1

>>> layer = nn.TransformerEncoderLayer(2, 4, 2)
>>> encoder = nn.TransformerEncoder(layer, 2)
>>> pad_mask = torch.tensor([[1, 1, 0, 0]], dtype=torch.uint8)
>>> inputs = torch.cat([torch.randn(1, 2, 2), torch.zeros(1, 2, 2)], dim=1)
# works before 1.13
>>> outputs = encoder(inputs, src_key_padding_mask=pad_mask)

1.13

>>> layer = nn.TransformerEncoderLayer(2, 4, 2)
>>> encoder = nn.TransformerEncoder(layer, 2)
>>> pad_mask = torch.tensor([[1, 1, 0, 0]], dtype=torch.bool)
>>> inputs = torch.cat([torch.randn(1, 2, 2), torch.zeros(1, 2, 2)], dim=1)
>>> outputs = encoder(inputs, src_key_padding_mask=pad_mask)

Updated torch.floor_divide to perform floor division (#78411)

Prior to 1.13, torch.floor_divide erroneously performed truncation division (i.e. truncated the quotients). In this release, it has been fixed to perform floor division. To replicate the old behavior, use torch.div with rounding_mode='trunc'.

1.12.1

>>> a = torch.tensor([4.0, -3.0])
>>> b = torch.tensor([2.0, 2.0])
>>> torch.floor_divide(a, b)
tensor([ 2., -1.])

1.13

>>> a = torch.tensor([4.0, -3.0])
>>> b = torch.tensor([2.0, 2.0])
>>> torch.floor_divide(a, b)
tensor([ 2., -2.])
# Old behavior can be replicated using torch.div with rounding_mode='trunc'
>>> torch.div(a, b, rounding_mode='trunc')
tensor([ 2., -1.])

Fixed torch.index_select on CPU to error that index is out of bounds when the source tensor is empty (#77881)

Prior to 1.13, torch.index_select would return an appropriately sized tensor filled with random values on CPU if the source tensor was empty. In this release, we have fixed this bug so that it errors out. A consequence of this is that torch.nn.Embedding which utilizes index_select will error out rather than returning an empty tensor when embedding_dim=0 and input contains indices which are out of bounds. The old behavior cannot be reproduced with torch.nn.Embedding, however since an Embedding layer with embedding_dim=0 is a corner case this behavior is unlikely to be relied upon.

1.12.1

>>> t = torch.tensor([4], dtype=torch.long)
>>> embedding = torch.nn.Embedding(3, 0)
>>> embedding(t)
tensor([], size=(1, 0), grad_fn=<EmbeddingBackward0>)

1.13

>>> t = torch.tensor([4], dtype=torch.long)
>>> embedding = torch.nn.Embedding(3, 0)
>>> embedding(t)
RuntimeError: INDICES element is out of DATA bounds, id=4 axis_dim=3

Disallow overflows when tensors are constructed from scalars (#82329)

Prior to this PR, overflows during tensor construction from scalars would not throw an error. In 1.13, such cases will error.

1.12.1

>>> torch.tensor(1000, dtype=torch.int8)
tensor(-24, dtype=torch.int8)

1.13

>>> torch.tensor(1000, dtype=torch.int8)
RuntimeError: value cannnot be converted to type int8 without overflow

Error on indexing a cpu tensor with non-cpu indices (#69607)

Prior to 1.13, cpu_tensor[cuda_indices] was a valid program that would return a cpu tensor. The original use case for mixed device indexing was for non_cpu_tensor[cpu_indices], and allowing the opposite was unintentional (cpu_tensor[non_cpu_indices]). This behavior appears to be rarely used, and a refactor of our indexing kernels made it difficult to represent an op that takes in (cpu_tensor, non_cpu_tensor) and returns another cpu_tensor, so it is now an error.

To replicate the old behavior for base[indices], you can ensure that either indices lives on the CPU device, or base and indices both live on the same device.

1.12.1

>>> a = torch.tensor([1.0, 2.0, 3.0])
>>> b = torch.tensor([0, 2], device='cuda')
>>> a[b]
tensor([1., 3.])

1.13

>>> a = torch.tensor([1.0, 2.0, 3.0])
>>> b = torch.tensor([0, 2], device='cuda')
>>> a[b]
RuntimeError: indices should be either on cpu or on the same device as the indexed tensor (cpu)
# Old behavior can be replicated by moving b to CPU, or a to CUDA
>>> a[b.cpu()]
tensor([1., 3.])
>>> a.cuda()[b]
tensor([1., 3.], device='cuda:0')

Remove deprecated torch.eig, torch.matrix_rank, torch.lstsq (#70982, #70981, #70980)

The deprecation cycle for the above functions has been completed and they have been removed in the 1.13 release.

torch.nn

Enforce that the bias has the same dtype as input and weight for convolutions on CPU (#83686)

To align with the implementation on other devices, the CPU implementation for convolutions was updated to enforce that the dtype of the bias matches the dtype of the input and weight.

1.12.1

# input and weight are dtype torch.int64
# bias is torch.float32
>>> out = torch.nn.functional.conv2d(input, weight, bias, ...)

1.13

# input and weight are dtype torch.int64
# bias is torch.float32
>>> with assertRaisesError():
>>>    out = torch.nn.functional.conv2d(input, weight, bias, ...)

# Updated code to avoid the error
>>> out = torch.nn.functional.conv2d(input, weight, bias.to(input.dtype), ...)

Autograd

Disallow setting the .data of a tensor that requires_grad=True with an integer tensor (#78436)

Setting the .data of a tensor that requires_grad with an integer tensor now raises an error.

1.12.1

>>> x = torch.randn(2, requires_grad=True)
>>> x.data = torch.randint(1, (2,))
>>> x
tensor([0, 0], requires_grad=True)

1.13

>>> x = torch.randn(2, requires_grad=True)
>>> x.data = torch.randint(1, (2,))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: data set to a tensor that requires gradients must be floating point or complex dtype

Added variable_list support to ExtractVariables struct (#84583)

Prior to this change, C++ custom autograd Function considers tensors passed in TensorList to not be tensors for the purposes of recording the backward graph. After this change, custom Functions that receive TensorList must modify their backward functions to also compute gradients for these additional tensor inputs. Note that this behavior now differs from that of custom autograd Functions in Python.

1.12.1

struct MyFunction : public Function<MyFunction> {
    static Variable forward(AutogradContext* ctx, at::Tensor t, at::TensorList tensors) {
      return 2 * tensors[0] + 3 * t;
    }

    static variable_list backward(
        AutogradContext* ctx,
        variable_list grad_output) {
      return {3 * grad_output[0]};
    }
};

1.13

struct MyFunction : public Function<MyFunction> {
    static Variable forward(AutogradContext* ctx, at::Tensor t, at::TensorList tensors) {
      return 2 * tensors[0] + 3 * t;
    }

    static variable_list backward(
        AutogradContext* ctx,
        variable_list grad_output) {
      return {3 * grad_output[0], 2 * grad_output[0]};
    }
};

Don't detach when making views; force kernel to detach (#84893)

View operations registered as CompositeExplicitAutograd kernels are no longer allowed to return input tensors as-is. You must explicitly create a new tensor (e.g., using .alias()).

1.12.1

torch::Tensor view_op(const torch::Tensor& self) {
  return self;
}

1.13

torch::Tensor view_op(const torch::Tensor& self) {
  return self.alias();
}

ONNX

torch.onnx.register_custom_op_symbolic now only registers the symbolic function at the specified opset version (#85636)

This updates register_custom_op_symbolic's behavior to only register the symbolic function at a single version. This is more aligned with the semantics of the API signature. Previously the API registers a symbolic function to all versions up to the specified version. As a result of this change, users will need to register a symbolic function to the exact version when they want to override an existing symbolic function. Users are not affected if (1) an implementation does not exist for the op, or (2) the symbolic function is already registering to the exact version for export.

1.12.1

# Assuming an implemented symbolic function `custom_op_function`
torch.onnx.register_custom_op_symbolic("aten::foo", custom_op_function, 16)

1.13

# Assuming an implemented symbolic function `custom_op_function`
for opset in range(1, 17):
    torch.onnx.register_custom_op_symbolic("aten::foo", custom_op_function, opset)

Default ONNX opset is updated to 14 (#83284)

The update is done in regularly to ensure we are in sync with the onnx updates. Users can specify opset_version in torch.onnx.export to maintain opset version 13.

torch.onnx.symbolic_registry is removed (#84382)

We removed the symbolic_registry module and hid it as an internal implementation detail. Users previously relying on the register_op function to register custom symbolic functions should move to use the torch.onnx.register_custom_op_symbolic API.

ScalarType and global variables in torch.onnx.symbolic_helper are removed (#82995)

The ScalarType class in torch.onnx.symbolic_helper, along with the global variables cast_pytorch_to_onnx, pytorch_name_to_type, scalar_name_to_pytorch, scalar_type_to_onnx and scalar_type_to_pytorch_type are removed from the module. Users previously using these global variables for PyTorch JIT-ONNX type conversion in symbolic functions should move to use the torch.onnx.JitScalarType class.

1.12.1

# 1
torch.onnx.symbolic_helper.scalar_type_to_onnx[
    symbolic_helper.scalar_type_to_pytorch_type.index(x.dtype)
].value

# 2
torch.onnx.symbolic_helper.scalar_name_to_pytorch[element_type] in cast_pytorch_to_onnx.keys()

# 3
torch.onnx.symbolic_helper.cast_pytorch_to_onnx["Long"]

# 4
torch.onnx.symbolic_helper.cast_pytorch_to_onnx[tensor.type().scalarType()]

1.13

# 1
torch.onnx.JitScalarType.from_dtype(x.dtype).onnx_type()

# 2
torch.onnx.JitScalarType.from_name(element_type).onnx_compatible()

# 3
torch.onnx.TensorProtoDataType.INT64

# 4
torch.onnx.JitScalarType.from_name(tensor.type().scalarType()).onnx_type()

Distributed

In c10d collectives, input tensors dtype must now be the same (#84664)

We added a check to validate all dtype across all input tensors. Previously, users were allowed to pass in tensors with diferent dtypes for c10d collectives. Now, passing in tensors with different dtypes will throw a RuntimeError with the following message: “Invalid usage of tensors with different dtypes Found torch.float and torch.half”. Users can use tensor.to(dtype={some_dtype}) to fix this.

1.12.1

# users could pass inputs having different dtypes
>>> tensor = torch.ones(2, 2) * 7
>>> tensor_h = tensor.half()
>>> tensor_list = [torch.zeros(2, 2) for _ in range(4)] # Assume world_size = 4
# Both cases work.
>>> dist.all_gather(tensor_list, tensor)
>>> dist.all_gather(tensor_list, tensor_h)
...

1.13

# all inputs of c10d collectives need to have the same dtype
>>> tensor = torch.ones(2, 2) * 7
>>> tensor_h = tensor.half()
>>> tensor_list = [torch.zeros(2, 2) for _ in range(4)] # Assume world_size = 4
# Only allow same dtype for all input tensors.
>>> dist.all_gather(tensor_list, tensor) # RuntimeError thrown
...

Users doing wildcard imports of torch.distributed.distributed_c10d will no longer get non-public symbols (#84872)

We limit the usage of c10d APIs to public APIs, so if a user does a wildcard import and calls an internal API, it will fail. Please see the example below:

1.12.1

# users could import both public and non-public symbols:
from torch.distributed.distributed_c10d import *
>>> is_nccl_available() # public API
>>> _check_single_tensor(...) # Non-public API
...

1.13

# users can only import public symbols
from torch.distributed.distributed_c10d import *
is_nccl_available() # public API
_check_single_tensor(...) # Non-public API, this will fail now
...

Process Group C++ extensions must use absolute path when importing ProcessGroup.hpp (#86257), ProcessGroup::Work object moved out of work to its own Work class (#83680):

Details of the changes and the updated tutorial can be found in the PyTorch tutorial PR #2099

1.12.1

// users use relative path to import C++ headers and Work resides in ProcessGroup class
#include <c10d/ProcessGroup.hpp>
#include <c10d/Store.hpp>
#include <c10d/Types.hpp>
#include <c10d/Utils.hpp>
...
class WorkDummy : public ProcessGroup::Work {
    ...
}

1.13

// users must use absolute path of import C++ files and Work is its own class
#include <torch/csrc/distributed/c10d/ProcessGroup.hpp>
#include <torch/csrc/distributed/c10d/Store.hpp>
#include <torch/csrc/distributed/c10d/Types.hpp>
#include <torch/csrc/distributed/c10d/Utils.hpp>
...
#include <torch/csrc/distributed/c10d/Work.hpp>
class WorkDummy : public Work {
    ...
}

Quantization

Add required example_args argument to prepare_fx and prepare_qat_fx (#249) (#77608)

We added an additional required example_inputs argument to prepare_fx and prepare_qat_fx APIs, this can be used to do type inference to figure out the type information for each of the fx Node in the graph.

1.12.1

m = resnet18(...)
m = prepare_fx(m, qconfig_dict)
# or
m = prepare_qat_fx(m, qconfig_dict)

1.13

m = resnet18(...)
m = prepare_fx(m, qconfig_dict, example_inputs=(torch.randn(1, 3, 224, 224),))
# or
m = prepare_qat_fx(m, qconfig_dict, example_inputs=(torch.randn(1, 3, 224, 224),))

Stop moving models to CPU in quantization convert (#80555)

Previously, we automatically moved the model to CPU in torch.ao.quantization.fx.convert to work around the issue where certain functions called by convert expect CPU arguments. This commit pushes this responsibility to the caller since it is the user's decision of which device to use.

1.12.1

model = resnet18(...)
model = prepare_fx(model, qconfig_mapping, example_inputs)
# calibrate
model = convert_fx(model)

1.13

model = resnet18(...)
model.cpu()  # if needed
model = prepare_fx(model, qconfig_mapping, example_inputs)
# calibrate
model = convert_fx(model)

Replace the is_reference flag of the torch.ao.quantize_fx.convert_fx function with the convert_to_reference function (#80091, #81326)

This PR removes the is_reference flag from the existing convert_fx API and replaces it with a new convert_to_reference function. This separates (1) converting the prepared model to a reference model from (2) lowering the reference model to a quantized model, enabling users to call their custom lowering function for custom backends.

1.12.1

from torch.ao.quantization.quantize_fx import (
    prepare_fx,
    convert_to_reference,
)

prepared = prepare_fx(model, ...)
reference = convert_to_reference(prepared, ...)

1.13

from torch.ao.quantization.quantize_fx import (
    prepare_fx,
    convert_to_reference_fx,
)

prepared = prepare_fx(model, ...)
reference = convert_to_reference_fx(prepared, ...)

Add default configs for fixed qparams ops (#80184)

This commit adds qconfigs with special observers for fixed qparams ops (operators whose corresponding quantized version has fixed quantized parameters for output) like sigmoid in get_default_qconfig_mapping and get_default_qat_qconfig_mapping. For correctness, we also require users to use these special observers if we detect these fixed qparams ops in prepare.

1.12.1 (fails after this PR):

from torch.ao.quantization.quantize_fx import prepare_fx

model = ModelWithFixedQParamsOps()
qconfig_mapping = QConfigMapping()
example_inputs = ...
prepare_fx(model, qconfig_mapping, example_inputs)

1.13

from torch.ao.quantization import get_default_qconfig_mapping
from torch.ao.quantization.quantize_fx import prepare_fx

model = ModelWithFixedQParamsOps()
qconfig_mapping = get_default_qconfig_mapping()
example_inputs = ...
prepare_fx(model, qconfig_mapping, example_inputs)

Replace qconfig_dict with a typed QConfigMapping object (#78452, #79618)

Previously, FX graph mode quantization configurations were specified through a dictionary of qconfigs. However, this API was not in line with other core APIs in PyTorch. This commit replaces this dictionary with a config object that users will create and pass to prepare and convert. This leads to better type safety and better user experience in notebook settings due to improved auto completion.

1.12.1 (deprecated)

from torch.ao.quantization.quantize_fx import prepare_fx

qconfig_dict = {
    "": qconfig,
    "object_type": [
        (torch.nn.Linear, qconfig),
    ],
    "module_name_regex": [
        ("foo.*bar", qconfig),
    ],
    "module_name": [
        ("mod", qconfig),
    ],
}

prepare_fx(model, qconfig_dict)

1.13

from torch.ao.quantization import QConfigMapping
from torch.ao.quantization.quantize_fx import prepare_fx

qconfig_mapping = QConfigMapping()
    .set_global(qconfig)
    .set_object_type(torch.nn.Linear, qconfig)
    .set_module_name_regex("foo.*bar", qconfig)
    .set_module_name("mod", qconfig)

prepare_fx(model, qconfig_mapping)

Replace *custom_config_dict with typed config objects (#79066)

This commit replaces the following config dicts with python objects:

• prepare_custom_config_dict → PrepareCustomConfig
• convert_custom_config_dict → ConvertCustomConfig
• fuse_custom_config_dict → FuseCustomConfig

This leads to better type safety and better user experience in notebook settings due to improved auto completion. 1.12.1

from torch.ao.quantization.quantize_fx import prepare_fx, convert_fx

prepare_custom_config_dict = {
  "float_to_observed_custom_module_class": {
     "static": {
         FloatClass: ObservedClass
     }
  },
  "non_traceable_module_name": ["mod1", "mod2"],
  "non_traceable_module_class": [class1, class2],
  "input_quantized_idxs": [0, 1],
  "output_quantized_idxs": [0],
  "preserved_attributes": ["attr1", "attr2"],
}

convert_custom_config_dict = {
  "observed_to_quantized_custom_module_class": {
     "static": {
         FloatClass: ObservedClass
     }
  },
  "preserved_attributes": ["attr1", "attr2"],
}

model = prepare_fx(
    model,
    qconfig_mapping,
    example_inputs,
    prepare_custom_config_dict=prepare_custom_config_dict)

model(data)

model = convert_fx(model, convert_custom_config_dict=convert_custom_config_dict)

1.13

from torch.ao.quantization.fx.custom_config import (
    PrepareCustomConfig,
    ConvertCustomConfig,
)
from torch.ao.quantization.quantize_fx import prepare_fx, convert_fx

prepare_custom_config = PrepareCustomConfig() \
    .set_float_to_observed_mapping(float_class, observed_class) \
    .set_non_traceable_module_names(["mod1", "mod2"]) \
    .set_non_traceable_module_classes([class1, class2]) \
    .set_input_quantized_indexes([0, 1]) \
    .set_output_quantized_indexes([0]) \
    .set_preserved_attributes(["attr1", "attr2"])

convert_custom_config = ConvertCustomConfig() \
    .set_observed_to_quantized_mapping(observed_class, quantized_class) \
    .set_preserved_attributes(["attr1", "attr2"])

model = prepare_fx(
    model,
    qconfig_mapping,
    example_inputs,
    prepare_custom_config=prepare_custom_config)

model(data)

model = convert_fx(model, convert_custom_config=convert_custom_config)

Remove remove_quant_dequant_pairs and fix tests (#84203)

This PR removed some passes in convert_fx, and also fixes the way we quantize layer_norm operator, so the qconfig for layer_norm op needs to be updated as well.

1.12.1

import torch
from torch.ao.quantization.qconfig_mapping import QConfigMapping, QConfig
from torch.ao.quantization.observer import default_weight_observer
from torch.ao.quantization.backend_config import (
    DTypeConfig,
    ObservationType,
)
from torch.ao.quantization.quantize_fx import prepare_fx, convert_fx

qconfig = QConfig(activation=qconfig.activation, weight=default_weight_observer)
qconfig_mapping = QConfigMapping().set_object_type(torch.nn.LayerNorm, q_config) \
.set_object_type(torch.nn.functional.layer_norm, q_config)

# assuming mymodel contains a LayerNorm layer or torch.nn.functional.layer_norm
m = MyModel()
example_inputs = (torch.rand(3, 3),)
m = prepare_fx(m, qconfig_mapping, example_inputs)

1.13

import torch
from torch.ao.quantization.qconfig_mapping import QConfigMapping, QConfig
from torch.ao.quantization.observer import default_placeholder_observer
from torch.ao.quantization.backend_config import (
    DTypeConfig,
    ObservationType,
)
from torch.ao.quantization.quantize_fx import prepare_fx, convert_fx

qconfig = QConfig(activation=qconfig.activation, weight=default_placeholder_observer)
qconfig_mapping = QConfigMapping().set_object_type(torch.nn.LayerNorm, q_config) \
.set_object_type(torch.nn.functional.layer_norm, q_config)

# assuming mymodel contains a LayerNorm layer or torch.nn.functional.layer_norm
m = MyModel()
example_inputs = (torch.rand(3, 3),)
m = prepare_fx(m, qconfig_mapping, example_inputs)

Align observer dtype with reference model spec (#85345)

Before this PR, the dtype attribute of observers was not clearly defined. It originally meant interface_dtype in the eager mode workflow, which is how the codebase before this PR is using it. In the new reference model spec, dtype attribute of an observer represents the dtype value which needs to be passed into a quantize function in the reference model spec. This PR aligns the codebase to this definition of dtype.

1.12.1

dynamic_quant_observer = PlaceholderObserver.with_args(
    dtype=torch.float, compute_dtype=torch.quint8)

1.13

dynamic_quant_observer = PlaceholderObserver.with_args(
    dtype=torch.quint8, compute_dtype=torch.quint8)

Composability

Changed the backend C++ kernel representation for some operators that take in lists of tensors (#73350)

If an operator in ATen takes in a list of tensors, and is marked as “structured” in native_functions.yaml (example), then previously, TensorList was represented as at::TensorList, or c10::ArrayRef<at::Tensor>. Now, it is represented as a more efficient type: const ITensorListRef&.

1.12.1

at::Tensor cat_kernel(at::TensorList tensors,int64_t dim) {
    ...
}
TORCH_LIBRARY_IMPL(aten, dispatch_key, m) {
    ...
    m.impl("cat", &cat_kernel);
}

1.13

at::Tensor cat_kernel(const at::ITensorListRef& tensors,int64_t dim) {
    ...
}
TORCH_LIBRARY_IMPL(aten, dispatch_key, m) {
    ...
    m.impl("cat", &cat_kernel);
}

C++ API

Lowered randint default dtype to the C++ API (#81410)

Prior to 1.13, the default for the dtype argument of torch.randint, torch.long, was set via manual python binding. However, in the C++ API, torch::randint would default to the global default data type, which is usually float. In 1.13 we changed the default for dtype in the C++ API to int64 in order to match the python API. To reproduce the old behavior, one can set the dtype argument.

1.12.1

torch::randint(/*low=*/0, /*high=*/10, {2, 3});

1.13

// assuming default dtype is float
torch::randint(/*low=*/0, /*high=*/10, {2, 3}, torch::kFloat);

Enabled dim=None for torch.{std, var, std_mean, var_mean} (#81845, #82765, #82912)

Prior to 1.13, a C++ API call that has argument types torch::{std, var, std_mean, var_mean}(Tensor, OptionalIntArrayRef, int64_t, bool) used to resolve to the {std, var, std_mean, var_mean}.correction overload. In this release, it resolves to the {std, var, std_mean, var_mean}.dim overload. With the .correction overload, the third argument of type int64_t could be used to pass a correction δN other than 1. In order to call the {std, var, std_mean, var_mean}.correction overload in 1.13, the old int64_t argument can be wrapped in a c10::optional.

1.12.1

// using std as an example
int64_t correction = 2;
torch::std(t, /*dim=*/dim, /*correction=*/correction, /*keepdim=*/True);

1.13

// To replicate in 1.13 using std as an example
auto correction = c10::make_optional<int64_t>(2);
torch::std(t, /*dim=*/dim, /*correction=*/correction, /*keepdim=*/True);

Deprecations

Distributed

We are deprecating the following APIs of c10d: *_coalesced APIs (#85959), *_multigpu APIs (#85961) and ProcessGroupRoundRobin (#85158)

We added warnings when users call c10d’s *_coalesced, *_multigpu and ProcessGroupRoundRobin APIs. Previously, users can use these APIs without any warnings but now they will see warnings like “torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions”. There are still workarounds for *_coalesced APIs but no workarounds will be provided for the other two.

1.12.1

# users could use the following APIs with no warnings:
all_reduce_coalesced(...)
all_gather_coalesced(...)
broadcast_multigpu(...)
all_reduce_multigpu(...)
reduce_multigpu(...)
all_gather_multigpu(...)
reduce_scatter_multigpu(...)
...

1.13

# users can still use these APIs but it will come with warnings:
all_reduce_coalesced(...)
# Warnings:
# torch.distributed.all_reduce_coalesced will be deprecated. If you must
# use it, please revisit our documentation later at
# https://pytorch.org/docs/master/distributed.html#collective-functions"

# Potential workaround:
reqs = []
with dist._coalescing_manager(group, reqs):
    reqs.append(dist.all_reduce(tensor1, async_op=True))
    reqs.append(dist.all_reduce(tensor2, async_op=True))
for req in reqs:
    req.wait()
...

We are deprecating passing optim_input into the FSDP optimizer state checkpointing APIs. The user can simply not pass the optim_input argument, and all behavior is preserved. No fix is needed from users side for now.

1.12.1

# the user can use the following APIs with no warnings
full_optim_state_dict(...)
sharded_optim_state_dict(...)
shard_full_optim_state_dict(...)
flatten_sharded_optim_state_dict(...)
scatter_full_optim_state_dict(...)
rekey_optim_state_dict(...)

1.13

# users can still use these APIs, but they will come with warnings
# The `optim_input` argument is deprecated and will be removed after PyTorch 1.13.
# You may remove it from your code without changing its functionality.

LinAlg

Deprecate torch.lu in favor of linalg.lu_factor (#77636)

The new operation has a cleaner API and better docs. The update rule is as follows:

1.12.1

LU2, pivots2, info = torch.lu(A, compute_pivots, get_infos=True)
LU1, pivots1, info = torch.lu(A, compute_pivots)

1.13

LU2, pivots2, info = torch.linalg.lu_factor_ex(A, compute_pivots)
LU1, pivots1 = torch.linalg.lu_factor(A, compute_pivots)

Deprecate torch.lu_solve in favor of linalg.lu_solve(#77637)

The new operation has a notation consistent with linalg.solve, and has an extra parameter adjoint=False. The update rule is as follows:

1.12.1

X = torch.lu_solve(B, LU, pivots)

1.13

X = linalg.lu_solve(LU, pivots, B)

ONNX

Monkey patched convenience method on torch._C.Graph, torch._C.Block and torch._C.Node are deprecated. (#83006)

Deprecated methods include Graph.op(), Graph.constant(), Graph.at(), Block.op(), and Node.__getitem__(). Previously, these methods are patched into the classes above when users call torch.onnx.export() and are typically used in custom symbolic functions. Users can continue to expect g.op() and g.at() in symbolic functions to work. The g parameter has been substituted by the GraphContext object (#84728). The methods are now exposed by the GraphContext class with APIs unchanged. Users should not rely on the Graph.op(), Graph.constant(), Graph.at(), Block.op(), Node.__getitem__() methods when they are directly interacting with the C classes. Users should use only the op() and at() methods of the GraphContext object, as other fields in the class will change in future releases.

New features

Python API

• Added a deterministic implementation of scatter_add on CUDA for all input sizes (#79466)
• Added torch.concatenate that aliases torch.cat (#85073)
• Added Tensor.is_cpu() that returns whether a tensor is on CPU (#78887)
• Added a force kwarg to Tensor.numpy() that enables returning a numpy ndarray that does not share storage with the tensor (#78564)
• Added torch.special.{airy_ai, bessel_j0, bessel_j1, bessel_y0, bessel_y1, modified_bessel_i0, modified_bessel_i1, modified_bessel_k0, modified_bessel_k1, scaled_modified_bessel_k0, scaled_modified_bessel_k1, spherical_bessel_j0} (#78900), (#78901), (#78902), (#78912), (#78451)
• Added torch.special.{chebyshev_polynomial_t, chebyshev_polynomial_u, chebyshev_polynomial_v, chebyshev_polynomial_w, hermite_polynomial_h, hermite_polynomial_he, laguerre_polynomial_l, legendre_polynomial_p, shifted_chebyshev_polynomial_t, shifted_chebyshev_polynomial_u, shifted_chebyshev_polynomial_v, shifted_chebyshev_polynomial_w} (#78196), (#78293), (#78304), (#78366), (#78352), (#78357)
• Added weights_only option to torch.load that restricts load to state_dict only, enabling safe loading. This can also be set using the TORCH_FORCE_WEIGHTS_ONLY_LOAD environment variable (#86812)

Build

• Added -Werror=unused-but-set-variable build flag (#79305)
• Added ability to get release versions based on the current tag (#78584)
• Added -Werror=type-limits in Bazel CPU build (#79139)
• Added -Werror=unused-variable in Bazel CPU build (#79156)
• Added —config=shell to bazelrc file for easier debugging (#79350)
• Added clang -Wconstant-conversion to catch errors detected in #75400 (#80461)
• Added -Werror=non-virtual-dtor build flag (#81012)
• Turned on pocketfft flag for third-party pocket_fft library (#81670)
• Updated NCCL to v2.13.4-1 (#82775)
• Added -Wunused-local-typedef build flag (#86154)
• Increased max python version to include 3.10 (#84815)

Complex

• Added complex half support for:
  • [CPU] torch.{index_select, index_add} (#79217), (#79897).
  • [CUDA] torch.roll (#79970), torch.fft.{fftshift, ifftshift} (#79970), torch.{acos, acosh, asinh, atanh}, (#80030), torch.{cos, sinh, cosh, tanh} (#78718), torch.sqrt, rsqrt (#77490), torch.{triu, tril, diag, trace}(#78062).
  • [CPU and CUDA] torch.where (#78665), torch.{where, pow, masked_fill, sgn, tan, angle}(#78665)
• Added complex support for torch.nn.ConvTranspose1d (#79694).

torch.nn

• Added pop function to nn.Sequential and nn.ModuleList (#81601)
• Added deepcopy support for parametrized nn.Module (#80811)

torch.optim

• Added maximization support via the maximize kwarg for optim.SparseAdam (#80336), optim.ASGD (#81875), optim.Rprop (#81864), optim.RMSprop (#80326)
• Added support for differentiable optimizers via the differentiable kwarg optim.SGD (#80938), optim.Adam (#82205), optim.RMSprop (#83578)
• Added support for complex number for optim.Adam (#80279), optim.AdamW (#80280), optim.Adamax (#80319), optim.RMSprop (#83860), optim.Rprop (#83858),
• Handled complex params as independent real params in optim.{RMSprop, ASGD} (#83860), (#84472)
• Added optim.lr_scheduler.PolynomialLR (#82769)

BetterTransformer

• Allowed user to assert no mask contiguous check is necessary (#82533)
• Added support for norm_first in nn.TransformerEncoderLayer fast path (#78269)
• Added ustom scaled dot product implementations dense (#85984)
• Added Better Transformer fastpath diagnostics (#81013)

ForEach

• Implemented inplace foreach maximum and minimum (#82523)

LinAlg

• Added linalg.lu_solve, linalg.solve_ex, linalg.vecdot, linalg.vander (#77634, #80073, #70542, #76303)

Sparse

• Added torch.sparse.spdiags for easier creation of diagonal sparse matrices (#78439)

torch.fx

• Enabled symbolic shapes (#82063, #82317, #82209, #83380, #85808, #84113, #84829, #84918, #85185, #85261, #85260, #85754, #85768, #86050, #86098, #86067)
• Created an improved version of subgraph matcher (#82090, #82853, #85444, #85456, #85617)
• Rewrite subgraph_rewriter with subgraph_matcher (#83717)
• Added PassBase for writing passes, PassResult for the return value of passes, and a PassManager for managing the workflow of passes (#79878, #81366, #80531, #82485, #83933, #84094, #84425, #84232)
• Added an FX graph partitioner and fuser (#79439, #80292)
• Added a reinplacing FX pass (#80897, #83626, #83845, #83846)
• Added a CSE pass to the common passes (#81512, #81530, #81742)
• Created DecompositionInterpreter for decomposing aten → prims after an initial make_fx call (#79989)
• Created a Backend for NvFuser based graph partitioner + Prims (#80591, #81311, #81436, #81911)
• Created a Backend for Cudagraphs from dynamo (#80566)
• Created a type constraint generator to Z3 (#79912, #80084, #80095, #80102, #80110, #80147, #80744, #80799, #80823, #80847, #80909, #80925, #80976, #81159, #81175, #81189, #81190, #81265, #81274, #81344, #81360, #81376, #81445, #81516, #81527, #81714, #82163, #82590, #82597, #82614, #82742, #82856, #82923,#82938,#83087, #83109, #83194, #83334, #83682, #83945)

JIT

• Added new NVFuser Python Frontend Record Keeping for Cache enablement. (#81578)
• Added torch.ops.nvprims namespace for nvFuser-specific prims (#82155)
• Enabled fusion of conv with elementwise OP in NNC (#77157)
• Added symbolic shape functions for conv_transpose2d.input, convolution, convolution_backward (#77283, #83557, #80860)
• Added support in symbolic shapes for generalized lists of tensor shapes, tuple outputs, optional None, upper and lower bounds (#77389, #83092, #83222, #78679)
• Added support for aten::_convolution when it is 2D conv in NNC (#84038)
• Exposed ProcessGroup::Work.wait() API to TorchScript (#83303)

ONNX

• Inlined prim::PythonOp for Autograd Function Export (#74765)

AMD

• Enabled nvfuser (#82498)

CUDA

• Added CUDA trace Python hooks (#82824)
• Added CUDA Sanitizer (#83984)
• Added support for multiple outputs in python jiterator (#77921, #78139)

Intel

• Added a launch script with Best Recipe of Deep Learning on Intel Xeon CPU (#63932)
• Enabled Intel® VTune™ Profiler's Instrumentation and Tracing Technology APIs (ITT) to PyTorch (#63289)
• Added unified x86 quantization backend (#84329)

MPS

• Added aten::index_add.out operator for MPS backend (#79935)
• Added aten::prelu operator for MPS backend (#82401)
• Added aten::bitwise-not operator native support for MPS backend (#83678)
• Added aten::tensor::index_put operator for MPS backend (#85672)
• Added aten::upsample_nearest1d operator for MPS backend (#81303)
• Added aten::bitwise_{and|or|xor} operators for MPS backend (#82307)
• Added aten::index.Tensor_out operator for MPS backend (#82507)
• Added aten::masked_select operator for MPS backend (#85818)
• Added aten::multinomial operator for MPS backend (#80760)

Profiler

• Integrated Execution Graph Observer into PyTorch Profiler (#75358, #79753, #82895, #84285)
• TorchTidy: experimental tool to identify anti-patterns from traces (#79631, #79874, #79993, #80094, #80108, #80572, #81056, #81273, #81501, #81733, #81740, #81921, #82421, #82248, #82261, #82782)
• Added reporting for OOM events to the Pytorch Profiler. (#80050)

Vulkan

• Added Vulkan support for the following operators:
  • torch.cumsum (#78554, #81107)
  • torch.nn.LSTM (#78943, #79702)
  • torch.nn.ReplicationPad2d (#79057, #79291)
  • torch.nn.threshold (#78654, #79717)
  • torch.nn.BatchNorm2d (#80510)
  • torch.nn.LayerNorm (#80980)
  • torch.nn.GLU (#80910, #81729)
  • torch.select (#81771)
  • torch.stack (#81064)
• Prototype implementations for Quantized Tensors were added (#81491). These implementations still need to be exposed to Torchscript, but so far prototype implementations for the following ops have been added:
  • torch.quantize_per_tensor (#81492)
  • torch.dequantize (#81493)
  • Quantized arithmetic ops (#81494, #81632, #81640, #81641)
  • Quantized 2D convolution (#81495, #81496, #81497)
  • Quantized Upsample2D (#81720)

Mobile

• Added support for dtypes and custom classes in model tracer (#84795)
• Extended Flatbuffer to get mobile_info for NMLML workflows (#78306)
• Added serialization/deserialization of Sparse Quantize Linear Packed Params (#80474)
• Added qnnpack bcsr matrix unpacking and use unpacking in Linear module (#80475)
• Added OwnedOrBorrowedVector for QNNPack BCSR Indices/Values (#80476)

Distributed

Distributed Checkpointing (Prototyping)

• This is a prototyping effort which enables loading and saving PyTorch models from one or more hosts. Models can use features such as DDP, FSDP and ShardedTensor and they can have a different configuration between saving and loading - for example, save from 4 hosts and load from a single host. Distributed checkpointing has an extensibility API that enables full control of how a model is saved; and a pluggable IO backend. (#83781, #83419, #84952, #84881)

Distributed(c10d)

• Made c10d collective ops dispatcher passable. It allows tracing mechanisms such as LazyTensor and AOTAutograd to observe communications, e.g., : broadcast(#76722), allreduce(#79582), allgather (#79669), reduce_scatter (#79683), reduce (#79686), gather (#79687), scatter (#79688), alltoall (#79691), barrier (#79777), send/recv (#79779).
• Added UCC process group (#79918)
• Enabled uneven input support for all_gather (#83713) and uneven output support for reduce_scatter (#87010)
• Added NCCL PreMul Sum to c10d ReduceOp (#84243)

DistributedDataParallel

• Made DDP work with Python process group (#79176)
• Enabled Zero1's ddp_with_overlap for hpu backend (#80438)

FullyShardedDataParallel

• Added forward prefetching option in FSDP API (#85177)
• Added fp16 and bf16 hooks for FSDP (#81711)
• Implemented sharded_optim_state_dict and flatten_sharded_optim_state_dict. (#77628)
• Added rate limiter (#83917) Thanks to IBM Research team, @lchu-ibm for his contributions to FSDP and @hfwen0502 for the experimental testbed that identified the issues.
• Added an option to keep grads in lower prec (#85223)

torch.distributed.elastic

• Added watchdog to TorchElastic agent and trainers (#84081)

Activation Memory Management (Prototyping)

• We offer a new API, torch.distributed.algorithms.checkpoint.checkpoint_wrapper to wrap nn.Modules with activation checkpointing or activation offloading to easily use and experiment with activation checkpoint techniques without modifying model code. This makes it simpler to leverage activation checkpointing to reduce memory footprint of your training applications and train larger models. (#83035, #78704, #78854, #79830, #80089, #84907, #84908, #85448, #85449)

Infra (RelEng)

• Enabled multigpu unittests on FSDP (#77947)
• Added feature to do rebase (via comment) onto any branch (#78772)
• Added implementation to allow PR collaborators to revert their PRs (#82360)
• Added torchvision onto the commit pins file (#79151)
• Turned on -Werror=all with a few exceptions in Bazel build for CUDA (#79306)
• Prepared for running PyTorch tests with TorchDynamo and skips for known failing tests (#80106)
• Added ROCm build to pull request jobs (#80149)
• Added dynamo test configuration (#80342)
• Enabled ROCm CI for trunk test (#80920)
• Added linux cuda 11.7 workflows (#81089)
• Updated CI docker images and jobs to ROCm5.2 (#81168)
• Added UCC PG build in CI (#81583)
• Enabled periodic builds for CUDA 11.7 (#81688)
• Enabled distributed tests for ROCm (#81751)
• Added New TORCH_UCC_BLOCKING_WAIT env variable (#81791)
• Change functorch pin mechanism to test functorch in pytorch/pytorch now that functorch is inside pytorch/pytorch (#81918)
• Added Python 3.11 nightlies for Linux PyPi (Please note that 3.11 binaries are not fully functional) (#82302)
• Updated ROCm nightly builds to rocm5.2 (#82353)
• Add functorch target to cmake (#83464)
• Upgraded CUDNN version for cuda 11.7 (#84964)
• Enabled pytest-shard for functorch (#85321)
• Enabled CI to run test_ops in parallel (#85528)
• Updated trunk CUDA-10.2 to CUDA-11.7 (#85943)
• Added support for building and running Metal tests in CI (#86073)
• Bumped nvidia docker version and using python 3.10 for cuda11.7 (#82472)

Improvements

Python API

• Added float16 support for torch.{arange, linspace} (#80492)
• Added integer support to torch.index_reduce (#80464)
• Added a stable kwarg to torch.argsort that controls the relative order of equivalent elements (#75162)
• Improved stability of torch.distributions.kl_divergence for two Bernoulli distributions (#79944)
• Improved type annotations for torch.{as_tensor, as_subclass} (#86105)
• Added type promotion support for torch.{addcmul, addcdiv} (#74234)
• Added bfloat16 support for torch.save with XLA/HPU tensors (#77534)
• Improved wrapper subclass detection for serialization (#81105)
• Updated python API TensorOption signatures for consistency with JIT schemas (#82241)
• Allowed disabling oftorch.library.Library with PYTORCH_DISABLE_LIBRARY (#85190)
• Enabled dim=None for torch.{mean, sum, nanmean, nansum} (#81286), (#79881), (#82912)
• Added feature to enable registration of extension device modules as a native module under the torch namespace (#78329)
• Added logsumexp to amp.autocast (#76330)

C++ API

• Allowed const T& access to ListElementReference (#83177)
• Redirected print messages to stderr in torch.utils.cpp_extension (#82097)
• Updated CUDA compiler matrix in torch.utils.cpp_extension (#82860)
• Added __all__ to torch.utils.cpp_extension, torch.utils.hooks and torch.utils.show_pickle (#85331)

Autograd

• Added forward AD coverage for torch.{amin, amax, nansum, nanmean} (#80082), torch.scatter_reduce (except reduction=prod) (#85000), torch.linalg.det (#79487), torch.{elu_, celu_, selu_} (#83080)
• Added forward-over-reverse AD coverage for nn.functional.{binary_cross_entropy} (#77852) , nn.functional.{embedding}(#79699), nn.functional.{mse_loss, softplus, l1_loss, smooth_l1_loss, prelu, hardswish} (#78740), nn.functional.{nll_loss, batch_norm, layer_norm, group_norm, cross_entropy, soft_min} (#84976) torch.{log_softmax, softmax}(#84976), torch.amin, amax, nansum (#80082)
• Added support a stable double backward on torch.linalg.det for real inputs (#80217)
• Added support for kwargs input to function when torch.utils.checkpoint with use_reentrant=False (#80987)
• Added context manager to disable saved tensor hooks: torch.autograd.graph.disable_saved_tensors_hooks (#85971)
• Added new cpp custom function API to inform the backward function whether a gradient is necessary to compute: ctx->needs_input_grad(idx) (#82544)
• Added all device types in the pybinded DeviceType enum (#83676)
• Added check_nan flag to torch.autograd.detect_anomaly which enables users to run anomaly mode without nan checking (#83481)

Build

• Specify "Generic" BLAS library name to ensure PyTorch can find the BLAS llibrary (#74269)
• Generate CUDAConfig.h only for CUDA builds (#78218)
• Moved build_variables.bzl and ufunc_defs.bzl from pytorch-root/tools/ to PyTorch root directory (#78542)
• Made lintrunner compatible with M1 (#78628)
• BLAS library is linked privately instead of being linked publicly (#78883)
• Updated build targets to include generated enum_tag.cpp (#79668)
• Use miopen_LIBRARIES and rccl_LIBRARIES directly, when they are valid target for RCCL (#80446)
• Deleted Win specific case for CMake older than 3.1 (#81411)
• Split .cu to improve compile times (#81193)
• Added append_cxx_flag_if_supported macro (#82883)

torch.nn

• Improved groups argument validation for nn.Conv{1,2,3}d modules (#77919)
• Improved error message for convolution backward fallback kernel (#81538)
• Reduced memory usage of nn.Module full backward hooks by removing reference cycles (#80139)
• Improved kl_div at boundary and its general implementation (#80334)
• Improved input shape validation for MKL-backed convolution operations (#76526)
• Improved input validation for nn.AdaptiveAvgPool2d (#84061)
• Improved groups argument validation for nn.Conv{1,2,3}d (#85248)
• Improved input index validation for nn.MaxUnpool{2,3}d (#78280)
• Improved listing of public APIs for optim and nn (#80237)
• Added new operator for nn.Sequential: + (#81170), extend (#81179), insert (#81402), +=, * and *= (#81279),
• Added deepcopy support for unitialized parameter (#83809)
• Added nondeterministic alert for nn.MaxUnpool{1,2,3}d (#84766)
• Added Bfloat16 support for the backward pass of nn.functional.kl_div on CUDA (#77676)

torch.optim

• Added support for optimizers with more than 2 betas for LRScheduler (#84486)
• Added fused kwarg to optim.Adam to enable a fused implementation on CUDA (#85739)

Composability

• Significant hardening and improvements to the functionalize() API that lives with functorch (#77129, #77126, #77125, #78199, #77132, #77713, #77714, #78819, #78820, #82008, #82009, #81702, #80416, #80418, #80251, #80526, #82326, #81454, #81471, #83542, #83701, #85975)
• Allow __torch_dispatch__ subclasses and modes to override more tensor metadata: device/size/stride/dim (#77684, #77970, #78646, #78691)
• Improvements to the torch.library API, for registering python functions to the pytorch dispatcher:
  • Improved error checking in torch.library (#77990)
  • Make torch.library decorators return function, to allow for chaining (#78996)
• Ported cholesky, linalg_qr, linalg_eigh and linalg_eighvalsh to structured kernels, giving them support with meta tensors (#79300, #79054, #79072)
• Added python decompositions for many torch operators. This adds meta tensor coverage for a large number of pytorch operators (#77930, #79768, #79808, #84062, #84350, #80219, #78350, #79667, #81003, #81420, #81113, #81241, #81765, #82284, #80497, #80358, #80182, #80737, #81734, #81826, #78461, #78468, #78525, #78914, #78919, #79900, #79225, #80964, #83235, #84108, #84451, #78602, #78603, #78527, #78604, #78992, #78993, #78997, #79278, #79341, #79311, #79411, #79581, #81800, #79834, #82309, #79975, #82587, #82603, #83191, #84349, #84460, #85793, #86057)
• Beefed up API for printing out operators registered to the dispatcher (#78995)
• Trued up c10::FunctionSchema::operator<< to print native_functions.yaml syntax (#79645)
• Made it so that it is valid to set metadata after detach calls, like x.detach().resize_(...) (#83590)
• Optimized torch.ops.ns.opname.overload accessor in __torch_dispatch__ (#85132)

Dataloader

• Added shape checking on argument weights for WeightedRandomSampler (#78585)
• Added support for radom_split to accept percentages as lengths (#78877)
• Extended collate function that can register collate functions to handle specific batch types (#85748)

Functorch

• functorch.jacfwd now accepts a randomness kwarg (#84220)
• Improved the error message when using vmap on a function with no Tensor inputs (#83016)
• Relaxed the Tensor.as_strided batching rule. This is a primitive used in forward-mode AD (among other things) and improves composability of vmap with other transforms (like jvp).
• functorch.functionalize: added support for in-place views on inputs (#83993)
• functorch.functionalize: moved this API out of the functorch.experimental namespace (#85742)
• Added vmap support for linalg.cholesky, linalg.eigvals, linalg.eigvalsh, linalg.matrix_norm, linalg.matrix_power, linalg.norm, linalg.tensorinv, linalg.solve_triangular (#82177)
• Added vmap support for linalg.solve (#82814)
• Added vmap support for linalg.cross (#83759)
• Added vmap support for linalg.matrix_rank (#83760)
• Added vmap support for linalg.pinv (#83761)
• Added vmap support for Tensor.fill_ (#84015)
• Added vmap support for linalg.lstsq (#82325)
• Added vmap support for linalg.lu_solve (#85175)

LinAlg

• Added a driver= kwarg to torch.linalg.svd and svdvals. Add cusolver gesvdaStridedBatched driver to linalg.svd (#74521)
• Added opteinsum backend to torch.einsum (#86219)
• Added path optimize kwarg to einsum (#84890)
• Call view instead of sum in einsum to remediate MPS regression (#87135)
• Ensure that we contract left to right in einsum (#87199)
• Fixed opt_einsum defaults to be more reasonable (#86985)

Sparse

• Added sparse_dim and dense_dim for batched, hybrid CSR/CSC/BSR/BSC (#80565, #80901)
• Added support for conversion between batched CSR/CSC/BSR/BSC and dense Tensors (#80781, #83084, #83086, #78025, #80354, #82120)
  • Conversion between SparseBsr and Strided (#78025)
  • Added support for BSR <-> Strided Conversion (#80354)
• Added support for conversion between CSR and CSC (#85091)
• Added support for conversion between BSR and BSC (#85091)
• Added partial support for CSR/CSC/BSR/BSC inputs to mm, addmm, matmul and F.linear (#85551, #85308, #85379, #85307)
• Added support for COO to permute (#79707)
• Added support for ComplexHalf to torch.nonzero and add(dense, CSR) (#79062)
• Added support for CSC/BSR/BSC to unary zero-preserving functions. (#78173, #85031)
• Added support for batched BSR/BSC to transpose (#82122)
• Added support for scalar together with COO inputs to mul (#82962)
• Added support for CSC/BSR/BSC to empty_like (#82310)
• Added support for batch dims of CSR/CSC/BSR/BSC to select (#82119)

torch.fx

• In constant folding, added device_for_folded_attrs parameter and sets the requires_grad option for a folded tensor (#79067)
• Mode-based tracing in make_fx (#79638, #84238)
• Made executor handle kwargs (#79858)
• Added ignore_parameters_and_buffers flag to FxGraphDrawer (#79982)
• Enabled an is_fx_tracing flag in the FX tracer (#80255)
• Attached ProxyTorchDispatchMode to ProxyTensor and use it in __torch_dispatch__ (#82549)
• Used enable_tracing flag for ProxyTorchDispatchMode instead of modifying torch dispatch mode stack inner attributes (#82643)
• Improved legalize_graph pass in FX (#82874)
• Implemented __deepcopy__ for fx.Tracer (#83130)
• Hackde up make_fx to natively support varargs (#83210)
• Updated proxy_tensor.py to support List input/output (#83302)
• Added *_only and all/any pytree utilities (#83316)
• Deleted ProxyTensor wrapper subclass (#83330, #83646)
• Added support for partial decompositions in make_fx (#83770)
• Added metadata field to fx.GraphModule (#84378)
• Added option to maintain the FX graph execution order after splitting_module (#85188)

JIT

• Added PReLU to MKLDNN convertible Ops in JIT optimize_for_inference (#79011)
• Enabled torch._refs.var for nvFuser executor (#79517)
• Fixed nvFuser's where (tensor, python_scalar, tensor) type promotion (#80347)
• Added ComplexDouble scalar creation bindings to nvFuser's Python API (#80522)
• Added real and imag to NVFuser and its python frontend (#79824)
• Added Nvfuser opt in for decomposition (#81134)
• Added torch.jit.fuser() option for disabling all fusers (#81731)
• Added support for symbolic diff for silu (#81724)
• Added NVFuser support for (prims.sign, refs.sign, squeeze, native_batch_norm, transpose) (#83167, #85562, #84629, #84117)
• Use high precision accumulate buffer for bf16 accumulation in NNC (#84402)

Quantization

• Improved quantization support for masked_fill (#78368, #85108)
• Improved quantization support for index_put (#78384, #85685)
• Improved quantization support for LSTM and MultiHeadAttention (#79959, #79956, #79960, #83304, #85068)
• Added support for quantized matmul (#83885)
• Introduced a more stable conv_bn fusion for QAT training (#85744)
• Removed warnings from using torch.tensor(value) (#84277)

ONNX

• Added operator support for torch.tensor_split (#77437), torch.lerp (#78891), torch.movedim and torch.moveaxis (#78931), torch.scatter_add (#79103), torch.argsort (#80234), aten::native_dropout (#81743), aten::native_layer_norm (#81754), aten::convolution (#81815), aten::_log_softmax (#81804), aten::layer_norm for ONNX opset version 17 using LayerNormalization (#84293), nn.init.normal (#84149)
• Added quantization support to more single output ops (#83008) aten::reshape, aten::reshape_as, aten::t, aten::transpose, aten::numpy_T, aten::expand, aten::expand_as, aten::embedding, aten::embedding_bag, aten::view, aten::select, aten::eq, aten::ne, aten::gt, aten::lt, aten::le, aten::ge, aten::elu, aten::selu, aten::hardtanh, aten::hardswish, aten::as_strided, quantized::sigmoid, quantized::layer_norm, quantized::group_norm, quantized::leaky_relu, quantized::instance_norm
• ONNX operators are exported with names containing their associated scope from nn.module (#82038), (#82039), (#82040)
• Introduced runtime type checking with the beartype library in all public APIs (#83673), (#84091)
• All torch.onnx APIs now support runtime type checking when @beartype is present in the Python environment. A warning is emitted when a type mismatch is detected.
• This feature is experimental. To turn all warnings into errors, set the environment variable TORCH_ONNX_EXPERIMENTAL_RUNTIME_TYPE_CHECK=ERRORS. To disable this behavior, set TORCH_ONNX_EXPERIMENTAL_RUNTIME_TYPE_CHECK=DISABLED which effectively makes it a no-op.
• Improved shape type inference (#78999)
• Turn on ONNX shape inference by default (#82767)
• Enabled data propagation from ONNX (#80730)
• Introduced SARIF (#85428) for torch.onnx submodule
• Improved warnings and errors (#78441), (#78309), (#83332), (#85179), (#83007)
• Updated ONNX submodule to 1.12 (#79585)
• Apply Common Subexpression Elimination pass to ONNX export (#85665)

AMD

• Support benchmark flag for MIOpen (#77438)
• Correctly handle the error codes of hipGetDeviceCount (#80405)
• Use torch._C._cuda_getArchFlags to get list of gfx archs pytorch was built for (#80498)
• torch.cuda.is_bf16_supported() returns True (#80410)
• Workaround missing hipProfilerStart/Stop (#82778)
• Enabled jiterator on ROCm (#77982)
• Enabled MIOpen fused convolution relu (#82002)
• Restore MIOpen benchmark flag default to true (#82656)
• embedded_interpreter_hip to enable torch::deploy on AMD (#83329)
• Add HIP libs into torch deploy init list & corresponding dependency for CURE benchmark running on AMD (#83434)

CUDA

• Added synchronize hooks (#84427)
• Added CSAN support for CPU synchronizations (#84428)
• Return device count using nvml (#84879)
• Reworked printing tensor aliases in CSAN error message (#85008)
• Added jiterator support when dtype is complex32 for tan, atan, sin, asin (#77802),(#77606)
• Added jiterator support when dtype is complex for logical_{or, xor} (#75947)
• Reduced overhead of get_current_stream (#78066)
• Added an argument to specify warmup iterations in make_graphed_callables (#78124)
• Small improvements to device_count (#85192)
• Memoize torch.cuda.device_count (#84878)
• Remove the construction of unused tensors in fallback convolution implementation (#79183)
• __launch_bounds__ for torch.mode with CUDA 11.7 (#79710)
• Removed synchronization for D2H copy with a different dtype (#80607)
• Added nondeterministic alert to CUDA cumsum (#75693)
• Annotated CUDACAchingAllocator snapshots (#82146)
• CUDACachingAllocator snapshots from C++ (#86190)
• Propagate CUDAOutOfMemoryError to Python. (#83146)
• Set cublas workspace size to 4M (#74159)
• Allow changing the cuda allocator settings even after the process started (#84970)
• Fixed exception handling, improve overheads and avoid constructing storage for element size for DLPack (#84612)
• Added BFloat16 for fast layernorm (#83971)
• Added BFloat16 support for torch.{im2col,col2im} on CUDA (#84372)
• Added Bfloat16 support for ReflectionPad (#84949)
• Added explicit __all__ to torch.cuda (#85193)
• Set CUDA_MODULE_LOADING to LAZY when not set by the user (#85692)
• Support cuDNN Errata Filter (#73934)
• Allow the number of kernels profiled under torch.backends.cudnn.benchmark = True to be limitedCudnnv8 benchmark limit (#78299)
• Update tests and dispatching for CUDNN V8 API behavior for bfloat16 convs (#81139)

Intel

• [RFC] Enable oneMKL&oneDNN on-demands verbose functionality (#63212)
• Updated ideep for NNC post-op (#82705)
• Enabled native 1d spatial input for Intel xpu (#82301)
• Added loss operators to fp32 cast policy of AutocastCPU (#81689)
• Added bfloat16 support for lerp on CPU (#84327)
• Added prelu op and module for quantized CPU backend (#73491)
• Enabled mkldnn matmul for aarch64 bf16 devices (#85546)

MPS

• Added ranked tensors for addcmul ops in MPS instead of constants and update MacOS version check (#78354)
• Moved MPS compat check into common comparison machinery of TensorLikePair (#77836)
• Made MPS buildable with either XCode or CommandLineTools (#79430)
• Improved MPS aten::softplus operator by adding RankedPlaceholder for graph nodes instead of constants (#81169)
• Extended MPS Conv1D operation for NHWC format (#83121)
• Added support for 1D weights in MPS linear layer (#85752)
• Added full support for serialization of MPS Tensors (#79465)
• Added support for 1D bias in MPS operation torch.addmm (#81519)
• Added torch dispatch stub code for MPS backend (#82612)
• Use convenience helper function dispatch1DJob for MPS native implementations (#82982)
• Enabled support in MPS for torch.adaptive_avgpool_2d for larger output sizes (#85726)
• Extended support in MPS for torch.constant_pad_nd for 4D+ padding (#85991)

Profiler

• Propagate metadata into Engine::evaluate_function event. (#77696)
• Switched to nanoseconds for Result's internal representation (#77697)
• Made profiler table column widths changeable via arguments (#85203)

Vulkan

• Enabled higher dimensional input in torch.nn.linear (#81773)
• Vulkan tensor views now infers dim size when -1 is provided as input (#81668)
• Vulkan prepacked op contexts will now release the deserialized CPU tensors from memory upon construction (#83587)
• Vulkan shader codegen is now Windows compatible (#85241)

Mobile

• Allowed tracing multiple input models at once (#84833)
• Leaky relu in metal shader (#78544)
• Added detailed error message for iOS test (#79140)
• Remove dcode duplications and refactor (#79184)
• Optionally run fbgemm in tracer (#83531)
• Added hardshrink op to metal backend (#82224)
• New flatbuffer_loader functions that do not depend on flatbuffers.h (#82618)
• Added max_pool2d, linear, conv2d FP32 operator tests for XNNPACK (#83131)
• Removed flatbuffer types/headers from flatbuffer_serializer[_jit].h (#82619)
• Migrated remaining pytorch code to use new flatbuffer_loader.h APIs (#82620)
• Remove flatbuffer types/headers from flatbuffer_loader.h (#82893)
• Use flatbuffer of alternate namespace (#82952)
• Hide flatbuffer build dependencies (#82953)
• Renamed flatbuffer_all to flatbuffers_jit (#82826)
• Renamed flatbuffer_serializer to _mobile or _full_jit (#82827)
• Created flatbuffers_mobile (#82828)
• Added API for profiling backend memory events for Edge CPU profiler (#80350)
• Switched mobile targets to flatbuffers_mobile (#82829)
• Added an option to avoid adding base ops to static op library for Edge (#84360)
• Fixed load_extra_only api for flatbuffers and enable flatbuffers in mobile for OSS properly (#83855)
• Remove unused field 'order_' in nnapi.h (#84067)

Distributed

Distributed(c10d)

• c10d API improvements:
  • Introduced util functions in c10d get_local_rank, get_global_rank and get_global_ranks (#82134, #84363)
  • Replaced internal API _all_gather_base with a public API all_gather_into_tensor (#85686)
  • Replaced internal API _reduce_scatter_base with a public API reduce_scatter_tensor (#85867)
• Improvements to c10d error messages:
  • Added ncclGetLastError (#83724, #85825, #85850)
  • Added closing parentheses to the CollectiveFingerprint (#79723)
  • Added tensor deserializer and included rank and collective type to the error messages (#79724)
  • Adopted ncclRemoteError (#85887)
• Passed group ranks and options to third party distributed backends (#73164)
• Enabled NCCL_DESYNC_DEBUG when TORCH_DISTRIBUTED_DEBUG is set to DETAIL (#83881)
• Added a soft error handling mode NCCL_ASYNC_ERROR_HANDLING=2 that does not crash the process (#84386)
• Upgraded NCCL to 2.14.3 (#85367)

Distributed Optimizer

• Added functionality for save and restore step counter for model averanger in PostLocalSGDOptimizer (#78988)

DistributedDataParallel

• Enabled the static graph to print unused parameters in debug mode for DDP. (#81929)
• Enabled stateful PowerSGD communication hook now can be saved and reloaded to resume training (#79334)

FullyShardedDataParallel

• Allowed different optim_input orders across ranks (#78599)
• Added profiling range for FSDP.backward (#78479)
• Enabled NamedTuple support for FSDP (#83055)
• Added FSDP communication hook interface for NO_SHARD strategy (#79833)
• Moved the sharded_state_dict logic to the post hook to avoid OOM (#82613)
• Added ability to iterate through dataclasses in fsdp.utils (#82638)
• Enabled passing kwargs to load_state_dict (#83309)
• Used _init_from_local_tensor to create ShardedTensor to avoid communication overhead (#82911)
• Added communication hook for sharded strategies (#83254)
• Changed to print exec order only in debug mode (#83868)
• Ensured that all ranks use the same order to iterate through optimizer states (#84654)
• Optimizer states may be on CPU, copied them to GPU before gathering (#84708)
• Handled the state_dict on CPU cases (#85640)
• Add FSDPExtensions for TP support (#85039)
• Ignored buffers that are non-persistent. (#85740)
• Delayed moving tensor to CPU until necessary for optim_state_dict() (#85761)
• Dequeue one event instead of flushing for rate limit (#86165)

torch.distributed.elastic

• Implemented a named pipe based watchdog timer (#83695)

Infra (RelEng)

• Consolidated all python targets in the tools folder (#80408)
• Improved ios simulator test in CI (#80459)
• Add functorch testing shard in CI (#81283)
• Added functorch shards for windows CI (#82161)
• Added functorch shard for mac x86 tests, linux cu102 tests (#82000)
• Added CI workflow to build official docker images with multiarch (#83437)
• Sharded trunk / linux-bionic-cuda10.2-py3.9-gcc7 / test (default from 2 -> 4 (#83424)
• Migrated workflows from 18.04 to 22.04 (#83861)

Bug fixes

Python API

• Fixed dim out of range check for logcumsumexp on CUDA when the source tensor is empty(#78284)
• Added missing __init__.py for torch.utils.jit (#78629)
• Fixed backward crash for gather with an empty index tensor when sparse_grad=True (#78698)
• Added type annotations to torch.distributions.kl_divergence (#78432)
• Fixed erroneous inclusion of end in the output of torch.arange for some inputs (#80758)
• Fixed torch.distributions.Transform to be pickle-able (#81707)
• Added check that self and mask are on the same device for torch.masked_fill (#82737)
• Fixed potential ref cycle creation in torch.utils.checkpoint (#82776)
• Fixed Tensor.__hash__ for Tensor subclasses (#83174)
• Fixed torch.cat for 0-dim tensors with different dtypes (#83391)
• Fixed torch.equal on CPU when inputs have different dtypes (#83350)
• Fixed data-dependent shapes in torch.districutions.{HalfCauchy, HalfNormal} (#84322)
• Added check that the size of the last dimension of tau is less than or equal to that of input in torch.ormqr (#85278)
• Added check that weights is a 1D tensor in torch.bincount (#85881)
• Fixed segfault for out arguments that have a large number of dims (#85294)
• Fixed comparison ops with scalar arguments by removing overflow check (#78881)
• Normalized torch.utils.dlpack strides to 1 where size of corresponding dimensions < 2 (#83158)
• Added a check in torch.empty_strided that sizes has the same dimensionality as strides (#82422)
• Fixed torch.istft default output length to prevent trimming of last element (#80031)

C++ API

• Fixed missing antialiasing path to the interpolation for bicubic mode (#84599)
• Added IListRefTag::Materialized to IListRefIterator destructor. (#85467)
• Fixed im2col by adding a check that pad_width and pad_height are non-negative (#85541)
• Fixed check_compiler_ok_for_platform on non-English locales in torch.utils.cpp_extension (#85891)

Autograd

• Corrected the forward AD formula of torch.sgn which fixed forward-over-backward for torch.linalg.svd and other spectral decompositions, and torch.norm, torch.linalg.{norm, matrix_norm}(#80082)
• Fixed derivatives of convolution overridable backward (#80840)
• Updated setting non-float non-complex values for forward AD dual tensor to properly error(#78361)
• Fixed forward AD to not set tangent as-is in some situations (#79664, #79653)
• Fixed cpp hooks, retains grad, and backward(inputs=) behavior in-place (#79996)
• Relaxed storage layout checks for forward AD when zero-numel tensor (#81055)
• Fixed leak when create_graph=True and full backward hook registered (#82788)
• Fixed view and in-place interaction when grad_fn is first accessed in no-grad mode (#83872)
• Updated backward of torch.stack to correctly handle implicit real->complex casting (#84993)
• Fixed gradients for torch.nn.functional.{leaky_relu, threshold} when inplace=True (#85634)
• Corrected autocasting behavior in torch.utils.checkpoint when use_reentrant=False (#81766)
• Fixed gradcheck when outputs that don't require grad precede those that do (#77743)
• Fixed backward and double backward for nn.functional.binary_cross_entropy_with_logits (#80083)
• Fixed derivatives of norm(p=inf) (#78105)
• Fixed forward AD when conj-ness of primal and tangent of the dual tensor tensor do not match (#78358)

Build

• Use C++17 for RocksDB 7 header. (#75741)
• Fixed Windows builds with _DEBUG flag (bbe8d019f2)
• Pass WITH_BLAS option from environment to CMake (#78037)
• Remove -Wno-unused-but-set-variable for clang 13.0.0 (#79666)
• Fixed variable typo for USE_SYSTEM_PYBIND11. (#80272)
• Fixed compilation errors during build with clang13 (#80916)
• Added missing -fexceptions flags during PyTorch build (#81394)
• Fixed CMake dev warning (#81580)
• Fixed false positive AVX, AVX2 and AVX512 detection with MSVC (#82554)
• Fixed NCCL detection issues of the Gloo library (#82773)
• Fixed objcopy version detection in NCCL cmake process (#82774)
• Fixed build error by changing COLORIZE_OUTPUT option to USE_COLORIZE_OUTPUT in cmake file (#83716)
• Set default value for NCCL make to MAX_JOBS if ProcessorCount returns 0 (#84231)
• Fixed intermittent link errors in NCCL build (#84245)
• Deleted torch._dl extension (#84361)
• Used unified source file list for BUCK build (#84770)

Complex

• Fixed the derivative of torch.acosh for complex numbers (#80841).
• Removed unused conjugate kernels for real dtypes (2.2MB reduction in CUDA binary size) (#80374).

torch.nn

• Fixed nn.Embedding ‘s max_norm argument when forward mode AD is used (#78560)
• Fixed nn.ChannelShuffle when given empty Tensors (#77029)
• Fixed nn.RReLU backward on CUDA (#80434)
• Fixed spurious warnings in torch.nn.parallel.* APIs (#81476)
• Fixed nn.Conv2d fallback implementation for single channel inputs and channels last weight (#82392)
• Fixed segfault in adaptive pooling for specific index values (#84010)
• Fixed type annotation in nn.Conv{1,2,3}d for in_channels (#84302)
• Fixed nn.GeLU for empty inputs (#84926)
• Fixed correctness issues for nn.Conv2d on ARM-based machines (#85711)
• Fixed nn.ParameterList printing of Tensors on the “meta” device (#78529)
• Fixed channels-first behavior for nn.MaxPool3D on CUDA (#80748)
• Fixed input shape validation nn.MaxPool1d (#85594)
• Fixed nn.Softmax for large input tensors (#84182)
• Fixed lower and upper bound checks for nn.RReLU (#84996)
• Fixed edge cases in torch.nn.grad by calling into the c++ backward kernel directly (#81839)
• Fixed torch.nn.PixelShuffle for empty inputs (#86262)
• Fixed consistency of output and input dtypes for torch.nn.BatchNorm (#84410)

torch.optim

• Fixed optim.SGD maximize flag when momentum is involved (#81859)
• Fixed temporary bug where checkpoints from optimizers created with older PyTorch version could not be loaded (#83588)
• Fixed memory leak in optim.lr_scheduler.CyclicLR (#85462)
• Fixed initialization of lr in optim.lr_scheduler.SequentialLR (#72856)

BetterTransformer

• Cleaned up native transformer implementation (#78265)
• Added fastpath test for mask check flag (#82999)
• Added check for contiguous well-formed mask (#79927)
• Introduced mask contiguity check function (#79186)
• Fixed issue in softmax.cu with transformer error when mask seqlen > 1024 (#83639)
• Disabled Transformer/MHA fast path when autocast is enabled (#84722)
• Moved odd num_head in TransformerEncoder to slow_path (#83483)

Composability

• Fixed __torch_function__ bug in getindex that causes an error not set exception (#78781)
• Fixed __torch_dispatch__ usage with inplace views (#79902)

Dataloader

• Fixed NoneType object has no attribute python_exit_status when DataLoader exits (#83985)

Functorch

• functorch.grad: fixed silent correctness issue from calling a view operation on a captured tensor followed by an in-place operation (#85374)
• functorch.jacrev, functorch.jacfwd: fixed loud in-place errors when passing in inputs to the transforms and mutating them (#84914, #84915)
• functorch.vmap: Fixed support for in-place view operations (Tensor.unsqueeze_, Tensor.transpose_, Tensor.t_, Tensor.squeeze_) (#82899, #82903, #82972)
• functorch.vmap: added an error on incorrect weight shape to torch.nn.functional.prelu (#83106)
• functorch.vmap: fixed support for multinomial (#83838)
• functorch.vmap: fixed incorrect support for conv_transpose with groups > 1 (#84938)
• Fixed vmap x vjp x vjp composition for torch.nn.functional.prelu (#84939)
• Fixed printing tensors that are not being transformed over inside functorch transforms (#85556)
• Disallowed saved tensor hooks in functorch transforms to avoid silently incorrect behavior(#85972)
• Fixed cross to match unbatched behavior (#86926)

LinAlg

• Strengthen the preconditions of linalg.cross (#83798)
• Fix memory issues in linalg.lstsq (#85357)
• Fix linalg.lu_solve/torch.unpack to prevent bad memory usage on CPU (#85922)
• Preserve the dim of the input in matrix_exp. (#81330)

Sparse

• Fixed COO Tensors with less than two non-zero elements to always be marked coalesced. (#82426, #82085)
• Fixed CUDA kernel launch misconfiguration for mul on tiny COO tensors (#80254)
• Fixed silent type promotion bug by select if given all zero integer COO tensors(#82215)
• Fixed CUDA kernel coverage on 0-sized dense inputs for torch.sparse.sampled_addmm (#85194)

torch.fx

• Fixed bug where curly brackets were not properly escaped in FxGraphDrawer (#83604)
• Fixed torch.fx.wrap to use the callable function.__name__ rather than function.__code__.co_name (#84373)
• Added strictness check and made tensors into leaves if input tensors were leaves (#77474)
• Used getattr_recursive instead of getattr when splitting (#80011)
• Stopped ProxyTensor from turning aten::lift tensors into proxy objects (#81024)
• Fixed named_modules to be subscriptable (#81258)
• Fixed to_folder by adding custom_builtins to dump (#81433)
• Correctly unpacked constants when used in multi-return output (#82568)
• Replaced module name for torch.ops (#82395)
• Removed unnecessary import warnings (#82760)
• Don't constant propagate through nondeterministic functions (#83650)
• Don't extract tensor metadata from sparse tensors (#83669)
• Skipped folding side-effectful functions (#84016)
• Fixed make_fx issue by introducing get_attr into symbolic tracing (#84011)
• Disabled autocast cache during aotdispatch (#84035)
• Modified split_by_tags to retain output order (#84136)
• Made NormalizeArgs preserve node type (#85637)
• Fixed PyTree unpacking carrying forward type annotations (#81906)

JIT

• Fixed conv-batchnorm folding for previously-broken datatype inputs during JIT freezing (#78241)
• Fixed lightweight dispatch OOM error by introducing selective build (#79215)
• Used signed integers in CalculatedNecessaryArgs to avoid underflow with schemas where all args have defaults. (#79331)
• Fixed indexing into a tensor with a tuple (#79335)
• Propagate map_location arg to torch.jit.load in torch.load (#78733)
• Improved JIT autodiff heuristics for determining whether outputs require gradients (#78392, #79498)
• Used streams for import_ir_module for pickle case to reduce memory usage (#80131)
• Added scripting support for "start" kwarg in enumerate() (#80585)
• Turned off arc in CoreML backend, because throwing exceptions in arc code leaks memory (#79928)
• Suppressed virtual-dtor check on llvm_jit to fix NNC build (#81449)
• Fixed annotation extraction for python 3.10 (#81334) (#81334, #81506)
• Fixed std::out_of_range when using NNC and ConstantChunk input shapes are unknown (#82698)
• Limits constant chunk propagation for pw-node-only in NVFuser (#83083)
• When encountering dynamic types, one should cast it recursively. (#83218)
• Fixed handling of empty dim list in sum_mean_dim symbolic shape fn (#83357)
• Check existence of the array ref when tracing resize_ to avoid _MapBase::at runtime error (#81422)
• Fixed define_constant pybind signature to match std::complex scalar in NVFuser (#83684)
• Cast to signed char to fix aarch64 build (#84429)
• Support torch.ScriptObject in torch::jit::as_object (#84398)
• NVFuser torchbench patch to take nvprim fallback when no cuda tensors are provided as inputs (#84411)
• Fixed coreml gpu flag not set (#84725)
• Print the real type for function schema arguments (#85103)
• Fixed torch.jit.trace check that was causing tracing to fail for MPS inputs (#84850)
• Throw an error instead of segfaulting when passing None to futures (#85304)
• Cherry pick sorting patch for NVFuser fusion segmented (#85620)
• Support freezing modules that don't have a forward method (#85779)

Quantization

• Added channel axis bound checking in fused_moving_avg_obs_fake_quant_* (#78148)
• Disable use of qnnpack with ceil_mode of the avgpool op (#79028)
• Improve sub

Note truncated.

This release is meant to fix the following issues (regressions / silent correctness):

Optim

• Remove overly restrictive assert in adam https://github.com/pytorch/pytorch/pull/80222

Autograd

• Convolution forward over reverse internal asserts in specific case https://github.com/pytorch/pytorch/issues/81111
• 25% Performance regression from v0.1.1 to 0.2.0 when calculating hessian https://github.com/pytorch/pytorch/pull/82504

Distributed

• Fix distributed store to use add for the counter of DL shared seed https://github.com/pytorch/pytorch/pull/80348
• Raise proper timeout when sharing the distributed shared seed https://github.com/pytorch/pytorch/pull/81666

NN

• Allow register float16 weight_norm on cpu and speed up test https://github.com/pytorch/pytorch/pull/80600
• Fix weight norm backward bug on CPU when OMP_NUM_THREADS <= 2 https://github.com/pytorch/pytorch/pull/80930
• Weight_norm is not working with float16 https://github.com/pytorch/pytorch/issues/80599
• New release breaks torch.nn.weight_norm backwards pass and breaks all Wav2Vec2 implementations https://github.com/pytorch/pytorch/issues/80569
• Disable src mask for transformer and multiheadattention fastpath https://github.com/pytorch/pytorch/pull/81277
• Make nn.stateless correctly reset parameters if the forward pass fails https://github.com/pytorch/pytorch/pull/81262
• torchvision.transforms.functional.rgb_to_grayscale() + torch.nn.Conv2d() don`t work on 1080 GPU https://github.com/pytorch/pytorch/issues/81106
• Transformer and CPU path with src_mask raises error with torch 1.12 https://github.com/pytorch/pytorch/issues/81129

Data Loader

• [Locking lower ranks seed recepients https://github.com/pytorch/pytorch/pull/81071

CUDA

• os.environ["CUDA_VISIBLE_DEVICES"] has no effect https://github.com/pytorch/pytorch/issues/80876
• share_memory() on CUDA tensors no longer no-ops and instead crashes https://github.com/pytorch/pytorch/issues/80733
• [Prims] Unbreak CUDA lazy init https://github.com/pytorch/pytorch/pull/80899
• PyTorch 1.12 cu113 wheels cudnn discoverability issue https://github.com/pytorch/pytorch/issues/80637
• Remove overly restrictive checks for cudagraph https://github.com/pytorch/pytorch/pull/80881

ONNX

• ONNX cherry picks https://github.com/pytorch/pytorch/pull/82435

MPS

• MPS cherry picks https://github.com/pytorch/pytorch/issues/80898

Other

• Don't error if _warned_capturable_if_run_uncaptured not set https://github.com/pytorch/pytorch/pull/80345
• Initializing libiomp5.dylib, but found libomp.dylib already initialized. https://github.com/pytorch/pytorch/issues/78490
• Assertion error - _dl_shared_seed_recv_cnt - pt 1.12 - multi node https://github.com/pytorch/pytorch/issues/80845
• Add 3.10 stdlib to torch.package https://github.com/pytorch/pytorch/pull/81261
• CPU-only c++ extension libraries (functorch, torchtext) built against PyTorch wheels are not fully compatible with PyTorch wheels https://github.com/pytorch/pytorch/issues/80489

PyTorch 1.12 Release Notes

• Highlights
• Backwards Incompatible Change
• New Features
• Improvements
• Performance
• Documentation

Highlights

We are excited to announce the release of PyTorch 1.12! This release is composed of over 3124 commits, 433 contributors. Along with 1.12, we are releasing beta versions of AWS S3 Integration, PyTorch Vision Models on Channels Last on CPU, Empowering PyTorch on Intel® Xeon® Scalable processors with Bfloat16 and FSDP API. We want to sincerely thank our dedicated community for your contributions.

Summary:

• Functional Module API to functionally apply module computation with a given set of parameters
• Complex32 and Complex Convolutions in PyTorch
• DataPipes from TorchData fully backward compatible with DataLoader
• Functorch with improved coverage for APIs
• nvFuser a deep learning compiler for PyTorch
• Changes to float32 matrix multiplication precision on Ampere and later CUDA hardware
• TorchArrow, a new beta library for machine learning preprocessing over batch data

Backwards Incompatible changes

Python API

Updated type promotion for torch.clamp (#77035)

In 1.11, the ‘min’ and ‘max’ arguments in torch.clamp did not participate in type promotion, which made it inconsistent with minimum and maximum operations. In 1.12, the ‘min’ and ‘max’ arguments participate in type promotion.

1.11

>>> import torch
>>> a = torch.tensor([1., 2., 3., 4.], dtype=torch.float32)
>>> b = torch.tensor([2., 2., 2., 2.], dtype=torch.float64)
>>> c = torch.tensor([3., 3., 3., 3.], dtype=torch.float64)
>>> torch.clamp(a, b, c).dtype
torch.float32

1.12

>>> import torch
>>> a = torch.tensor([1., 2., 3., 4.], dtype=torch.float32)
>>> b = torch.tensor([2., 2., 2., 2.], dtype=torch.float64)
>>> c = torch.tensor([3., 3., 3., 3.], dtype=torch.float64)
>>> torch.clamp(a, b, c).dtype
torch.float64

Complex Numbers

Fix complex type promotion (#77524)

Updates the type promotion rule such that given a complex scalar and real tensor, the value type of real tensor is preserved

1.11

>>> a = torch.randn((2, 2), dtype=torch.float)
>>> b = torch.tensor(1, dtype=torch.cdouble)
>>> (a + b).dtype
torch.complex128

1.12

>>> a = torch.randn((2, 2), dtype=torch.float)
>>> b = torch.tensor(1, dtype=torch.cdouble)
>>> (a + b).dtype
torch.complex64

LinAlg

Disable TF32 for matmul by default and add high-level control of fp32 matmul precision (#76509)

PyTorch 1.12 makes the default math mode for fp32 matrix multiplications more precise and consistent across hardware. This may affect users on Ampere or later CUDA devices and TPUs. See the PyTorch blog for more details.

Sparse

Use ScatterGatherKernel for scatter_reduce (CPU-only) (#74226, #74608)

In 1.11.0, unlike scatter which takes a reduce kwarg or scatter_add, scatter_reduce was not an in-place function. That is, it did not allow the user to pass an output tensor which contains data that is reduced together with the scattered data. Instead, the scatter reduction took place on an output tensor initialized under the hood. Indices of the output that were not scattered to were filled with reduction inits (or 0 for options ‘amin’ and ‘amax’).

In 1.12.0, scatter_reduce (which is in beta) is in-place to align with the API of the related existing functions scatter/scatter_add. For this reason, the argument input in 1.11.0 has been renamed src in 1.12.0 and the new self argument now takes a destination tensor to be scattered onto. Since the destination tensor is no longer initialized under the hood, the output_size kwarg in 1.11.0 that allowed users to specify the size of the output at dimension dim has been removed. Further, in 1.12.0 we introduce an include_self kwarg which determines whether values in the self (destination) tensor are included in the reduction. Setting include_self=True could, for example, allow users to provide special reduction inits for the scatter_reduction operation. Otherwise, if include_self=False, indices scattered to are treated as if they were filled with reduction inits.

In the snippet below, we illustrate how the behavior of scatter_reduce in 1.11.0 can be achieved with the function released in 1.12.0.

Example:

>>> src = torch.arange(6, dtype=torch.float).reshape(3, 2)
>>> index = torch.tensor([[0, 2], [1, 1], [0, 0]])
>>> dim = 1
>>> output_size = 4
>>> reduce = "prod"

1.11

>>> torch.scatter_reduce(src, dim, index, reduce, output_size=output_size)
`tensor([[ 0., 1., 1., 1.],
        [ 1., 6., 1., 1.],
        [20., 1., 1., 1.]])`

1.12

>>> output_shape = list(src.shape)
>>> output_shape[dim] = output_size
# reduction init for prod is 1
# filling the output with 1 is only necessary if the user wants to preserve the behavior in 1.11
# where indices not scattered to are filled with reduction inits
>>> output = src.new_empty(output_shape).fill_(1)
>>> output.scatter_reduce_(dim, index, src, reduce)
`tensor([[ 0., 1., 1., 1.],
        [ 1., 6., 1., 1.],
        [20., 1., 1., 1.]])`

torch.nn

nn.GroupNorm: Report an error if num_channels is not divisible by num_groups (#74293)

Previously, nn.GroupNorm would error out during the forward pass if num_channels is not divisible by num_groups. Now, the error is thrown for this case during module construction instead.

1.11

m = torch.nn.GroupNorm(3, 7)
m(...)  # errors during forward pass

1.12

m = torch.nn.GroupNorm(3, 7)  # errors during construction

nn.Dropout2d: Return to 1.10 behavior: perform 1D channel-wise dropout for 3D inputs

In PyTorch 1.10 and older, passing a 3D input to nn.Dropout2D resulted in 1D channel-wise dropout behavior; i.e. such inputs were interpreted as having shape (N, C, L) with N = batch size and C = # channels and channel-wise dropout was performed along the second dimension.

1.10

x = torch.randn(2, 3, 4)
m = nn.Dropout2d(p=0.5)
out = m(x)  # input is assumed to be shape (N, C, L); dropout along the second dim.

With the introduction of no-batch-dim input support in 1.11, 3D inputs were reinterpreted as having shape (C, H, W); i.e. an input without a batch dimension, and dropout behavior was changed to drop along the first dimension. This was a silent breaking change.

1.11

x = torch.randn(2, 3, 4)
m = nn.Dropout2d(p=0.5)
out = m(x)  # input is assumed to be shape (C, H, W); dropout along the first dim.

The breaking change in 1.11 resulted in a lack of support for 1D channel-wise dropout behavior, so Dropout2d in PyTorch 1.12 returns to 1.10 behavior with a warning to give some time to adapt before the no-batch-dim interpretation goes back into effect.

1.12

x = torch.randn(2, 3, 4)
m = nn.Dropout2d(p=0.5)
out = m(x)  # input is assumed to be shape (N, C, L); dropout along the second dim.
            # throws a warning suggesting nn.Dropout1d for 1D channel-wise dropout.

If you want 1D channel-wise dropout behavior, please switch to use of the newly-added nn.Dropout1d module instead of nn.Dropout2d. If you want no-batch-dim input behavior, please note that while this is not supported in 1.12, a future release will reinstate the interpretation of 3D inputs to nn.Dropout2d as those without a batch dimension.

F.cosine_similarity: Improve numerical stability (#31378)

Previously, we first compute the inner product, then normalize. After this change, we first normalize, then compute inner product. This should be more numerically stable because it avoids losing precision in inner product for inputs with large norms. Because of this change, outputs may be different in some cases.

Composability

Functions in torch.ops.aten.{foo} no longer accept self as a kwarg

torch.ops.aten.{foo} objects are now instances of OpOverloadPacket (instead of a function) that have their __call__ method in Python, which means that you cannot pass self as a kwarg. You can pass it normally as a positional argument instead.

1.11

>>> torch.ops.aten.sin(self=torch.ones(2))
    tensor([0.8415, 0.8415])

1.12

# this now fails
>>> torch.ops.aten.sin(self=torch.ones(2))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: __call__() got multiple values for argument 'self'
# this works
>>> torch.ops.aten.sin(torch.ones(2))
tensor([0.8415, 0.8415])

torch_dispatch now traces individual op overloads instead of op overload packets (#72673)

torch.ops.aten.add actually corresponds to a bundle of functions from C++, corresponding to all over the overloads of add operator (specifically, add.Tensor, add.Scalar and add.out). Now, __torch_dispatch__ will directly take in an overload corresponding to a single aten function.

1.11

class MyTensor(torch.Tensor):
    ....
    def __torch_dispatch__(cls, func, types, args=(), kwargs=None):
        # Before, func refers to a "packet" of all overloads
        # for a given operator, e.g. "add"
        assert func == torch.ops.aten.add

1.12

class MyTensor(torch.Tensor):
    ....
    def __torch_dispatch__(cls, func, types, args=(), kwargs=None):
        # After, func refers to an individual operator overload,
        # e.g. "add.Tensor"
        assert func == torch.ops.aten.add.Tensor
        # you can recover the old behavior with func.overloadpacket
        assert func.overloadpacket == torch.ops.aten.add

Profiler

Disable forward-backward correlation (#72904)

The forward-backward correlation is no longer captured as to workaround a profile crash. This feature may be reenabled in a future release after the underlying issue is fixed.

with torch.profiler.profile() as p:
    loss = model(inputs)
    loss.backward()  # Invoke autograd

# The exported chrome trace will not have forward-backward flow events. (arrows)
p.export_chrome_trace(...)

Mobile

Remove support for bytecode version 3 (#57775)

The minimum supported bytecode version is being bumped from 3 to 4. We no longer support version 3 bytecode models because the bytecode version was bumped from 3 to 4 more than half a year ago, and there was code in operator loading that performed differently on one operator on the global bytecode version 3.

If the model is generated before Oct 5, 2020, please use the following lines to update the model to the latest version:

1.12

import torch
from torch.jit.mobile import _get_model_bytecode_version

old_model_path = "old_model.ptl"
new_model_path = "new_model.ptl"

# Load full jit model
jit_model = torch.jit.load(old_model_path)
# Save model for mobile 
jit_model._save_for_lite_interpreter(new_model_path)
# Verify the model can be loaded
mobile_m = _load_for_lite_interpreter(new_model_path)

# Get bytecode version from the new model
bytecode_version = _get_model_bytecode_version(new_model_path)
print(f"bytecode version is {bytecode_version}")

Remove redundant FSDP prefix and change default auto wrap policy name to avoid confusion (#76858, #73791)

FullyShardedDataParallel's optional param name ‘fsdp_auto_wrap_policy’ (1.11) changed to ‘auto_wrap_policy’ (1.12). ‘default_auto_wrap_policy’ (1.11) is changed to ‘size_based_auto_wrap_policy’ (1.12).

In 1.11, when wrapping a model with FSDP instead of:

model = MyModel()
wrapped_model = FullyShardedDataParallel(
    model,
    **fsdp_auto_wrap_policy**=functools.partial(
        default_auto_wrap_policy,
        min_num_params=0,  # wrap all modules
    )
   ...

1.12

model = MyModel()
wrapped_model = FullyShardedDataParallel(
    model,
   **auto_wrap_policy**=functools.partial(
        size_based_auto_wrap_policy,
        min_num_params=0,  # wrap all modules
    )
   ...

Quantization

TorchScript models exported prior to PyTorch 1.6 using quantized Linear, GRU and LSTM operators will no longer work (#72680, #72522)

TorchScript models created with PyTorch 1.5 or earlier and using the operators quantized::linear_prepack_legacy, linear_prepack_fp16_legacy, quantized::linear_unpack.legacy, or quantized::linear_unpack_fp16.legacy will no longer work and need to be re-exported. Please use PyTorch Quantization to quantize the Linear module instead.

ONNX

Infra (Releng)

• Bump minimum CMake version to 3.13 (#76312)

Deprecations

Python API

Deprecated torch.testing.make_non_contiguous (#72705)

torch.testing.make_non_contiguous is being deprecated and will be removed in a future release. Depending on the use case there are different replacement options: If you are using make_non_contiguous in the PyTorch test suite, you can use torch.testing._internal.common_utils.noncontiguous_like

1.11

a = torch.randn(1, 2, 3)
torch.testing.make_non_contiguous(a)

1.12

a = torch.randn(1, 2, 3)
torch.testing._internal.common_utils.noncontiguous_like(a)

If you are using make_non_contiguous in combination with a creation function to create a noncontiguous tensor with random values, you can use make_tensor.

1.11

a = torch.randn(1, 2, 3)
torch.testing.make_non_contiguous(a)

1.12

torch.testing.make_tensor(..., noncontiguous=True)

If you are using make_non_contiguous with a specific tensor, you can use torch.repeat_interleave

1.11

a = torch.tensor([[1., 2.], [1., 2.]])
torch.testing.make_non_contiguous(a)

1.12

a = torch.tensor([[1., 2.], [1., 2.]])
torch.repeat_interleave(input, 2, dim=-1)[..., ::2]

Build

LinAlg

Deprecate torch.lu (#73804)

torch.lu() is deprecated in favor of torch.linalg.lu_factor() and torch.linalg.lu_factor_ex(). torch.lu() will be removed in a future PyTorch release. If you were previously using get_infos=False (this is the default), you should use torch.linalg.lu_factor instead:

1.11

LU, pivots = torch.lu(A, compute_pivots) 

1.12

LU, pivots = torch.linalg.lu_factor(A, compute_pivots) 

If you were previously using get_infos=True you should use torch.linalg.lu_factor_ex:

1.11

LU, pivots, info = torch.lu(A, compute_pivots, get_infos=True)

1.12

LU, pivots, info = torch.linalg.lu_factor_ex(A, compute_pivots) 

Deprecate torch.lu_solve (#73806)

torch.lu_solve() is deprecated in favor of torch.linalg.lu_solve(). torch.lu_solve() will be removed in a future PyTorch release.

1.11

X = torch.lu_solve(B, LU, pivots)

1.12

X = torch.linalg.lu_solve(LU, pivots, B) 

Remove deprecated torch.solve (#70986)

torch.solve which was deprecated in a previous release is now being removed. You should use torch.linalg.solve. instead. Note that torch.linalg.solve has its arguments reversed and does not return the LU factorization. To get the LU factorization see torch.lu, which can be used with torch.lu_solve or torch.lu_unpack.

1.11

X = torch.solve(B, A).solution

1.12

X = torch.linalg.solve(A, B)

torch.nn

nn.Module: Deprecate positional args for state_dict() (#72780)

state_dict can currently be called in two ways: destination, prefix, and keep_vars can be passed as positional arguments, or as kwargs. The ability to do the former is being deprecated and will be removed in a future release. You should pass the arguments in as kwargs only.

Composability

Deprecated __torch_function__ as instance method for more functions (#74829)

__torch_function__ should be defined as a class method. Defining __torch_function__ as a plain method has already been previously deprecated for the functions handling __torch_function__ in Python. This change makes it so that that is also the case for functions that handle __torch_function__ in c++.

1.11

class Bad():
    def __torch_function__(self, *args, **kwargs):
        pass
t = Bad()
torch.abs(t)

1.12

class Good():
    @classmethod
    def __torch_function__(cls, *args, **kwargs):
        pass
t = Good()
torch.abs(t)

Quantization

Deprecate torch.jit.quantized (#72690)

Instead of using functions defined in torch.jit.quantized, please use PyTorch Quantization to dynamically quantize Linear/RNNCell/LSTMCell/GRUCell/LSTM modules. It’s both supported in Eager Mode Quantization and FX Graph Mode Quantization

1.11

>> torch.jit.quantized.QuantizedLSTMCell(...)

1.12

>> torch.jit.quantized.QuantizedLSTMCell(...)
   "torch.jit.QuantizedLSTMCell is deprecated and will be removed in an upcoming
    PyTorch release. Please use the torch.nn.quantized.dynamic.LSTMCell instead."

Infra (Releng)

• Removed CUDA 11.1 binary builds (#73376)
• Removed CUDA 11.5 binary builds (#76257)

New features

Python API

• Added new device mps that can be used to leverage GPU acceleration on macOS platform with Apple Native Silicon (M1) or discrete AMD GPUs. (blogpost with details)
• Added torch.special.log_ndtr (#74795)
• Added torch.distributions.transforms.{SoftplusTransform,CumulativeDistributionTransform} (#52300, #72495)
• Promoted torch.testing to stable (#73348)
• Added maximize flag for optim.Adadelta(#75330)

Build

• Distributed torchgen as part of PyTorch package (#76306)
• Added BUILD_LAZY_CUDA_LINALG option (#73447)
• Introduced an environment variable to change c10 log level (#71746)

Complex Numbers

• Added a new data-type torch.complex32 to help computing with complex datatype with lower memory usage at the cost of lower precision. Note that this is an experimental feature (#78245) and the major focus in this release was to support operators under torch.fft on CUDA. Besides those operators we have added support and testing for following limited set of ops (NOTE: few operators are only supported on CUDA): Tensor.copy_, torch.complex, torch.testing.make_tensor, cat, Tensor.fill_, Tensor.item, torch.atleast_1d, torch.atleast_2d, torch.atleast_3d, torch.dsplit, torch.vsplit, torch.hsplit, torch.hstack, torch.dstack, torch.vstack, Tensor.conj, torch.add, torch.sub, torch.mul, torch.sub, torch.div, torch.view, torch.view_as, torch.real, torch.imag, torch.neg, Tensor.__getitem__, torch.sum, torch.prod, torch.abs, torch.sgn, torch.exp, torch.log, torch.eq, torch.masked_fill, torch.index_put, torch.rand, torch.randn, torch.full, torch.empty, torch.ones, torch.zeros, torch.block_diag, Tensor.chunk, Tensor.clone, Tensor.contiguous, torch.diag_embed, torch.diagonal, torch.as_strided, torch.column_stack, Tensor.T, Tensor.H, Tensor.mT, Tensor.mH, Tensor.narrow, torch.isfinite, torch.isinf, torch.isreal, torch.flatten, Tensor.chalf, torch.empty_like, torch.movedim ( #73847, #74667, #74854, #75010,#75156, #75311, #75498, #76132, #76158, #75592, #76615, #77179, #77339, #77446, #77483, #77479, #77192, #76724, #77404).
• Operators in torch.fft now support tensors with torch.complex32 dtype (CUDA only) (#74857).
• torch.complex32 tensor now participate in type-promotion (#76893)
• Added torch.chalf alias for torch.complex32 and Tensor.chalf method (#75320).
• Added proper print support for torch.chalf tensors (#76614).
• Added support for complex convolution (data-types supported: torch.complex32, torch.complex64, torch.complex128)
  • torch.nn.functional.conv1d and torch.nn.Conv1d (#75310)
  • torch.nn.functional.conv2d and torch.nn.Conv2d (#75412)
  • torch.nn.functional.conv3d and torch.nn.Conv3d (#75581)

LinAlg

• Added torch.linalg.ldl_factor_ex and torch.linalg.ldl_solve (#69828)
• Added linalg.vander (#76303)
• Added linalg.lu (#67833)
• Added linalg.lu_solve (#72935)

Meta API

• Added meta tensor kernels for the following operators:
  • mse_loss (#72294), amax (#72124), normal (#70089) squeeze() + unsqueeze (#73440), unfold (#75717), clamp_min/max, (#76926), index_copy (#67329), linalg_cross (#72413), amin (#73581)
• Enabled the ability to register Python decompositions for operators as meta kernels, get meta support for where and huber_loss (#77353)
• Registered meta functions through Python for dot/group_norm/instance_norm/var_mean/index_reduce/matmul/bernoulli/adaptive_avg_pool (#77499) index_select/abs/min/max (#76916), reflection_pad2d (#77681), square (#77682), log_sigmoid_forward (#77739), several more ops (#77362)

torch.nn

• nn.Dropout1d: New module for 1D channel-wise dropout (#79545)
• nn.Module: Public API for stateless / functional module computation (#75834)
• nn.Module: Support for hooks that run after state dict loading (#76823, #77392)
• Added support for tensor subclasses as parameters (#73459, #77655)

torch.fx

• Core
  • Allowed Tracer to record usages of Buffers (#73612)
  • Introduced experimental MetaTensorTracer (#76003)
  • Introduced Tracer the ability to trace different forward functions (#77502)

Composability

• Many features, improvements and fixes to Python tensor subclasses based on __torch_function__ and __torch_dispatch__
  • Added __torch_function__ mode, which allows you to override the meaning of all __torch_function__ overrideable functions within a dynamic scope. (#75154)
  • Added enable_torch_dispatch_mode, which allows nesting of different __torch_dispatch__ modes. (#75965)
  • Added a default implementation of __torch_dispatch__ (#73684)
  • Added support super().__torch_dispatch__ with arguments list (#74509, #74720)
  • Miscellaneous __torch_function__ fixes (#75484, #75110)
  • Added __torch_function__ override protocol supporting to some factory functions (#75639)
  • Fixed propagation of warnings when using __torch_dispatch__. (#74357)
  • Removed spurious warning when using disabled torch function (#75826)
  • Added the ability to snapshot TLS for “has-a” use cases of __torch_dispatch__ (#72623, #74577)
  • Fixed serialization and deep copying for wrapper subclasses (#73078)
  • Allowed is_contiguous() to be overridden in __torch_dispatch__ (#77906)
• Added a “functionalization” program transform, that can be used to remove mutation + aliasing ops from PyTorch programs, while maintaining program semantics. Currently while most of the logic for the pass lives in core, the pass is exposed as an API through functorch. You can run it with functorch.experimental.functionalize(). Example usages can be found here. (#75913, #76083, #76084, #73442, #77285, #73441, #75302, #75818, #75819, #76125, #76318, #77358)
• Added a new torch.library API to allow users to override kernels for existing C++ ops through Python (#75905, #76892)
• Allowed creating new libraries and defining new operators from Python (#76250, #77690)
• Added experimental API’s for registering and looking up Python decompositions for many aten operators: from torch._decomp import register_decomposition, get_decompositions. (#76311, #76814)
  • Many decompositions have also been added to this table (#76621, #77329, #77219, #76633, #76855, #76714, #76763, #77473, #77807, #77500)

Sparse

• Added factory functions for sparse CSC, BSR, and BSC tensors (#76634, #76623, #75946, #75961, #75831, #76651)
• Added ccol_indices and row_indices methods for CSC and BSC tensors. (#77503)
• Added to_sparse_csc with support for 2D Strided and 2D CSC input (#77521)
• Added to_sparse_bsr with support for 2D CSR input (#77366)
• Added index_reduce (#76997, #75981, #76296)

CUDA

• Add Jiterator support when dtype is complex for sigmoid, exp, sqrt, rsqrt, log, log10, log2, addcmul, abs, addcdiv, sgn, neg , logical_and, angle(#73643, #73776, #73781, #74160, #74161, #74533, #74455, #74827, #74814, #74863, #75123, #76692)
• Add Jiterator support when dtype is complex for the backward of sigmoid and tanh (#76289, #74948)
• Add Jiterator support for kaiser_window , prod (#73734, #75231)
• Enable simple reductions with Jiterator (#75231)
• Updated to cuDNN v8 API with cuDNN benchmark, convolution bwd / transposed convolution fwd, bfloat16, conv-bias-activation fusion (#60755)
• Added Python Interface for Jiterator (#76394)
• Added Jiterator with Python Registration (#77121)
• Prepared Jiterator code template for multiple outputs (#77902)
• For CUDA graphs, added torch.cuda.is_current_stream_capturing (#77789)

Vulkan

• Added Vulkan support for Gated Recurrent Units (torch.nn.GRU) (#72692, #73599)
• Added Vulkan support for the linear interpolation op (torch.lerp) (#76544)

Profiler

• Added support both global (experimental) and thread local profiling (#75525, #76078, #76239)

Mobile

• Added support for different memory formats of Tensors in NNC (#72873)
• Upgraded mobile model bytecode to V9 and provide backporting to previous versions (#71662)

Distributed

• ShardedTensor and tensor parallel
  • This is a prototyping effort which consists of having a new class to represent how one torch.tensor is being sharded across multiple GPUs or hosts and a high level APIs for users to specify how to shard, enabling basic tensor ops for ShardedTensor and enabling optimizer for ShardedTensor. In addition, we have added PartialTensor, ReplicatedTensor and checkpoint with ShardedTensor (#63997, #65511, #65671, #65855, #66012, #66351, #66464, #66603, #66604, #67057, #67188, #67199, #67799, #68021, #68096, #68607, #68771, #68786, #68806, #69226, #69493, #69569, #69725, #69874, #69945, #69946, #70145, #70228, #70266, #70331, #70476, #71445, #72062, #72130, #73309, #76360, #76477, #72733, #73392, #76199, #75374, #71624, #74040, #73529, #74941, #73703, #75712, #73873, #75991, #75844, #76824, #76897, #77185, #77191, #76758, #77209, #77214, #77367, #77580, #77626, #77800, #77707, #78056)
    • Design proposal: ShardedTensor and Sharding APIs. Example for a Megatron-LM style tensor parallel.
• FullyShardedDataParallel
  • Added FlatParameter to track the information of a flat parameter (#69241)
  • Enabled summon_full_params for FSDP. (#71225)
  • Added no_sync() context manager (#72446)
  • Implemented apply() (#72925)
  • Implemented local_state_dict and load_local_state_dict (#73300)
  • Implemented full_state_dict (#73324)
  • Implemented clip_grad_norm for FSDP (#73405)
  • Added grad accumulation without no_sync() (#73535)
  • Added full_optim_state_dict (#74215)
  • Implemented reshard_flatten_tensor (#75192)
  • Added scatter_full_optim_state_dict() (#75517)
  • Implemented sharded_state_dict and load_sharded_state_dict (#77356)
  • Enabled mixed precision in FSDP (#75024)
  • Changed to allow specification of modules to ignore when wrapping with FSDP(#75431)
  • Added FullStateDictConfig to allow full state dict checkpoint with rank0 only and CPU offload (#75908)
  • Added validation to ensure FSDP units execute consistently across ranks (#75902)
  • Added support for initialization of modules on meta device (#75880)
  • Added support for no sharding config for DDP-style parallelism (#76736)
  • Changed to allow FSDP to specify device sharded wrapped module should be placed on (#77321)
  • Enabled FSDP parameter sync (#77492)
  • Made sharding strategy configurable and support zero2 algorithm (#73819)
  • Added a shard aware grad scaler for FSDP+MixedPrecision (#76918)
  • Enabled FSDP full state dict to work for non root FSDP modules via post load hooks (#76912)
  • Added always_wrap policy (#73687)
  • Provided an auto wrap policy for common transformer models (#76455)
• DistributedDataParallel
  • Added support for hierarchical model averaging (#73285)
• torch.distributed.rpc
  • Changed to allow for optional world_size argument in init_rpc (#73372)
  • Changed to allow newly joined ranks to communicate with existing ranks (#73373)
  • Changed to allow existing ranks to communicate with newly joined ranks (#74035)
  • Added graceful shutdown for dynamic RPC members (#74561)

JIT/TorchScript

• Added autocasting of values from fp32 to lower precision floats in torch.jit.freeze (#74178)
• torch.jit.set_fusion_strategy is now a public API, allowing one to set if they want fusion based on static or dynamic tensor sizes (#72639)
• Added support for compiling tensor.__getitem__() (#73952)
• TorchScript uses a fuser to combine multiple operator calls into a single kernel. In 1.12 the default fuser for NVIDIA GPUs is switched to NVFuser, which supports a wider range of operators and has demonstrated improved throughput compared to NNC, the previous fuser. (#74361, #77010, #77395, #72127, #73627, #75235, #75539, #75558, #75646, #76226, #76459, #76604, #76563, #77001, #77017, #77471, #77777, #77884, #76790, #76343, #76605, #76769, #77158, #74339)
• Added option to save extra files in torch.jit.save_jit_module_to_flatbuffer (#77870)

Quantization

• Added oneDNN quantization backend (#69820)
• Added oneDNN quant backend (#74137)

ONNX

• Added support to exporting additional ops:
  • Cross, Cdist and Pairwise Distance (#75278)
  • bucketize (#74856)
  • pixel unshuffle (#72499)
  • embedding_renorm (#72738)
  • aminmax (#75714)
  • amax and amin (#75268)
  • grid_sample (#76159)
• Added support to exporting quantized models (#72986, #73102, #75697, #76002, #76055, #73336,#77393, #75920, #75921)
• Added support to optional type. See tests in PR for examples. (#73284)
• Added support to ATen fallback for non-Caffe2 implementations of ONNX (#74759, #75468, #74680, #73954)

Infra (Releng)

• Added support for ROCm 5.0 (#72895)
• Added LibTorch builds for ROCm (#57506)
• Added support for CUDA 11.6 (#75518)

Improvements

Python API

• Improved numerical stability of torch.distributions.wishart.Wishart (#72993)
• Added mode property to torch.distributions.Distribution (#76690)
• Added foreach flag for torch.optim.{Adadelta, Adagrad, Adamax, Adam, ASGD, NAdam, RAdamSGD, Rmsprop, Rprop, AdamW} (#69980, #69981, #69982, #70295, #70481, #70229, #70230, #70231, #70482, #70483, #70484)
• Added out variant for torch.softmax and torch.log_softmax (#75833)
• Added handling for r=0 case for torch.combinations (#70270)
• Added XPU support for torch.autocast (#75250)
• Added support for Tensor source for .set_(storage, offset, size, strides) (#77007)
• Changed to register torch.return_types.* as pytree nodes (#75915)
• Added typing for torch.return_type (#74199)
• Set correct module for APIs in the torch module (#75801)
• Improved NotImplementedError verbosity for torch.distributions.kl_divergence (#72845)
• Added maximize flag to torch.optim.Adagrad (#75968)
• optim.{Adagrad, Adam, Adamax, AdamW, RAdam}: Updated step in functional optimizers and pass state_steps instead of state (#71333)
• Improved torch.lerp numerical precision by doing intermediate math in opmath_t (#76062)
• Changed to alias torch.finfo.tiny to torch.finfo.smallest_normal (#76292)

C++ API

• Added catch for overflows in calculating storage byte size for col2im (#73719)
• Implemented center padding for stft (#73432)

Autograd

• Added forward AD support for torch.{atan2, dist, logsumexp, log_softmax, norm, polar, put softmax} (#73741, #74205, #75027, #75326, #77421)
• Added forward AD support for torch.nn.functional.{cross_entropy, pairwise_dist, nll_loss, normalize} (#73741, #74205)
• Added forward AD support for torch.cholesky_inverse (#75033)
• Added forward AD and forward-over-reverse support for FFTs (#75326)
• Added forward AD support for torch.nn.functional.{embedding,prelu, bilinear, rrelu, logsigmoid} (#77421)
• Added forward AD support for torch.nn.BCELoss (#77755)
• Added forward AD support for Tensor.__rsub__ (#75326)
• Added forward AD support for torch.clamp when bounds are tensors (#74042)
• Added forward AD support for torch.nn.functional.{dropout, glu}(#75288, #77186)
• Added forward-over-reverse for torch.nn.functional.{leaky_relu, glu, elu, selu, celu} (#75294, #77309, #75297)
• Improved forward and backward derivative torch.{linalg.cholesky, cholesky} (#76032)
• Improved forward and backward derivative of torch.linalg.qr (#76115)
• Added complex autograd support for torch.cholesky_inverse (#75033)
• Added double backward support for torch.nn.functional.binary_cross_entropy wrt target (#77416)
• Improved error message for torch.nn.functional.batch_norm when running_{mean,var} have forward grad defined (#73655)
• Improve error message when forward AD is not supported (#75105)
• Added forward AD and forward-over-reverse support for torch.nn.functional.max_unpool (#68625)
• Added autograd support for masked_softmax (#71502)

Build

• Fixed pybind deprecation warnings (#72376)
• Enabled win-arm64 (#72424)
• Moved magma utils to its own header (#73058)
• Turned on -Wsign-compare (#74996)
• Made all .pyi.in files exportable from torch/_C/ folder (#74962)
• Improved Jinja2 for docs/cpp build set to version 3.0 (#74718)
• Added CMake option for using static MKL libraries (#73069)
• CPU Kernel: Changed to use per-operator headers (#71137)
• CUDA Kernels: Changed to use per-operator headers (#71212)

Dataloader

• Added pin_memory_device to Dataloader to pin Tensor to the corresponding GPU device (#65402)

ForEach

• Improved numerical precision for ForEach L1 and L2 norm by using OpMathType tensor for intermediate results (#68107)

Meta API

• Changed to skip superfluous storage allocations while constructing meta tensors (#65331)

torch.nn

• Made nn.init.orthogonal_ no-op for empty input (#75553)
• nn.{Conv1d, Conv2d, Conv3d}: Added support for complex datatypes (#75310, #75412, #75581)
• nn.Conv2d: Added bfloat16 support for mkl-dnn backend (#55864)
• nn.Conv2d: Added support for channels last memory format on CPU for mkl-dnn backend, naive algorithm, and dilated algorithm (#55584, #68101, #70665)
• nn.EmbeddingBag: Added half precision support on CPU (#74844)
• nn.FractionalMaxPool*d: Added support 0s in out_size (#73634)
• nn.Module: Changed to throw error for non-dict inputs to load_state_dict() (#77197)
• nn.{PixelShuffle, PixelUnshuffle}: Added support for channels last memory format (#50573)
• nn.PReLU: Enabled fp32/bfloat16 forward and backward for mkl-dnn backend (#60427)
• F.elu: Improve numerical precision by using opmath and expm1 (#77062)
• F.{hardshrink, hardsigmoid, hardswish, logsigmoid, smooth_l1_loss, softplus, softshrink}, nn.{BatchNorm, GLU, Upsample}: Add bfloat16 support on CPU (#62558, #63134, #77496, #61944, #76935)

torch.fx

• FX/graph_drawer
  • Added args/kwargs and users (#73464)
  • Added skip_node_names_in_args option, default to True (#73815)
• Core
  • Refactor FX codegen into extensible Codegen object (#72566)
  • Modified replace_all_uses_with to allowing filtering of nodes to update(#73763)
  • Made immutable_list and immutable_dict work with pytrees (#73766)
  • Added Assert None concrete_args and improve error messages (#74662)
• In minimizer, made args work in the uru10x10_to_trt_eval script (#74707)
• For split_module, changed to return mapping of qualified names from split_module() (#73564)
• For shape propagation, made shapes and args/kwargs concrete for minimizer (#75291)

Sparse

• Added CUDA support for scatter_reduce (#74606,#74607)
• Added 2D Strided, 2D CSR, 2D CSC, 2D COO support to to_sparse_csr (#77521)
• Added ND Strided, 2D CSC support to to_dense (#74486, #77521)
• Added 2D CSC support to to_sparse (#73642, #77521)
• Added support for batched CSR to sparse_csr_tensor (#74542)
• Added support for __str__ for CSC, BSR, and BSC tensors (#77530, #76650)
• Updated transpose to return CSC when given CSR (#77615)
• Added support for CSR gradients for CSR tensors (#75435)
• Added CSC support to addmm, addmv, mm (#77615)
• Added autograd for CSR inputs to torch.sparse.sampled_addmm (#68084)
• Added autograd for CSR inputs to torch.sparse.addmm and torch.sparse.mm (#76591)
• Added Half/BFloat16 support for to_dense and coalesce methods. (#72397)
• Added CSR support to mul (#74266, #77177)
• Added CSR support to sum (#74766)
• Added BSR support to addmm, addmv, triangular_solve (#77255)
• Added batched CSR support to torch.sparse.sampled_addmm on CUDA (#77243)
• Added CSR support for torch.sparse.sampled_addmm on CPU (#76589)
• Added CSR support to torch.select (#76228)
• Added CSR support to Tensor.to (#76400)
• Added CSC support to torch.empty (#77508)
• Added CSC, BSR, BSC support to torch.clone (#77512)
• Added CSC, BSR, BSC support for copy_ (#77605)
• Added (Strided, CSR) input support to torch.mm (#73686)
• Added CSR support to torch.sparse.mm (#73075)
• Added (Strided, CSR, CSR) support to addmm on CPU (#73076)
• Added runtime beta support warning to CSR, CSC, BSR, BSC tensors (#75594, #75865)
• Added bool support to coalesce and to_dense (#74495)
• Added half support to sparse_mask (#76862)
• Added AMD Navi 21 support to coalesce (#73548)

AMD

• Enabled atomicAddNoRet() for all gfx targets. (#75451)
• Enabled miopen for RNNs with dropout. (#75429)
• Used ncclAllToAll for ROCm (#75128)
• Navi21 Enablement: fix TI num_threads for ROCm, Depthwise kernels, Embedding kernels, Normalization kernels, Softmax kernels, Tensor kernels, Index, Repeat and Sort kernels, Range and Multinomial Kernels (#69942, #72682, #72809, #73543, #73545, #73546, #73549, #73550)
• Added ROCm version api within CMake (#69481)
• Enabled sort operator BF16 support (#72854)
• Enabled HIP IPC (#74383)
• Enabled topk operator for bfloat16 dtype (#71913)
• Added HIP_HOME/include.lib in cpp_extensions (#75548)

CUDA

• PyTorch: added support to NVTX range_start and range_end (#70030)
• Show friendly error message when forgetting init in torch.cuda (#72404)
• PyTorch GPU Allocator: better use of blocks with rounding of allocation sizes (#74213)
• CUDACachingAlloc/GPUInference: implemented garbage collection without GPU sync (#74261)
• CUBLAS/TF32: added environment variable to allow override of allow_tf32_cublas (#77114)

Intel

• Bfloat16
  • Added BFloat16 support for torch.{nn.PReLU, nn.Upsample,nn.GLU, randperm, multinomial, poisson, nn.ELU, nn.SELU, nn.CELU, nn.LogSigmoid, nn.Hardsigmoid, nn.Hardshrink, nn.Softshrink, nn.Hardswish, nn.Softplus, nn.SmoothL1Loss, histc, atan2, logcumsumexp, diag, fmod, cumsum, cumprod, nn.utils.weight_norm , nn.BatchNorm2d} and allow autocast enabled (#63634, #58297, #61944, #63215 , #62546, #63134, #72694, #61897, #73845, #74410, #68725)
    • Improved torch.nn.functional.log_softmax on CPU when dim != -1 on both float32 and bfloat16 (#64726)
    • Improved torch.nn.functional.layer_norm bfloat16 performance on CPU (#71376)
    • Improved autocast cpu documentation (#68567)
• Channels last
  • Add channels-last support for torch.nn.{conv2D(kernel slow_conv_dilated2d and thnn_conv2d, mkldnn as backend), GroupNorm, PixelShuffle, PixelUnshuffle}(#70665, #68101, #55584, #50573, #555864)
• OneDNN
  • Upgraded oneDNN to v2.6.0, (#75398)
  • Added JIT graph fuser for oneDNN Graph API (v0.5) (#76622)
• Quantization
  • Improve {qcat_nhwc, qupsample_bilinear2d, qupsample_nearest2d, qbatch_norm2d, qmax_pool2d, qavg_pool2d} performance on multi-core (#69667, #69601, #69600, #69599, #69598, #69517)
  • Add oneDNN as backend for quantization (#69820)
• Improved torch{norm,argmax,argmin, scatter, gather} performance on CPU (#64479, #64478)
• Improved torch.nn.functional{log_softmax``, softmax} performance on CPU (#73953)
• Expanded graph rewrite to handle conv_transpose3d (#76888)
• Expanded coverage of convolution folding in conv→mul→add→bn (#75724)
• Added MKLDNN support for PReLU (#60427)

Composability

• Added torch.nn.init to list of functions overridable by __torch_function__ (#76014)
• Relaxed dtype restrictions on torch.Tensor(#73850)

Profiler

• Enabled iteration tracking for kineto (#72292)
• Added support for input sequence ID tracking for NVTX profiler (#70264)
• Re-enabled user-annotations in PyTorch (#75601)
• Added support to configure Kineto CUPTI profiler from PyTorch profiler interface (#75616)

Vulkan

• Added an interface to obtain execution time data for GPU shader kernels when executing Vulkan operators (#75829)

Mobile

• Improved Android instrumentation test and update README (#72736)
• Improved unsupported scalar type error message for Android (#74660)

JIT/TorchScript

• torch.jit.trace now treats tensor.numel() as aten::numel, instead of a constant value (#74081)
• When printing out the types of a JIT Dict, with a tuple key, we now print out the types of the tuple if it is simple (#76164)
• Added support for basic ops support for complex numbers in JIT, We now support op(complex, Tensor) for the following: add (+), mul (*), eq (==), ne (!=), sub (-), div (/) (#73286)
• TorchScript now preserves the original exception message when rethrowing a Python-based exception (#77093)
• Modified the conditions for conv folding in torch.jit.freeze to allow for folding arguments that can be promoted to floating point (eg integer tensor arguments) (#73278)
• Reduced size of JIT debug.pkl files by only storing unique traces (#76688)
• torch.jit.save and torch.jit.load are now supported for meta tensors ( aka torch.Tensor(device="meta")) (#73435)

Architecture Optimization

• Added default symmetric qconfig for QNNPACK (#74396)

Quantization

• Core (Quantized Tensor, Operator, Modules)
  • Added QAT fused Linear-Bn1d (#72431, #72796)
  • Added 4 bit support for embedding quantized module (re-land PR 69769) (#72276)
  • Enabled slicing on per-channel quantized tensors (support is limited to the a contiguous sliced tensor) and corresponding test case (#71269)
  • Added qint32 quantization support (#72472)
  • Added explicit entries for for functional and module conv and linear support into get_default_qconfig_dict&get_default_qat_qconfig_dict (#73528)
  • Added default symmetric QAT qconfig for QNNPACK (#74507)
  • Added Quantized Matmul Op (Naive Implementation) (#71783)
  • Added Quantized Softmax Op (Naive Implementation) (#75415)
  • Using QNNPACK in Quantized Softmax Op (#75799)
• Eager Mode Quantization
  • Added 4 bit support for eager mode quantization flow (#72277)
• FX Graph Mode Quantization
  • Added workflow support for torch.matmul quantization (#72444)
  • Added support conv1d and its fusion variants in QAT (#74506)
  • Decoupled prepare_*fx from training/eval modes (#75401)
  • Added quantized Softmax workflow integration (#75106)
  • Renamed default_affine_fixed_qparams_observer and default_symmetric_fixed_qparams_observer (#76637)

ONNX

• Updated default opset_version to 13. The previous default was 9. To get the old behavior, just specify opset_version=9 when calling torch.onnx.export. Going forward we plan to update the default regularly to "latest as of 18 months ago". (#73898)
• De-duplicated initializers to reduce ONNX model size for shared parameters (#69547, #74247)
• Changed to capture annotated attributes for local function (#72883)
• Improve error and warning messages (#71342, #73255, #73770, #73265)
• Added support to exporting torch.minimum with different dtype combinations (#76022)
• Improved Expand shape inference (#72985)
• Added broadcast to matmul shape inference (#72990)
• Rewrote linspace symbolic to improve numerical stability (#73610)
• Enabled topk export with non-int64 k (#73761)
• Enabled numel tracing (#74081)
• Added constant folding for onnx::ReduceProd (#74082)
• Added support to equality checks on devices (#77203)
• Added support to dynamic dimensions in Squeeze and Unsqueeze (#73104)

torch.package

• Added Python Version to Torch.Package metadata (#74610)
• Added utility for determining where bad modules may come from (#74998)

Distributed

• torch.distributed
  • Refactored TORCH_DISTRIBUTED_DEBUG implementation (#73166)
  • Set default value of TCPStore world_size to None in pybind definition (#77277)
  • Added orthogonalization with QR factorization (#72043)
  • Added pickling support for WorkerInfo (#73371)
  • Added support for RRefs that contain threading.Thread (#74462)
  • Added check for mismatch in number of parameters in verify_params_across_processes (#74113)
  • Added support for backend to register reducer timer (#71700)
  • Made ProcessGroupNCCL load torch_ucc.so when TORCH_UCC_LIBRARY_PATH is set (#69552)
  • Added support for non-contiguous inputs for nn.functional.all_gather/reducescatter/gather (#75276)
  • Added the use of batched operations for PowerSGD (#76041)
  • Changed to create UCC ProcessGroup when ucc_lib available (#69564)
  • Changed to generalize param verification and broadcast (#76374)
  • Changed to use a more reliable signaling mechanism to stop TCPStore background threads (#76973)
  • Added support to disabling post-local gradient sync (#76723)
  • Removed call into Python API without GIL being held in c10d (#72928)
• FullyShardedDataParallel
  • Fixed summon_full_params when not sharded (#72572)
  • Fixed 0-dim tensor optim state device (#75243)
  • Fixed the synchronization of all_gather stream in summon_full_params (#73314)
  • Added state_dict() save/reload in parity test (#73366)
  • Changed to use unflatten_parameter in _summon_full_parameters (#72467)
  • Changed to use summon_full_params in get_full_params (#73242)
  • Added generic arguments for state_dict (#73323)
  • Added generic argument forward for load_local_state_dict (#73325)
  • Made summon_full_params a public method (#73116)
  • Generalized fsdp_modules() (#73553)
  • Introduced a utility API to allow users easily to set state_dict_type (#73716)
  • Added an option to summon on rank 0 only in summon_full_params (#73903)
  • Enabled offload full params to CPU in summon_full_params (#73904)
  • Removed _lazy_init() in rebuild full params (#74263)
  • Changed to override named_parameters() for clean names in summon_full_params() (#74333)
  • Changed to strip FSDP info in summon_full_params context, similar to named_params in named_buffers (#74517)
  • Change to use param name as key in full_optim_state_dict (#74879)
  • Enabled re-key between param names/IDs for full_optim_state_dict (#74912)
  • Changed to register state_dict hooks for FlatParamsWrapper even if params_list is empty (#74860)
  • Made apply_to_tensors support OrderedDict type (#75560)
  • Added rank0_only to full_optim_state_dict() (#75516)
  • Made summon_full_params a static method (#75423)
  • Added support for PackedSequence type for apply_for_tensors (#76265)
  • Made mixed precision API configurable (#76423)
  • Validated exec order using compute_device (#76664)
  • Improved dict inversion in _get_param_name_to_param to be faster(#76665)
  • Changed to ignore params if not in Optim state dict (#76671)
  • Changed to include buffers in ignored_modules (#76784)
  • Moved param/buffer name computation to constructor for ignored_modules (#76994)
  • Changed to not clone buffers and ensure that we offload buffers to CPU if specified (#77000)
  • Profiling range for FSDP.forward (#76899)
  • Disabled the default behavior of moving CPU module to GPU (#77720)
  • Fixed _get_param_to_unflat_param_names() for shared params (#75430)
• ShardedTensor (prototype)
  • Changed to use absolute imports for ShardMetadata instead (#73678)
  • Fixed the metadata error in init_from_local_shards with deepcopy (#73400)
  • Fixed view op and matrix ops unit test (#77706)
• torch.distributed.rpc
  • Improved logging from 'unknown destination worker' (#75811)
  • Improved logging for store.wait error (#76548)
  • Added support for RPC Meta device (#76882)
  • Changed to k

Note truncated.

PyTorch 1.11 Release Notes

• Highlights
• Backwards Incompatible Change
• New Features
• Improvements
• Performance
• Documentation

Highlights

We are excited to announce the release of PyTorch 1.11. This release is composed of over 3,300 commits since 1.10, made by 434 contributors. Along with 1.11, we are releasing beta versions of TorchData and functorch. We want to sincerely thank our community for continuously improving PyTorch.

• TorchData is a new library for common modular data loading primitives for easily constructing flexible and performant data pipelines. View it on GitHub.
• functorch, a library that adds composable function transforms to PyTorch, is now available in beta. View it on GitHub.
• Distributed Data Parallel (DDP) static graph optimizations available in stable.

You can check the blogpost that shows the new features here.

Backwards Incompatible changes

Python API

Fixed python deepcopy to correctly copy all attributes on Tensor objects (#65584)

This change ensures that the deepcopy operation on Tensor properly copies all the attributes (and not just the plain Tensor properties).

<p align="center"> <table align="center"> <tr><th>1.10.2</th><th>1.11.0</th></tr> <tr valign="top"> <td><sub><pre lang="python"> a = torch.rand(2) a.foo = 3 torch.save(a, "bar") b = torch.load("bar") print(b.foo)

Raise AttributeError: "Tensor" object has no attribute "foo"

  </pre></sub></td>
  <td><sub><pre lang="python">

a = torch.rand(2) a.foo = 3 torch.save(a, "bar") b = torch.load("bar") print(b.foo)

3

  </pre></sub></td>
</tr>

</table> </p>

steps argument is no longer optional in torch.linspace and torch.logspace

This argument used to default to 100 in PyTorch 1.10.2, but was deprecated (previously you would see a deprecation warning if you didn’t explicitly pass in steps). In PyTorch 1.11, it is not longer optional.

<p align="center"> <table align="center"> <tr><th>1.10.2</th><th>1.11.0</th></tr> <tr valign="top"> <td><sub><pre lang="python">

Works, but raises a deprecation warning

Steps defaults to 100

a = torch.linspace(1, 10)

UserWarning: Not providing a value for linspace's steps is deprecated

and will throw a runtime error in a future release.

This warning will appear only once per process.

(Triggered internally at ../aten/src/ATen/native/RangeFactories.cpp:19

  </pre></sub></td>
  <td><sub><pre lang="python">

In 1.11, you must specify steps

a = torch.linspace(1, 10, steps=100) </pre></sub></td> </tr> </table> </p>

Remove torch.hub.import_module function that was mistakenly public (#67990)

This function is not intended for public use. If you have existing code that relies on it, you can find an equivalent function at torch.hub._import_module.

C++ API

We’ve cleaned up many of the headers in the C++ frontend to only include the subset of aten operators that they actually used (#68247, #68687, #68688, #68714, #68689, #68690, #68697, #68691, #68692, #68693, #69840)

When you #include a header from the C++ frontend, you can no longer assume that every aten operators are transitively included. You can work around this by directly adding #include <ATen/ATen.h> in your file, which will maintain the old behavior of including every aten operators.

Custom implementation for c10::List and c10::Dict move constructors have been removed (#69370)

The semantics have changed from "make the moved-from List/Dict empty" to "keep the moved-from List/Dict unchanged"

<p align="center"> <table align="center"> <tr><th>1.10.2</th><th>1.11.0</th></tr> <tr valign="top"> <td>_{<pre lang="cpp"> c10::List<string> list1({"3", "4"}); c10::List<string> list2(std::move(list1)); std::cout << list1.size() // 0 </pre>}</td> <td>_{<pre lang="cpp"> c10::List<string> list1({"3", "4"}); c10::List<string> list2(std::move(list1)); // calls copy ctr std::cout << list1.size() // 2 </pre>}</td> </tr> </table> </p>

CUDA

Removed THCeilDiv function and corresponding THC/THCDeviceUtils.cuh header (#65472)

As part of cleaning up TH from the codebase, the THCeilDiv function has been removed. Instead, please use at::ceil_div, and include the corresponding ATen/ceil_div.h header

Removed THCudaCheck (#66391)

You can replace it with C10_CUDA_CHECK, which has been available since at least PyTorch 1.4, so just replacing is enough even if you support older versions

Removed THCudaMalloc(), THCudaFree(), THCThrustAllocator.cuh (#65492)

If your extension is using THCThrustAllocator.cuh, please replace it with ATen/cuda/ThrustAllocator.h and corresponding APIs (see examples in this PR).

This PR also removes THCudaMalloc/THCudaFree calls. Please use c10::cuda::CUDACachingAllocator::raw_alloc(size)/raw_delete(ptr), or, preferably, switch to c10:cuda::CUDaCachingAllocator::allocate which manages deallocation. Caching allocator APIs are available since PyTorch 1.2, so just replacing it is enough even if you support older versions of PyTorch.

Build

Stopped building shared library for AOT Compiler, libaot_compiler.so (#66227)

Building aot_compiler.cpp as a separate library is not necessary, as it’s already included in libtorch.so. You can update your build system to only dynamically link libtorch.so.

Mobile

Make typing.Union type unsupported for mobile builds (#65556)

typing.Union support was added for TorchScript in 1.10. It was removed specifically for mobile due to its lack of use and increase in binary size of PyTorch for Mobile builds.

Distributed

torch.distributed.rpc: Final Removal of ProcessGroup RPC backend (#67363)

ProcessGroup RPC backend is deprecated. In 1.10, it threw an error to help users update their code, and, in 1.11, it is removed completely.

The backend type “PROCESS_GROUP” is now deprecated, e.g. torch.distributed.rpc.init_rpc("worker0", backend="PROCESS_GROUP", rank=0, world_size=1) and should be replaced with: torch.distributed.rpc.init_rpc("worker0", backend="TENSORPIPE", rank=0, world_size=1)

Quantization

Disabled the support for getitem in FX Graph Mode Quantization (#66647)

getitem used to be quantized in FX Graph Mode Quantization, and it is no longer quantized. This won’t break any models but could result in a slight difference in numerics.

<p align="center"> <table align="center"> <tr><th>1.10.2</th><th>1.11.0</th></tr> <tr valign="top"> <td><sub><pre lang="python"> from torch.ao.quantization.quantize_fx import convert_fx, prepare_fx class M(torch.nn.Module): def init(self): super().init() self.linear = torch.nn.Linear(5, 5) def forward(self, x): x = self.linear(x) y = torch.stack([x], 0) return y[0] m = M().eval() m = prepare_fx(m, {"": torch.ao.quantization.default_qconfig}) m = convert_fx(m) print(m)

prints

GraphModule(

(linear): QuantizedLinear(in_features=5, out_features=5,

scale=1.0, zero_point=0, qscheme=torch.per_tensor_affine)

)

def forward(self, x):

linear_input_scale_0 = self.linear_input_scale_0

linear_input_zero_point_0 = self.linear_input_zero_point_0

quantize_per_tensor = torch.quantize_per_tensor(x,

linear_input_scale_0, linear_input_zero_point_0, torch.quint8)

x = linear_input_scale_0 = linear_input_zero_point_0 = None

linear = self.linear(quantize_per_tensor)

quantize_per_tensor = None

stack = torch.stack([linear], 0); linear = None

getitem = stack[0]; stack = None

dequantize_2 = getitem.dequantize(); getitem = None

return getitem

  </pre></sub></td>
  <td><sub><pre lang="python">

from torch.ao.quantization.quantize_fx import convert_fx, prepare_fx class M(torch.nn.Module): def init(self): super().init() self.linear = torch.nn.Linear(5, 5) def forward(self, x): x = self.linear(x) y = torch.stack([x], 0) return y[0] m = M().eval() m = prepare_fx(m, {"": torch.ao.quantization.default_qconfig}) m = convert_fx(m) print(m)

prints

GraphModule(

(linear): QuantizedLinear(in_features=5, out_features=5, scale=1.0,

                zero_point=0, qscheme=torch.per_tensor_affine)

)

def forward(self, x):

linear_input_scale_0 = self.linear_input_scale_0

linear_input_zero_point_0 = self.linear_input_zero_point_0

quantize_per_tensor = torch.quantize_per_tensor(x, linear_input_scale_0,

                 linear_input_zero_point_0, torch.quint8)

x = linear_input_scale_0 = linear_input_zero_point_0 = None

linear = self.linear(quantize_per_tensor); quantize_per_tensor = None

stack = torch.stack([linear], 0); linear = None

dequantize_2 = stack.dequantize(); stack = None

getitem = dequantize_2[0]; dequantize_2 = None

return getitem

  </pre></sub></td>
</tr>

</table> </p>

Users should now use fuse_modules for PTQ fusion and fuse_modules_qat for QAT fusion (#69878, #71956)

There are two types of fusion supported by fuse_modules api: PTQ and QAT fusion. Previously we relied on module.training to decide which mode user wanted, but this was a misuse of the training attribute since that is not the intended purpose. This PR removes the dependency on module.training and uses separate APIs to make the fusion requested by the user explicit.

Previously, fuse_module used to support both cases and distinguished PTQ/QAT fusion based on module.training, but now fuse_module only supports the PTQ fusion. So, in the case when user wants to do QAT fusion, they need to change the call to fuse_modules_qat, instead of using fuse_modules, otherwise, they would silently get unwanted fusion results (PTQ fusion), or if the model is in training mode, it might result in error.

Note: Currently it is still enforced that if the model is in eval mode, only PTQ fusion can be used; if the model is in training mode, then only QAT fusion can be used. In the future this constraint will be relaxed.

<p align="center"> <table align="center"> <tr><th>1.10.2</th><th>1.11.0</th></tr> <tr valign="top"> <td>_{<pre lang="python"> import torch from torch.ao.quantization import fuse_modules class M(torch.nn.Module): def init(self): super().init() self.conv = torch.nn.Conv2d(3, 3, 3) self.bn = torch.nn.BatchNorm2d(3) def forward(self, x): return self.bn(self.conv(x)) m = M().train() m = fuse_modules(m, ["conv", "bn"]) print(type(m.conv)) m = M().eval() m = fuse_modules(m, ["conv", "bn"]) print(type(m.conv)) <class 'torch.nn.intrinsic.modules.fused.ConvBn2d'> <class 'torch.nn.modules.conv.Conv2d'> </pre>}</td> <td><sub><pre lang="python"> import torch from torch.ao.quantization import fuse_modules class M(torch.nn.Module): def init(self): super().init() self.conv = torch.nn.Conv2d(3, 3, 3) self.bn = torch.nn.BatchNorm2d(3) def forward(self, x): return self.bn(self.conv(x)) m = M().train()

For Quantization Aware Training, use fuse_modules_qat()

m = fuse_modules_qat(m, ["conv", "bn"]) print(type(m.conv)) m = M().eval() m = fuse_modules(m, ["conv", "bn"]) print(type(m.conv))

Result (doesn't change):

<class 'torch.nn.intrinsic.modules.fused.ConvBn2d'> <class 'torch.nn.modules.conv.Conv2d'> </pre></sub></td> </tr> </table> </p>

ONNX

Removed f arg from onnx.export_to_pretty_string (#69546)

The arg has always been ignored. Simply remove it from your code.

<p align="center"> <table align="center"> <tr><th>1.10.2</th><th>1.11.0</th></tr> <tr valign="top"> <td>_{<pre lang="python"> torch.onnx.export_to_pretty_string(model, inputs, "file_name") </pre>}</td> <td>_{<pre lang="python"> torch.onnx.export_to_pretty_string(model, inputs) </pre>}</td> </tr> </table> </p>

Removed use_external_data_format arg from onnx.export (#67809)

The arg has been deprecated and ignored since 1.10. The external data format is now used automatically if and only if the exported file would exceed protocol buffer’s file size limit. Simply remove it from your code.

<p align="center"> <table align="center"> <tr><th>1.10.2</th><th>1.11.0</th></tr> <tr valign="top"> <td>_{<pre lang="python"> torch.onnx.export(model, inputs, f_name, use_external_data_format=True) </pre>}</td> <td>_{<pre lang="python"> torch.onnx.export(model, inputs, f_name) </pre>}</td> </tr> </table> </p>

Removed example_outputs arg from torch.onnx.export (#67809)

The arg has been deprecated and ignored since 1.10. The provided model is instead executed once to produce example outputs. Simply remove it from your code.

<p align="center"> <table align="center"> <tr><th>1.10.2</th><th>1.11.0</th></tr> <tr valign="top"> <td>_{<pre lang="python"> torch.onnx.export(model, inputs, f_name, exaple_outputs=(foo,)) </pre>}</td> <td>_{<pre lang="python"> torch.onnx.export(model, inputs, f_name) </pre>}</td> </tr> </table> </p>

Removed enable_onnx_checker arg from onnx.export (#67276)

The arg has been deprecated and ignored since 1.10. The ONNX checker is always enabled. If it fails, onnx.CheckerError will be raised. Users can catch and ignore that exception.

<p align="center"> <table align="center"> <tr><th>1.10.2</th><th>1.11.0</th></tr> <tr valign="top"> <td>_{<pre lang="python"> torch.onnx.export(model, inputs, f_name, enable_onnx_checker=False) </pre>}</td> <td>_{<pre lang="python"> try: torch.onnx.export(model, inputs, f_name) except torch.onnx.CheckerError: pass # ignore error </pre>}</td> </tr> </table> </p>

Moved and renamed onnx.utils.ONNXCheckerError to onnx.CheckerError (#66644)

Previously the documentation was incorrect and stated ONNXCheckerError was in the onnx module, so this moves the class to the originally intended module and brings the code in line with the documentation. The new name is shorter and less redundant with the module name.

<p align="center"> <table align="center"> <tr><th>1.10.2</th><th>1.11.0</th></tr> <tr valign="top"> <td>_{<pre lang="python"> except torch.onnx.utils.ONNXCheckerError: </pre>}</td> <td><sub><pre lang="python"> except torch.onnx.CheckerError: </tr> </table> </p>

Removed _retain_param_name arg from onnx.export (#67276)

The arg has been deprecated and ignored since 1.10. Param names are now always retained. Simply remove it from your code. If you want to remove param names, you can do so by editing the exported ONNX model.

<p align="center"> <table align="center"> <tr><th>1.10.2</th><th>1.11.0</th></tr> <tr valign="top"> <td><sub><pre lang="python">

NOTE: No way to get same behavior as _retain_param_name=False.

torch.onnx.export(model, inputs, f_name, _retain_param_name=True) </pre></sub></td> <td><sub><pre lang="python"> torch.onnx.export(model, inputs, f_name) </tr> </table> </p>

Deprecations

Python API

Deprecated x.T on tensors of dimension other than 0 or 2 (#64180)

x.T only accepts tensors with 0 or 2 dimensions. Calling x.T on tensors with a different number of dimensions has been deprecated.

<p align="center"> <table align="center"> <tr><th>1.10.2</th><th>1.11.0</th></tr> <tr valign="top"> <td><sub><pre lang="python"> a = torch.ones(2, 3, 4) a.T.size()

torch.Size([4, 3, 2])

  </pre></sub></td>
  <td><sub><pre lang="python">

a = torch.ones(2, 3, 4) a.T.size()

UserWarning: The use of x.T on tensors of dimension other than 2

to reverse their shape is deprecated and it will throw an error in a future release.

Consider x.mT to transpose batches of matrices or x.permute(*torch.arange(x.ndim - 1, -1, -1))

to reverse the dimensions of a tensor. (Triggered internally at

aten/src/ATen/native/TensorShape.cpp:2386.)

torch.Size([4, 3, 2])

</tr>

</table> </p>

Quantization

torch.ao.quantization.QConfigDynamic is deprecated and going to be removed in next the release, please use torch.ao.quantization.QConfig instead (#69875, #69864)

<p align="center"> <table align="center"> <tr><th>1.10.2</th><th>1.11.0</th></tr> <tr valign="top"> <td>_{<pre lang="python"> qconfig = torch.ao.quantization.QConfigDynamic(...) </pre>}</td> <td><sub><pre lang="python"> qconfig = torch.ao.quantization.QConfig(...) </tr> </table> </p>

New features

Python API

• Added set_deterministic_debug_mode and get_deterministic_debug_mode (#67778, #66233)
• Added n-dimensional Hermitian FFT: torch.fft.ifftn and torch.fft.hfftn (#63890)
• Added Wishart distribution to torch.distributions (#70377)
• Preliminary support for the Python Array API standard has been added to the torch and torch.linalg modules. PyTorch implements over 90% of the operators defined by the Python Array API, including the torch.from_dlpack operation for improved DLPack support (#60627)
• Moved torch.testing from prototype to beta (#69668)

Autograd

• Added new torch.utils.checkpoint implementation that does not use reentrant autograd (can be toggled with the new use_reentrant flag) (#69508)
• Added batched_grad parameter to autograd.grad to allow batched gradient computation (#65564)
• Forward mode AD:
  • Added support for most ops (and many of their backwards as well) (#71026, #69956, #70355, #71901, #69908, #69884, #67837, #68566, #69661, #69384, #68631, #70468, #70460, #67820, #70460, #65546, #67043, #67268, #67837, #69727)
    • Check the following issue (#71117) to see the list of ops that do not yet support forward AD. Please comment there if you run into any ops that don’t support forward AD that you want prioritized or are missing from that list.
  • Added ctx.save_for_forward function to autograd.Function (#71569)
  • autograd.forward_ad.unpack_dual returns a named tuple instead of plain tuple (#68062, #68628)
• Linear algebra operation support:
  • Added forward AD support for torch.linalg.{eig, inverse, householder_product, qr} and torch.*_solve (#65546, #67043, #67268, #67837)
  • Added forward and backward AD support for torch.linalg.lstsq (#65054)
  • Added support for a wider range of inputs for linalg.pinv (#66092)

Build

• Added FlexiBLAS build support (#64815)
• Added IS_LINUX and IS_MACOS global vars for cpp extensions building (#69093)
• Added ARC for iOS CMake builds (#67884)
• Added support for IBM z14/15 SIMD (#66407)

Complex Numbers

• Added complex number support to Adagrad and Adadelta optimizers (#66671, #66587)

Dataloader

• TorchData library is going to provide modular data loading primitives for easily constructing flexible and performant data pipelines. Beta release will be provided after the release of PyTorch Core (https://github.com/pytorch/data)

LinAlg

• Added an experimental flag that allows specifying a preferred linear algebra library (see the docs here) (#67980)
• Added the linalg.matrix_exp operation (see the docs here) (#62715)
• Added the linalg.cross operation (see the docs here) (#63285)
• Added the linalg.diagonal operation, an alias for torch.diagonal (see the docs here) (#70599)
• Added the linalg.lu_factor operation (see the docs here) (#66933)

torch.nn

• Added torch.nn.utils.rnn.{unpack_sequence,unpad_sequence} functions (#66550)

Sparse

• Added torch.sparse.sampled_addmm for CSR Tensors on GPU (#68007)

CUDA

• The Jiterator - enables compiling rarely used CUDA kernels at runtime (#69439)
  • Low precision supported for jiterator (#70157) - enables runtime-compilation of ops on low precision tensors (half and bfloat16)
  • Enable cpu scalar arguments for jiterator (#69861) - enables passing cpu scalars as an argument to the jit-compiled kernels at runtime
  • The Cacherator (#71350) - caches the jit-compiled kernels on disk, so that they can be reused between different processes
  • Added complex support for Jiterator, port sinc to Jiterator (#71577)
  • Jiterates lcm, i0e, i1e, ndtri, efcx, digamma, trigamma, lgamma (#70663)
  • Jiterates exp2, erfc, erfinv and entr (#71295)
  • Fixes jiterator cache macro include + updates CUDA note with cache variables (#71452)
  • Jiterates polygamma (#71162)
• Added cuSPARSE descriptors and updated CSR addmm (#60838)
• Sparse CSR CUDA: added addmv_out (#61407)
• Added nvidia-smi memory and utilization as native Python API (#69104)

Vulkan

• Added Vulkan support for several torch operators:
  • torch.cat (support for concatenation along the height and channel dimensions for 4-D tensors) (#66669, #66103, #67207)
  • torch.nn``.ConvTranspose2d (#67104, #67358)
  • torch.permute (#68274)
  • Tensor indexing (at::slice) (#69382)
  • torch.clone (#69551)
• Added the vulkan_perf_test benchmark binary to benchmark Vulkan ops under various input conditions. (#67230)

Mobile

• Tracing Based Selective Build (PyTorch Mobile Build Size Reduction) is a new feature that reduces a mobile model’s binary size by only including the operators that the model uses.
  • Build tracer for tracing based workflow (#66267)
  • Used operator.yaml to build LibTorch library (#66237)
  • Unified tracer between internal and external (#64152)
  • Reorganized model tracer dependency (#63421)
  • Added support for the bool and int dtypes in the copy kernel by default when using Tracing Based Selective Build (#69106, #69297)
  • Generic build features for selective build (#67817)
  • Made more classes selective (#67397)
  • Added custom classes to selective build and compatibility APIs (#67004, #66972, #67340)

Distributed

• FullyShardedDataParallel
  • FSDP is a type of data-parallel training but unlike traditional data-parallel it shards model’s parameters, gradients and optimizer states across data parallel workers and can optionally offload the sharded model parameters to the CPUs. This new API can help users to scale their large model training with minimal code change when switching from DDP to FSDP. (#63881, #64964, #66578, #66904, #66956, #66957, #67117, #67292, #67249, #67135, #67813, #68308, #68155, #68417, #68776, #69356, #69357, #69358, #70340, #71803, #71804, #70341, #70235, #72084)
• DistributedDataParallel
  • Made static graph to be stable (#71459, #68413)
  • Made LocalSGD beta, cleaned up some docs (#71621)
  • Support custom buffer reduction in DDP via hooks (#64513)

TorchScript

• Enabled running torch.jit.freeze() and torch.jit.optimize_for_inference on functions that are not forward (#68668, #69367)
• Enabled torch.jit.freeze to work on for sparse COO tensors (#69614)
• Enabled torch.jit.script(), torch.jit.freeze() and serialization for tensors in Compressed Sparse Row (CSR) format (#69555)
• Allowed users to set the fusion strategy for torch.jit.fuser through the now public torch.jit.set_fusion_strategy . (#72937)
• Enabled Dynamic Shape Fusion For GPU & CPU, configurable via torch.jit.set_fusion_strategy (#72036)

Quantization

• Added bilinear quantized implementation of torch.nn.functional.grid_sample 2d operator (#66879)
• Added the torch.quantize_per_tensor_dynamic operator (#68004)
• Added Quantization Aware Training support for torch.nn.Embedding and torch.nn.EmbeddingBag
  • Added basic EmbeddingBag QAT fakeQuant workflow (#65443)
  • Added support for quantization of Embedding{Bag} in dynamic quant APIs (#65674)
  • Eager mode QAT for Embeddings (#66429)
  • Add benchmarks for QAT Embedding+EmbeddingBag (#66560)
  • Supported Embedding QAT via FX API (#69333)
  • Add FX support for QAT EmbeddingBag (#69334)
• Added support for depthwise quantized torch.nn.Conv3d in qnnpack, for use in quantization
  • Depthwise Conv3d Indirection Buffer Setup (#69311)
  • Depthwise Conv3d Weight Packing (#69312)
  • Depthwise Conv3d mp8x27 (per channel) Neon Kernel (#69313)
  • Depthwise Conv3d mp8x27 (per-channel) Sse2 Kernel (#69314)
  • Tightened Step Height for Indirection Buffers (#70530)
  • Enabled Depthwise Specific Conv3d Kernel for Kernel Size 3x3x3 (#69315)
  • Implemented 3d convolution in qnnpack (#66350)

ONNX

• Supports opset version 15 (#67805)
• Supports exporting nn.Module calls as ONNX local functions (#66140, #67803)
• Supports for exporting new ops:
  • tanhshrink, hardshrink, softshrink (#68492)
  • __xor__ (#64581)
  • isfinite (#64754)
  • log10 (#64374)
  • diagonal (#66144)
• Added BFloat16 type support (#66788)
• Supports exporting with Apex O2 (#66700)

Infra (Releng)

• Added support for ROCm 4.3.1 (#65624)
• Added support for ROCm 4.5.2 (#71064)
• Added support for CUDA 11.5 (#69262)
• Added support for CUDA enabled Bazel builds (#66241)
• Added support for Python 3.10 (#71132, #71419)

Improvements

Python API

• NumPy compatibility:
  • Improved torch.searchsorted to be more consistent with NumPy (#66818)
  • Added torch.argwhere to match NumPy (#64257)
  • Added an alias for torch.special.softmax (#62251)
• Improved torch.Tensor.view(dtype): enable all dtype combinations (#66493)
• Improved torch.diff by adding support for n greater than 1 (#67260)
• Improved torch.movedim to handle scalar as no-op (#69537)
• Improved cartesian_prod: fixed a warning in the docs example (#68753)
• Improved error messages for max_unpool{}d operators (#67328)
• torch.distributions
  • Implemented positive-semidefinite constraint in torch.distributions (#71375)
  • Implemented Entropy methods for Binomial and Multinomial distributions (#67609)
  • Implemented support for non-negative constraint in exponential distribution (allowing it to include zero). (#67184)
  • Implemented kl divergence between normal and laplace distribution. (#68807)
• Improved meta tensor support for operators:
  • max (#61449) min (#61450) tril, triu (#67055) mv (#67373) range, arange, linspace, logspace (#67032) lerp (#68924) smooth_l1_loss (#67404) fractional_max_pool2d_backward (#68245) linalg.lu_factor (#66934) fractional_maxpool3d: port to structured kernel (#70414)
• Added support for torch.Tensor.real for real-valued tensors (#71718)
• torch.logaddexp, torch.logaddexp2, torch.remainder: added BFloat16 support on CPU (#63621)
• torch.bucketize and searchsorted: added Half precision support (#67077)
• Added new torch.slice_scatter,torch.select_scatter, torch.diagonal_scatter ops (#64430)
• Made torch.scatter_reduce a public API (#68580, #73125)

C++ API

• Added C++ API and docs for hfftn (#66127)
• Added support for MaybeOwned<IValue> (#68157)
• Added set_to_none option for zero_grad() to C++ API (#68801)
• Added an environment variable, TORCH_CPP_LOG_LEVEL, that you can use to toggle the log level in the c10 library (#71746)

Autograd

• Added nesting support for torch.autograd.graph.saved_tensor_hooks (#70932)
• Delayed all warnings encountered during the backward pass until the end of backward execution (#66235)
• Added complex autograd support to torch.{col2im,im2col} (#68199)
• Added new reduce options and autograd support for torch.scatter_reduce (#71788)
• Added derivatives wrt the second argument for torch.{remainder,fmod} (#69908)
• Added new strategy flag to autograd.functional.{Jacobian, Hessian} to enable vectorized computation (#67041, #66292)
• Added check_backward_ad flag to torch.autograd.gradcheck to be able to skip backward mode AD checks (#65040)
• Relaxed forward AD layout check to allow primal and tangent stride to differ when their size is 1 (#66294)

Build

• Improved incremental build times of PyTorch core by removing a dependency on native_functions.yaml in many core files (#64499, #66914, #64172, #64171, #66620, #66793, #66913, #66794, #64169, #64173, #64170, #67735)
• Enabled bazel build without glog and gflags (#70850)
• Added support for C++ frontend wrapper on Linux (#69094)
• Added support for dynamic codegen outputs in CMake (#68246)
• Max CMake version is now used by default with setup.py (#69355)
• Upgraded oneDNN to v2.3.3 and package oneDNN Graph API together (#63748)
• Code base should now be -Wno-unused-variable compliant (#66041)
• Added lazy import for packaging in torch_version (#71345)

Dataloader

• Support custom Sequence and Mapping for utils.data.default_collate (#68779)
• Allowed specifying num_samples to RandomSampler when replacement is False (#71568)
• Fixed the warning of shape inconsistency utils.data.default_collate (#71065)

ForEach

• Implemented ForEach L1 & L2 norm (#62646)

LinAlg

• The linalg.matrix_rank (docs) and linalg.pinv (docs) operations now support specifying absolute and relative tolerances for better handling of singular values (#63102)

torch.nn

• Added channels_last support for ChannelShuffle (#50247)
• Added no-batch-dim support for nn.{AdaptiveLogSoftmaxWithLoss, Bilinear, Conv*d, ConvTranspose*d, CrossEntropyLoss, CTCLoss, Fold, FractionalMaxPool3d, GaussianNLLLoss, GRU, GRUCell, InstanceNorm*d, LSTM, LSTMCell, MarginRankingLoss, MultiheadAttention, MultiLabelSoftMarginLoss, RNN, RNNCell, Transformer, TransformerDecoderLayer, TransformerEncoderLayer} (#69054, #69539, #70506, #71055, #70092, #64909, #69732, #69783, #70236, #65323, #71056, #64975, #67176, #70590, #65690, #70977, #70597, #70322, #69291)
• Added BFloat16 support on CPU to nn.{AdaptiveAvgPool2d, AdaptiveMaxPool2d, AvgPool2d, MaxPool2d} (#56902, #66929, #66927, #56903)
• Added maximize support to optim.{Adam, AdamW, SGD} (#68164, #70146, #67847, #68733, #71023)
• F.interpolate: Add nearest-exact mode to fix off-by-one error in nearest mode (#64501)
• F.interpolate: Added support for anti-aliasing to bilinear and bicubic algorithms (#70930, #68819, #65142, #69318)
• F.interpolate: Improved error message for invalid shapes (#66417)
• nn.Conv*d: Accepts 0-sized channel inputs (#66256)
• nn.LogSigmoid: Used log1p for improved precision (#66441)
• nn.Module: Added flag for removing duplicates from parameters (#71542)
• nn.Module: Added register_module alias for registering a sub-module (#65174)
• nn.ModuleList: Supported concatenation (#70887)
• nn.MultiheadAttention: Added flag to optionally average output attention weights across heads (#70055)
• nn.ParameterDict: Supported full set of dict methods (#69403)
• nn.{RNN, GRU}: Allowed hidden_size to be 0 (#70556)
• nn.Sequential: Added append method (#71326)
• nn.Upsample: Exposed recompute_scale_factor (#66419)
• nn.ZeroPad2d: Added extra_repr for printing purposes (#69206)
• optim.{ChainedScheduler, SequentialLR}: Added optimizer attribute (#67406, #69817)
• optim.swa_utils.AveragedModel: Added use_buffers flag for averaging buffers in addition to parameters (#65921, #71763)

torch.fx

• Improved the customizability of fx.Graph’s code generation function, including support for setting a breakpoint in the generated code (#67139)
• Supported printing inplace operators in FX (#71887)

Sparse

• Add CSR support for several operators:
  • torch.triangular_solve, torch.addmv, torch.addmm, torch.add for all arguments on CPU (#62180, #61536, #65606, #64391)
  • torch.triangular_solve, torch.addmv, torch.addmm, torch.add for all arguments on GPU (#61407, #61858, #63511, #63948)
  • zero-preserving unary functions (#68123, #69292)
  • torch.empty, torch.resize_, torch.copy_, torch.randn_like, torch.clone (#63509, #63510, #68083, #70581)
  • transpose (#70582)
• Added torch.sparse_coo Layout support to zeros_like (#68108)
• Added Half, BFloat16, and Complex dtype support for matrix multiplication of two COO Tensors on GPU (#59980)
• Added support for conversion of CSR to COO Tensor to to_sparse (#66774)
• Added support for empty COO Tensors to sparse.sum (#71091)

AMD

• Added sparse mappings for CUDA->HIP translation (#67323)
• Enabled frexp support for ROCm builds (#67226)
• Used hipCUB/rocPRIM scan algorithms for large index support (#68487)

CUDA

• Allows external CUDA streams to be set as current (#66324)
• Added an option to disable reduced precision reductions for FP16 GEMM (#67946)
• Improved CUDA memory usage of nanmedian result (#68591)
• Reduced number of igamma kernel instantiations (#70666)
• Reduced number of compare kernels by unifying them (#69111)
• Reduced number of bernoulli tensor tensor kernel instantiations (#70169)
• Used cub::FutureValue to simplify 64bit indexing split of cub scan (#66711)
• Added hascuSOLVER flag to Context (#69825)
• Improved error message from CUDACachingAllocator (#69174)
• Fixed masked_softmax perf for element_size is not 8 (#70271)
• Reduced binary size of TensorCompare.cu (#68835)
• Improved error message for interpolation (#72066)
• Doesn't compile pow kernels for non-existent case (#70017)

Profiler

• Added flop count formulas for bmm and baddbmm (#66636)

Vulkan

• Allowed Vulkan models to return multiple outputs by improving Vulkan tensor lifecycle management to release GPU resources when the tensor is destroyed, instead of being released at the end of every inference (#66477, #66478)
• Enabled multiple Vulkan models to execute concurrently in parallel threads, by moving components of the Vulkan global context into thread local objects (#67733, #69576)

Mobile

• Introduced multiple improvements for NNAPI
  • Added converters for torchscript ops quantized::mul and quantized::convtranspose2d to converter (torch.backends._nnapi.prepare.convert_model_to_nnapi) (#63913, #63914)
  • Supported int32 and qint16 type in Torchscript expressions (#70197, #70621)
  • Supported runtime flexible shapes and return shapes (#70334)
• Improved Model Tracer Coverage and Selective Metal Ops (#68134, #69492, #69328)
• Introduced multiple improvements for CoreML
  • Fixed error messages (#67410)
  • Assigned computationUnit to executor (#67411)
  • Cleaned up shape information from TensorSpec (#67412)
• Type Support in Mobile Lite Interpreter
  • Extended type_parser to handle NamedTuple type (#63130, #62612)

Distributed

• torch.distributed

  • Improvements to error handling in TCPStore’s socket implementation (#68225)
  • Enabled ncclAvg for reductions (#62835)
  • Init dummy NCCL comms in constructor (#65173, #66393)
  • Added pybind trampoline for ProcessGroup and Work (#66338)
  • Setup c10d extension Backend class attr the same way as builtin ones (#66991)
  • Added barrier to ProcessGroup trampoline (#67236)
  • Raised warning when calling collectives on non-member group objects (#67639)
  • Patched bfloat16 support for NCCL (#67843)
  • Fixed c10d TCP store race condition with mutex (#68499)
  • Surfaced ncclUniqueId store broadcast error (#68597)
  • Checks for file existence before invoking cleanup logic in FileStore destructor (#68603)
  • Implemented gather primitive for ProcessGroupNCCL (#66745)
  • Implemented scatter primitive for ProcessGroupNCCL (#70029)
  • Enabled gather_object on NCCL (#71623)
  • Implemented allreduce_coalesced for ProcessGroupNCCL (#62140)
  • Set non-default backend names to lower case (#69400)
  • Added support for deleteKey for FileStore (#69953)
  • Fixed TSAN issue in TCPStore (#69590)

• DistributedDataParallel

  • Refactored and removed sync_params (#64514)
  • Used named_params and named_buffers explicitly (#65181)
  • Allow await of custom buffer reduction in backward (#64515)
  • Profiling range for bucket copy (#65769)
  • Logs iteration in debug mode (#65770)

• torch.distributed.rpc

  • Added a timeout argument to RPC shutdown() (#65425)
  • Released GIL during RPC shutdown. (#69586)
  • Updated RPC shutdown() logic to remove process group usage. (#65946)
  • Removal of Process Group dependency for TensorPipe Agent. (#68128)

• torch.distributed.autograd

  • Made Kineto + distributed a warning rather than an error (#71120)

• torch.distributed.elastic

  • Added ability to override sys.executable for torch.distributed.run (#66179)

TorchScript

• Several improvements to NVFuser, which is an optimization that speeds up all JIT graphs with a CUDA Tensors on Nvidia GPUs. This includes extending fusing support to normalization and reduction kernels, enabling multiple kernel launch for single CudaFusionGroup, and addition of a graph segmentation cache to the hierarchical caching system. (#63745, #65137, #63745, #65137)
• Enabled profile_ivalue to convert dynamic scalar into compile time constants in NVFuser. (e.g. reduction axes). (#63745, #65137)
• Added support in torch.jit.trace for tracing already JITted subgraphs(#59949)
• We now provide full types on graph inputs when tracing graphs that are already JITted(#67424)
• torch.jit.freeze now can preserve attributes of submodules - previously, it was only possible to prevent inlining of attributes of the top level module.(#66102)
• The peephole optimizer, which is used in torch.jit.freeze now coalesces consecutive calls to torch.concat into a single call (#67000)
• Added ability for Torch.JIT C dispatch to convert python None into an undefined Tensor(#67793)
• torch.jit.script now recognizes union of scalars as a JIT NumberType (#66591)
• No longer adds a tensor in a returned list to the wildcard alias set in AliasDB, allowing for additional optimizations in JIT optimization passes. (#71170)
• In torch.jit.optimize_for_inference, there is a new graph pass to precompute transposes for linear layers. (#65631, 68024)
• In torch.jit.freeze, there is a new pass where we concat together multiple linear layers with same input Tensor (different weight/bias) (#63198, #68024)
• Added support for normalizing torch.Tensor.__rsub__ in normalize_ops JIT pass(#65014)

Quantization

• Quantized op improvements
  • torch.ao.FakeQuantize now supports fp32/fp16 zero_point. (#65836)
  • torch.ops.quantized.add now supports broadcasting (#66049)
  • torch.Tensor.dequantize now supports fp16 + cuda (#67234)
  • Added quantized CPU support for torch.nn.GELU (#69968)
  • torch.nn.quantized.functional.hardsigmoid supports an inplace flag (#65740)
• Workflow improvements
  • FX graph mode quantization: enable torch.nn.Linear + torch.nn.BatchNorm1d fusion for PTQ (#66484)
  • Added an option in torch.ao.quantization.quantize_fx.convert_fx to accept qconfig_dict to skip quantization (#66878)
  • Added torch.nn.qat.dynamic.modules.Linear module (#67325)
  • Added torch.nn.ConvTranspose{n}d + torch.nn.BatchNorm{n}d fusion support (#70022)
  • Extended torch.ao.quantization.prepare_qat with allow_list argument, to allow custom mapping and custom QAT module (#65119)
  • Added torch.ao.quantization.default_replay_qconfig which allows observer reuse for torch.reshape in FX graph mode quantization (#69249)

ONNX

• Set ir_version of the exported model based on opset_version. This increases the odds that the exported ONNX model will be usable. Before this change, we were setting the IR version to a hard-coded value which may be higher than what the model consumer supports. (#67803)
• Preserved op input names when op just passes through the input to the output (#67275)
• Shape inference improvements:
  • Updated slice process shape to support rank only inference (#66149)
  • Represent symbolic shape as value (#69545)
• Included op type in exported models’ input and output names (#68976)
• Supports Conv-BatchNorm fusion inside blocks (#67272)
• Exported torch.reciprocal to ONNX Reciprocal operator instead of Div(1, x) (#67271)
• Supports beta!=1 in softplus (#66146)
• Added warning for inplace updates on tensor.shape in tracing mode (#66142)
• Supports instance_norm in training mode (#64375)
• Allow registration of custom symbolics for ops specifying aten namespace (i.e. aten::foo is allowed as well as “foo”). (#67810)
• Allow registration of custom symbolics for prim namespace (#66139)
• Supports dynamic inputs for OneHot, bool for Einsum (#66147)

Infra (Releng)

• Build with BUILD_SPLIT_CUDA for all 11.X Windows builds (#70899)

torch.package

• Add ability to retrieve the dependency graph via all_path function(#65602)
• Add support for pickle v4 (#70642)
• Add better testing support for Package Exporter (#70641)

Bug fixes

Python API

• Fixed scalar inputs for aliased binary ops {multiply, subtract, divide} (#65937)
• Fixed torch.save when saving storages that view same data with different type (#66949)
• Fixed torch.save error if storages are unallocated (#68787)
• Fixed k out-of-bounds in torch.kthvalue (cpu kernel) (#68863)
• Fixed inference_mode decorator: with inference_mode(mode=False) used to ignore the mode argument and always set inference mode. (#68617)
• Fixed cdist_backward in the case when cdist inputs are not contiguous (#70016)
• Fixed cdist error message typo (#70178)
• Fixed scatter for empty indexes (#70662)
• Fixed torch.{unique, unique_consecutive} out of bound (#71540)
• Fixed torch.isin in the case when inputs are non-contiguous on CPU (#70659)
• Fixed hsplit vsplit dsplit crash when section is 0 (#69342)
• Fixed: torch.gradient ignores dim argument when checking edge_order (#67926)
• Fixed: TransformedDistribution.icdf should perform validation after applying the inverse transformation rather than before. (#71393)
• Fixed torch.all and torch.any internal assert error with requires_grad=True (#65714)
• Fixed torch.logsumexp type promotion: promote integral inputs to floating for(#63393)

C++ API

• Fixed libtorch at::Tensor::print() linking error (#69615)
• Avoided UB when indexing into size-0 tensors (#65878)
• Fixed an ICE when compiling PyTorch from source on MacOS with clang-1300 (#65655)

Autograd

• Fixed autocast state propagation in the torch.utils.checkpoint API (#71169)
• Fixed torch.nn.functional.conv_transpose3d backward when grad_out is non-contiguous (#67829)
• Forward mode AD:
  • Fixed a case where forward AD in-place-over-view silently copies the view (#67816)
  • Fixed deadlock in forward AD for functions that return multiple outputs (#67995)
  • Fixed forward AD codegen for functions that have multiple formulas (#68535)
  • Fixed deadlock when forward and backward AD are used at the same time (#67360)
  • Fixed Tensor.copy_ forward AD to handle broadcasting (#69592)
  • Do not generate not_implemented error for forward AD when input with tangent passed to non-differentiable function (#66926)
• Fixed autograd.Function when non-Tensor argument precedes tensor argument (#71530)
• Fixed autograd.Function forward AD when forward is a no-op to no longer raise an internal error (#71531)

Build

• Stopped reporting CPU Capability as AVX512 on machines with AVX512 support but without AVX512 kernels (#66703)
• Disabled SVE when cross-compiling for M1 (#67114)
• Added failure if pocketfft is not found and at_mkl is not enabled (#67909)
• Fixed clang issues when compiling with _GLIBCXX_USE_CXX11_ABI (#72081)

Complex Numbers

• Fixed torch.autograd.gradcheck to generate valid inputs for forward AD computation for complex functions (#68001)
• Fixed torch.Tensor.copy_ transpose path for tensors with conjugate or negative bit set (#69026)
• Fixed torch.Tensor.copy_ behavior for the case when two conjugated or negated tensors of the same dtype (one or both of which are non-contiguous) are copied into each other (#68963)

Dataloader

• Made ProcessException picklable (#70118)
• Fixed persistent worker exiting before pin_memory_thread (#71579)

torch.nn

• nn.AdaptiveAvgPool*d: Throws an error for negative output_size (#70488)
• nn.Conv1d: Fixed for 1D convolution on MKL-DNN backend (#68166)
• nn.CrossEntropyLoss: Fixed for usage of weight, ignore_index, and label_smoothing together (#69511)
• nn.Fold: Checked that block height and width are positive (#69048)
• nn.LayerNorm: Fixed incorrect result on CUDA when gamma or bias are missing (#69210)
• nn.LayerNorm: Avoided overflow by doing computation in float for half (#66920)
• nn.Module: Throws a proper error message from load_state_dict for non-tensor values (#70596)
• nn.ModuleList: Fixed incorrect return type in __getitem__ (#69083)
• nn.MultiheadAttention: Used query dtype for mask type (#68077)
• nn.NLLLoss: Fixed backward computation with negative weights (#64572)
• nn.{RNN, GRU}: Fixed RNN modules with input shapes containing-0 in CUDA (#71696)
• nn.utils.rnn.pad_sequence: Fix regression to support tuples for padding (#72436)
• optim._LrScheduler: Fixed print formatting (#68338)
• optim.ChainedScheduler: Fixed get_last_lr() (#69112)
• optim.CosineAnnealingWarmRestarts: Fixed ordering bug when last_epoch > 0 (#64758)
• optim.SequentialLR: Updated _last_lr on step (#70558)

torch.fx

• Supported torch.layout as arg (#66048)
• Specified a default value when possible for placeholders created from concrete_args (#59569)
• Fixed issue where GraphModule.delete_all_unused_submodules deletes submodules from called leaf modules (#66430)
• Fixed torch.fx.subgraph_rewriter.replace_pattern mechanism so that multiple one-liner instances of the pattern are captured correctly (#66442)
• Fixed bug in graph matcher that caused certain nodes to be matched twice (#69238)
• Ensured node stack trace survives copying (#69368)
• Fixed to_folder not saving dtype (#69983)
• Added a default_value arg to fx.Graph.placeholder and fix split_module (#71016)

Sparse

• Fixed CSR storage access to throw when used (#70072)
• Fixed multiplication of 0-D sparse tensors (#70749)
• Fixed result dtype for neg if given sparse Tensor (#68885)

CUDA

• Fixed CUDA vs CPU consistency for index_put_ when accumulating (#66790)
• Fixed CUDA vs CPU consistency for index_put_ when accumulating (part 2) (#67189)
• Fixed error in warning about unsupported GPU (#67900)
• Disabled TF32 in pinv_jvp and pinv_backward (#67948)
• Fixed DLPack CUDA stream convention (#67618)
• Sets device guard in _cudnn_impl functions (#70406)
• Fixed mem_get_info when querying on a device other than the current device (#69640)

Benchmark

• Fixed divide-by-zero errors in torch.utils.benchmark.Timer (#70050)

Dispatcher

• Added explicit OperatorHandle destructor, so that the symbol shows up in windows builds (#70033)

Profiler

• Fixed race condition in profiler (#65812)
• Fixed TensorBoard memory profiling (#71417)

Visualization

• Fixed torch.utils.tensorboard parsing JIT graph incorrectly (#65692)

Vulkan

• Greatly reduced memory usage of the Vulkan backend by updating the configuration of the Vulkan Memory Allocator (#69088)
• Addressed several warnings raised by the Vulkan Validation layers:
  • Updated all texture resources to have the same dimensionality (#67647)
  • Added image format qualifier to shader files (#69330)
  • Disabled SPIR-V compiler size optimization (#69331)

Mobile

• Fixed quantized logistic converter for NNAPI (#70847)
• Fixed potential crash if MTLCreateSystemDefaultDevice returns nil (#66859)
• Used full name to look for the promoted prim operator table (#66081)
• Fixed function name bug in mobile export (#66915)
• Fixed issues with irange not having a header included in Metal (#66877)
• Fixed backward compatibility issue for UnionType on mobile in type_parser. (#71341)
• Fixed forward flatbuffer type handling with dynamic type in flatbuffer_loader. (#71500)
• Fixed type equalities issue in pytorch_jni_common (#71508)
• Fixed missing properties to the executor in CoreML (#67737)
• Fixed memory computation when both constants and data tensors are present in model_dump (#66006)
• Ensured that function participating in bundled inputs have their “name" attribute set (#65856)

Distributed

• torch.distributed
  • Fixed bug on empty GLOO_SOCKET_IFNAME_ENV (#68933)
• DistributedDataParallel
  • Fixed “Cannot modify in-place due to DDPSink” (#66015)
• torch.distributed.elastic
  • Fixed scale down bug caused by calling rdzv_handler.shutdown() on premature agent failures (#67749)

TorchScript

• Fixed a race condition in the JIT interpreter when unpickling source ranges (5525e9a591)
• Fixed a ref counting loop for CompilationUnit, resulting in memory leaks when class objects were in JIT graphs. (#65442)
• Fixed bug where output type was discarded after calling SubgraphRewriter in C++ (#65453)
• Fixed bug where torch.jit.optimize_for_inference did not torch.jit.freeze a module when passed a a non-frozen module (#71436)
• Fixed bug where running module.forward() on a torch.jit.freeze ed module ran the wrong graph (#68316)
• Fixed bug where alias analysis in the JIT optimizer was incorrect for the int[] version of torch.split , resulting in invalid optimizations in various JIT optimization passes (#69745)
• Fixed places where using torch.autocast together with autodiff (module.backwards()) in a JIT graph had the wrong number of arguments and would error out. (#67648)
• Forbid propagating gradients through views in JIT graphs as currently it is broken (#67732)
• Fixed bug where graph input types were incorrect after running torch.jit.trace (#68242)
• Fixed case where BroadcastMKLDNN breaks the stack invariant by pushing more than 2 tensors to the stack for when torch.jit.freeze ops are converted to MKLDNN(#66628)
• Raised error instead of segfaulting when passing None into torch.jit.Graph.create (#68253)
• Raised error instead of crashing when a JIT ScriptFunction is pickled with an incompatible Python pickle version.(#69807)
• Fixed bug where torch.jit.script fails when comments in function has less indent than surrounding code (#70227)
• Fixed incorrect device type when torch.device is called inside scripted (torch.jit.script) code (#69645)
• Fixed warning: overloaded virtual function torch::jit::Function::call is only partially overridden in class torch::jit::GraphFunction (4bf1be898d)

Quantization

• Fixed applying non-zero offset 1 to null pointer in torch.nn.functional.interpolate for quantized tensors (#65570)
• Doesn't assume bias is a keyword argument to torch.nn.Conv{n}d (#61647, #71426)
• Made error message when trying to use torch.quantize_per_tensor on non floats more specific (#66050)
• Quantized torch.nn.Embedding conversion with unsupported dtype: make error message clearer (#66051)
• Fixed torch.nn.qat.EmbeddingBag from_float error message (#66989)
• Fixed bug enforcing quant_min <= zero_point <= quant_max for float zeropo

Note truncated.

This release is meant to deploy additional fixes not included in 1.10.1 release:

• fix pybind issue for get_autocast_cpu_dtype and get_autocast_gpu_dtype #66396
• Remove fgrad_input from slow_conv2d #64280
• fix formatting CIRCLE_TAG when building docs #67026

This release is meant to fix the following issues (regressions / silent correctness):

• torch.nn.cross_entropy silently incorrect in PyTorch 1.10 on CUDA on non-contiguous inputs #67167
• channels_last significantly degrades accuracy #67239
• Potential strict aliasing rule violation in bitwise_binary_op (on ARM/NEON) #66119
• torch.get_autocast_cpu_dtype() returns a new dtype #65786
• Conv2d grad bias gets wrong value for bfloat16 case #68048

The release tracker should contain all relevant pull requests related to this release as well as links to related issues

1.10.0 Release Notes

• Highlights
• Backwards Incompatible Change
• New Features
• Improvements
• Performance
• Documentation

Highlights

We are excited to announce the release of PyTorch 1.10. This release is composed of over 3,400 commits since 1.9, made by 426 contributors. We want to sincerely thank our community for continuously improving PyTorch.

PyTorch 1.10 updates are focused on improving training and performance of PyTorch, and developer usability. Highlights include:

• CUDA Graphs APIs are integrated to reduce CPU overheads for CUDA workloads.
• Several frontend APIs such as FX, torch.special, and nn.Module Parametrization, have moved from beta to stable.
• Support for automatic fusion in JIT Compiler expands to CPUs in addition to GPUs.
• Android NNAPI support is now available in beta.

You can check the blogpost that shows the new features here.

Backwards Incompatible changes

Python API

torch.any/torch.all behavior changed slightly to be more consistent for zero-dimension, uint8 tensors. (#64642)

These two functions match the behavior of NumPy, returning an output dtype of bool for all support dtypes, except for uint8 (in which case they return a 1 or a 0, but with uint8 dtype). In some cases with 0-dim tensor inputs, the returned uint8 value could mistakenly take on a value > 1. This has now been fixed.

<p align="center"> <table align="center"> <tr><th>1.9.1</th><th>1.10.0</th></tr> <tr valign="top"> <td><sub><pre lang="python">

torch.all(torch.tensor(42, dtype=torch.uint8)) tensor(1, dtype=torch.uint8) torch.all(torch.tensor(42, dtype=torch.uint8), dim=0) tensor(42, dtype=torch.uint8) # wrong, old behavior </pre></sub></td> <td>_{<pre lang="python"> torch.all(torch.tensor(42, dtype=torch.uint8)) tensor(1, dtype=torch.uint8) torch.all(torch.tensor(42, dtype=torch.uint8), dim=0) tensor(1, dtype=torch.uint8) # new, corrected and consistent behavior </pre>}</td> </tr> </table> </p>

Remove deprecated torch.{is,set}_deterministic (#62158)

This is the end of the deprecation cycle for both of these functions. You should be using torch.use_deterministic_algorithms andtorch.are_deterministic_algorithms_enabled instead.

Complex Numbers

Conjugate View: tensor.conj() now returns a view tensor that aliases the same memory and has conjugate bit set (#54987, #60522, #66082, #63602).

This means that .conj() is now an O(1) operation and returns a tensor that views the same memory as tensor and has conjugate bit set. This notion of conjugate bit enables fusion of operations with conjugation which gives a lot of performance benefit for operations like matrix multiplication. All out-of-place operations will have the same behavior as before, but an in-place operation on a conjugated tensor will additionally modify the input tensor.

<p align="center"> <table align="center"> <tr><th>1.9.1</th><th>1.10.0</th></tr> <tr valign="top"> <td><sub><pre lang="python">

import torch x = torch.tensor([1+2j]) y = x.conj() y.add_(2) print(x) tensor([1.+2.j]) </pre></sub></td> <td>_{<pre lang="python"> import torch x = torch.tensor([1+2j]) y = x.conj() y.add_(2) print(x) tensor([3.+2.j]) </pre>}</td> </tr> </table> </p>

Note: You can verify if the conj bit is set by calling tensor.is_conj(). The conjugation can be resolved, i.e., you can obtain a new tensor that doesn’t share storage with the input tensor at any time by calling conjugated_tensor.clone() or conjugated_tensor.resolve_conj() .

Note that these conjugated tensors behave differently from the corresponding numpy arrays obtained from np.conj() when an in-place operation is performed on them (similar to the example shown above).

Negative View: tensor.conj().neg() returns a view tensor that aliases the same memory as both tensor and tensor.conj() and has a negative bit set (#56058).

conjugated_tensor.neg() continues to be an O(1) operation, but the returned tensor shares memory with both tensor and conjugated_tensor.

<p align="center"> <table align="center"> <tr><th>1.9.1</th><th>1.10.0</th></tr> <tr valign="top"> <td><sub><pre lang="python">

x = torch.tensor([1+2j]) y = x.conj() z = y.imag z.add_(2) print(x) tensor([1.+2.j]) </pre></sub></td> <td>_{<pre lang="python"> x = torch.tensor([1+2j]) y = x.conj() z = y.imag print(z.is_neg()) True z.add_(2) print(x) tensor([1.-0.j]) </pre>}</td> </tr> </table> </p>

tensor.numpy() now throws RuntimeError when called on a tensor with conjugate or negative bit set (#61925).

Because the notion of conjugate bit and negative bit doesn’t exist outside of PyTorch, calling operations that return a Python object viewing the same memory as input like .numpy() would no longer work for tensors with conjugate or negative bit set.

<p align="center"> <table align="center"> <tr><th>1.9.1</th><th>1.10.0</th></tr> <tr valign="top"> <td><sub><pre lang="python">

x = torch.tensor([1+2j]) y = x.conj().imag print(y.numpy()) [2.] </pre></sub></td> <td>_{<pre lang="python"> x = torch.tensor([1+2j]) y = x.conj().imag print(y.numpy()) RuntimeError: Can't call numpy() on Tensor that has negative bit set. Use tensor.resolve_neg().numpy() instead. </pre>}</td> </tr> </table> </p>

Autograd

Raise TypeError instead of RuntimeError when assigning to a Tensor’s grad field with wrong type (#64876)

Setting the .grad field with a non-None and non-Tensor object used to return a RuntimeError but it now properly returns a TypeError. If your code was catching this error, you should simply update it to catch a TypeError instead of a RuntimeError.

<p align="center"> <table align="center"> <tr><th>1.9.1</th><th>1.10.0</th></tr> <tr valign="top"> <td>_{<pre lang="python"> try: # Assigning an int to a Tensor's grad field a.grad = 0 except RuntimeError as e: pass </pre>}</td> <td>_{<pre lang="python"> try: a.grad = 0 except TypeError as e: pass </pre>}</td> </tr> </table> </p>

Raise error when inputs to autograd.grad are empty (#52016)

Calling autograd.grad with an empty list of inputs used to do the same as backward. To reduce confusion, it now raises the expected error. If you were relying on this, you can simply update your code as follows:

<p align="center"> <table align="center"> <tr><th>1.9.1</th><th>1.10.0</th></tr> <tr valign="top"> <td>_{<pre lang="python"> grad = autograd.grad(out, tuple()) assert grad == tuple() </pre>}</td> <td>_{<pre lang="python"> out.backward() </pre>}</td> </tr> </table> </p>

Optional arguments to autograd.gradcheck and autograd.gradgradcheck are now kwarg-only (#65290)

These two functions now have a significant number of optional arguments controlling what they do (i.e., eps, atol, rtol, raise_exception, etc.). To improve readability, we made these arguments kwarg-only. If you are passing these arguments to autograd.gradcheck or autograd.gradgradcheck as positional arguments, you can update your code as follows:

<p align="center"> <table align="center"> <tr><th>1.9.1</th><th>1.10.0</th></tr> <tr valign="top"> <td>_{<pre lang="python"> torch.autograd.gradcheck(fn, x, 1e-6) </pre>}</td> <td>_{<pre lang="python"> torch.autograd.gradcheck(fn, x, eps=1e-6) </pre>}</td> </tr> </table> </p>

In-place detach (detach_) now errors for views that return multiple outputs (#58285)

This change is finishing the deprecation cycle for the inplace-over-view logic. In particular, a few things that were warning are updated:

* `detach_` will now raise an error when invoked on any view created by `split`, `split_with_sizes`, or `chunk`. You should use the non-inplace `detach` instead.
* The error message for when an in-place operation (that is not detach) is performed on a view created by `split`, `split_with_size`, and `chunk` has been changed from "This view is an output of a function..." to "This view is the output of a function...".

<p align="center"> <table align="center"> <tr><th>1.9.1</th><th>1.10.0</th></tr> <tr valign="top"> <td>_{<pre lang="python"> b = a.split(1)[0] b.detach_() </pre>}</td> <td>_{<pre lang="python"> b = a.split(1)[0] c = b.detach() </pre>}</td> </tr> </table> </p>

Fix saved variable unpacking version counter (#60195)

In-place on the unpacked SavedVariables used to be ignored. They are now properly detected which can lead to errors saying that a variable needed for backward was modified in-place. This is a valid error and the user should fix this by cloning the unpacked saved variable before using it.

No internal formula will trigger this, but it might be triggered by user custom autograd.Function if the backward modifies a saved Tensor inplace and you do multiple backwards. This used to silently return the wrong result and will now raise the expected error.

torch.nn

Added optional tensor arguments to __torch_function__ handling checks (#63967)

This fixes the has_torch_function*() checks throughout torch.nn.functional to correctly pass in optional tensor arguments; prior to this fix, handle_torch_function() was not called for these optional tensor arguments. Previously, passing a tensor-like object into a function that accepts an optional tensor might not trigger that object's __torch_function__. Now, the object's __torch_function__ will be triggered as expected.

<p align="center"> <table align="center"> <tr><th>1.9.1</th><th>1.10.0</th></tr> <tr valign="top"> <td><sub><pre lang="python"> import torch import torch.nn.functional as F class TestTensor(object): def init(self, weight): self.weight = weight def torch_function(self, func, _, args=(), kwargs=None): print(func) print(func == F.group_norm)

Call F.group_norm with a custom Tensor as the non-optional arg 'features'

features = TestTensor(torch.randn(3,3)) F.group_norm(features, 3)

...prints "group_norm" and True

Call F.group_norm with a custom Tensor as the optional arg 'weight'

features = torch.randn(3,3) weight = TestTensor(torch.randn(3)) F.group_norm(features, 3, weight=weight)

...prints "group_norm" and False because weight's torch_function is

called with func as torch.group_norm instead of F.group_norm

  </pre></sub></td>
  <td><sub><pre lang="python">

import torch import torch.nn.functional as F class TestTensor(object): def init(self, weight): self.weight = weight def torch_function(self, func, _, args=(), kwargs=None): print(func) print(func == F.group_norm)

Call F.group_norm with a custom Tensor as the non-optional arg 'features'

features = TestTensor(torch.randn(3,3)) F.group_norm(features, 3)

...prints "group_norm" and True

Call F.group_norm with a custom Tensor as the optional arg 'weight'

features = torch.randn(3,3) weight = TestTensor(torch.randn(3)) F.group_norm(features, 3, weight=weight)

...prints "group_norm" and True

  </pre></sub></td>
</tr>

</table> </p>

CUDA

Removed post-backward syncs on default stream (#60421)

Calls to backward() or grad() synced only the calling thread's default stream with autograd leaf streams at the end of backward. This made the following weird pattern safe:

with torch.cuda.stream(s):
    # imagine forward used many streams, so backward leaf nodes may run on many streams
    loss.backward()# no sync
use grads

but a more benign-looking pattern was unsafe:

with torch.cuda.stream(s):
    # imagine forward used a lot of streams, so backward leaf nodes may run on many streams
    loss.backward()
    # backward() syncs the default stream with all the leaf streams, but does not sync s with anything,
    # so counterintuitively (even though we're in the same stream context as backward()!)
    # it is NOT SAFE to use grads here, and there's no easy way to make it safe,
    # unless you manually sync on all the streams you used in forward,
    # or move "use grads" back to default stream outside the context.
    use grads

Note: this change makes it so that backward() has same user-facing stream semantics as any cuda op.** In other words, the weird pattern is unsafe, and the benign-looking pattern is safe. Implementation-wise, this meant backward() should sync its calling thread's current stream, not default stream, with the leaf streams. This PR deletes syncs on the default stream.

torch.package

• Removed verbose mode from PackageExporter (#61145)
  • PackageExporter is losing “verbose” mode argument as we have found it is not useful and sometimes confusing. See following examples on how to modify your code to accommodate this change.

<p align="center"> <table align="center"> <tr><th>1.9.1</th><th>1.10.0</th></tr> <tr valign="top"> <td>_{<pre lang="python"> with PackageExporter(buffer, verbose=False) as e: e.intern("") e.save_pickle("res", "mod1.pkl", mod1) e.save_pickle("res", "mod2.pkl", mod2) </pre>}</td> <td>_{<pre lang="python"> with PackageExporter(buffer) as e: e.intern("") e.save_pickle("res", "mod1.pkl", mod1) e.save_pickle("res", "mod2.pkl", mod2) </pre>}</td> </tr> </table> </p>

Quantization

Added extra observer/fake_quant (the same observer/fake_quant instance as the input) for some operators in prepare_fx, e.g. maxpool, add_scalar and mul_scalar (#61687, #61859)

Previously the way we insert observers/fake_quants are specific to fbgemm/qnnpack backend, as we work on making FX Graph Mode Quantization extensible to custom backends, we are changing some behaviors for the fbgemm/qnnpack path as well. The above changes are adding extra observer/fake_quant to the output of some operators to make sure we model the quantized operator more accurately in quantization aware training, the comprehensive list of operators where the behavior changes are the following:

• modules: torch.nn.MaxPool1d, torch.nn.MaxPool2d, torch.nn.MaxPool3d, torch.nn.Identity
• torch functions: torch.nn.functional.max_pool1d, torch.nn.functional.max_pool2d, torch.nn.functional.max_pool3d, torch.chunk, torch.flatten, torch.transpose, torch.repeat_interleave, torch.sort, torch.squeeze, torch.stack, torch.unsqueeze, operator.getitem,
• Tensor methods: chunk, contiguous, detach, detach_, numel, permute, repeat, repeat_interleave, reshape, resize_, shape, size, squeeze, squeeze_, transpose, unsqueeze, unsqueeze_, view
• Tensor operations: add scalar and mul scalar (add/mul with a Tensor and a Scalar input)

We will show an example with torch.nn.MaxPool2d:

class M(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.maxpool2d = torch.nn.MaxPool2d(kernel_size=3)

    def forward(self, x):
        x = self.maxpool2d(x)
        return x
m = M().eval()        
m = prepare_fx(m, {"": torch.quantization.default_qconfig})
print(m.code)

<p align="center"> <table align="center"> <tr><th>1.9.1</th><th>1.10.0</th></tr> <tr valign="top"> <td>_{<pre lang="python"> def forward(self, x): x_activation_post_process_0 = self.x_activation_post_process_0(x); x = None maxpool2d = self.maxpool2d(x_activation_post_process_0); x_activation_post_process_0 = None return maxpool2d </pre>}</td> <td>_{<pre lang="python"> def forward(self, x): x_activation_post_process_0 = self.x_activation_post_process_0(x); x = None maxpool2d = self.maxpool2d(x_activation_post_process_0); x_activation_post_process_0 = None maxpool2d_activation_post_process_0 = self.maxpool2d_activation_post_process_0(maxpool2d); maxpool2d = None return maxpool2d_activation_post_process_0 </pre>}</td> </tr> </table> </p>

Note that self.maxpool2d_activation_post_process_0 and self.x_activation_post_process_0 will refer to the same observer/fake_quant instance, this is to simulate the numerics for the quantized maxpool implementation, where the output would reuse the quantization parameter of the input. Simple illustration with graph:

Before:

observer_0 - maxpool - ...

After:

observer_0 - maxpool - observer_0 (same observer instance as input observer) - ...

ONNX

Removed aten arg from torch.onnx.export(). (#62759)

The new OperatorExportTypes.ONNX removes the need for an explicit aten argument. If Pytorch was built with -DPYTORCH_ONNX_CAFFE2_BUNDLE the a None value means OperatorExportTypes.ONNX_ATEN_FALLBACK

<p align="center"> <table align="center"> <tr><th>1.9.1</th><th>1.10.0</th></tr> <tr valign="top"> <td>_{<pre lang="python"> torch.onnx.export(..., aten=True) </pre>}</td> <td>_{<pre lang="python"> torch.onnx.export(..., operator_export_type=torch.onnx.OperatorExportTypes.ONNX_ATEN) </pre>}</td> </tr> </table> </p>

Deprecations

Python API

Deprecate __torch_function__ as a plain methods (#64843)

The __torch_function__ function used to create Tensor like objects did not have any constraint whether it should be a method, class method or static method.

To make it compatible with newer features on Tensor-like objects, we are deprecating setting it as a plain method. You can define it as a class method to get the current class and scan the argument list if you need an object that is an instance of this class.

Mobile

Removed API torch.utils.bundled_inputs.run_on_bundled_input (#58344)

This API caused many issues and is not really necessary. The functionality (run model with bundled input) can be achieved by using get_all_bundled_inputs. For example:

1.9.1:

model.run_on_bundled_input(0)

1.10.0:

model(*model.get_all_bundled_inputs()[0])

Distributed

torch.distributed.rpc: Removed ProcessGroup RPC backend (#62411 , #62985)

ProcessGroup RPC backend has been deprecated and 1.9 was the last release which carried it. The default RPC backend is TensorPipe which is the recommended backend for RPC. Users who use torch.distributed.rpc.BackendType.PROCESS_GROUP will be given an error message to switch to torch.distributed.rpc.BackendType.TENSORPIPE.

ONNX

Removed following arguments in torch.onnx.export(): enable_onnx_checker, strip_doc_string, _retain_param_name (#64369, #64371, #64370)

enable_onnx_checker argument is removed. ONNX checker will now always run by default. Users can catch exceptions to ignore raised failures. strip_doc_string has been rolled into the verbose arg in torch.onnx.export(). _retain_param_name argument has been removed in torch.onnx.export() will default to True . There is no way to get the old behavior of _retain_param_name=False. Users should stop setting this arg.

1.9.1:

torch.onnx.export(..., enable_onnx_checker=False, strip_doc_string=False)

1.10.0:

try:
    torch.onnx.export(verbose=True)
except torch.onnx.utils.ONNXCheckerError:
   pass

Infra (Releng)

Disable ParallelTBB (#65092)

ParallelTBB config/codepath is no longer actively tested by PyTorch CI and as result is subject to code/functionality degradation

New features

Python API

• Added new functions:
  • torch.isin() (#53125), torch.bitwise_{left/right}_shift, __rlshift__, __rrshift__ (#59544), torch.Tensor.{__rand__, __ror__,__rxor__} (#59240), torch.aminmax (#62401), torch.new_ones (#58405)
  • For numpy compatibility torch.cov (#58311), torch.frombuffer (#59077), torch.corrcoef (#60420), torch.nanmean (#62671), torch.cumulative_trapezoid (#61615)
• The torch.special module is now stable! This module, consistent with SciPy’s special module, has 30 operations including the Hurwitz zeta function and various gamma functions. (#59623, #56352, #58126, #59141, #59143, #58650, #55878, #58838, #60512, #60641, #61633, #60519, #59691, #58194)
• Added support for slots and subclass magic getstate/setstate method for Tensor serialization (#62745)
• torch.optim:
  • Added Nesterov Adam as NAdam (#59009)
  • Added lr_scheduler.ChainedScheduler (#63491, #63457, #65034))
  • Added lr_scheduler.SequentialLR (#64037, #65035)
  • Added lr_scheduler.{ConstantLR,LinearLR} (#64395)
• torch.cpu.amp.autocast: enable new API for CPU autocast (#57386, #63534)
• Added BFloat16 support for torch.{cross, tril, triu, tril_indices, triu_indices, cumsum, cummax, cummin, median, kthvalue, nansum, nextafter, range, sinh, cosh, frexp, nan_to_num, sigmoid, sigmoid_backward, tanh_backward, addcmul, addcdiv, bucketize, bernoulli, dropout, fold, unfold, MaxPool2D, AdaptiveAvgPool2D, topk} on CPU (#62454, #63307, #55210, #60074, #61083, #61829, #55221, #61826, #55588, #56372, #62880, #55202, #59547)
• Added BFloat16 support for torch.{ceil, floor, frac, round, trunc, sort, topk, aminmax, cumsum, logcumsumexp, cumprod, cummin, cummax} on CUDA (#57910, #58196, #59977, #62767, #57904).
• Added torch.cuda.is_bf16_supported (#63798)
• Added zero rate to Poisson distribution (#61511)
• Added torch.segment_reduce (#59951, #60018, #61141, #61266, #59521, #60379, #60379)
• Added boolean support to torch.isclose (#61271)
• Added torch.trapezoid (#61475).
• Added torch.gradient support for second order central differences (edge_order=2) (#58165)
• torch.sigmoid: CUDA support and complex autograd support (#48647)
• Added channels-last support for torch.bilinear and torch.nn,MaxUnpool2d (#56322, #49984)

Autograd

• [Experimental] Forward mode AD:
  • NOTE: In addition to operators listed below, many simple ops are already supported. If you encounter an operator that does not have a forward-mode AD formula implemented, please file an issue. As a workaround, you can use custom autograd.Function to implement your own forward-mode-AD-supported operator.
  • Added forward-mode AD support for custom autograd.Function (#64061, #63434)
  • Added forward-mode AD support for torch.{acos, add, addbmm, addcdiv, addcmul, addmm, addmv, addr, angle, acosh, asinh, atanh, asin, atan, conj, baddbmm, bmm, cat, ceil, clamp, clamp_min, clamp_max, complex, copy_sign, cos, cosh, cross, cumprod, cumsum, cummax, cummin, deg2rad, div, dot, vdot, exp, exp2, expm1, expand, floor, frac, frexp, gather, hardswish, hstack, hypot, index_add_, index_copy_, index_put_, index_select, kthvalue, lerp, lgamma, digamma, polygamma, log, log10, log1p, log2, logaddexp, logaddexp2, xlogy, masked_fill_, masked_fill_, masked_scatter_, masked_select, max, maximum, fmax, mean, min, mininum, fmin, mm, mode, mul, lu, lu_solve, vstack} (#57768, #57863 #59711, #64742)
  • Added Forward AD support for the following element-wise and linear operators torch.{mvlgamma, nan_to_num, permute, pow, reciprocal, remainder, repeat, round, rsqrt, sigmoid, logit, sign, sgn, sin, sinc, sinh, sqrt, squeeze, sub, sum, t, flip, roll, rot90, take, tan, tanh, trace, transpose, tril, triu, trunc, unfold, unsqueeze, view, zero_, hardshrink} (#59993)
  • Added Forward AD support for torch.special.{xlog1py, entr} (#59711, #59993)
  • Added forward AD support for torch.linalg.{cholesky, cholesky_ex, eigh, inv, inv_ex, solve} (#62160, #64646, #62163, #62159)
  • Added forward AD support for torch.functional.leak_relu (#59993)
• Added saved tensor hooks to customize packing/unpacking behavior of tensors saved for backward (#60685, #60663, #62564, #60975, #62909, #62717)
• Exposed raw saved tensors for custom autograd.Function to use with the saved tensor hooks (#60551)
• Added default saved tensor hooks (#61834, #62563, #62361)
• Added context manager using default saved tensor hooks to automatically move saved tensors on CPU and back (#61928, #62410)
• Added C++ and python bindings for .is_inference() method (#58729)
• torch.lu_solve: Implement support for backward AD (#61681).

torch.nn

• Added new modules: nn.{ReflectionPad3d, LazyInstanceNorm*d} (#59791, #60837, #61308, #60982)
• nn.CrossEntropyLoss: Added support for class probability targets (#61044)
• nn.CrossEntropyLoss: Added support for label smoothing (#63122)
• nn.Module: Added support for arbitrary objects in state_dicts via get_extra_state() / set_extra_state() (#62976)
• nn.utils.skip_init(): Added function to skip module parameter / buffer initialization (#57555)

Profiler

• Added profiler support for mobile (#62419, #62418, #62417,#62228,#62191,#61792)
• Ported Nvtx support to new profiler (#61634)
• Added Tensor core usage stats and recommendations in Tensorboard (#364,#368,#383, #422)

CUDA

• Allow enabling warnings on CUDA synchronization (#62092)
• Added CUDA graph Prototype API and documentation (#63269)
• Make stream semantics of backward calls consistent with other cuda ops (#57833, #60230, #60127)
• Enabled autocast support for user-specified device and dtype (#61002, #63416)

C++ API

• Added C++ API for meta functions. They are available in the at::meta:: namespace (#58570)
• Exposed interface to set grain size on cpu_kernel, cpu_kernel_vec and cpu_kernel_multiple_outputs (#58949)
• Added at::native::resize_bytes_cpu to resize Storage in ATen (#60324)
• Added transpose to PackedTensorAccessor (#61114)
• Added torch::linalg::qr as the C++ API (#60529)
• Exposed amin and amax to aten symbols (#61550)
• Added support to invoke callable activation function for Transformer modules (#62342)
• Added support for c10::optional to compare with different but comparable types (#62890)
• Added a unified API c10::util::check_env to check environment variable (#59052)

TorchScript

• Added reference semantics to TorchScript classes (#44324)
• Conservatively moved all suitable prim ops from full-jit to mobile, and make them selective. (#58353)
• Added change to predicate uses of RPC APIs on torch.distributed.rpc.is_available() (#58887)
• Added a phase to perform inplace<->functional conversion for activation operators (#57477)
• Enabled Profile-Directed Typing in torch.jit.script (#62420)
• Introduced enhancement for smart serialization for operator schemas with out arg (#63096)
• Added a pass to transform better handle concatenation ops (#59881)
• Added a new operator for concat that takes in variadic parameters (#59880)
• Added support for union in TorchScript (#64234)

torch.package

• Added basic tooling to enable users to see what is inside of a PackageExporter (#61147)
• Added hasattr to torch::deploy C++ API (#62669)
• Added support to re-save a PackageImporter module (#65101)
• Added support to make frozen symbol name customizable in torch::deploy. (#63817)

Mobile

• Enabled kineto profiler on mobile via EdgeKinetoProfiler (#62419)
• Added support of loading lite interpreter module from assets in Android (#61609)
• Enabled tracing based selective build (#63421, #64087, #66237, #66395)
  • built tracer in OSS (#64087)
  • used operator.yaml to build libtorch library (#66237)
  • Built tracer and enabled tracing-based build with tracer output (#66395)
• NNAPI
  • Android NNAPI delegate implementation of runtime initialization (compilation) and execution (#62272)
  • Added aten::{avgpool2d,softmax,to,div,flatten,detach,slice,log_softmax,conv2d_transpose} to NNAPI converter (#58538, #58539, #58540, #58541, #60885, #58543, #59364, #61378, #59529
  • Added Int32 support for NNAPI (#59365)
  • Made nnapi aten::{conv2d,linear,cat,flatten} converter accept flexible batch (#61021, #61022, 76c0f223d3, #61024)
  • Added option to specify custom NNAPI serializer (#61025)
  • Made Android NNAPI preprocess to accept both single Tensor inputs and Tensor List inputs (#61752)
  • Added a few improvements in NNAPI delegation (#63489)
  • Added support const values in binary ops (2d58f3f56d)
• Added unary/binary ops necessary and more shape functions for mobilenet (#56828, #58932)
• Added aten::{hardswish,tanh,clamp} for iOS Metal (#64588, #61383)
• Added CoreML support (#64521, #64522, #64523)
• Added compatibility API (#61477, #57501)
• Added support operators with default argument in front of out argument (#63651, #63540)

Distributed

DistributedDataParallel

• Local SGD and variants for DDP communication optimization (#60303, #60320, #60632, #60891, #61206, #61207, #62105, #62111, #62131, #62132, #62392, #63277, #63340, #64885, #65197)
• Provided a noop hook for performance debugging (#64344, #64352)
• Implemented BF16 allreduce gradient communication hook (#63260)
• Allowed retrieval of model parameters in communication hook (#61637)

torch.distributed

• Added a function to create new subgroups of a given size (#59111)
• Introduced a new torchrun entry point for elastic (#64049)

torch.fx

• Added APIs to mutate specific args/kwargs (#58571)
• Introduced EngineHolder for serializing and running TRT Engines with PyTorch (06399d441d)
• Introduced __fx_create_arg__ dunder method for controlling custom classes are handled as node args (#61780)
• Added autowrap_functions kwarg to Tracer (#62106)
• Gradual typing
  • Added type annotation field to nodes (#60621)
  • Added experimental gradual typechecker (#60805)
  • Extended all experimental type-checking operations to support conv2d, BatchNorm2D, ReLU, maxpool2D, AdaptiveAvgPooling2D, flatten (#61093, #61012, #61150, #61188, #61239, #61265)
  • Added experimental refinement types and unification for symbolic shape inference (#61776)
  • Changed output node handling for typechecker to deal with tuples (#62582)
  • Added handle of get_attr operations in typechecker (#62682)
  • Added equality constraints for some acc operations for symbolic inference (#63689)
  • Added inference for algebraic expressions (#63822)
• Provided function interface for remove_duplicate_output_args (#65134)
• Introduced helper function to generate an unique name for an attr in a module (#64970)

ONNX

• Added support for ONNX op set 14 (#59486)
• Added support for GRU RNNs with packed input in scripting mode (#58691)
• Enhanced shape inference (#64585)
• Added support for torch.{linspace, new_ones, nn.LSTMCell, bernoulli, dot, nn.utils.spectral_norm,bernoulli, distributions.normal.Normal, roll} (#58854, #59255, #62757, #62765, #59536,#61560,#58697)