PyTorch 1.8.1 Release Notes

• New Features
• Improvements
• Bug Fixes
• Documentation

New Features

Revamp of profiling tools in torch.profiler

The torch.profiler submodule is now available. It leveraged the newly released kineto library for profiling. You can find more details in this blogpost: https://pytorch.org/blog/introducing-pytorch-profiler-the-new-and-improved-performance-tool/

Enable use of autocast for pytorch xla (#48570)

The torch.cuda.autocast package can now be used in conjunction with torch xla to provide easy mixed-precision training.

Improvements

• Make torch. submodule import more autocomplete-friendly (#52339)
• Add support in ONNX for torch.{isinf,any,all} (#53529)
• Replace thrust with cub in GPU implementation of torch.randperm for performance (#54537)

Bug fixes

Misc

• Fixes for torch.distributions validation checks (#53763)
• Allow changing the padding vector for nn.Embedding (#53447)
• Fix TensorPipe for large copies and interoperability with CUDA (#53804)
• Properly de-sugar Ellipsis in TorchScript (#53766)
• Stop using OneDNN for group convolutions when groups size is a multiple of 24 (#54015)
• Use int8_t instead of char in {load,store}_scalar (#52616)
• Make ideep honor torch.set_num_thread (#53871)
• Fix dimension out of range in pixel_{un}shuffle (#54178)
• Update kineto to fix libtorch builds (#54205)
• Fix distributed autograd CUDA stream synchronization for send/recv operations (#54358)

ONNX

• Update error handling in ONNX to avoid ValueError (#53548)
• Update assign output shape for nested structure and dict output (#53311)
• Update embedding export wrt padding_idx (#53931)

Documentation

• Doc update for torch.fx (#53674)
• Fix distributed.rpc.options.TensorPipeRpcBackendOptions.set_device_map (#53508)
• Update example for nn.LSTMCell (#51983)
• Update doc for the padding_idx argument for nn.Embedding (#53809)
• Update general doc template (#54141)

PyTorch 1.8.0 Release Notes

• Highlights
• Backwards Incompatible Change
• New Features
• Improvements
• Performance
• Documentation

Highlights

We are excited to announce the availability of PyTorch 1.8. This release is composed of more than 3,000 commits since 1.7. It includes major updates and new features for compilation, code optimization, frontend APIs for scientific computing, and AMD ROCm support through binaries that are available via pytorch.org. It also provides improved features for large-scale training for pipeline and model parallelism, and gradient compression. A few of the highlights include:

1. Support for doing python to python functional transformations via torch.fx;
2. Added or stabilized APIs to support FFTs (torch.fft), Linear Algebra functions (torch.linalg), added support for autograd for complex tensors and updates to improve performance for calculating hessians and jacobians; and
3. Significant updates and improvements to distributed training including: Improved NCCL reliability; Pipeline parallelism support; RPC profiling; and support for communication hooks adding gradient compression. See the full release notes here.

Along with 1.8, we are also releasing major updates to PyTorch libraries including TorchCSPRNG, TorchVision, TorchText and TorchAudio. For more on the library releases, see the post here. As previously noted, features in PyTorch releases are classified as Stable, Beta and Prototype. You can learn more about the definitions in the post here.

You can find more details on all the highlighted features in the PyTorch 1.8 Release blogpost.

Backwards Incompatible changes

Fix Tensor inplace modulo in python (#49390)

Inplace modulo in python %= was wrongfully done out of place for Tensors. This change fixes the behavior. Previous code that was relying on this operation being done out of place should be updated to use the out of place version t = t % other instead of t %= other.

<p align="center"> <table align="center"> <tr><th>1.7.1</th><th>1.8.0</th></tr> <tr valign="top"> <td><sub><pre lang="python">

a = torch.arange(0, 10) b = a b %= 3 print(a) tensor([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) print(b) tensor([0, 1, 2, 0, 1, 2, 0, 1, 2, 0]) </pre></sub></td> <td>_{<pre lang="python"> a = torch.arange(0, 10) b = a b %= 3 print(a) tensor([0, 1, 2, 0, 1, 2, 0, 1, 2, 0]) print(b) tensor([0, 1, 2, 0, 1, 2, 0, 1, 2, 0]) </pre>}</td> </tr> </table> </p>

Standardize torch.clamp edge cases (#43288)

For ease of exposition let a_min be the value of the "min" argument to clamp, and a_max be the value of the "max" argument to clamp.

This PR changes the behavior of torch.clamp to always compute min(max(a, a_min), a_max). torch.clamp currently computes this in its vectorized CPU implementation but uses different approaches for other backends. These implementations are the same when a_min < a_max, but divergent when a_min > a_max. This divergence is easily triggered:

>>> t = torch.arange(200).to(torch.float)
>>> torch.clamp(t, 4, 2)[0]
tensor(2.)

>>> torch.clamp(t.cuda(), 4, 2)[0]
tensor(4., device='cuda:0')

>>> torch.clamp(torch.tensor(0), 4, 2)
tensor(4)

This PR makes the behavior consistent with NumPy's clip. C++'s std::clamp's behavior is undefined when a_min > a_max. Python has no standard clamp implementation.

Tensor deepcopy now properly copies the .grad field (#50663)

The deepcopy protocol will now properly copy the .grad field of Tensors when it exists. The old behavior can be recovered by setting the .grad field to None after doing the deepcopy.

<p align="center"> <table align="center"> <tr><th>1.7.1</th><th>1.8.0</th></tr> <tr valign="top"> <td><sub><pre lang="python">

t.grad tensor([0.8883, 0.5765]) deepcopy(t).grad None </pre></sub></td> <td>_{<pre lang="python"> t.grad tensor([0.8883, 0.5765]) deepcopy(t).grad tensor([0.8883, 0.5765]) </pre>}</td> </tr> </table> </p>

Fix torch.fmod type promotion (#47323, #48278)

1.7.1 Raises RuntimeError for integral tensor and floating-point tensor. The dtype of output is determined by the first input.

>>> x = torch.arange(start=1, end=6, dtype=torch.int32) # tensor([1, 2, 3, 4, 5])
>>> y = torch.arange(start=1.1, end=2.1, step=0.2, dtype=torch.float32) # tensor([1.1, 1.3, 1.5, 1.7, 1.9])
>>> torch.fmod(x, y)
RuntimeError: result type Float can't be cast to the desired output type Int
>>> z = torch.arange(start=0.2, end=1.1, step=0.2, dtype=torch.float64) # tensor([0.2, 0.4, 0.6, 0.8, 1.], dtype=torch.float64)
>>> torch.fmod(y, z).dtype
torch.float32
>>> torch.fmod(z, y).dtype
torch.float64
>>> torch.fmod(x, 1.2)
tensor([0, 0, 0, 0, 0], dtype=torch.int32)

1.8.0: Support integral tensor and floating-point tensor as inputs. The dtype of output is determined by both inputs.

>>> x = torch.arange(start=1, end=6, dtype=torch.int32) # tensor([1, 2, 3, 4, 5])
>>> y = torch.arange(start=1.1, end=2.1, step=0.2, dtype=torch.float32) # tensor([1.1, 1.3, 1.5, 1.7, 1.9])
>>> torch.fmod(x, y)
tensor([1.0000, 0.7000, 0.0000, 0.6000, 1.2000])
>>> z = torch.arange(start=0.2, end=1.1, step=0.2, dtype=torch.float64) # tensor([0.2, 0.4, 0.6, 0.8, 1.], dtype=torch.float64)
>>> torch.fmod(y, z).dtype
torch.float64
>>> torch.fmod(z, y).dtype
torch.float64
>>> torch.fmod(x, 1.2)
tensor([1.0000, 0.8000, 0.6000, 0.4000, 0.2000])

Preserve non-dense or overlapping tensor's layout in *_like functions (#46046)

All the *_like factory functions will now generate the same striding as out of place operations would. This means in particular that non-contiguous tensors will produce non-contiguous outputs. If you require a contiguous output, you can pass the memory_format=torch.contiguous keyword argument to the factory function. Such factory functions include clone, to, float, cuda, *_like, zeros, rand{n}, etc.

Make output of torch.norm and torch.linalg.norm consistent for complex inputs (#48284)

Previously, when given a complex input, torch.linalg.norm and torch.norm would return a complex output. torch.linalg.cond would sometimes return a complex output and sometimes return a real output when given a complex input, depending on its p argument. This PR changes this behavior to match numpy.linalg.norm and numpy.linalg.cond, so that a complex input will result in a real number type, consistent with NumPy.

Make torch.svd return V, not V.conj() for complex inputs (#51012)

torch.svd added support for complex inputs in PyTorch 1.7, but was not documented as doing so. The complex V tensor returned was actually the complex conjugate of what's expected. This PR fixes the discrepancy. Users that were already using the previous version of torch.svd with complex inputs can recover the previous behavior by taking the complex conjugate of the returned V.

torch.angle: properly handle pure real numbers (#49163)

This PR updates PyTorch's torch.angle operator to be consistent with NumPy's. Previously torch.angle would return zero for all real inputs (including NaN). Now angle returns pi for negative real inputs, zero for non-negative real inputs, and propagates NaNs.

Enable distribution validation by default for torch.distributions (#48743)

This may slightly slow down some models. Concerned users may disable validation by using torch.distributions.Distribution.set_default_validate_args(False) or by disabling individual distribution validation via MyDistribution(..., validate_args=False).

This may cause new ValueErrors in models that rely on unsupported behavior, e.g. Categorical.log_prob() applied to continuous-valued tensors (only {0,1}-valued tensors are supported). Such models should be fixed but the previous behavior can be recovered by disabling argument validation using the methods mentioned above.

Prohibit assignment to a sparse tensor (#50040)

Assigning to a sparse Tensor did not work properly and resulted in a no-op. The following code now properly raises an error:

>>> t = torch.rand(10).to_sparse()
>>> t[0] = 42
TypeError: Cannot assign to a sparse tensor

C++ API: operators that take a list of optional Tensors cannot be called with ArrayRef<Tensor> anymore (#49138)

This PR changes the C++ API representation of lists of optional Tensors (e.g. in the Tensor::``index method) from ArrayRef<Tensor> to List<optional<Tensor>>. This change breaks backwards compatibility, since there is no implicit conversion from ArrayRef<Tensor> to List<optional<Tensor>>.

A common call pattern is tensor.index({indices_tensor}), where indices_tensor is a Tensor. This will continue to work because the {} initializer_list constructor for List<optional<Tensor>> can take Tensor elements that are implicitly converted to optional<Tensor>.

However, another common call pattern is tensor.index(indices_tensor), where previously the Tensor got implicitly converted to an ArrayRef<Tensor>. To implicitly convert Tensor -> optional<Tensor> -> List<optional<Tensor>> would chain two implicit conversions, which C++ doesn't allow. So those call sites should be rewritten to use the tensor.index({indices_tensor}) pattern.

Autograd view creation informations are now properly propagated when views are chained

After this fix, an error will properly be thrown to avoid wrong gradients when an in-place operation is performed on a view of a view, when in-place operation were not allowed on the first view. This means that code that used to return wrong gradients in 1.7.1 (such as t.unbind()[0].select(0, 0).add_(1)) will now properly raise an error.

End of deprecation cycle for spectral ops in the torch. namespace (#48594)

This PR removes the deprecated torch.{fft,rfft,ifft,irfft} and their corresponding methods on torch.Tensor. PyTorch programs using these functions must now update to use the torch.fft namespace.

torch.digamma : properly handle all inputs (#48302)

This PR updates PyTorch's torch.digamma function to be consistent with SciPy's special.digamma function. This changes the result of the torch.digamma function on the nonpositive integers, where the gamma function is not defined. Since the gamma function is undefined at these points, the (typical) derivative of the logarithm of the gamma function is also undefined at these points, and for negative integers this PR updates torch.digamma to return NaN. For zero, however, it returns -inf to be consistent with SciPy.

Interestingly, SciPy made a similar change, which was noticed by at least one user: scipy/scipy#9663

SciPy's returning of negative infinity at zero is intentional: https://github.com/scipy/scipy/blob/59347ae8b86bcc92c339efe213128f64ab6df98c/scipy/special/cephes/psi.c#L163

This change is consistent with the C++ standard for the gamma function: https://en.cppreference.com/w/cpp/numeric/math/tgamma

Fix torch.remainder type promotion (#48668)

1.7.1: In the case where the second argument is a python number, the result is casted to the dtype of the first argument.

>>> torch.remainder(x, 1.2)
tensor([0, 0, 0, 0, 0], dtype=torch.int32)

1.8.0 In the case where the second argument is a python number, the dtype of result is determined by type promotion of both inputs.

>>> torch.remainder(x, 1.2)
tensor([1.0000, 0.8000, 0.6000, 0.4000, 0.2000])

Changes to onnx export API to better handle named arguments (#47367)

The args input argument of the torch.onnx.export function is updated to better support optional arguments. An optional dictionary can be passed in addition as the last argument in the args tuple, specifying inputs with the corresponding named parameter. Note that this is backward breaking for cases where the last input is also of a dictionary type. In the new API, for such cases, it is mandatory to have an empty dictionary as the last argument in the args tuple. More details can be found at: https://pytorch.org/docs/1.8.0/onnx.html?highlight=onnx#using-dictionaries-to-handle-named-arguments-as-model-inputs.

Update signature of torch.quantization.quantize function #48537

The run_args argument must now contain a list or tuple containing the positional arguments, even if there is only a single argument. In particular, code like: qmodel = quantize(float_model, default_eval_fn, img_data) that was working in 1.7.1 will now raise the error: TypeError: default_eval_fn() takes 2 positional arguments but 3 were given. You should update this code to provide the image in a list for example: qmodel = quantize(float_model, default_eval_fn, [img_data])

Change the way we quantize relu, leaky relu and sigmoid(#47415, #48038, #45702,#45711, #45883 #45883, #45882, #47660)

Starting with version 1.8.0, in the eager mode quantization flow, relu is not observed anymore as it is not needed. In previous versions, quantized leaky_relu and sigmoid did not require observation and just inherited the quantization parameters from their input, but that does not work very well in eager mode quantization. Starting with version 1.8.0, they are observed operator so that they work better in eager mode quantization.

Update direction numbers to 21201 dims in the SobolEngine (#49710)

This update is BC-breaking because the values drawn by the engine will be different from the ones drawn in 1.7.1 even with the same seed.

<p align="center"> <table align="center"> <tr><th>1.7.1</th><th>1.8.0</th></tr> <tr valign="top"> <td><sub><pre lang="python">

from torch.quasirandom import SobolEngine eng = SobolEngine(1) eng.draw(3) tensor([[0.5000], [0.7500], [0.2500]]) </pre></sub></td> <td>_{<pre lang="python"> from torch.quasirandom import SobolEngine eng = SobolEngine(1) eng.draw(3) tensor([[0.0000], [0.5000], [0.7500]]) </pre>}</td> </tr> </table> </p>

Deprecations

Python API

Deprecate old style nn.Module backward hooks (#46163)

Old style nn.Module backward hooks have been broken for a long time (they do not behave as advertised in the documentation). We now have new nn.Module.register_full_backward_hook that provide a fully working implementation of these hooks. The old function should not be used and migrated to the new full version.

An example of this discrepancy is shown in the example below where a Linear layer takes as input a single Tensor of size 5 and returns a single Tensor of size 5 but old style hook would return two gradients with respect to the input for only one input.

1.7.1:

import torch
from torch import nn

mod = nn.Linear(5, 5)
def hook(mod, grad_inp, grad_out):
    print(f"grad input size: " + " ".join(str(g.size()) for g in grad_inp))
    print(f"grad output size: " + " ".join(str(g.size()) for g in grad_out))
mod.register_backward_hook(hook)

mod(torch.rand(5, requires_grad=True)).sum().backward()
>>> `grad input size: torch.Size([5]) torch.Size([5]) # One too many
>>> grad output size: torch.Size([5])`

1.8.0: Old style hooks are deprecated and will warn when providing wrong result.

import torch
from torch import nn

mod = nn.Linear(5, 5)
def hook(mod, grad_inp, grad_out):
    print(f"grad input size: " + " ".join(str(g.size()) for g in grad_inp))
    print(f"grad output size: " + " ".join(str(g.size()) for g in grad_out))
mod.register_backward_hook(hook)

mod(torch.rand(5, requires_grad=True)).sum().backward()
>>> grad input size: torch.Size([5]) torch.Size([5]) # One too many
>>> grad output size: torch.Size([5])
>>> `UserWarning: Using a non-full backward hook when the forward contains multiple
autograd Nodes is deprecated and will be removed in future versions. This hook
will be missing some grad_input.`

Full hooks should be used to get the proper result all the time and avoid warnings

mod.register_full_backward_hook(hook)

mod(torch.rand(5, requires_grad=True)).sum().backward()
>>> grad input size: torch.Size([5])
>>> grad output size: torch.Size([5])

torch.stft: Deprecate default value of the require_complex argument (#49022, #50102)

Previously torch.stft took an optional return_complex parameter that indicated whether the output would be a real tensor or a complex tensor. return_complex has the default value of False. This default value is deprecated (meaning that this optional argument is becoming mandatory) and will be removed in future versions. You can pass this argument explicitly to avoid this deprecation.

Deprecate torch.set_deterministic in favor of torch.use_deterministic_algorithms (#49904)

This beta feature is being renamed for improved clarity. Users should migrate to use the new name.

Deprecate torch.* linear algebra functions in favor of the torch.linalg.* variant for cholesky (#51460), slogdet (#51354), inverse (#51672), pinverse (#51671)

All the linear algebra functions are being moved to the torch.linalg submodule that provided a compatible API with NumPy. These new functions have the same set of features as the torch. ones and should be used instead.

New features

Python API

• New functions (most of them to improve numpy compatibility): torch.nan_to_num (#44592), torch.tensor_split (#45168), torch.nanmedian (#45847), torch.ravel (#46098), torch.igamma (#46183), torch.igammac (#48171), torch.{column_stack,row_stack} (#46313), torch.kron (#45358), torch.copysign (#46396), Tensor.new_empty_strided (#47225), torch.{swapdims,swapaxes} (#46041), torch.tile (#47974), torch.float_power (#44937), torch.moveaxis (#48581), torch.inner (#46716), torch.msort (#48440), torch.sinc (#48740), torch.broadcast_to (#48997), torch.xlogy (#48777), torch.f{max,min} (#49312), torch.diff (#50569), torch.ldexp (#45370), torch.broadcast_shapes (#43935),
• torch.fft new features: 2D FFT functions (#45164), use new FFT operators in stft (#47601), helper functions (#44877), fuzzing benchmark (#47872)
• torch.linalg new features: linalg.tensorsolve (#46142), linalg.cholesky (#46083), linalg.tensorinv (#45969), linalg.{eigh,eigvalsh} (#45526), linalg.matrix_rank (#48206), linalg.solve (#48456), linalg.qr (#47764, #50046), linalg.svd (#45562), linalg.inv (#48261), linalg.pinv (#48399), linalg.slogdet (#49194), linalg.cond (#45832)
• New torch.nn Modules: nn.PixelUnshuffle (#49334), nn.GaussianNLLLoss (#50886)
• Automatic shape inference in torch.nn: new nn.LazyLinear (#44538), nn.LazyConv{1,2,3}d and nn.LazyConvTranspose{1,2,3}d (#47350)
• Add channels last support for torch.nn.AdaptiveAvgPool2d (#48916)
• Add option to produce standalone executable with cpp_extensions (#47862)
• Add sparse-sparse matrix multiplication support (#39526)
• Add torch.futures.Future.add_done_callback (#45675)
• Add three_phase optional argument to torch.optim.lr_scheduler.OneCycleLR (#42715)
• Add bicubic option for the mode argument of torch.nn.functional.grid_sampler (#44780)
• Add new distributions to torch.distributions: Kumaraswamy (#48285), LKJCholesky (#48798)
• Add reparameterization support to torch.distributions.OneHotCategorical (#46610)
• Add new transforms to torch.distributions: CorrCholeskyTransform (#48041)
• Add new constraint to torch.distributions: independent (#50547, #50302)
• Add zero annealing epochs to SWA optimizer (#47579)
• Add close method to torch.hub.tqdm mock (#46040)
• Add support for pruning based on custom importance scores via the importance_scores keyword argument (#48378)
• Add torch vitals (#51047)

Complex Numbers

• Complex Number support on CPU and CUDA for torch.symeig (#45121), torch.pinverse (#45819), torch.det (#45980), torch.diagflat (#47564), torch.{addcmul, addcdiv} (#46639), torch.lu_solve (#48028), torch.matrix_exp (#48363), torch.eig (#49168), torch.{acosh, asinh, atanh} (#50387), torch.masked_scatter (#51281), torch.bmm and torch.baddbmm (#42553), torch.orgqr (#50502), torch.index_fill_ (#50578), torch.cholesky_inverse (#50269)
• Complex Number support on CUDA for torch.qr (#45032), torch.lu (#45898), torch.prod(#45980), torch.triangular_solve (#46916), torch.solve (#47045), torch.cholesky_solve (#47047), torch.mean (#47048), torch.svd (#45795), torch.inverse (#47595), torch.Tensor.index_put_ (#51148)
• Complex Number support on CPU for torch.trace (#50380)
• Complex Number support for torch.nn.DataParallel (#48686), torch.nn.L1Loss (#49912), Padding functions (#50594)
• Complex Number support for torch.distributed.{all_reduce, all_gather} (#45879, #46270)
• Complex Autograd support for torch.{atan, log, log10, log1p, log2, reciprocal, tan, pow, rsqrt, tanh, asinh, acosh} (#46275), torch.{cholesky, triangular_solve, mm, mv, ger} (#45737), torch.take(), torch.Tensor.fill_() (#46860), torch.matrix_exp (#48363), torch.{baddbmm, addbmm, addmm, addmv} (#50632), torch.qr (#48489), torch.svd and torch.pinverse (#47761), torch.sqrt (#49461), torch.diag (#51268), torch.trace (#51537), torch.exp (#47194), torch.mean (#47566), torch.addr (#50667), torch.{stack, gather, index_select}, torch.Tensor.index_add_(#49552), torch.{masked_scatter, masked_select} (#51281), torch.{addcmul, addcdiv} (#46639), torch.{acosh, asinh, atanh} (#50387), torch.solve (#47045), torch.cholesky_solve (#47047), torch.inverse (#47595)
• Add complex autograd support for named tensors (#47289)
• Allow converting parameters and buffers of torch.nn.Module to complex dtypes (#44788)
• Add complex support to IValues (#50883, #51476)
• Add TorchScript type annotation logic for complex numbers (#50884)
• Add serialization logic for complex numbers (#51287)
• Add support for complex number lists in JIT (#51145)
• Add support for complex valued keys for dict in TorchScript (#51472)
• Add scalar.conj() (#46596)
• Add Tensor.copy_() for ComplexHalf tensors (#45339)

Profiler

• New profiler API (#48280)
• Use libkineto in profiler (#46470)
• Add FLOPS computation support to the new profiler API (#51734)
• Add high level profiling trace for dataloading and optimizer (#47655)
• Add support for SVG visualization (#48438)

Autograd

• Add inputs argument to autograd.backward() both in python and c++ (#46855, #47214)
• Add support for Tensor-like objects in torch.autograd.gradcheck (#45732)
• Add experimental vectorize flag to torch.autograd.functional.{jacobian, hessian} (#50915, #51638)
• Add anomaly mode in C++ API (#46981, #47164)
• Make torch.lu differentiable. (#46284)
• Add support for generators in autograd decorators like torch.no_grad (#49017)

Dataloader

• Add BufferedShuffleDataset (#45290)
• Add warning if DataLoader is going to create excessive number of thread (#46867)
• Add prototype of BatchIterDataPipe (#49186, #51880)
• Add prototype of SamplerIterDataPipe (#49363, #52104)
• Implement BucketBatchIterDataPipe (#51126, #51880)
• Add Tar DataPipe-s (#51398)
• Add MapIterDataPipe (#51488#51879)

CUDA

• Allow user to specify a fraction of the GPU memory with set_per_process_memory_fraction. (#48172)
• CUDA BFloat16 TopK (#44755)
• Add LazyNVRTC (#45674)
• Enable CUDA Fuser for ROCm (#45965)
• Define the record_stream method in native_functions.yaml (#44301)
• Add CUDA 11.1 docker build (#46283)
• Add nvtx.range() context manager (#42925)
• CUDA BFloat16 gelu, hardswish, hardsigmoid (#44997)
• [ROCm] enable stream priorities (#47136)
• Add bfloat support for torch.randn and torch.norm (#47143)
• CUDA BFloat16 Dropout (#45005), batchnorm (non-cuDNN) (#44994), backwards (#48809), sparse (#48807), indexing (#48801), embedding (#44848), signal windows (#45155), norm (#48806), isinf and isfinite (#49356), gemms on arch other than ampere (#50442), clamp, remainder, lshift, rshift (#45247)
• Make CUDAGeneratorImpl capturable (#48694)
• Adding support for CuDNN-based LSTM with projections (#47725)
• Add torch.cuda.can_device_access_peer (#50446)
• Add torch::cuda::ncll::all2all (#45900)

C++ API

• Add distance-agnostic triplet margin loss (#45377)
• Add torch::nn::ModuleDict (#47707)
• Add torch::cuda::synchronize (#50072)
• Add new XPU backend type for Intel heterogeneous computation platform. (#49786)

TorchScript

• torch::jit::freeze C++ api introduced (#52337, #52392)
• Add API for ignoring arbitrary module attributes during compilation (#45262)
• Support tracing tensor __setitem__ with dynamic shape (#45828)
• Expose script_if_tracing as public API (#46494)
• Support %-based string formatting (#45976)
• Add torch.jit.isinstance support for typed containers (#46062)
• Allow for source code comments at any level of indentation (#46548)
• Support hashing of various data types by implementing generic hashing for IValues (#46441)
• Support doc string for TorchBind custom classes (#46576)
• Add API for selective lowering of modules to custom JIT backend (#43613)
• add list() support (#42382)
• Support using lambda function as TorchBind constructor (#47819)
• Support user defined classes as constants (#45556)
• Allow del statements with multiple targets (#48876)
• Tuple Slice with both negative and positive stepped size (#48660)
• Expose run_async function on torch::jit::Method (#48607)
• Add flag torch_jit_disable_warning_prints to allow disabling all warnings.warn (#49313)
• Add dict comprehension (#47774)
• Adding support for bitwise augassignment operators (+= style statements) (#44621)
• Support the in operator with str (#47057)
• Adding JIT support for cuda streams and events (#48020)
• Add Type::{castRaw,expectRef} (#50061)
• Allow arbitrary docstrings to be inside torchscript interface methods (#50271)
• Change list striding parameters to take optional integer (#48719)
• Add support for scripting and running module level hooks in JIT (#49544, #49975, #49545, #49546, #49547)
• Support default argument values of a method (#48863)
• Graceful invalidation of Python Node/Value/Block when C++ object is deleted (#50326)
• Support Union[NoneType, T] as input type (#51605)
• Allow implicit boolean conversion of lists, strings, and dictionaries (#51683)

Mobile

• Add instance_key into mobile stats logging. (#45517)
• Profiling allocator for mobile. (#43951)
• [Metal] Add Metal/MPSCNN support on iOS (#46112)
• [Metal] Introduce USE_PYTORCH_METAL (#46383)
• [Metal] Support Resnet models (b63ddd6f57)
• PyTorch NNAPI integration prototype (#46780)
• [Metal] Enable Metal on macosx (#47635)
• [Metal] Enable optimize_for_mobile on Linux (#46384)
• [Android] Fix YUV camera image to tensor (#50871)
• [Android] turn on USE_VULKAN for android builds by default (#51291)
• Add windows JNI support (#44257)
• Enable partial loading of GPU models on linux CPU machines (#51236)

Distributed

• Support send and recv in c10d NCCL backend (#44921, #44922)
• Add support for NCCL alltoall (#44374)
• Upstream fairscale.nn.Pipe into PyTorch as torch.distributed.pipeline (#44090)
• Add a --logdir option to log subprocess output to files in DDP launcher. (#33193)
• Support RRef.backward() for local RRefs. (#46568) and Owner RRefs. (#46641)
• Support C++ implementation for DDP communication hook. (#46566)
• Provide 2 default C++ comm hooks for DDP (#46701)
• Support remote device format "worker_name/device" (#46773)
• Enable creation and transfer of ScriptModule over RPC (#48293)
• Enable TCPStore on Windows (#47749)
• Support torch.distributed.irecv(src=None, ...) as recv_anysource (#49383)
• Implement layer-wise PowerSGD as a DDP comm hook (#49639)
• Support alltoall_single in TorchScript (#48345)
• Enable GPU-to-GPU comm in TensorPipeAgent (#44418)
• Support timeout in rref._get_type() (#50498)
• Support timeout for RRef proxy functions (#50499)
• Add optimizer state sharding as ZeroRedundancyOptimizer (#46750)
• Add distributed functional Adam optimizer (#50624), sgd optimizer (#50618), Adadelta optimizer (#50623), RMSprop optimizer (#50619), l AdamW optimizer (#50620)
• Create a DDPLoggingData struct and expose it to python interface (#50622)
• Implement autograd functions for c10d communication operations (#40762)
• Enable TensorPipe's SHM transport (#50760)
• Support device map for distributed autograd while using TensorPipe. (#44859)
• Create PyTorch DDP logging APIs for applications to use (#50637)
• Add set_exception API in torch.futures.Future (#50983)
• Add scatter_object_list API for c10d (#43930)
• Provide parameter to pass GPU ID in barrier function (#49069)
• Enable TensorPipe CUDA fallback channel (#50675)
• Enable TensorPipe's InfiniBand transport (#50761)

torch.fx

• allow custom behavior for args, kwargs, and bool (#45193)
• Mutable Graph APIs (#45227)
• Make output a non-special Node (#45599)
• Make Tracer.trace() just return a Graph (#45704)
• Preserve type annotations on generated code in Graph (#45880)
• Make graph_copy examine existing values in val_map (#46104)
• Allow tracing free functions (#46268)
• Make sure args/kwargs are immutable (#46325)
• Make wrapped functions traceable (#46692)
• Added GraphModule.to_folder (#47544)
• Support default args in symbolic tracing (#47615)
• Add Node.all_input_nodes (#48270)
• Support torchbind as attribute in torch.fx symbolic tracing (#48732)
• Create subgraph rewriter API (#49540)
• Make len traceable and scriptable with wrap (#50184)
• Add Interpreter and Transformer APIs (#50420)
• Add alternative prettyprinting method to Graph (#50878)
• Move some heavily used passes out of experimental (#51392)
• Added partial concrete values for symbolic tracing (#51609)

Quantization

• Quantized Operators and Modules
  • Embedding and EmbeddingBag operator support
    • creating quint4x2 dtype for quantized tensors (#44678)
    • PerChannelFloatQParams support for quint4x2 dtype (#45594)
    • Add 4-bit embedding_bag prepack/unpack support using quint4x2 (#45751)
    • Support 4-bit embedding_bag operators using the dtype quint4x2 (#45752)
    • Support for 4-bit quantized EmbeddingBag module (#45865)
    • Refactor qembeddingbag to remove duplicate code (#45881)
    • Rename the sparse argument for embedding_bag ops (#46003)
    • Add support for pruned weights in embedding_bag_byte lookup (#47329)
    • fp16 -> fp32 EmbeddingBag moved into CPU impl (#47076)
    • Add non-fbgemm fallback implementation for embedding lookup ops (#50706)
    • Out variant for embedding_bag_4bit_rowwise_offsets (#51324)
    • Using int32 as indices for embedding_bag operators (#45878)
  • Add transposed conv support for fbgemm backend for 1d, 2d, 3d (#46607, #46608)
  • Add quantized flip dispatch (#46235)
  • Add support for ReflectionPad2d (#48036)
  • Dynamic GRU quantization support (#49448)
  • Quantizable LSTM (#49671)
• Quantization Flow/API
  • quantization: Linear + BatchNorm1d fusion (#50748)
  • compare_model_stub_fx API implementation (#48951)
  • Add additional_fuser_method_mapping to config (#46355)
  • Compare Weights FX Implementation (#48056)
  • Numeric Suite: Swap with shadow modules only for quantized part of model (#51052)
• FX Graph Mode Quantization
  • Add prepare_custom_config_dict and convert_custom_config_dict (#46223, #46364)
  • Add FixedQParamsFakeQuantize module (#46657)
  • Add support for additional_fuse_method_mapping (#46345), additional_{fusion/quant}_pattern (#46346)
  • Support in qat sigmoid/hardsigmoid/tanh (#46871), convbn{relu}1d (#47248), FloatFunctional (#46634)
  • custom_module support static/dynamic/weight_only quant (#46786)
  • Support standalone_module_class (#47705)
  • Embedding/EmbeddingBag works in static quant qconfig (#48062)
  • Add MatchAllNode in pattern matching (#48979)
  • Add support for dynamic quant for RNN and RNNCell (#49126), ConvTranspose{n}d (#49717), quantizing functional linear + {functional relu/module relu} (#50975), functional conv2d + relu (#51079), functional conv1d and conv3d (#51155) (#51254), Scalar as first input for add/mul (#46751), leaky relu (#45712), Embedding (#46677), EmbeddingBag (#46678)
  • Remove inplace option for convert_fx (#46955)
  • Support non_traceable_module/module_class (#46298)
  • Add additional_object_mapping argument to convert (#46338)
  • Keep linear op unchanged when qconfig is not supported (#48067)
  • Move {input|output}_quantized_idxs cfg from convert to prepare (#49238)
  • Allow user to specify qconfig for call_method (#49621)
  • Do not observe bias on F.conv and F.linear (#49623, #49628)
  • Linear work with float_qparam_dynamic_qconfig (#47068)
  • Fix error that DefaultQuantizer is not inserted after a module configured with None qconfig (#47316)
  • Scope support for call_method in QuantizationTracer (#50173)
  • Support preserved_attributes in prepare_fx (#50306)
  • Add option to leave graph inputs and/or outputs quantized (#48624)
  • Support quantization for custom module (#44074)
  • Remove inplace option for fuse_fx (#46953) and prepare_fx (#46954)
  • Scope support for call_function in QuantizationTracer (#51086)

ONNX

• Preprocess index_put with bool inputs to torch.masked_{scatter,fill} (#45584)
• Export torch.{var,var_mean,std_mean} ops (#45678)
• Enable NoneType inputs to export API (#45792)
• Add export of prim::dtype, prim::tolist (#46019)
• Enable onnx shape inference in export by default (#46629)
• Add torch.silu operator support for onnx (#51519)
• Support list remove for onnx export (#51526)
• Added torch.hardswish symbolic in opset 9 (#48423)
• Add export of aten::is_floating point (#46442)
• Add torch.logical_{and,or,xor} torch op support in pytorch exporter (#50909)
• Add torch.binary_cross_entropy_with_logits op to ONNX opset version 12 (#50908)
• Support opset13 nn.Squeeze and nn.Unsqueeze (#50906)
• Add export of prim::data (#45747)
• Support torch.nonzero(*, as_tuple=True) export (#47421)
• Update Reducesum operator for opset 13 (#50907)

Misc

• Enable python code coverage on windows (#44548) and onnx (#47387)
• Fix PyTorch compilation on Apple M1 chips (#48275, #49701)

Improvements

Python API

• Add integer support (by promoting integer to float) to torch.{cos,sin,tan} (#45733, #46706), torch.log{2,10} (#46810), torch.{a}tanh (#47064), torch.a{cos, tan} (#47005), torch.a{cosh, sinh} (#47152), torch.sqrt (#47293), torch.log1p (#48002). torch.erf{c} (#48472), torch.asin (#48461), torch.sigmoid (#47551), torch.sinh (#48644), torch.cosh (#48923), torch.exp{2, m1}(#48926), torch.reciprocal (#49102), torch.erfinv (#49155), torch.rsqrt (#47909), torch.exp (#50093), torch.lgamma (#50140)
• Add optional dtype argument to Tensor.view (#47951)
• Add out optional arguments to torch.{reshape,flatten} (#51249), torch.tensordot (#47278), torch.fft.* (#49335), torch.narrow_copy (#49502)
• Add support for int32 indices and offset in nn.Embedding and nn.EmbeddingBag (#46758)
• Add boolean type support to torch.where (#47454), torch.mul and Tensor.__mul__ (#48637), torch.diag (#47455), torch.{all,any} (#44790), Tensor.to_dense (#50019)
• Add inplace version of torch.cum{sum,prod}_ (#47651)
• Add sparse support to torch.sqrt (#50088)
• Add support for both dtype and ord arguments in torch.linalg.norm (#46637)
• Make torch.nn Module accept batch size of 0: nn.ReplicationPad (#39137), nn.Unfold (#40689), nn.PixelShuffle (#49187), nn.AvgPool{1,2,3}d (#50008), nn.MultiLabelMarginLoss and nn.MultiMarginLoss (#50007)
• utils.cpp_extensions Ensure default extra_compile_args are properly handled (#45956)
• torch.LongTensor legacy construction improved error message (#46147)
• torch.utils.checkpoint allow having Tensors that don’t require gradients (#45934)
• torch.nan_to_num: fix deprecated warnings (#46309)
• Remove more use of “blacklist” (#45512, #45781)
• Add type annotation to submodules: torch.nn.cpp (#46490), torch.nn.parallel.comm (#46736), torch.nn.modules.* (#46828, #45772, #46013, #49957, #49479, #49045, #49035, #49494, #48969), autograd functions from c++ (#46622), torch.distributed functions from c++ (#46623), torch.storage (#46876), torch._tensor_str (#48463, #48584), torch.nn.modules.pooling (#48412), common_nn (#48190), torch.lobpcg (#47680), torch.nn.functional (#50106), torch.overrides (#50824), torch.generate_torch_version (#51637), torch.distributions (#45689), torch.quantization.quantize_jit (#45548), torch.utils.tensorboard (#49834), torch.multiprocessing (#47756), torch.cuda (#47134), torch._C._distributed_rpc (#46624), torch.distributed.* (#47531, #47532, #47533, #47534), torch.nn.parallel._functions (#49687)
• Make comparison fail when dtypes don’t match (#47288)
• Allow large inputs for torch.svd (#47440)
• Add nondeterministic alerts to torch.index_copy, torch.median on CUDA and torch.kthvalue on CUDA (#46942)
• Add float16 and bfloat16 support to torch.where (#49004), torch.matmul (#47873)
• Add float16 support for CPU and bfloat16 support for CPU & CUDA to torch.flip and torch.flip{lr, ud} (#49895)
• Add support for providing indices as a Tensor for torch.tensor_split (#49169)
• Add support for SELU activation in torc.nn.init.calculate_gain (#50664)
• Add function version of torch.optim optimizers and refactor existing classes to use the functional version: SGD (#45597), Adadelta (#50409), RMSProp (#50410), AdamW (#50411)
• Improve error message when window is on wrong device for torch.fft.stft (#51128)
• Add rounding_mode selection to torch.div (#51706, #52242)
• Remove spurious numpy writable warning (#47271)
• Enable deterministic mode for rocBLAS (#48654)
• Hipify submodule revamp and improved integration with cpp_extensions (#48715)
• Remove warning about saving state in torch.optim.lr_scheduler.LambdaLR (#46813)
• Improve typing of torch.nn.Unflatten (#49838)
• Add exception classification to torch.multiprocessing.spawn

Autograd

• Add double backward checks for the torch.fft submodule (#46004)
• Detect inplace modifications of views of leaf Tensors earlier to improve error (#46204)

torch.utils

• data.TensorDataset: Add more specific error message (#46905)
• data.DistributedSampler: Additional validation (#48865)

Complex Numbers

• Improve error message thrown by torch.sign for complex tensors (#43280)
• Remove unnecessary dtype checks for complex types and disable complex dispatch for CPU torch.{min,max} pointwise ops (#50465)

CUDA

• Allow consumer ops to sync on autograd engine base gradient (#45787)
• Add torch::cuda::nccl::{send,recv} (#45926)
• Cusolver inverse check info (#46625)
• Make numpy optional dependency for torch.cuda.amp (#48154)
• Support all visible cards when building a cuda extension (#48891)
• Enable using torch.utils.checkpoint.checkpoint and torch.cuda.amp at the same time (#49757)
• Make DeviceCachingAllocator's error handling more defensive and a bit easier to read (#51158)

Distributed

• Create NCCL communicator for send/recv on demand (#44922)
• Reduce the peak memory of fp16 compression DDP comm hook by avoiding converting to fp32 (#46078)
• Allow RPC framework to use rank in addition to WorkerInfo and name. (#46221)
• Add to the HashStore getNumKeys() (#46048) and deleteKey() (#46049)
• Print exception message on both RPC caller and callee (#46372)
• Add RRef proxy support for ScriptModule methods (#48339)
• Support retrieving the RRef to the remote module (#48983)
• Add a c++ interface in processGroup to get its backend name (#51066)
• Enable NamedTuple data type to work with DDP (#44220)
• Support send/recv to/from self when communicator is created on demand (#45873)
• Add Error log when ProcessGroupNCCL takes down a process (#44988)
• Provide additional information about NCCL error codes. (#45950)
• Avoid scatter for single-device case in DDP (#46304)
• Use Blocking Wait if both Blocking Wait and Async Error Handling Are Set (#47926)
• Providing more information while crashing a process in async error handling (#47246)
• Add PowerSGD comm hook (#48060)
• Define a customized state for PowerSGD comm hook (#48348)
• Add a random generator to PowerSGD state for initializing low-rank matrix Q (#48507)
• Replace the key of error_dict in PowerSGD state with bucket index (#48867)
• Make CUDAFuture remember and restore current device in callback (#48789)
• Update pipeline API to accept arbitrary sequence of Tensors and not just Tuple (#48467)
• Use group.WORLD appropriately in process group initialization. (#48767)
• Add error feedback to layerwise PowerSGD (#49418)
• Warm-start of PowerSGD by reusing states from previous iteration is possible (#49451)
• Change wait() to value() in some callbacks of PowerSGD communication hook (#49709)
• Ensure DDP + Pipe works with find_unused_parameters. (#49908)
• Enable TensorPipe CUDA sending to self (#50674) and GDR channel (#50763)
• Add warning to distributed optimizer (#50630)
• Make python object collective API args consistent (#50625)
• Add option to make rref.get_type non-blocking. (#50977)
• Unescape string in RPC error message (#49373)
• Event Logging for NCCL Async Error Handling Process Crash (#47244)
• Remove balance and devices parameter from Pipe. (#48432)
• Error feedback for PowerSGD DDP comm hook (#48670)
• Add an index field to GradBucket for PowerSGD (#48757)
• Have FutureNCCL record streams w/ allocator in addCallback (#48496) and events in current stream (#48497)
• Use fresh stream from pool for each FutureNCCL callback (#48498)
• Record CUDA events for "follow-up" FutureNCCL inside markCompleted() (#48499)
• Fix FutureNCCL's completed() disagreeing with wait() (#48503)
• Fix FutureNCCL not recording DataPtrs with caching alloc in wait() (#48563)
• Add multi-GPU support to FutureNCCL (#48500)
• Don't store device indices separately on FutureNCCL (#48501)
• Support wider range of types in FutureNCCL (#48502)
• Split FutureNCCL's CUDA-specific parts from generic future logic (#48504)
• Merge common parts of FutureNCCL into at::ivalue::Future (#48505)
• Split out reusable CUDAFuture from FutureNCCL (#48506)
• Cache the DataPtrs in CUDAFuture (#48788)
• Modify Pipe to return an RRef. (#47829)
• Cleanup APIs for pipeline parallelism. (#48630)
• Fix TCPStore type coercion (#49685)
• Simplify the implementation of error feedback and warm-start (#50981)
• Explicitly specify the dtype of the error tensor (#50985)
• Check start_PowerSGD_iter > 1 and add guidance on tuning PowerSGD configs. (#51427)
• Check if the backend is NCCL when a DDP communication hook is registered (#51759)

TorchScript

• Add multiline string dedent support (#45580)
• Add string versions of argument funcs in jit Node (#45464)
• Make sure each warnings.warn only executes once inside TorchScript. (#45382)
• Allow slicing multiple dimensions with indexes if not Tuple (#45239)
• Change type inferred from empty annotation (#45360)
• Fix stride printing/parsing formatting (#45156)
• Make objects throw Python AttributeError on nonexistant attr access (#45911)
• Make InsertInstruction overflow check a warning instead of fatal (#46369)
• Add an option to getWriteableTensorData to avoid copy CUDA tensor to CPU (#46524)
• Add error messages and workaround for RET failure of containers with a torch class type (#46543)
• Correctly mark unannotated NamedTuple field to be inferred TensorType (#46969)
• Enable ModuleDict non-literal indexing (#45716)
• Add an attribute to the torchscript model exported by metal (#47174)
• Print out interface mismatch for prim::ModuleDictIndex (#47300)
• better message for bad type annotation (#47464)
• Resolve string literal type annotations using Resolver::resolveType (#47731)
• Resolve torch.device in recursive compilation of classes (#47734)
• Metacompile boolean constants (#46721)
• Allow JIT unpickler to accept CUDA DataPtr from read_record_ (#46827)
• Skip None submodule during JIT-tracing (#49765)
• Add __prepare_scriptable__ duck typing to allow replacing nn.Modules with scriptable preparations (#45645) (#49242)
• Fix deprecation warning in scalar_type_analysis (#50218)
• Support scripting classmethod called with object instances (#49967)
• Use FileStore in TorchScript for store registry (#50248)
• Treat has_torch_function and object_has_torch_function as static False when scripting (#48966)
• Print better error when class attribute IValue conversion fails (#50255)
• Clean up some type annotations in test/jit/...../test_class_type.py (#50156)
• Type annotations in test/jit (#50293)
• Eliminate static default_extra_files_mobile from header import.h (#50832)
• Dump torch::jit::AliasDb objects as Graphviz files (#50452)
• Fix test_jit_cuda_archflags on machine with more than one arch (#50405)
• Provide more info when attribute fails to convert (#50870)
• Adding correct error message for for..else (#51258)
• Handle error during dict expansion (#51374)

Mobile

• Update default output extension in optimize_for_mobile.cc (#45598)
• Add named tuple's error message and workaround for RET failure (#46347)
• [Metal] Add metal backend type (#46455)
• [Metal] Add the Python binding for optimize_for_mobile (#46456)
• [Metal] Add pin_memory check in empty_strided (#47228)
• [Metal] Calculate strides for metal tensors (#50309)
• [Metal] Clean up the operator tests (#50311)
• Add an overload for deserialize() that doesn't accept the extra_files map. (#50932)
• bundled_inputs: Preserve bundled input related methods when calling optimize_for_mobile (#49170)
• bundled_inputs: Preserved all functions generated by bundled inputs (#51496)
• bundled_inputs: Expanded Bundled Inputs To Any Public Function (#51153)
• Expose _export_operator_list to python (#51312)

Quantization

• Quantized Operators and Modules
  • Add reflection padding to conv (#49011)
  • Add support for 2D indices for quantized embedding operators (#47766)
  • quantize_tensor_per_channel ARM implementation (#46018)
  • Support either min or max in qclamp (#45937)
  • Add preliminary support for advanced indexing (#49346)
  • Add backend_independent option for quantized linear module (#48192)
  • Add out-variant for the reflection pad (#48037)
  • Support 2 dim input in quantized batchnorm 1d (#51597)
• Typing, Formatting, Error Messages, Logging and Tests
  • numeric suite: add types to eager (#51168)
  • Enable type check for torch.quantization.fake_quantize (#45701)
  • Type check for torch.quantization.observer (#45630), torch.quantization._numeric_suite (#46330), torch.quantization.stubs (#46475), quantization.fx.Quantizer (#48343), quantization.fx.Quantizer (#48350), quantization_mappings.py (#49179), fusion_patterns.py (#49606), torch/nn/quantized/modules (#49941), quantization-related files in torch/jit (#49939), fuser (#48844), quantization_patterns (#48851), observed_module.py (#49607), quantization (#49942)
  • Enable mypy on torch/quantization/fx/* (#48331)
  • Make each line of fx/quantize.py <=80 chars (#48357)
  • Add more typehints (#48774, #48794, #48792)
  • Nice error message on convtranspose with per-channel weight (#49899)
  • Throw a nice error message for allclose with quantized inputs (#49802)
  • Add type annotations to torch.nn.quantized.modul

Note truncated.

PyTorch 1.7.1 Release Notes

• New Features
• Critical Fixes
• Other Fixes

New Features

Add Python 3.9 binaries for linux and macOS (#48133) and Windows (#48218)

NOTE: Conda installs for Python 3.9 will require the conda-forge channel, example: conda install -y -c pytorch -c conda-forge pytorch.

Upgrade CUDA binaries to use cuDNN 8.0.5 (builder repo #571)

This upgrade fix regressions on Ampere cards introduced in cuDNN 8.0.4. It will improve performance for 3090 RTX cards, and may improve performance in other RTX-30 series card.

Critical Fixes

Python 3.9

• Use custom version of pybind11 to work around Python 3.9 issues (#48312)
• Fix jit Python 3.9 parsing (#48744)
• Fix cpp_extension to work with Python 3.9 (#48768)

Build

• Fix cpp_extension to properly handle env variable on Windows (#48937)
• Properly package libomp.dylib for macOS binaries (#48337)
• Fix build for statically linked OpenBLAS on aarch64 (#48819)

Misc

• torch.sqrt: fix wrong output values for very large complex input (#48216)
• max_pool1d: fix for discontiguous inputs (#48219)
• collect_env: fix detection of DEBUG flag (#48319)
• collect_env: Fix to work when PyTorch is not installed (#48311)
• Fix amp memory usage when running in no_grad() mode (#48936)
• nn.ParameterList and nn.ParameterDict: Remove spurious warnings (#48215)
• Tensor Expression fuser bugfixes (#48137)

Other Fixes

• Tensor Expression fix for CUDA 11.0 (#48309)
• torch.overrides: doc fix (#47843)
• torch.max: Fix output type for Tensor subclasses (#47735)
• torch.mul: Add support for boolean Tensors (#48310)
• Add user friendly error when trying to compile from source with Python 2 (#48317)

PyTorch 1.7.0 Release Notes

• Highlights
• Backwards Incompatible Change
• New Features
• Improvements
• Performance
• Documentation

Highlights

The PyTorch 1.7 release includes a number of new APIs including support for NumPy-Compatible FFT operations, profiling tools and major updates to both distributed data parallel (DDP) and remote procedure call (RPC) based distributed training. In addition, several features moved to stable including custom C++ Classes, the memory profiler, the creation of custom tensor-like objects, user async functions in RPC and a number of other features in torch.distributed such as Per-RPC timeout, DDP dynamic bucketing and RRef helper.

A few of the highlights include:

• CUDA 11 is now officially supported with binaries available at PyTorch.org
• Updates and additions to profiling and performance for RPC, TorchScript and Stack traces in the autograd profiler
• (Beta) Support for NumPy compatible Fast Fourier transforms (FFT) via torch.fft
• (Prototype) Support for Nvidia A100 generation GPUs and native TF32 format
• (Prototype) Distributed training on Windows now supported

To reiterate, starting PyTorch 1.6, features are now classified as stable, beta and prototype. You can see the detailed announcement here. Note that the prototype features listed in this blog are available as part of this release.

Front End APIs

[Beta] NumPy Compatible torch.fft module

FFT-related functionality is commonly used in a variety of scientific fields like signal processing. While PyTorch has historically supported a few FFT-related functions, the 1.7 release adds a new torch.fft module that implements FFT-related functions with the same API as NumPy.

This new module must be imported to be used in the 1.7 release, since its name conflicts with the historic (and now deprecated) torch.fft function.

Example usage:

>>> import torch.fft
>>> t = torch.arange(4)
>>> t
tensor([0, 1, 2, 3])

>>> torch.fft.fft(t)
tensor([ 6.+0.j, -2.+2.j, -2.+0.j, -2.-2.j])

>>> t = tensor([0.+1.j, 2.+3.j, 4.+5.j, 6.+7.j])
>>> torch.fft.fft(t)
tensor([12.+16.j, -8.+0.j, -4.-4.j,  0.-8.j])

• Documentation | Link

[Beta] C++ Support for Transformer NN Modules

Since PyTorch 1.5, we’ve continued to maintain parity between the python and C++ frontend APIs. This update allows developers to use the nn.transformer module abstraction from the C++ Frontend. And moreover, developers no longer need to save a module from python/JIT and load into C++ as it can now be used it in C++ directly.

• Documentation | Link

[Beta] torch.set_deterministic

Reproducibility (bit-for-bit determinism) may help identify errors when debugging or testing a program. To facilitate reproducibility, PyTorch 1.7 adds the torch.set_deterministic(bool) function that can direct PyTorch operators to select deterministic algorithms when available, and to throw a runtime error if an operation may result in nondeterministic behavior. By default, the flag this function controls is false and there is no change in behavior, meaning PyTorch may implement its operations nondeterministically by default.

More precisely, when this flag is true:

• Operations known to not have a deterministic implementation throw a runtime error;
• Operations with deterministic variants use those variants (usually with a performance penalty versus the non-deterministic version); and
• torch.backends.cudnn.deterministic = True is set.

Note that this is necessary, but not sufficient, for determinism within a single run of a PyTorch program. Other sources of randomness like random number generators, unknown operations, or asynchronous or distributed computation may still cause nondeterministic behavior.

See the documentation for torch.set_deterministic(bool) for the list of affected operations.

• RFC | Link
• Documentation | Link

Performance & Profiling

[Beta] Stack traces added to profiler

Users can now see not only operator name/inputs in the profiler output table but also where the operator is in the code. The workflow requires very little change to take advantage of this capability. The user uses the autograd profiler as before but with optional new parameters: with_stack and group_by_stack_n. Caution: regular profiling runs should not use this feature as it adds significant overhead.

• Details | Link
• Documentation | Link

Distributed Training & RPC

[Stable] TorchElastic now bundled into PyTorch docker image

Torchelastic offers a strict superset of the current torch.distributed.launch CLI with the added features for fault-tolerance and elasticity. If the user is not be interested in fault-tolerance, they can get the exact functionality/behavior parity by setting max_restarts=0 with the added convenience of auto-assigned RANK and MASTER_ADDR|PORT (versus manually specified in torch.distributed.launch).

By bundling torchelastic in the same docker image as PyTorch, users can start experimenting with torchelastic right-away without having to separately install torchelastic. In addition to convenience, this work is a nice-to-have when adding support for elastic parameters in the existing Kubeflow’s distributed PyTorch operators.

• Usage examples and how to get started | Link

[Beta] Support for uneven dataset inputs in DDP

PyTorch 1.7 introduces a new context manager to be used in conjunction with models trained using torch.nn.parallel.DistributedDataParallel to enable training with uneven dataset size across different processes. This feature enables greater flexibility when using DDP and prevents the user from having to manually ensure dataset sizes are the same across different process. With this context manager, DDP will handle uneven dataset sizes automatically, which can prevent errors or hangs at the end of training.

• RFC | Link
• Documentation | Link

[Beta] NCCL Reliability - Async Error/Timeout Handling

In the past, NCCL training runs would hang indefinitely due to stuck collectives, leading to a very unpleasant experience for users. This feature will abort stuck collectives and throw an exception/crash the process if a potential hang is detected. When used with something like torchelastic (which can recover the training process from the last checkpoint), users can have much greater reliability for distributed training. This feature is completely opt-in and sits behind an environment variable that needs to be explicitly set in order to enable this functionality (otherwise users will see the same behavior as before).

• Documentation | Link
• RFC | Link

[Beta] TorchScript remote and rpc_sync

torch.distributed.rpc.rpc_async has been available in TorchScript in prior releases. For PyTorch 1.7, this functionality will be extended the remaining two core RPC APIs, torch.distributed.rpc.rpc_sync and torch.distributed.rpc.remote. This will complete the major RPC APIs targeted for support in TorchScript, it allows users to use the existing python RPC APIs within TorchScript (in a script function or script method, which releases the python Global Interpreter Lock) and could possibly improve application performance in multithreaded environment.

• Documentation | Link
• Usage examples | Link

[Beta] Distributed optimizer with TorchScript support

PyTorch provides a broad set of optimizers for training algorithms, and these have been used repeatedly as part of the python API. However, users often want to use multithreaded training instead of multiprocess training as it provides better resource utilization and efficiency in the context of large scale distributed training (e.g. Distributed Model Parallel) or any RPC-based training application). Users couldn’t do this with with distributed optimizer before because we need to get rid of the python Global Interpreter Lock (GIL) limitation to achieve this.

In PyTorch 1.7, we are enabling the TorchScript support in distributed optimizer to remove the GIL, and make it possible to run optimizer in multithreaded applications. The new distributed optimizer has the exact same interface as before but it automatically converts optimizers within each worker into TorchScript to make each GIL free. This is done by leveraging a functional optimizer concept and allowing the distributed optimizer to convert the computational portion of the optimizer into TorchScript. This will help use cases like distributed model parallel training and improve performance using multithreading.

Currently, the only optimizer that supports automatic conversion with TorchScript is Adagrad and all other optimizers will still work as before without TorchScript support. We are working on expanding the coverage to all PyTorch optimizers and expect more to come in future releases. The usage to enable TorchScript support is automatic and exactly the same with existing python APIs, here is an example of how to use this:

import torch.distributed.autograd as dist_autograd
import torch.distributed.rpc as rpc
from torch import optim
from torch.distributed.optim import DistributedOptimizer

with dist_autograd.context() as context_id:
  # Forward pass.
  rref1 = rpc.remote("worker1", torch.add, args=(torch.ones(2), 3))
  rref2 = rpc.remote("worker1", torch.add, args=(torch.ones(2), 1))
  loss = rref1.to_here() + rref2.to_here()

  # Backward pass.
  dist_autograd.backward(context_id, [loss.sum()])

  # Optimizer, pass in optim.Adagrad, DistributedOptimizer will
  # automatically convert/compile it to TorchScript (GIL-free)
  dist_optim = DistributedOptimizer(
     optim.Adagrad,
     [rref1, rref2],
     lr=0.05,
  )
  dist_optim.step(context_id)

• Documentation | Link
• RFC | Link

[Beta] Enhancements to RPC-based Profiling

Support for using the PyTorch profiler in conjunction with the RPC framework was first introduced in PyTorch 1.6. In PyTorch 1.7, the following enhancements have been made:

• Implemented better support for profiling TorchScript functions over RPC
• Achieved parity in terms of profiler features that work with RPC
• Added support for asynchronous RPC functions on the server-side (functions decorated with rpc.functions.async_execution).

User are now able to use familiar profiling tools such as with torch.autograd.profiler.profile() and with torch.autograd.profiler.record_function, and this works transparently with the RPC framework with full feature support, profiles asynchronous functions, and TorchScript functions.

• Design doc | Link
• Usage examples | Link

[Prototype] Windows support for Distributed Training

PyTorch 1.7 brings prototype support for DistributedDataParallel and collective communications on the Windows platform. In this release, the support only covers Gloo-based ProcessGroup and FileStore. To use this feature across multiple machines, please provide a file from a shared file system in init_process_group.

# initialize the process group
dist.init_process_group(
    "gloo",
    # multi-machine example:
    # Shared files need six "/"
    # init_method = `"file://////{machine}/{share_folder}/file"`
    # Local file need three "/"
    init_method="file:///{your local file path}",
    rank=rank,
    world_size=world_size
)

model = DistributedDataParallel(local_model, device_ids=[rank])

• Design doc | Link
• Documentation | Link
• Acknowledgement | gunandrose4u

Mobile

PyTorch Mobile supports both iOS and Android with binary packages available in Cocoapods and JCenter respectively. You can learn more about PyTorch-Mobile here.

[Beta] PyTorch Mobile Caching allocator for performance improvements

On some mobile platforms, such as Pixel, we observed that memory is returned to the system more aggressively. This results in frequent page faults as PyTorch being a functional framework does not maintain state for the operators. Thus outputs are allocated dynamically on each execution of the op, for the most ops. To ameliorate performance penalties due to this, PyTorch 1.7 provides a simple caching allocator for CPU. The allocator caches allocations by tensor sizes and, is currently, available only via the PyTorch C++ API. The caching allocator itself is owned by client and thus the lifetime of the allocator is also maintained by client code. Such a client owned caching allocator can then be used with scoped guard, c10::WithCPUCachingAllocatorGuard, to enable the use of cached allocation within that scope.

Example usage:

#include <c10/mobile/CPUCachingAllocator.h>
.....
c10::CPUCachingAllocator caching_allocator;
  // Owned by client code. Can be a member of some client class so as to tie the
  // the lifetime of caching allocator to that of the class.
.....
{
  c10::optional<c10::WithCPUCachingAllocatorGuard> caching_allocator_guard;
  if (FLAGS_use_caching_allocator) {
    caching_allocator_guard.emplace(&caching_allocator);
  }
  ....
  model.forward(..);
}
.....

NOTE: Caching allocator is only available on mobile builds, thus the use of caching allocator outside of mobile builds won’t be effective.

• Documentation | Link
• Usage examples | Link

Backwards Incompatible changes

Python API

torch.conj now returns the input as-is for real Tensors (#43270)

Previously, torch.conj and Tensor.conj were making a clone for Tensors of real dtype. It now returns the Tensor as-is to improve performance. You can recover the original behavior by adding a .clone() for real Tensors. Note that this behavior is different from numpy for which np.conj returns a new ndarray and ndarray.conj returns the ndarray as-is.

<p align="center"> <table align="center"> <tr><th>1.6.0</th><th>1.7.0</th></tr> <tr valign="top"> <td><sub><pre lang="python">

t.is_complex() False t.conj() is t False
</pre></sub></td> <td>_{<pre lang="python"> t.is_complex() False t.conj() is t True t.conj().clone() is t False
</pre>}</td> </tr> </table> </p>

torch.tensor, torch.as_tensor, and torch.sparse_coo_tensor now use the input Tensor’s device when it is not specified (#41984)

This will change the device on which the Tensor is created and so the user can start seeing device mismatch errors. It also means for sparse Tensors that both of the provided Tensors must be on the same device if the device is not specified. You can recover the original behavior by passing the device argument.

<p align="center"> <table align="center"> <tr><th>1.6.0</th><th>1.7.0</th></tr> <tr valign="top"> <td><sub><pre lang="python">

t.device device(type=‘cuda:0’)

tensor constructor

torch.tensor(t, dtype=torch.float32).device device(type=‘cpu’)

sparse constructor

torch.sparse_coo_tensor( torch.tensor(([0], [2]), device="cpu"), torch.tensor(([1.],), device="cuda"), size=(3, 3, 1)).device device(type='cuda', index=0)
</pre></sub></td> <td><sub><pre lang="python"> t.device device(type=‘cuda:0’)

tensor constructor

torch.tensor(t, dtype=torch.float32).device device(type=‘cuda:0’)

Specify the device to get the same behavior as 1.6

torch.tensor(t, dtype=torch.float32, device='cpu').device device(type=‘cpu’)

sparse constructor

torch.sparse_coo_tensor( torch.tensor(([0], [2]), device="cpu"), torch.tensor(([1.],), device="cuda"), size=(3, 3, 1)).device RuntimeError: backend of indices (CPU) must match backend of values (CUDA)

Specify the device to get the same behavior as 1.6

torch.sparse_coo_tensor( torch.tensor(([0], [2]), device="cpu"), torch.tensor(([1.],), device="cuda"), size=(3, 3, 1), device="cuda:0").device device(type='cuda', index=0)
</pre></sub></td> </tr> </table> </p>

torch.nn.utils.pack_padded_sequence: remove hidden cross-device copy for lengths (#41984)

In previous versions, when the lengths argument was a CUDA tensor, it would incorrectly be moved to the CPU silently. This can lead to surprising performances and CPU/GPU sync when using CUDA so this has been removed. You need to make sure that the provided lenghts is a CPU Tensor when it is provided as a Tensor.

<p align="center"> <table align="center"> <tr><th>1.6.0</th><th>1.7.0</th></tr> <tr valign="top"> <td><sub><pre lang="python">

inp = torch.rand(10, 2, 3, device="cuda") lengths = torch.tensor([10, 7], device="cuda") torch.nn.utils.rnn.pack_padded_sequence(inp, lengths)

Implicitly move lengths to the CPU and runs fine

  </pre></sub></td>
  <td><sub><pre lang="python">

inp = torch.rand(10, 2, 3, device="cuda") lengths = torch.tensor([10, 7], device="cuda") torch.nn.utils.rnn.pack_padded_sequence(inp, lengths) RuntimeError: 'lengths' argument should be a 1D CPU int64 tensor, but got 1D cuda:0 Long tensor

Ensure the lenghts is already on the right device

lengths = lengths.cpu() torch.nn.utils.rnn.pack_padded_sequence(inp, lengths)

Runs fine with no implicit move across device

  </pre></sub></td>
</tr>

</table> </p>

Improve torch.norm handling of keepdim=True (#41956)

Before this change, when calling torch.norm with keepdim=True and p='fro' or p=number, leaving all other optional arguments as their default values, the keepdim argument would be ignored. It is now properly respected. Also, any time torch.norm was called with p='nuc' and keepdim=True, the result would have one fewer dimension than the input, and the dimensions could be out of order depending on which dimensions were being reduced. It is now properly keeping all the dimensions. You can recover the original behavior by setting keepdim=False. NOTE: this function is now deprecated (see below) and we recommend you use torch.linalg.norm, which follows NumPy’s conventions.

<p align="center"> <table align="center"> <tr><th>1.6.0</th><th>1.7.0</th></tr> <tr valign="top"> <td><sub><pre lang="python">

t.size() torch.Size([4, 4]) t.norm(p=‘fro’, keepdim=True).size() torch.size([]) t.norm(p=3, keepdim=True).size() torch.size([]) t.norm(p=‘nuc’, keepdim=True).size() torch.size([1]) </pre></sub></td> <td>_{<pre lang="python"> t.size() torch.Size([4, 4]) t.norm(p=‘fro’, keepdim=True).size() torch.size([1, 1]) t.norm(p=3, keepdim=True).size() torch.size([1, 1]) t.norm(p=‘nuc’, keepdim=True).size() torch.size([1, 1])
</pre>}</td> </tr> </table> </p>

torch.split and torch.chunk: Fix view tracking for the autograd (#41567)

The autograd system is able to correctly handle modifications through views of Tensors by explicitly tracking known view operations. In prior releases, torch.split and torch.chunk were not marked as known view operations, which could lead to silently wrong gradients.

Note that since v1.5, inplace modification of views created by functions that return multiple views is deprecated. Such case is not properly handled by the autograd and can lead to internal errors or wrong gradients. So, as a side effect of this view fix, inplace modifications of the outputs of torch.split and torch.chunk will now raise a warning and can lead to internal errors or wrong gradients while they were previously silently computing wrong gradients. If you see such a warning, you should replace the inplace operation with an out of place one. You can recover the original behavior by using the new torch.unsafe_split and torch.unsafe_chunk. Note that these functions are only here to ease the transition and will also be removed in a future version.

torch.{argmin,argmax} now always return the first min/max index (#42004)

torch.argmin (torch.argmax) now always returns the index of the first minimum (maximum) element. This choice is consistent with NumPy. Previously if there were multiple minima (maxima) the index returned could be the index of any of them. You cannot recover the original behavior as it was platform dependent and not guaranteed. If your code was relying on a specific index for your specific platform, you should update it to work with the first index and this new code will work on all platforms.

torch.{min,max,median}: Update backward formula when doing full reduction (dim argument not provided) (#43519)

When no dimension is specified, full reduction is performed and the gradient will now flow back evenly towards all the input that realized the output value. The old behavior was to propagate the gradient only for one of such input selected arbitrarily. This should improve stability of training by gradient descent. To recover the previous behavior, you can perform the reduction with the dim= argument. It will ensure that the gradient only flows back for the input whose index was returned.

<p align="center"> <table align="center"> <tr><th>1.6.0</th><th>1.7.0</th></tr> <tr valign="top"> <td><sub><pre lang="python">

a tensor([3, 2, 3]) a.max().backward() a.grad tensor([0, 0, 1])
</pre></sub></td> <td>_{<pre lang="python"> a tensor([3, 2, 3]) a.max().backward() a.grad tensor([0.5, 0, 0.5]) a.max(dim=0).max(dim=0).max(dim=0).backward() a.grad tensor([0, 0, 1])
</pre>}</td> </tr> </table> </p>

nn.BCELoss size mismatch warning is now an error (#41426)

This is the end of the deprecation cycle for this op to make sure it does not have different broadcasting semantic compared to numpy’s broadcasting semantic used everywhere else in PyTorch’s codebase. You need to make sure all inputs are the same size to avoid the error.

<p align="center"> <table align="center"> <tr><th>1.6.0</th><th>1.7.0</th></tr> <tr valign="top"> <td><sub><pre lang="python">

bceloss = nn.BCELoss() a = torch.rand(25) b = torch.rand(25, 1) bceloss(a, b) UserWarning: Using a target size (torch.Size([25, 1])) that is different to the input size (torch.Size([25])) is deprecated. Please ensure they have the same size. tensor(1.0604)
</pre></sub></td> <td>_{<pre lang="python"> bceloss = nn.BCELoss() a = torch.rand(25) b = torch.rand(25, 1) bceloss(a, b) ValueError: Using a target size (torch.Size([25, 1])) that is different to the input size (torch.Size([25])) is deprecated. Please ensure they have the same size. b = b.reshape(25) bceloss(a, b) tensor(1.0604) </pre>}</td> </tr> </table> </p>

Custom autograd.Function stop materializing None output Tensors (#41490)

To improve performance, the custom autograd.Function will not create a Tensor full of zeros when an input is differentiable but the user’s backward function returns None for it. This means that code for which the .backward() or autograd.grad() final result will now be None while it used to be a Tensor full of zeros. You can recover the previous behavior by having your custom autograd.Function materialize the zero Tensor with torch.zeros_like(input) to replace the None output for the backward method.

import torch

# Custom Function that returns None for the gradient
class GetTwos(torch.autograd.Function):
    @staticmethod
    def forward(ctx, inp):
        return inp.clone().fill_(2)

    @staticmethod
    def backward(ctx, grad_out):
        # To recover the 1.6 behavior, replace the line below with `return torch.zeros_like(grad_out)`
        return None

a = torch.rand(10, requires_grad=True)
b = GetTwos.apply(a)
b.sum().backward()

print(a.grad)
# In PyTorch 1.6 this will print
# tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])
# In PyTorch 1.7 this will print
# None

Fix inplace detection for non-differentiable outputs (#41269)

We fixed a bug in the inplace detection code that was preventing the detection of some inplace operations for output that are not differentiable (like integer type Tensors). This can lead to code that used to run fine to throw the error “a Tensor that was needed for backward was modified in an inplace operation”. Such failure is true and the user code must be fixed to compute proper gradients. In general, this involves cloning the Tensor before modifying it inplace to make sure the backward pass can happen safely.

import torch

a = torch.rand(10, requires_grad=True)
with torch.no_grad():
    a[2] = 10

b, ind = a.max(dim=0)
# ind is 2 here

with torch.no_grad():
    t = torch.rand(10)
    t[4] = 10
    res = torch.max(t, dim=0, out=(torch.Tensor(), ind))
    # ind becomes 4 here

# This backward runs in 1.6 but will fail in 1.7
b.sum().backward()
print(a.grad)
# tensor([0., 0., 0., 0., 1., 0., 0., 0., 0., 0.])
# The value is wrong is at index 4 while it should be at index 2

# The issue is avoided by not modifying ind inplace by replacing the line
# above with:
# res = torch.max(t, dim=0, out=(torch.Tensor(), ind.clone()))

Add __torch_functions__ for methods (#37091)

Functions, slicing and Tensor methods will now properly preserve the subclass type when possible.

>>> class SubTensor(torch.Tensor):
...     pass
>>> type(torch.add(SubTensor([0]), SubTensor([1]))).__name__
'SubTensor'
>>> type(torch.add(SubTensor([0]), torch.Tensor([1]))).__name__
'SubTensor'

The old behavior of “any operations on your subclass produces a torch.Tensor instead of the subclass” can be recovered by doing:

from torch._C import _disabled_torch_function_impl
    
class SubTensor(torch.Tensor):
    __torch_function__ = _disabled_torch_function_impl

For all details on how to use this feature, please refer to the doc page for it.

tensor.__iter__: Use torch.unbind instead of a for loop (#40884)

This improves performances significantly but it changes the behavior of in-place operations on the value returned by the iterator. This happens only if either the input Tensor or any argument of the in-place operation is a Tensor that requires gradients. And it will fail with "Output X of UnbindBackward is a view and is being modified inplace". You can recover the previous behavior by manually slicing the Tensor: [t[i] for i in range(t.size(0))] as shown in the example below.

<p align="center"> <table align="center"> <tr><th>1.6.0</th><th>1.7.0</th></tr> <tr valign="top"> <td><sub><pre lang="python">

x = torch.randn(5, 10, requires_grad=True) for i, v in enumerate(x): v.fill_(i) </pre></sub></td> <td>_{<pre lang="python"> x = torch.randn(5, 10, requires_grad=True) for i, v in enumerate([x[j] for j in range(x.size(0))]): v.fill_(i) </pre>}</td> </tr> </table> </p>

Updated most function that take zero, one or two Tensor arguments and indexing op to check for memory overlap in the Tensor being worked on (#43418, #43419, #43420, #43421, #43423, #43422)

It fixes silent correctness errors: something that used to be silently incorrect now errors out. Code that raises this error must be updated to avoid doing such op that was returning wrong results as shown in the example below:

>>> x = torch.randn(1, 3)
>>> # Create a tensor that has internal memory overlap
>>> y = x.expand(2, 3)

# In 1.6, this would not error out, but in 1.7, this errors out
>>> torch.nn.functional.elu(y, inplace=True)
RuntimeError: unsupported operation: more than one element of the written-to tensor refers to a single m
emory location. Please clone() the tensor before performing the operation.

# Here is the fix in 1.7
>>> torch.nn.functional.elu(y, inplace=False)

c++ API: Any external users of TensorIterator now always get the memory overlap check. The previous behavior can be recovered by setting set_check_mem_overlap(false) when creating the iterator.

TorchScript

TorchScript now correctly supports various exception type and custom exception message (#41907)

• Exceptions raised in TorchScript was traditionally replaced with a generic runtime error that doesn’t carry exception type or message, leading to crashes that are difficult to pin-point and debug. We improved TorchScript to correctly parse exception types and messages and surface them to users.
• This change is backward incompatible because TorchScript now attempts to compile user code that creates custom exception messages instead of ignoring them. Any TorchScript-incompatible Python features used in those code snippets would lead to failures.
• There is no fixed formula to fix this backward incompatibility failure other than updating code that generates exceptions to be TorchScript-able.

TorchScript now supports properties of TorchScript classes and ScriptModules (#42389, #42390)

• TorchScript added support for @property of TorchScript classes and ScriptModules. Custom setters and getters are also supported. Custom deleters are not supported.
• This improvement is backward incompatible because TorchScript now attempts to script properties of existing classes and Modules. If these properties use Python or Pytorch features that are not supported in Torchscript, scripting will fail.
• There are two ways of fixing backward incompatibility failures introduced by this change. One is using @torch.jit.unused to annotate problematic properties, the other is to update the implementation of the property so that the getter and setter are scriptable.

Quantization

The convolution parameters now support versioning.

• This change means that any quantized convolution module saved using PyTorch 1.7+ cannot be loaded in v1.6 and lower.
• But this change is backward compatible: if the model (with conv layers) is saved in version 1.6, it can be safely loaded in version 1.7.

Some undocumented functions that were mistakenly made public have been removed

• torch.absolute_ has been removed, the Tensor method (Tensor.absolute_) should be used instead just like all other inplace ops.
• torch.ExtraFilesMap is an internal jit construct and should not be used.

TorchScript Compiler Update

In 1.7, we are enabling a Profiling Executor and a new Tensor-Expressions-based (TE) Fuser. All compilations will now go through one (an adjustable setting) profiling run and one optimization run. For the profiling run, complete tensor shapes are recorded and used by the new Fuser. For the optimization run, the focus is on finding (in torch.jit.ScriptModules) and fusing element-wise operations over CUDA tensors into a single CUDA kernel.

The TE fuser is expected to deliver performance similar to the old fuser used in 1.6. It however unlocks more opportunities for performance improvements in future releases. In rare cases, performance of some models may degrade 5-10%. If you experience any regressions please report it on Github, so we can address them as soon as possible! For 1.7, we are providing an option for our users to revert back to the old fuser by calling torch._C._jit_set_profiling_executor(False) in Python and torch::jit::getExecutorMode()`` = false; in C++. For more information, please see “Graph Executor” section in our documentation.

Deprecations

Python API

torch.norm and torch.functional.norm are deprecated in favor of torch.linalg.norm (#44321)

The new torch.linalg.norm has the same behavior as numpy.linalg.norm Both deprecated functions had odd behaviors for matrix and vector norms. You should refer to the doc here to find the exact behavior they had and how to replicate it with the new API.

Deprecate fft functions in torch. namespace in favor of torch.fft. namespace (#44876)

Please use torch.fft.foo as a drop-in replacement for torch.foo for the following functions: fft, ifft, rfft and irfft.

Warns when some out= functions need to resize an output which is not 0-size (#42079)

This behavior is dangerous and leads to an API that is hard to use. It is being deprecated to be able to fix that API in future versions. You should resize the output before-hand to avoid any issue in the future:

a = torch.rand(5)
b = torch.rand(25)

# This is deprecated
torch.add(a, a, out=b)

# This has the same behavior but will work in future versions
torch.add(a, a, out=b.resize_(0))

torch.optim: Warn for duplicate params in param group (#41597)

Providing multiple times the same Parameter in a single param group is most likely due to user error and is being deprecated. Please open an issue if you have a valid use case that require this feature.

torch.linspace and torch.logspace: Not giving the step argument is deprecated (#43860)

The default steps argument that has been used historically in PyTorch is not consistent with other libraries and so is being removed to avoid confusion. For both functions, passing steps=100 keyword argument can be used to recover the original behavior.

<p align="center"> <table align="center"> <tr><th>1.6.0</th><th>1.7.0</th></tr> <tr valign="top"> <td><sub><pre lang="python">

torch.linspace(0, 10).size() torch.Size([100])
</pre></sub></td> <td>_{<pre lang="python"> torch.linspace(0, 10).size() UserWarning: Not providing a value for linspace's steps is deprecated and will throw a runtime error in a future release. torch.Size([100]) torch.linspace(0, 10, steps=100).size() torch.Size([100]) </pre>}</td> </tr> </table> </p>

Distributed

• Make TensorPipe the default backend for RPC (#43246)
• Infer RPC backend type to preserve backward compatibility as we make TensorPipe the default (#45065)
• Add deprecation warning to ProcessGroup backend and make TensorPipe backend stable. (#45356)
• Add warnings on ProcessGroup and ProcessGroup::Work APIs which will be retired soon. (#46366)

New features

Python API

New namespaces:

• torch.fft added (#41911)
• torch.linalg added (#42664)
• torch.optim.functional added (#44715)

New operators:

• torch.count_nonzero added (#39992)
• nn.SiLU activation added (#41034)
• torch.logit added (#41062)
• torch.gcd, torch.lcm added (#40651, #41552, #42254)
• torch.functional.atleast_{1d/2d/3d} added (#41317)
• torch.isreal added (#41298)
• nn.Unflatten added (#41564)
• torch.movedim added (#41480)
• torch.isposinf, torch.isneginf added (#41588)
• torch.signbit added (#41589)
• torch.absolute added (#42586)
• torch.clip alias added (#42770)
• torch.quantile added (#42755)
• torch.linalg.det and torch.outer alias added (#42802)
• torch.nansum added (#38628)
• torch.hypot added (#42291)
• torch.nextafter added (#42580)
• torch.hstack, torch.vstack, torch.dstack added (#42799)
• torch.arccosh alias added (#43107)
• Tensor.movedim as a method added (#43122)
• torch.matrix_exp added (#40161)
• torch.fix alias added (#43326)
• torch.arccos, torch.arcsin, torch.arctan aliases added (#43319)
• torch.negative alias added (#43400)
• torch.maximum, torch.minimum added (#42579)
• torch.arctanh, torch.arcsinh aliases added (#43762)
• torch.linalg.norm added (#42749, #43907)
• torch.amax, torch.amin added (#43819)
• torch.heaviside added (#42523)
• torch.i0 added (#43132)
• torch.not_equal, torch.greater, torch.greater_equal, torch.less, torch.less_equal aliases added (#43870)
• torch.exp2 added (#44184)
• torch.kaiser_window added (#44271)
• torch.nanquantile added (#44393)
• torch.multiply, torch.divide aliases added (#44463)
• nn.TripletMarginWithDistanceLoss added (#43680)
• torch.fft.fft, torch.fft.ifft, torch.fft.rfft, torch.fft.irfft, torch.fft.hfft, torch.fft.ihfft added (#43011)
• torch.fft.fftn, torch.fft.ifftn, torch.fft.rfftn, torch.fft.irfftn added (#44550)
• optim.functional.adagrad added (#44715)
• optim.functional.adam added (#44791)
• torch.complex, torch.polar added (#39617)
• Tensor.__complex__ added (#43844)
• torch.vdot added (#43004)

API extension:

• torch.full added support for bool and integer dtypes (#41912)
• torch.lt and torch.masked_select added support for half dtype (#43704)
• torch.div, torch.true_divide, torch.atan2 added support for integer to float type promotion in (#42359)
• unflatten added support for non-named dimensions (#42563)
• torch.polygamma added support for n >= 2 (#42499)
• torch.qr added backward support for wide input matrices (#42216)
• nn.Linear for MKLDNN added support for no-bias (#43703)
• torch.lerp added support for half dtype (#43541)
• Updates torch.div to perform true division (end of deprecation cycle) (#42907)
• torch.scatter added support for reductions on CUDA (#41977)
• BFloat16 support type promotion (#41698, #43324)
• BFloat16 support on CUDA for torch.pow (#44760), unary ops and activations (#44813, #44824, #44834), torch.i0 (#44750), softmax (#44837), div, addcdiv, addcmul, mean, var (#44758), layernorm (#45002),all pooling layers (#44836, #45151)), torch.logspace (CPU and CUDA) (#44675), random kernels on Windows (#44918), torch.addmm, torch.addmv (#44986), loss functions (#45011), batched gemm (#45167), nccl path (#38515), binary logical operators (#42485), torch.neg (#45240), Conv (non-cuDNN) (#45007), torch.abs (#44804), torch.erfinv (#43399), comparison ops (#44748)
• torch.asin, torch.neg added support for sparse Tensors (#44028)
• torch.softmax added support for CUDA (#42307)
• Tensor.{real,imag} added setter for these attributes (#39860)
• torch.{addmm,addmv} added support for complex on CUDA (#40431, #43827)
• torch.bmm added support for complex on CPU #42383,
• torch.{dot, vdot} added support for complex (#42745)
• torch.stft, torch.istft added support for complex (#43886)
• torch.cholesky added support for complex (#44895, #45267)
• torch.sgn added (to support complex) (#39955)
• Binary ops added support for complex (#43174)
• Add allowlist for complex backward (#45461)

Autograd

• Don't automatically materialize output grads with zeros for autograd.Function (#41821)
• Benchmark tool for autograd.functional API (#43428)
• Added reset_grad API to remove gradient instead of setting them to zero (#44423)
• Allow Tensor-like objects in torch.autograd.gradcheck (#43877)
• Added support for nested call for @torch.no_grad() decorator (#44633)
• Added support for torch.lobpcg backward (#43002)

CUDA

• Added TF32 support (#41498)
• CUDA RTX30 series support (#45489, #45130)
  • **Note: **At the time of the 1.7 release, the currently available and stable Nvidia CUDA libraries are not fully tuned for the RTX 3080 and 3090 so users might see performance regressions.
• torch.cuda.amp.GradScaler now supports sparse gradients (#36786)
• Autocast support for cudnn RNNs (#42385)
• Support AMP in nn.parallel (#43102)
• Support for tf32 in cudnn and backends.cudnn.allow_tf32 flag to control it (#40737)
• Added torch.cuda.memory.list_gpu_processes to list running processes on a give GPU (#44616)
• Add env variable to bypass CUDACachingAllocator for debugging (#45294)
• Add non-deterministic alert to CUDA operations that use atomicAdd() (#41538)

C++ API

• nn::TransformerEncoderLayer added (#42633)
• nn::TransformerDecoderLayer added (#42717)
• nn::TransformerEncoder added (#43187)
• nn::TransformerDecoder added (#42886)
• nn::Transformer added (#44333)
• nn::Unflatten added (#42613)
• nn.ParameterList added (#41259)
• torch::cuda::manual_seed and torch::cuda::manual_seed_all added (#42638)

Mobile

• Support Tensor MemoryFormat in java wrappers (#40785)
• Add mobile_optimized boolean flag to optimized model. (#45479)

Vulkan

• Backend added (#36491, #43076)
• Add many operators adaptive_avg_pool2d (#41220), mm (#41221), reshape (#41223), max_pool2d (#41379), add_ and relu_ (#41380), cat (#41434), add and mul (#42674) and avg_pool2d (#42675).
• Model preparation via torch.utils.optimize_for_vulkan (#44903)
• Add to Java API option to load on Vulkan and test app (#44896, #44897)

Distributed

• Support alltoall collective in ProcessGroupGloo (#41424, #41690)
• Add a DDP Communication Hook providing the flexibility to completely override DDP gradient communication (#40848)
• Examples on how to use the DDP communication hook (#43310)
• Add NCCL Alltoall to NCCL process group (#42514)
• Support allgather and gather APIs for Python Objects (#42189)
• Join-based API to support uneven inputs in DDP (#42577)
• broadcast_object API for c10d (#43887)
• Async Error Handling support for ProcessGroupNCCL (#41050, #41051, #41052, #41053, #41054, #44163)
• Add a “gradient_as_bucket_view" parameter to DDP to reduce memory overhead (#44344)
• Add getNumKeys API to c10d TCPStore (#43962)
• Add DeleteKey API for c10d TCP Store (#45401)

Quantization

• New quantized ops
  • Adaptive average pooling (#40271)
  • Max pooling (#45152)
  • Embedding and EmbeddingBag quantization (8-bit + partial support for 4-bit): (#40076, #41293, #41612, #42924, #42762, #42881, #43077, #43088, #43090, #43176, #43296, #43433, #43989, #44008, #44207, #44208, #44217, #45149, #44845, #44048, #42690, #42612)
  • QNNPACK Transposed convolution2D and 3D (#39714, #40351, #40360, #40370, #40371, #44844, #45078, #45081)
  • Operations on quantized tensors
    • aten::repeat (#40644)
    • aten::apend (#40743)
    • stack (#42187)
    • fill_ (#43303)
    • clone for per channel affine quantized tensor (#44573)
    • append (graphmode) (#44641)
  • 1D batch normalization support (#42491)
  • N-Dimensional constant padding (#43304)
  • CELU operator (#39199)
• Support for FP16 quantization (#40708, #40709, #40710, #42147, #42221, #42222, #42348, #41049)
• Add Quantizer support to IValue (#42438)
• Custom module support (#44835)
• Preserving pre and post forward hooks (#37233)

Misc

• torch.set_deterministic and torch.is_deterministic: Raise error when the flag is set and a non-deterministic operation is used (#15359, #41377)
• Add CUDA 11 to nightly binaries (#44086, #43366)
• Dev Tool: Nightly checkout tool and doc in CONTRIBUTING.md (#42635, #43294)
• Website: Add docs for tagged version (include rc) on the general website (#45204)
• Build: Added BUILD_CAFFE2 flag to be able to disable caffe2 compilation (#43673)
• Dataloader: Add prefetch_factor argument to control the number of batch loaded ahead of time(#41130)
• Dataloader: Allow handling of np.memmap objects (#39847)
• ROCm: Add support torch utils.cpp_extension (#41257, #43528)
• ROCm: Enable complex BLAS (#43744)
• docker: Add torchelastic to docker image (#45438)
• docker: Add CUDA 11 support (#45071)
• docker: Use python 3.8 in pytorch docker image (#45466)

Improvements

Python API

• Use tree-based sum for floats to avoid numerical instability (#39516)
• nn.ReflectionPad: Add support for 0-dim batch sizes. (#39231)
• torch.scatter: Add reductions for CPU (#36447)
• Allow any valid ASCII python identifiers as dimnames (#40871)
• Improve Python warning prints when there is also an error (#41116)
• torch.iinfo, torch.finfo: Improve printing (#40488)
• torch.where: Add support for scalar input (#40336)
• torch.nonzero: Remove deprecation warning for as_tuple argument (#45413)
• torch.distributions.Categorical: Clamp logit to avoid -inf when calculating entropy (#41002)
• torch.futures.Future: Add done function to query the status of the future (#42013)

torch.nn

• nn.EmbeddingBag: Add support for incude_last_offset=True when reduction is mean or max (#42215)
• nn.AvgPooling{1,2,3}d: Ensure all cells are valid in ceil mode to avoid division by 0 (#41368)
• nn,[Adaptive]MaxPool{1,2,3}d: Handle edge case when input is filled with -inf (#40665)
• nn.Hardsigmoid, nn.Hardswish: Add inplace option (#42346)
• nn.MSELoss, nn.L1Loss, nn.SmoothL1Loss: Add support for target that requires gradients. (#44437, #44471, #44486)
• nn.Parameter{List,Dict}: Add warning when improperly used (with DataParallel or weight_norm) (#44405)
• nn.functional.smooth_l1: Add beta parameter (#44433)

Build

• Report error when ATEN_THEADING is OMP and USE_OPENMP is turned off. (#40146)
• Raise nice error when trying to build PyTorch on 32-bit Windows system (#40321)
• Make setup.py Python-2 syntactically correct and work for version >= 3.9 (#41960, #46388)
• Don't proceed into setup.py too far if Python version is unsupported (#42870)

Distributed

• Support profiling rpc_async in TorchScript (#40652)
• Allow RPC to be initialized again after shutdown. (#42723)
• Support rpc_sync, rpc.remote in TorchScript (#43043, #43046)
• Make async_execution compatible with RRef helpers (#44666)
• Extend RPC profiling to support async function execution over RPC. (#44664)
• Support record_shapes in RPC profiling (#44419)
• Add variants for cuda.comm.broadcast/gather/scatter which store the result in a provided “out” parameter (#39681)
• Explicitly abort NCCL Communicators on ProcessGroupNCCL Destruction (#40585)
• Helper function to print out all DDP-relevant env vars (#41297)
• Add timeout to ProcessGroup Work Wait (#40944)
• Support Wait Timeout in ProcessGroupNCCL (#40946)
• Support work-level timeouts in ProcessGroupGloo (#40948)
• Support for torch.bool in ProcessGroupNCCL (#41959)
• DDP.train() returns self to stay consistent with nn.Module (#42131)
• Add a drop_last option in DistributedSampler to drop tail of the data to ensure data is even across ranks (#41171)
• Additional error checking for torch.cuda.nccl APIs. (#43247)
• Support work.result() to get result tensors for allreduce for Gloo, NCCL backends (#43970)
• Add a device parameter to RemoteModule (#44254)
• Add remote_parameters() API for RemoteModule. (#43906)
• Add a warning log when there is high skew of uneven inputs in DDP training (#45238)

TorchScript

• Support string concatenation (cc29c192a6)
• Support using Python Enum in TorchScript (#41390,#41965,#42085,#42623,#42661,#42661,#42874,#43460,#43188,#44243,#44891)
• Support sorting list of strings (#42398)
• Support boolean key in dictionary (#42833)
• Support @torch.no_grad (#41371)
• Support del to TorchScript classes (#44352)
• Speed up saving modules in case of having many classes (#44589)
• Support Python Slice class in TorchScript (#44335)
• Support sorting a list of tuples (#43448)
• Enable @torch.jit.unused syntax for ignoring properties (#45261)
• Enable ProfilingExecutor + TensorExpression (#45546) (#45546)
• Support @torch.jit.unused on a @torch.no_grad decorated function (#41496)
• Improve ModuleList indexing error msg (#43361)
• Better match behavior of loaded `ScriptModule``s vs. freshly created ones (#43298)
• Support backend-lowered submodules (#41146)
• Allow freezing of modules containing interface attribute (#41860)
• to_backend API now accepts wrapped modules (#43612)
• Allow submodule methods inference rules to be different (#43872)
• Support default values for arguments of class type methods (#45098)
• Improve sugared value's error message when closing over global variables (#42889)
• Support backend-lowered submodules (#40841)
• Turn on non-ASCII string literals serialization (#40719)
• Better printing of Tensor stride information (#45156)

Mobile

• Allow specifying PYTHON executable to build_android (#41927)
• Include all overloads for OSS custom build (a01e91e6b2)

Quantization

• Change the whitelist to allowlist (#41771, #41802)
• dequantize now supports list and tuple of tensors (#41079)
• User now has a way to add a activation post process hook using register_activation_post_process_hook function (#42342)
• add/mul now support different variants (#42769)
• Fake quantizer now has more info when printed (#43031)
• OP_LIST_TO_FUSER_METHOD is exposed to the user (#43286)
• quantize_jit can handle new upsample overloads (#43407)
• Setter/getter method for quantization and fusion mappings (#43990)
• fake_quant and observer can be disabled in scriptmodule (#44773)
• convert_jit can now take preserved_attrs argument (#44490)
• SyncBN: preserve qconfig if it exists (#45317)
• Add quant APIs to save/load observer state_dict (#44846)
• Add version support for the conv parameters (#43524, #43086, #43651, #44671)

ONNX

In PyTorch 1.7, we have continued to add and improve PyTorch operator export to ONNX. We have enabled export of 10 new operators, and further enhanced and optimized export of 10+ torch operators to ONNX. We have also focused on improving export of TorchScript modules, in particular laying some groundwork required for better support in near future. We have also created an API (torch.onnx.utils._find_missing_ops_onnx_export) as a diagnostic tool (preview only) to get a list of operators in a model that are not supported or implemented by ONNX exporter. Support for export of torch.quantization.FakeQuantize has also been added to help enable some QAT workflows.

• Add support to export more torch ops torch.view_as (#40496), fake quantize functions (#39738), embedding_bag (#41234, #44693), torch.eye (#41357), Tensor.as_strided (#41569), torch.tensor (#41872), addition between list of tensors (#41888), Tensor.__floordiv__ (#43022), torch.nn.KLDivLoss (#41858), Tensor.new_empty and Tensor.new_zeros (#43506)

• Improves existing export logic and optimizing exported ONNX graph

  • Add warning in ONNX export when constant folding is on in training-amenable mode (#40546)
  • Fix export of torch.full_like (#40063)
  • Add pass that fuses Conv and BatchNormalization (#40547)
  • torch.where export, add support for ByteTensor (#42264)
  • Fix scalar type cast for comparison ops (#37787)
  • torch.scatter export, add support for src being scalar or different dtype (#42765, #43440)
  • Fix Squeeze operator when applied to a dimension with shape > 1 (#38476)
  • Extend support for torch.where (#41544)
  • Update ops torch.slice (#42935), torch.split (#43670), torch.repeat (#43430), torch.arange (#43777), len (#43824), torch.narrow (#44039), flatten (#40418), adaptive_pool (#46100)

• Update export to follow pytorch changes

  • Update div export to perform true divide (#44831)
  • Enable true_divide scripting export with ONNX shape inference (#43991)

Misc

• torch.utils.collect_env: Collect more informations (python 32/64bit, clang version, CPU architecture, ROCm version) (#42887, #42961, #44106)
• torch.hub.load_local: Allow to load models from any local directory (#44204)
• Add warning if import torch is called from the source root (#39995)
• Improve Dynamic Library loading for Windows (#40365)
• serialization: validate sparse tensors after loading (#34059)
• Add --continue-through-error option to run_test.sh script (#41136)
• Tensorboard: Support custom run_name and ``hparam_domain_discreteinadd_hparams` (#40660, #40720)
• MKLDNN: Enable conv3d, batchnorm3d, max_pool3d and avg_pool3d (#40691, #40995, #40996)
• Profiler: Do not record zero duration kernel events (#41540)
• Profiler: Improve cuda time counting (#45209)
• Profiler: Adding with_source parameter to enable tracking source code (#43898)
• Optim: Add verbose param for all schedulers (#41580)
• Pruning: check attributes before deleting (#41913)
• Autograd: In zero_grad, avoid using inpalce detach when it is not required (#41283)
• Autograd: Update the torch.div backward formula to improve numerical stability (#43627)
• Autograd: Print all traceback for higher order backwards in detect_anomaly (#43626)
• Autograd: Stop saving input of torch.repeat as only input.dim() is needed in backward (#40766)
• CUDA: Improve cuDNN error messages to include call parameters (#45023)
• CUDA: Improve device_count and cuda init error detection and messages (#42249)
• Improve Tensor layout propagation for pointwise ops to follow input layout more closely (#42922)
• Remove blacklist/whitelist references (#41447, #41644, #41636, #41777, #41822, #41691, #41789, #41979, #41627, #42011, #41796, #42067, #42091, #42097, #42071, #42089, #42279, #42047, #42088, #45260)

Python Type Annotations

• Update some types in top level torch/*.py (#40235, #40873)
• Added typing for Tensor attributes and methods: T and grad_fn (#40879), Tensor._version (#41125), ndim (#42909), nonzero (#43053), #40499)
• Added typing for torch.serialization (#40862)
• Added typing for torch.tensor (#45077)
• Added typing for torch.Size (#40879)
• Added typing for torch.futures (#41675)
• Added typing for torch.random (#42234)
• Added typing for torch.hub (#42252)
• Added typing for collect_env.py (#43062)
• Added typing for torch.utils (#39392, #42647, #42711, #42960, #43806, #44136, #44216)
• Added typing for torch.nn (#43044, #44093, #43080, #42231, #40669)
• Added typing for torch.sparse (#43108)
• Added typing for torch.cuda.nvtx (#43443)
• Added typing for torch.cuda.memory (#43444)
• Added typing for torch.functional (#43446)
• Added typing for torch.autograd (#44451, #46206)
• Added typing for torch.quantization.fuse_modules (#43786)
• Added typing for torch.nn.quantized (#43186, #44154, #43110)
• Added typing for torch.testing._internal submodules (#44575, #44805, #44832, #44911, #44927, #44985, #44971, #45107, #45368, #45375)
• Added typing for torch.backends.quantized ([#44794](https

Note truncated.

PyTorch 1.6.0 Release Notes

• Highlights
• Backwards Incompatible Changes
• Deprecations
• New Features
• Improvements
• Bug Fixes
• Performance
• Documentation

Highlights

The PyTorch 1.6 release includes a number of new APIs, tools for performance improvement and profiling, as well as major updates to both distributed data parallel (DDP) and remote procedure call (RPC) based distributed training.

A few of the highlights include:

1. Automatic mixed precision (AMP) training is now natively supported and a stable feature - thanks to NVIDIA’s contributions;
2. Native TensorPipe support now added for tensor-aware, point-to-point communication primitives built specifically for machine learning;
3. New profiling tools providing tensor-level memory consumption information; and
4. Numerous improvements and new features for both distributed data parallel (DDP) training and the remote procedural call (RPC) packages.

Additionally, from this release onward, features will be classified as Stable, Beta and Prototype. Prototype features are not included as part of the binary distribution and are instead available through either building from source, using nightlies or via compiler flag. You can learn more about what this change means in the post here.

[Stable] Automatic Mixed Precision (AMP) Training

AMP allows users to easily enable automatic mixed precision training enabling higher performance and memory savings of up to 50% on Tensor Core GPUs. Using the natively supported torch.cuda.amp API, AMP provides convenience methods for mixed precision, where some operations use the torch.float32 (float) datatype and other operations use torch.float16 (half). Some ops, like linear layers and convolutions, are much faster in float16. Other ops, like reductions, often require the dynamic range of float32. Mixed precision tries to match each op to its appropriate datatype.

• Design doc | Link
• Documentation | Link
• Usage examples | Link

[Beta] TensorPipe backend for RPC

PyTorch 1.6 introduces a new backend for the RPC module which leverages the TensorPipe library, a tensor-aware point-to-point communication primitive targeted at machine learning, intended to complement the current primitives for distributed training in PyTorch (Gloo, MPI, ...) which are collective and blocking. The pairwise and asynchronous nature of TensorPipe lends itself to new networking paradigms that go beyond data parallel: client-server approaches (e.g., parameter server for embeddings, actor-learner separation in Impala-style RL, ...) and model and pipeline parallel training (think GPipe), gossip SGD, etc.

# One-line change needed to opt in
torch.distributed.rpc.init_rpc(
    ...
    backend=torch.distributed.rpc.BackendType.TENSORPIPE,
)

# No changes to the rest of the RPC API
torch.distributed.rpc.rpc_sync(...)

• Design doc | Link
• Documentation | Link

[Beta] Memory Profiler

The torch.autograd.profiler API now includes a memory profiler that lets you inspect the tensor memory cost of different operators inside your CPU and GPU models.

Here is an example usage of the API:

import torch
import torchvision.models as models
import torch.autograd.profiler as profiler

model = models.resnet18()
inputs = torch.randn(5, 3, 224, 224)
with profiler.profile(profile_memory=True, record_shapes=True) as prof:
    model(inputs)

# NOTE: some columns were removed for brevity
print(prof.key_averages().table(sort_by="self_cpu_memory_usage", row_limit=10))
# ---------------------------  ---------------  ---------------  ---------------
# Name                         CPU Mem          Self CPU Mem     Number of Calls
# ---------------------------  ---------------  ---------------  ---------------
# empty                        94.79 Mb         94.79 Mb         123
# resize_                      11.48 Mb         11.48 Mb         2
# addmm                        19.53 Kb         19.53 Kb         1
# empty_strided                4 b              4 b              1
# conv2d                       47.37 Mb         0 b              20
# ---------------------------  ---------------  ---------------  ---------------

• PR | Link
• Documentation | Link

Distributed and RPC Features and Improvements

[Beta] DDP+RPC

PyTorch Distributed supports two powerful paradigms: DDP for full sync data parallel training of models and the RPC framework which allows for distributed model parallelism. Currently, these two features work independently and users can’t mix and match these to try out hybrid parallelism paradigms.

Starting PyTorch 1.6, we’ve enabled DDP and RPC to work together seamlessly so that users can combine these two techniques to achieve both data parallelism and model parallelism. An example is where users would like to place large embedding tables on parameter servers and use the RPC framework for embedding lookups, but store smaller dense parameters on trainers and use DDP to synchronize the dense parameters. Below is a simple code snippet.

// On each trainer

remote_emb = create_emb(on="ps", ...)
ddp_model = DDP(dense_model)

for data in batch:
   with torch.distributed.autograd.context():
      res = remote_emb(data)
      loss = ddp_model(res)
      torch.distributed.autograd.backward([loss])

• DDP+RPC Tutorial | Link
• Documentation | Link
• Usage Examples | Link

[Beta] RPC - Asynchronous User Functions

RPC Asynchronous User Functions supports the ability to yield and resume on the server side when executing a user-defined function. Prior to this feature, when an callee processes a request, one RPC thread waits until the user function returns. If the user function contains IO (e.g., nested RPC) or signaling (e.g., waiting for another request to unblock), the corresponding RPC thread would sit idle waiting for these events. As a result, some applications have to use a very large number of threads and send additional RPC requests, which can potentially lead to performance degradation. To make a user function yield on such events, applications need to: 1) Decorate the function with the @rpc.functions.async_execution decorator; and 2) Let the function return a torch.futures.Future and install the resume logic as callbacks on the Future object. See below for an example:

@rpc.functions.async_execution
def async_add_chained(to, x, y, z):
    return rpc.rpc_async(to, torch.add, args=(x, y)).then(
        lambda fut: fut.wait() + z
    )

ret = rpc.rpc_sync(
    "worker1", 
    async_add_chained, 
    args=("worker2", torch.ones(2), 1, 1)
)
        
print(ret)  # prints tensor([3., 3.])

• Tutorial for performant batch RPC using Asynchronous User Functions| Link
• Documentation | Link
• Usage examples | Link

[Beta] Fork/Join Parallelism

This release adds support for a language-level construct as well as runtime support for coarse-grained parallelism in TorchScript code. This support is useful for situations such as running models in an ensemble in parallel, or running bidirectional components of recurrent nets in parallel, and allows the ability to unlock the computational power of parallel architectures (e.g. many-core CPUs) for task level parallelism.

Parallel execution of TorchScript programs is enabled through two primitives: torch.jit.fork and torch.jit.wait. In the below example, we parallelize execution of foo:

import torch
from typing import List

def foo(x):
    return torch.neg(x)

@torch.jit.script
def example(x):
    futures = [torch.jit.fork(foo, x) for _ in range(100)]
    results = [torch.jit.wait(future) for future in futures]
    return torch.sum(torch.stack(results))

print(example(torch.ones([])))

• Documentation | Link

Backwards Incompatible Changes

Dropped support for Python <= 3.5 (#39879)

The minimum version of Python we support now is 3.6. Please upgrade your Python to match. If you use conda, instructions for setting up a new environment with Python >= 3.6 can be found here.

Throw a RuntimeError for deprecated torch.div and torch.addcdiv integer floor division behavior (#38762, #38620)

In 1.5.1 and older PyTorch releases torch.div , torch.addcdiv, and the / operator perform integer floor division. In 1.6 attempting to perform integer division throw a RuntimeError, and in 1.7 the behavior will change so that these operations always perform true division (consistent with Python and NumPy division).

To floor divide integer tensors, please use torch.floor_divide instead.

<p align="center"> <table align="center"> <tr><th>1.5.1</th><th>1.6.0</th></tr> <tr valign="top"> <td><sub><pre lang="python">

torch.tensor(3) / torch.tensor(2) ../aten/src/ATen/native/BinaryOps.cpp:81: UserWarning: Integer division of tensors using div or / is deprecated, and in a future release div will perform true division as in Python 3. Use true_divide or floor_divide (// in Python) instead. tensor(1) </pre></sub></td> <td><sub><pre lang="python">

NB: the following is equivalent to

torch.floor_divide(torch.tensor(3), torch.tensor(2))

torch.tensor(3) // torch.tensor(2) tensor(1) </pre></sub></td> </tr> </table> </p>

The fix for torch.addcdiv is similar.

<p align="center"> <table align="center"> <tr><th>1.5.1</th><th>1.6.0</th></tr> <tr valign="top"> <td><sub><pre lang="python">

input = torch.tensor(0) tensor = torch.tensor(1) other = torch.tensor(3) value = 1 torch.addcdiv(input, tensor, other, value=value) ../aten/src/ATen/native/PointwiseOps.cpp:81: UserWarning: Integer division with addcdiv is deprecated, and in a future release addcdiv will perform a true division of tensor1 and tensor2. The current addcdiv behavior can be replicated using floor_divide for integral inputs (self + value * tensor1 // tensor2) and division for float inputs (self + value * tensor1 / tensor2). The new addcdiv behavior can be implemented with true_divide (self + value * torch.true_divide(tensor1, tensor2). tensor(0) </pre></sub></td> <td>_{<pre lang="python"> input = torch.tensor(0) tensor = torch.tensor(1) other = torch.tensor(3) value = 1 (input + torch.floor_divide(value * tensor, other)) tensor(0) </pre>}</td> </tr> </table> </p>

Prevent cross-device data movement for zero-dimension CUDA tensors in binary pointwise PyTorch operators (#38998)

In previous versions of PyTorch, zero dimensional CUDA tensors could be moved across devices implicitly while performing binary pointwise operations (e.g. addition, subtraction, multiplication, division, and others). For example,

torch.tensor(5, device='cuda:0') + torch.tensor((1, 1), device='cuda:1')

would work, even though the tensors are on different CUDA devices. This is a frequent source of user confusion, however, and PyTorch generally does not move data across devices without it being explicit. This functionality is removed in PyTorch 1.6.

To perform binary pointwise operations on data of different devices, please cast the tensors to the correct device by using Tensor.to:

<p align="center"> <table align="center"> <tr><th>Version 1.5.1</th><th>Version 1.6.0</th></tr> <tr valign="top"> <td><sub><pre lang="python">

torch.tensor(5, device='cuda:0') + torch.tensor((1, 1), device='cuda:1') torch.tensor([6, 6], device='cuda:1') </pre></sub></td> <td><sub><pre lang="python"> torch.tensor(5, device='cuda:0').to('cuda:1') + torch.tensor((1, 1), device='cuda:1') torch.tensor([6, 6], device='cuda:1') </tr> </table> </p>

Dropped support for CUDA 9.2 on Windows

In previous versions of PyTorch, we provided an installation option for Windows environments running CUDA 9.2. Starting from PyTorch 1.6.0, we are no longer providing those binaries. Please upgrade your CUDA version to 10.1 or 10.2 and install a PyTorch binary for one of those CUDA versions instead.

PyTorch release binaries dropped dedicated bytecode for CUDA compute capability 6.1; removed PTX for CUDA compute capability 3.7

To check whether you are affected, please find your GPU in a table inthis link.

If you are using a Nvidia GPU with compute capability 6.1, you may notice a performance hit when using the release binaries (installed via pip or conda). We stopped building for CUDA compute capability 6.1 but PyTorch programs should still continue to work with those devices. If you do notice a performance hit, a workaround is to compile PyTorch from source.

If you are using a Nvidia GPU with compute capability 3.7 and relied on PTX, we have dropped support for that in our release binaries (installed via pip or conda). Potential workarounds are: install a previous version of PyTorch or to compile PyTorch from source.

Changed how bool tensors are constructed from non-bool values to match Python, C++, and NumPy (#38392)

In previous versions of PyTorch, when a bool tensor is constructed from a floating-point tensor, we would first convert the tensor to a long tensor, then to float tensor. This is not consistent with how bools are interpreted in Python, C++, and NumPy (just to name a few), which interpret 0 floating-point values as False and everything else as True.

If you were relying on the previous behavior, the following code will achieve the same effect.

<p align="center"> <table align="center"> <tr><th>Version 1.5.1</th><th>Version 1.6.0</th></tr> <tr valign="top"> <td><sub><pre lang="python">

torch.tensor([-2, -1, -0.9, 0, 0.9, 1, 2], dtype=torch.bool) tensor([ True, True, False, False, False, True, True]) </pre></sub></td> <td><sub><pre lang="python"> torch.tensor([-2, -1, -0.9, 0, 0.9, 1, 2]).long().bool() tensor([ True, True, False, False, False, True, True]) </tr> </table> </p>

Throw RuntimeError when torch.full would infer a float dtype from a bool or integral fill value (#40364)

In PyTorch 1.6 bool and integral fill values given to torch.full must set the dtype our out keyword arguments. In prior versions of PyTorch these fill values would return float tensors by default, but in PyTorch 1.7 they will return a bool or long tensor, respectively. The documentation for torch.full has been updated to reflect this.

Enabled thread parallelism for autograd on CPU (#33157)

In previous versions of PyTorch, running .backward() in multiple threads causes them to be serialized in a specific order, resulting in no parallelism on CPU. In PyTorch 1.6.0, running .backward() in multiple threads no longer serializes the execution and instead autograd will run those in parallel.

This is BC-breaking for the following two use cases:

• If any weights are shared among threads, gradient accumulation that was previously deterministic may become non-deterministic in 1.6 as two different threads will write to the .grad attribute in a non-deterministic order.
• If you use any C++ hooks, those are not guaranteed to be thread-safe. Please change them to be thread-safe.

In more detail, in 1.6.0, when you run backward() or grad() via python, TorchScript or the C++ API in multiple threads on CPU, you should expect to see extra concurrency. For example, you can manually write multithreaded Hogwild training code like:

# Define a train function to be used in different threads
def train_fn(model, input):
    # forward
    y = model(input)
    # backward
    y.sum().backward()
    # potential optimizer update

# define your model in python or in TorchScript
model = Model()
# User write their own threading code to drive the train_fn
threads = []
for _ in range(10):
    # define or load the data
    input = torch.ones(5, 5, requires_grad=True)
    p = threading.Thread(target=train_fn, args=(model, input))
    p.start()
    threads.append(p)

for p in threads:
    p.join()

Note when you use the same model and call backward() concurrently in multiple threads, model parameters are automatically shared across threads. The gradient accumulation might become non-deterministic as two backward calls might access and try to accumulate the same .grad attribute. Although we do proper locking to avoid data corruption, we don't guarantee the order in which the ops are executed, so non-determinism might arise, but this is an expected pattern in multithread training. You could use the functional API torch.autograd.grad() to calculate the gradients instead of backward() to avoid the non-determinism.

For thread safety:

• The custom Python/C++ Autograd Functions (both forward and backward) are properly protected and are guaranteed to be thread safe in 1.6.0.
• For hooks, both Python/C++ hooks will run concurrently. Note that in C++, just like in regular C++ threading, you will need to do proper locking when writing shared objects, so previous custom C++ hooks might not work nicely under a multithreaded environment in 1.6.0. In Python, just like in regular python threading, you can read/write objects safely but the order (and thus determinism) is not guaranteed.

Change autograd gradient accumulation logic to yield .grads that match the weights' memory layout (#40358)

In previous versions of PyTorch, autograd would yield contiguous gradients. Now, gradients have the same memory layout as their respective weights. This should result in silent performance improvements. Since PyTorch operators generally support non-contiguous tensors, this should have no functional effect on most PyTorch programs. A known exception is when accessing param.grad and performing an operation that requires a contiguous tensor, such as param.grad.view(-1). In this case, you will receive an error as follows: RuntimeError: view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead.

If a user wants to force accumulation into a grad with a particular layout, they can preset param.grad to a zeroed tensor with the desired strides or manually set grad to have the desired strides ( param.grad = param.grad.contiguous(desired format).)

See the below section on “Note: BC-breaking memory format changes” for more details.

Change memory format promotion rules of pointwise operators (#37968)

In previous versions of PyTorch, performing a binary pointwise operation between a Contiguous and a Channels Last tensor produced a Channels Last. In PyTorch 1.6, this now returns a tensor with the layout of the first operand.

See the below section on“Note: BC-breaking memory format changes” for more details.

Note: BC-breaking memory format changes

Operations that now return tensors in a different memory format generally should have no functional effect on most PyTorch programs because PyTorch operators generally support non-contiguous tensors.

The most common incompatibility with Python programs is with the view operator, which has specific stride requirements. If these requirements are no longer met as a result of this change, you will get an error message indicating that you should use reshape instead, i.e. "RuntimeError: view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead."

Another possible exception incompatibility is if you have a (usually) C++ operator implementation that works directly on memory (i.e. calls data_ptr and relies on the strides being contiguous).

nn.functional.interpolate: recompute_scale_factor default behavior changed from True to False (#39453)

In PyTorch 1.5.1 and older versions, nn.functional.interpolate(input, size, scale_factor, ..., recompute_scale_factor) has a default of recompute_scale_factor = True. In PyTorch 1.6, we’ve changed the default to recompute_scale_factor = False.

Depending on the precision of the scale_factor, this may result in an output tensor with different values than before. To retain the old behavior, simply change your code to use recompute_scale_factor = True.

More concretely, what recompute_scale_factor = True means is, if the user passes in a scale_factor:

1. We will first compute the new output size; and
2. Then, we will compute a new scale_factor by dividing the output size by the input size and sending it to an internal helper function.
3. The new scale_factor is used in the interpolate computation but in some cases is different from the scale_factor the user passed in.

This behavior resulted in loss of precision so we deprecated it in PyTorch 1.5.0. In PyTorch 1.6 and onward, recompute_scale_factor has a default of False, which means that we pass it directly to an internal helper function.

out= arguments of pointwise and reduction functions no longer participate in type promotion (#39655)

In PyTorch 1.5 passing the out= kwarg to some functions, like torch.add, could affect the computation. That is,

out = torch.add(a, b)

could produce a different result than

torch.add(a, b, out=out)

This is because previously the out argument participated in the type promotion rules. For greater consistency with NumPy, Python, and C++, in PyTorch 1.6 the out argument no longer participates in type promotion, and has no effect on the computation performed.

Changed torch.quasirandom.SobolEngine(..., scramble=True, seed=None) to respect torch.manual_seed when a seed has not been provided (#36427)

In previous versions of PyTorch, SobolEngine(..., scramble=True, seed=None) did not respect any calls to torch.manual_seed. The expected behavior for random number generation functions is to respect the seed set by torch.manual_seed, so we’ve changed SobolEngine to match.

If you were relying on the old behavior where SobolEngine ignores torch.manual_seed, please explicitly pass a different seed to SobolEngine:

<p align="center"> <table align="center"> <tr><th>Version 1.5.1</th><th>Version 1.6.0</th></tr> <tr valign="top"> <td><sub><pre lang="python">

torch.manual_seed(1337)

SobolEngine ignores the manual_seed and instead uses its own.

x1 = SobolEngine(dimension=1, scramble=True, seed=None).draw(3) </pre></sub></td> <td><sub><pre lang="python"> import time torch.manual_seed(1337)

To replicate the old behavior of, pass a seed to SobolEngine.

ms_since_epoch = int(round(time.now() * 1000)) x1 = SobolEngine(dimension=1, scramble=True, seed=ms_since_epoch).draw(3) </pre></sub></td> </tr> </table> </p>

Tensor.random_(to, from): Enforce check that from and to are within the bounds of the Tensor’s dtype (#37507)

In previous versions of PyTorch, to and from did not have to be within the bounds of the tensor’s dtype (this raised a warning). The behavior of random_ in that case can be unexpected. We are making this a hard error starting from PyTorch 1.6.0; please modify your code if you run into the error.

<p align="center"> <table align="center"> <tr><th>Version 1.5.1</th><th>Version 1.6.0</th></tr> <tr valign="top"> <td><sub><pre lang="python">

tensor = torch.zeros(10, dtype=torch.uint8)

256 is the maximum value for to for torch.uint8

tensor.random_(0, 257) UserWarning: to - 1 is out of bounds for unsigned char. </pre></sub></td> <td><sub><pre lang="python"> tensor = torch.zeros(10, dtype=torch.uint8)

256 is the maximum value for to for torch.uint8

tensor.random_(0, 256) </pre></sub></td> </tr> </table> </p>

Dropped support for CUDA < 9.2 from for source builds (#38977, #36846)

If you build PyTorch from source, we’ve dropped support for using CUDA < 9.2 (run nvcc --version to check your CUDA version). Users who install PyTorch packages via conda and/or pip are unaffected.

DataLoader’s __len__ changed to return number of batches when holding an IterableDataset (#38925)

In previous versions of PyTorch, len(<instance of dataloader holding an IterableDataset>) would return the number of examples in the dataset. We’ve changed it to be the number of batches (e.g., the number of examples divided by the DataLoader’s batch_size) to be consistent with the computation of length when the DataLoader has a BatchedSampler.

torch.backends.cudnn.flags: deleted unused verbose flag (#39228)

The verbose flag did nothing, so we deleted it. If you were passing a value to flags for verbose, please remove it.

RPC

RpcBackendOptions takes float instead of timedelta for timeout argument to stay consistent with timeout types in other TorchScriptable RPC APIs.

# v1.5
rpc.init_rpc(
    "worker1",
    rank=0,
    world_size=2,
    rpc_backend_options=rpc.ProcessGroupRpcBackendOptions(
        num_send_recv_threads=16,
        datetime.timedelta(seconds=20)
    )
)

# v1.6
rpc.init_rpc(
    "worker1",
    rank=0,
    world_size=2,
    rpc_backend_options=rpc.ProcessGroupRpcBackendOptions(
        num_send_recv_threads=16,
        20 # seconds
    )
)

TorchScript

The Default Executor Is Rolled Back To Legacy (#41017)

We rolled back to the old fuser and the legacy executor in this release in order to recover some reported performance regressions. In future releases we plan to reach the same or better performance with a new redesigned executor and fuser.

In order to switch back to the executor used in the 1.5 release one could use the following API:

• in Python: call torch._C._jit_set_profiling_executor(True) before you call your model for the first time,
• in C++: include #include <torch/csrc/jit/runtime/graph_executor.h> and set getExecutorMode() = true before you invoke your model for the first time.

Added dynamic versioning (#40279)

Note: this isn’t actually BC-breaking but we are listing it here because it is BC-Improving.

The PyTorch Team recommends saving and loading modules with the same version of PyTorch. Older versions of PyTorch may not support newer modules, and newer versions may have removed or modified older behavior. These changes are explicitly described in PyTorch’s release notes, and modules relying on functionality that has changed may need to be updated to continue working properly.

In this release, the historic behavior of torch.div and torch.full is preserved for models saved via torch.jit.save in previous versions of PyTorch. Modules saved with the current version of PyTorch will use the latest torch.div and torch.full behavior. See the notes above for the BC changes to those operators.

Internals

The following are a list of BC-breaking changes to some of PyTorch’s internal components.

Dispatcher C++ API has had some spring cleaning. This is still considered an “internal” API, but it is becoming more public facing as it stabilizes.

• Renamed callUnboxed() to call() in Dispatcher, OperatorHandle, KernelFunction (#37999)
• The TensorId suffix has been removed from most DispatchKey enum entries (#36240)
• Removed ::callOp(); use Dispatcher::call instead (renamed in #37797, removed in #38351, #38742)
• Removed KernelFunction::makeFromUnboxedFunctorFactory; use makeFromUnboxedFunctor directly instead (#35488)
• Renamed boxing/unboxing files and utilities in ATen/core/boxing (#35411)

autograd.gradcheck and autograd.gradgradcheck: Added a new default-true argument check_undefined_grad (#39400)

Internally, in the autograd engine, we use a special undefined Tensor value to represent zero-filled gradients and expect backward functions and user-defined torch.autograd.Functions to gracefully handle those values. When check_undefined_grad is True (the default for PyTorch 1.6+), gradcheck/gradgradcheck test that the operation in question supports undefined output gradients. This may cause a previously succeeding gradcheck to fail.

You can turn the check off by setting check_undefined_grad to False. As long as autograd does not error out due to an undefined gradient in your model, then everything should be fine.

<p align="center"> <table align="center"> <tr><th>Version 1.5.1</th><th>Version 1.6.0</th></tr> <tr valign="top"> <td><sub><pre lang="python">

torch.autograd.gradcheck(my_custom_function, inputs) True </pre></sub></td> <td><sub><pre lang="python">

To keep the previous behavior

torch.autograd.gradcheck(my_custom_function, inputs, check_undefined_grad=False) True </pre></sub></td> </tr> </table> </p>

[C++ API] Changed the TensorIterator API (#39803)

TensorIterator is an implementation detail for writing kernels that is exposed in our C++ API. We’ve modified how developers interact with TensorIterator, please see the Pull Request for more details.

Removed torch._min and torch._max(#38440)

torch._min and torch._max are undocumented and were intended to be an implementation detail; we expect very few users, if any at all, to be using it. We’ve deleted it in PyTorch 1.6.0. Please use torch.min/torch.max instead if you are using torch._min/torch._max.

Deprecations

Deprecated old torch.save serialization format (#39460, #39893, #40288, #40793)

We have switched torch.save to use a zip file-based format by default rather than the old Pickle-based format. torch.load has retained the ability to load the old format, but use of the new format is recommended. The new format is:

• more friendly for inspection and building tooling for manipulating the save files
• fixes a long-standing issue wherein serialization (__getstate__, __setstate__) functions on Modules that depended on serialized Tensor values were getting the wrong data
• the same as the TorchScript serialization format, making serialization more consistent across PyTorch

Usage is as follows:

m = MyMod()
torch.save(m.state_dict(), 'mymod.pt') # Saves a zipfile to mymod.pt

To use the old format, pass the flag _use_new_zipfile_serialization=False

m = MyMod()
torch.save(m.state_dict(), 'mymod.pt', _use_new_zipfile_serialization=False) # Saves pickle

Fixed missing deprecation warning for Tensor.nonzero() (#40187)

Calling torch.nonzero(tensor, as_tuple=False) with one argument or Tensor.nonzero(as_tuple=False) with no arguments is deprecated and will be removed in a future version of PyTorch. Please specify the as_tuple argument.

New Features

Python API

New Utilities

• Added global hooks to torch.nn.Module (#38972)
• Added option to enable cpp stack traces with TORCH_SHOW_CPP_STACKTRACES=1 (#38127)
• Added torch.utils.show_pickle for showing pickle contents in saved models (#35168)

New Operators

• torch.logcumsumexp added (#36308)
• torch.logaddexp added (#38384)
• torch.rad2deg, torch.deg2rad added (#38852)
• torch.arccosh, torch.arcsinh, torch.arctanh added (#38388)
• torch.flip{lr, ud} added (#38599)
• torch.bucketize, torch.searchsorted added (#34577)
• torch.istft (Inverse Short Time Fourier Transform) added (#35569)
• torch.vander: added support for generating Vandermonde matrices (#36725)
• torch.block_diag added (#33449)
• nn.Hardswish, nn.functional.hardswish added (#34747)
• torch.nn.init.trunc_normal_ (truncated normal initializer) added (#32397)
• Added Stochastic Weight Averaging. See torch.optim.AveragedModel and torch.optim.SWALR for more details.(#35032)

C++ API

• Added Optimizer AdamW to C++ frontend (#40009)
• Custom C++ autograd function now supports c10::optional<Tensor> as parameters (#37700)
• torch::Tensor now supports bitwise NOT(!), AND(&), OR(|), XOR(^) operators (#38691)
• Cpp extension now supports load and load_inline under ROCm (#35897)

[Beta] Complex Tensor support

The PyTorch 1.6 release brings beta-level support for complex tensors. The UX is similar to existing PyTorch tensors and the new complex-specific functionality is compatible with NumPy’s complex arrays. In particular, you’ll be able to create and manipulate complex tensors, interop with previously existing code that represented complex tensors as tensors of size (..., 2), and more.

While this is an early version of this feature, and we expect it to improve over time, the overall goal is provide a NumPy compatible user experience that leverages PyTorch’s ability to run on accelerators and work with autograd to better support the scientific computing and ML communities.

Please find the full documentation here.

Python API:

• Added torch.is_signed() for complex tensors. (#33773)
• Added dtype inference for complex tensors. (#33713)
• Added torch.randn and torch.normal_ for complex tensors. (#34037, #35056)
• Added complex type inference for torch.full. (#34709)
• Added type promotion logic for complex numbers. (#34093)
• Added is_complex tensor attribute for complex numbers. (#34093)
• Added torch.fill for complex tensors. (#34973)
• Added torch.rand for complex dtypes. (#34924, #35585)
• Fixed complex conversions, used in torch.copy_ , on cuda. (#35344)
• Added torch.from_numpy for complex dtypes. (#35531)
• Added a check to throw error for in place modification of non-complex tensors with complex number values. (#35883)
• Fixed torch.exp CPU implementation for complex tensors. (#35715)
• Added torch.masked_fill for complex tensors. (#36335)
• Updated torch.abs to return float tensors for complex tensors. (#35871)
• Added torch.isfinite and torch.isinf for complex tensors. (#36648)
• Added torch.isclose for complex tensors. (#36456)
• Updated torch.angle to return float tensors for complex tensors. (#36896)
• Enabled requires_grad for complex tensors. (#36932)
• Fixed reciprocal divide for complex tensors. (#37193)
• Added torch.reciprocal for complex tensors on CUDA. (#36749)
• Added Python API for Complex Storage. (#35771)
• Added torch.addmv for complex tensors. (#37924, #40238)
• Updated dtype inference for torch.tensor . (#38030)
• Added torch.pow for complex tensors on CUDA. (#36793)
• Added support for complex values as exponents in torch.pow .(#36793, #39117)
• Added torch.roll for complex tensors on CUDA. (#38664)
• Added torch.gather for complex tensors on CPU. (#36430)
• Added torch.tanh for complex tensors on CUDA. (#38786)
• Added complex dtypes to list of supported types in autograd. (#38325)
• Added torch.cumsum, torch.cumprod for complex tensors on CUDA. (#39063)
• Added real and imag views as tensor attributes. (#39033)
• Added torch.flip and torch.rot90 for complex tensors. (#37826)
• Added torch.view_as_real, torch.view_as_complex for complex tensors. (#39099)
• Added printing logic for complex tensors (#40513, #38031)
• Add torch.tan for complex tensors on CUDA (#38400)
• Added support for complex tensors for torch.tanh backward function (#37791, #38786)

C++ API:

• Added core of c10::complex. (#36626)
• Added overloads of std:: math functions in c10::complex (#37468, #37689)
• Added c10::complex as the C++ type for complex tensors (#37421, #39306)
• Added support for operations on c10::complex and integer scalars (#38418)
• Added overloads for complex math functions in both :: and std:: to fix ROCm bugs (#39829)
• Added at::tensor() and torch::tensor() for complex numbers (#39793)

Distributed

• torch.distributed: Add all_to_all API to the MPI backend in the distributed module (#32361).
• torch.distributed: Add c10d dynamic loading mechanism to support 3rd-party c10d implementations (#28068).
• torch.nn.parallel.DistributedDataParallel: Add distributed data parallel benchmark tool (#35198).
• torch.nn.parallel.DistributedDataParallel and torch.distributed.rpc: allow DDP to work with RPC (#37998, #39916, #40130, #40139, #40495).

Mobile

• Add torch.utils.mobile_optimizer.optimize_for_mobile to encapsulate several model optimizations appropriate for mobile models. (Note: currently broken on Windows.) (#35227) (#36357)

New operator registration API

PyTorch 1.6 has a new, pybind11-based operator registration API which replaces the torch::RegisterOperators() class.

Before:

static auto registry =
  torch::RegisterOperators("my_ops::warp_perspective", &warp_perspective);

After:

TORCH_LIBRARY(my_ops, m) {
  m.def("warp_perspective", warp_perspective);
}

You can read more about this API in the custom C++ operators tutorial or the reference documentation.

The new API was developed in PRs #35061, #35629, #35706, #36222, #36223, #36258, #36742, #37019. Internal code was ported to this API in #36799, #36800, #36389, #37834, #38014; you may find the code examples in these PRs helpful for your ports.

ONNX

In PyTorch 1.6, we have added support for ONNX Opset 12. We have also enhanced export of torchvision models, such as FasterRCNN, MaskRCNN, and KeypointRCNN to support dynamic input image size. Export support for several new ops have also been added. A new operator export mode, ONNX_FALLTHROUGH, has been added to the export API that allows exporting the model with non-standard ONNX operators. For large (> 2 GB) model export (using external_data_format=True argument), we now support models with large tensor data in attributes (not just model parameters).

New ONNX operator support:

• Update Dropout Export (#37641)
• Update Argmin/Argmax ONNX Export (#38329)
• Fix pow op export (#38065)
• Export Support for Celu (#38243)
• Add GreaterOrEqual and LessOrEqual to opset 12 ONNX export (#38311)
• ONNX Export Support for CrossEntropyLoss (#34830)
• Adding 'numel' and 'to' export for script module (#36501)
• Support clamp_min and clamp_max (#37872)
• Quantization: Add aten::max_pool2d to onnx jit pass (#34912)
• Quantization: Mark upsample_nearest2d, sigmoid and reshape as no scale in onnx (#36325)
• Quantization: export of quantized models with new conv and linear API in onnx (#38736)

Quantization

New quantization operators:

• quantized Conv1d (#35093, #36352, #38248, #38283, #38438, #38449, #38749)
• quantized hardsigmoid (#34959,#36351, #36698, #36699)
• quantized hardswish (#34820,#36350, #36252, #36320, #36545)
• quantized layernorm (#36593, #36690, #35693)
• quantized groupnorm (#36835, #39090)
• quantized instancenorm (#36847, #39091)
• quantized reflection_pad1d (#37452)
• quantized adaptive avgpool. (#36813)
• channel shuffle op fp32 + quantized. (#36815)
• qnnpack path for hardtanh (#35779)
• Quantized Threshold (#39352)

RPC

• torch.distributed.rpc: Add TensorPipe RPC backend (#36197, #35483, #37839, #37918, #37919,#37850,#37851, #37852,#37980, #38052, #38265, #38266, #40162, #40389, #37910, #38448, #38818, #38819, #38926, #38931, #38930, #38933, #38934, #39010, #39011, #39397)
• torch.distributed.rpc: Support per-RPC timeouts for rpc_sync and rpc_async (#34650)
• torch.distributed.rpc.functions.async_execution: Add an @async_execution decorator to allow pause and resume executions in RPC target functions (#39216, #39267, #39485, #39486, #39758).
• torch.futures.Future:Expose a Future type to Python API (#39008, #37311, #39119, #39597, #39964, #39950)
• torch.distributed.rpc: Allow profiler to be enabled remotely with RPC (#38748, #40066)
• torch.distributed.rpc: Implement TorchScript-compatible RemoteModule API (#37139, #40173)
• torch.distributed.rpc.RRef: enable retrying RRef control messages on communication failures (#33636)
• torch.distributed.rpc: Let RPC use torch._C.Future instead of exposing a dedicated future type. No impact on user side (#35039)
• torch.distributed.autograd: Add profiler support for backward of the distributed autograd engine (#35261)
• torch.distributed.rpc.RRef: Add TorchScript support for RRef.local_value() (#35433)
• torch.distributed.rpc.WorkerInfo: Add TorchScript support for WorkerInfo (#35447)
• torch.distributed.rpc: Allow profiling RPC with TorchScript target functions (#36275)
• torch.distributed.rpc.RRef: Add RRef Python Helper to launch function on the remotely referenced object (#36619)
• torch.distributed.rpc: Add timeout argument to TorchScriptable rpc_async (#37884)
• torch.distributed.rpc: Enable RPC Server Global Profiler (#38847)
• torch.distributed.rpc: Implement timeout support for rpc.remote and RRef.to_here() (#38590)
• torch.distributed.rpc: Enable RRef timeout for TensorPipe (#39531)
• torch.distributed.rpc.WorkerInfo: Add WorkerInfo python __repr__ magic method (#40004)

TorchScript

• Fork / Join Async Parallelism (#40438)
• ScriptModule Freezing (#40409, #37044, #38830, #34786, #34787)

Improvements

Python API

• Added long description to wheel packages (#39676)
• torch.add: Prevent unbounded growth while adding sparse tensors (#36030)
• torch.mv: enabled for sparse tensors (#21782)
• torch.bmm: enabled for sparse x dense tensor operations (#33430)
• torch.cat: improved error message (#38978)
• torch.masked_select: enabled bfloat16 support (#36859)
• torch.absolute: added as an alias for torch.abs (#36597)
• torch.device: improved error message to include xla as an acceptable device (#36446)
• torch.linspace, torch.logspace: improved precision (#35461)
• Tensor.true_divide method variant added (#34794)
• Tensor.isnan(), Tensor.isinf(), Tensor.isfinite() method variants added (#37942)
• Tensor.is_nonzero: improved error message (#38150)
• Tensor.cauchy_, Tensor.log_normal_, Tensor.exponential_: added support for bfloat16 (#38427)
• Tensor.as_subclass method added. (#34369)
• collect_env.py: improved to detect relevant conda-installed numpy and cudatoolkit (#35646)
• collect_env.py: made it more robust on Windows (#39136)
• torch.utils.data: Add generator= kwarg for DataLoader & random samplers (#39737)
• torch.utils.data.DataLoader: properly diagnose exceeding file descriptor limit (#34768)
• torch.utils.data.DataLoader: added repr for WorkerInfo (#39975)
• torch.utils.data.random_split: added option to pass a generator for determinism (#34043)
• torch.utils.data.IterableDataset: make the warning for when a DataLoader holds an IterableDataset clearer (#41185)
• torch.nn: Added support for non-persistent buffers that do not show up in a Module’s state dict (#37191)
• nn.Fold, nn.Unfold: added double backwards support (#36379)
• nn.MultiheadAttention: added support for bool/byte attn_mask tensor (#33763)
• nn.functional.upsample: enabled uint8 sampling support (#35029)
• nn.functional.kl_div: added option to accept target in log space (#34586)
• nn.functional.softmax: added support for sparse tensors (CPU) (#36305)
• nn.Softmin, nn.Softmax: improved repr (#39084)
• warnings: Changed warnings generated in cpp to show point of Python origination (#36052)
• warnings: Improve warnings to actually point at user code (#39143)
• Extend some of the basic ops to kHalf (#37121)
• Added a warning to a known autograd issue on XLA backend. (#35449, #35543)
• torch.cuda: Change DeprecationWarning to FutureWarning (#32142)
• Added torch.utils.cmake_prefix_path pointing to share/cmake folder (#38559)
• torch.hub: Added file_name argument to load_state_dict_from_url (#39749)
• Disable autograd while preparing Tensor for printing (#39420)
• Improved CUDA error message for MSVC (#39987)
• Improved reentrant autograd error message (#38625)
• Let >> and << support half on CUDA (#37670)
• dockerfile: Update miniconda installer download location & remove unnecessary flag (#37082)
• torch.cuda.get_arch_list() and torch.cuda.get_gencode_flags() added. These return the architecture list and gencode flags PyTorch was compiled with. (#41212)
• torch.min, torch.max: significantly improved CUDA performance (#38440, #39029)
• torch.multinomial with replacement=False: significantly improved performance (#39742)

Python Type Annotations

• torch.autograd: add type hints in-line (#38080)
• torch.finfo, torch.iinfo type annotations added (#38220)
• Moved torch.cuda annotations inline (#40075)
• Add typing for torch.cuda._CudaStreamBase and torch.cuda._CudaEventBase classes (#40256)
• Introduced torch.types.Device and stubbed all torch._C functions comprehensively (#38173)
• Move all torch.nn modules type annotations inline (#38211)
• Fixes type annotations for named tensors (#36890)
• Fix minor issue in type stub for Optimizer (#38067)
• Fixed some miscellaneous type hints (#36584)
• Fix multiple issues with type annotations (#36358)
• torch.autograd.anomaly_mode: fixed type hints stub (#39324)
• torch.backends.cudnn added type annotations (#38947)
• torch.channels_last, torch.preserve_format: added annotations (#39120)

AMD/ROCm

• torch.topk: enabled support for BFloat16 type on ROCm. (#34849)
• torch.dot: enabled fp16 support on ROCm (#30431, #30432)
• torch.add: enabled support for BFloat16 type on ROCm for sparse tensors(#35978)
• Enabled bfloat16 for operators in BERT model (#37634)
• torch.log: improved ROCm support (#40079)
• torch.pow, torch.exp, torch.erf: enabled support for BFloat16 type on ROCm (#40236)

C++ API

• Eliminate warnings for cpp extensions on Windows (#37400)
• Disable C4251 when compiling cpp_extensions on Windows (#35272) Note: Above two PRs eliminate unnecessary compile warnings for windows build, make build log more readable.

Distributed

• torch.distributed: Enhance error message for MPI unavailability. (#36781).
• torch.distributed: Expose torch.distributed.is_available() API (#37021).
• torch.utils.data: Only create torch.generator and seed in DistributedSampler when shuffling (#37604).
• ProcessGroup: Log incorrect device in ProcessGroupGloo (#38844).
• torch.utils.data: Improve DistributedSampler docs and add seed option (#39628).
• torch.cuda.comm.reduce: Avoid initializing unnecessary tensors in nccl.reduce (#39688).
• torch.nn.parallel.DistributedDataparallel: Remove obsolete warning message from DDP (#40190).

Distributions

• distributions.Cauchy: Implemented kl divergence (#36477)
• distributions.Transform: Add a .with_cache() method (#36882)
• distributions.Binary: Implemented BTRS algorithm for fast/efficient binomial sampling (#36858)

Internals

• New macro TORCH_FN for passing in compile time function pointers as regular function arguments rather than template arguments (#39823, #40110)
• Improved support for more types in registered custom kernels
  • Allow std::array as kernel argument and return (#34399)
  • Allow ArrayRef as kernel argument (#34335)
• Added FPGA DispatchKey, DeviceType, Backend for out-of-tree experimentation (#38938)
• Better type safety for calling the dispatcher; we now do a runtime test when casting OperatorHandle to TypedOperatorHandle that you’ve provided the correct type for kernels (#40251)
• OperatorHandle::callBoxed now works on all operators, you no longer need to manually go through JIT registry (#36010, #36850)
• Added Dispatcher::redispatch for performing a dispatch that bypasses the current key and all keys before it (#35476, subsequently renamed)
• More operators are fully supported by the dispatcher (#37273, #36564, #36398, #36666, #36838)
• Tracing is no longer done inside our autograd code; instead it has been factored into a separate Tracing dispatch key (#39514, #38467)
• DispatchKey computation no longer relies on TensorOptions; instead, factory functions and other functions with special dispatch key computation needs can register a BackendSelect kernel to compute the required key. (#36290, #36562, #37257)

ONNX

• Enable Constant Folding for ONNX Opset 12 (#34823)
• ONNX Update training ops and training amenable export API (#35567)
• Fix for constant folding: Slice, Added ReduceL1 and ReduceL2 (#35280)
• Added support for constant folding onnx::Add and onnx::Sub (#35869)
• Enable constant folding for Shape (#35386)
• Improve error checking for large model export (#37798)
• Remove Aten ops from ONNX export (#37239)
• Update pytoch/onnx doc (#39480)
• Update pytorch/onnx docs for new export API args (#39802)
• Support large attribute and subgraph for large model (#38793)

Operator Benchmark

• Added benchmark for quantized batchnorm (#35389)
• Added more quantized activation benchmarks and input sizes (#35729)
• Added __torch_function__ benchmarks (#36138)
• Aligned qconv benchmark to conv (#36673)
• Aligned the qlinear benchmark to linear (#36674)
• Added CUDA support for the observer benchmark (#39360)

Profiler

• torch.autograd.profiler: Make RecordFunction callbacks thread local and modernize interface (#37491)
• torch.autograd.profiler: Make profiler thread local (#36291)

Quantization

• Add ConvBn3d, ConvBnReLU3d, BNReLU2d, BNReLU3d to eager mode quantization (#33540)
• Enabled per channel quantized static linear/conv in QNNPACK (#37622)
• Enable per-channel quantization for LSTM Modules (#39666, #39041)
• Dynamic quantization support for LSTMCell, RNNCell and GRUCell (#40102)
• Quantization aware training now works with nn.DataParallel and nn.DistributedDataParallel
  • Make quantization modules work with nn.DataParallel (#37032)
  • fake_quant: move observer and fake_quant flags into buffers (#38368)
  • Make QAT Conv-BN work with nn.DistributedDataParallel and nn.SyncBatchNorm (#38478)
  • fake_quantize: respect device affinity (#39031)
• Add quantized tensor support on CUDA (#37081)
• Add reduce_range params for quantized_lstm (#39604)
• Use TorchBind for ConvPackedParams (#35923)
• Use TorchBind for Linear PackedParams" (#38101)

RPC

• torch.distributed.rpc.RRef: Throw an actionable error message on user call RRef.to_here() in TorchScript (#35369)
• torch.distributed.rpc.RRef: Handle exceptions returned via remote() calls (#35331)
• torch.distributed.rpc.RRef: Make RRef type hint mismatch exception message more actionable to users (#35943)
• torch.distributed.rpc:Allow abort RecvWork::wait() in ProcessGroupAgent::listenLoop (#36084)
• torch.distributed.autograd: Appropriately handle exceptions in autograd engine. (#36019)
• torch.distributed.autograd: Catch exception in distributed engine callbacks. (#36118)
• torch.distributed.autograd: Avoid some future callback self-captures. (#36502)
• torch.distributed.rpc: Propagate error from RPC retries to the original attempt (#35263)
• torch.distributed.autograd: Ensure future is complete when exiting Engine::mark_graph_task_completed() (#36856)
• torch.distributed.autograd: Trigger pre/post hooks of output function nodes under distributed autograd (#34501)
• torch.distributed.rpc: Supporting create an RPC gang of world size 1 (#32731)
• torch.distributed.autograd: Improve Error Message for Dist Autograd Context Cleanup Failure (#37255)
• torch.distributed.rpc: Guard against negative rpcTimeout being passed in to RpcBackendOptions (#38267)
• torch.distributed.rpc: Use infinite timeout for operations in ProcessGroup RPC backend (#38577)
• torch.distributed.rpc.WorkerInfo: Add stringify WorkerInfo (#39974)
• torch.distributed.rpc: Avoid using default process group in ProcessGroupAgent. (#39909)
• torch.distributed.rpc: Ignore expected errors in TensorPipe RPC backend (#39182)
• torch.distributed.rpc: Don't use separate heap allocation for metrics in TensorPipe RPC backend (#39183)
• torch.distributed.rpc: Bind to hostname's IP address instead of localhost in TensorPipe RPC backend (#39184)
• torch.distributed.rpc: Use PrefixStore to avoid conflicting keys in TensorPipe RPC backend (#39185)

TorchScript

Improvements

• Add id function (#34975)
• Add lazy script decorator (#34935)
• Make Future type annotation available in Python (#27637)
• Support converting str to float (#35352)
• Enable recording of TorchScript functions (#34710)
• Improve the error message when registering a custom class twice (#35568)
• Improve optimization of if statements with statically determinable predicates (#35834)
• Fix reporting of error message in toBool (#35570)
• Better error when types of default value and parameter do not match (#35888)
• Improve serialization for lists and dictionary (#35741)
• Add type hints on hardsigmoid, hardswish, and elu to make them scriptable (#35885)
• Add strict tracer flag to guard against risky behaviors (#36277)
• Add support of Dict as output when connecting script and tracing (#36265)
• Use current default dtype with torch.tensor when dtype is not specified (#36587)
• Add dictionary as output of tracer (#36696)
• Allowing casting str to int (#36016)
• Convert float Tensor argument to double in Tensor.tolist (#37465)
• Add a code_with_constants method to module printing (#37586)
• Support indexing using list literal as index (#37848)
• Support indexing using list variable as index (#37966)
• Support del statements with variables as targets in TorchScript (#37608)
• Recursively compile TorchScript class types (#38050)
• Better error message when missing init on custom C++ classes (#37474)
• Fix @staticmethod access from self on modules (#37702)
• Allow @torch.jit.unused to be used on TorchScript classes (#38522, #39336)
• Add support for %= operator in TorchScript (#38983)
• Provide error messages when JIT infers the type of an argument as Tensor (#38527)
• Allow self-referential type annotations in TorchScript classes ([#39821](https://github.com/pytorch/pytorch/pull/3982

Note truncated.

PyTorch 1.5.1 Release Notes

• Backwards Incompatible Changes
• Known Issues and Workarounds
• Critical Fixes
• Crashes and Error Fixes
• Other Fixes

Backwards Incompatible Changes

Autograd: Operations that return integer-type tensors now always returns tensors that don’t require grad (#37789).

This most notably affects torch.argmin, torch.argmax, and torch.argsort. This change is BC-Breaking because previously one could obtain an integer-type tensor that requires grad in 1.5.0. However, said tensors were not usable by autograd; calling .backward() on them resulted in an error, so most users are likely to not have been relying on this behavior.

<p align="center"> <table align="center"> <tr><th>Version 1.5.0</th><th>Version 1.5.1</th></tr> <tr valign="top"> <td><sub><pre lang="python">

tensor = torch.randn(3, requires_grad=True) torch.argmax(tensor).requires_grad True <td>_{<pre lang="python"> tensor = torch.randn(3, requires_grad=True) torch.argmax(tensor).requires_grad False </pre>}</td> </tr> </table> </p>

Known Issues and Workarounds

When using multiprocessing, PyTorch 1.5.1 and 1.5.0 may error out with complaints about incompatibility between MKL and libgomp (#37377)

You may see error messages like the following when using the torch.multiprocessing package. This bug has primarily affected users with AMD CPUs.

`Error: mkl-service + Intel(R) MKL: MKL_THREADING_LAYER=INTEL is incompatible with libgomp.so.1 library.
        Try to import numpy first or set the threading layer accordingly. Set MKL_SERVICE_FORCE_INTEL to force it.`

You can get rid of the error and the error message by setting the environment MKL_THREADING_LAYER=GNU. This can be done either by including the following in your python code:

import os
os.environ['MKL_THREADING_LAYER'] = 'GNU'

or by specifying the environment variable when running your script:

MKL_THREADING_LAYER=GNU python my_script.py

To learn more about what triggers this bug and other workarounds if the above isn’t working, please read this comment on the issue.

Critical Fixes

torch.multinomial: Fixed a bug where CUDA multinomial generated the same sequence over and over again with a shift of 4. (#38046)

nn.Conv2d: Fixed a bug where circular padding applied padding across the wrong dimension (#37881)

<p align="center"> <table align="center"> <tr><th>Version 1.5.0</th><th>Version 1.5.1</th></tr> <tr valign="top"> <td><sub><pre lang="python">

circular = nn.Conv2d(6, 1, (3, 3), padding=(0, 1), padding_mode='circular') circular(torch.zeros(1, 6, 10, 10)).shape

Notice the padding is incorrectly on the H dimension, not the W dimension.

torch.Size([1, 1, 10, 8]) </pre></sub></td> <td><sub><pre lang="python">

tensor = torch.randn(3, requires_grad=True) other = tensor + 1 output = nn.LeakyReLU(0, inplace=True)(other) output.sum().backward() torch.Size([1, 1, 8, 10]) </pre></sub></td> </tr> </table> </p>

Fixed bug where asserts in CUDA kernels were mistakingly disabled, leading to many silent kernel errors. (#38943, #39047, #39218)

torch.gather, torch.scatter: added checks for illegal input dtypes that caused silently incorrect behaviors (#38025, #38646)

torch.argmin, torch.argmax: Fixed silently incorrect result for inputs with more than 2^32 elements (#39212)

C++ Custom Operators: fixed a bug where custom operators stopped working with autograd and ignored the requires_grad=True flag. (#37355)

Crashes and Error Fixes

Fixed CUDA reduction operations on inputs with more than 2^32 elements (#37788)

<p align="center"> <table align="center"> <tr><th>Version 1.5.0</th><th>Version 1.5.1</th></tr> <tr valign="top"> <td><sub><pre lang="python">

torch.zeros(5, 14400, 14400, device='cuda').sum(0) RuntimeError: sub_iter.strides(0)[0] == 0 INTERNAL ASSERT FAILED at /pytorch/aten/src/ATen/native/cuda/Reduce.cuh:706, please report a bug to PyTorch. </pre></sub></td> <td><sub><pre lang="python"> torch.zeros(5, 14400, 14400, device='cuda').sum(0)

No problem

  </pre></sub></td>
</tr>

</table> </p>

Fixed pickling of PyTorch operators (#38033)

<p align="center"> <table align="center"> <tr><th>Version 1.5.0</th><th>Version 1.5.1</th></tr> <tr valign="top"> <td><sub><pre lang="python">

pickle.dumps(torch.tanh) PicklingError: Can't pickle <class 'torch._C._VariableFunctions'>: it's not the same object as torch._C._VariableFunctions </pre></sub></td> <td><sub><pre lang="python"> pickle.dumps(torch.tanh)

No problem

  </pre></sub></td>
</tr>

</table> </p>

nn.LeakyReLU: Fixed a bug where using autograd with in-place nn.LeakyReLu with a slope of 0 incorrectly errored out. (#37453, #37559)

<p align="center"> <table align="center"> <tr><th>Version 1.5.0</th><th>Version 1.5.1</th></tr> <tr valign="top"> <td><sub><pre lang="python">

tensor = torch.randn(3, requires_grad=True) other = tensor + 1 output = nn.LeakyReLU(0, inplace=True)(other) output.sum().backward() RuntimeError: In-place leakyReLu backward calculation is triggered with a non-positive slope which is not supported. This is caused by calling in-place forward function with a non-positive slope, please call out-of-place version instead. </pre></sub></td> <td><sub><pre lang="python"> tensor = torch.randn(3, requires_grad=True) other = tensor + 1 output = nn.LeakyReLU(0, inplace=True)(other) output.sum().backward()

No error

  </pre></sub></td>
</tr>

</table> </p>

torch.as_strided : Fixed crash when passed sizes and strides of different lengths. (#39301)

nn.SyncBatchNorm.convert_sync_batchnorm: Fixed bug where it did not respect the devices of the original BatchNorm module, resulting in device mismatch errors (#39344)

nn.utils.clip_grad_norm_: Fixed ability to operate on tensors on different devices (#38615)

torch.min, torch.max: added check for illegal output dtypes (#38850)

MacOS: Fixed import torch error (#36941).

C++ Extensions: fixed compilation error when building with older versions of nvcc (#37221)

This bug mainly affected users of ubuntu 16.04. We’re certain it affected the following configurations:

• ubuntu 16.04 + cuda 9.2 + gcc 5
• ubuntu 16.04 + cuda 9.2 + gcc 7
• ubuntu 16.04 + cuda 10.0 + gcc 5

C++ Extensions: fixed ability to compile with paths that include spaces (#38860, #38670)

C++ Extensions: fixed ability to compile with relative include_dirs for ahead-of-time compilation (#38264)

Other Fixes

nn.Conv1d, nn.Conv2d, nn.Conv3d: Fixed a bug where convolutions were using more memory than previous versions of PyTorch. (#38674)

Fixed in-place floor division magic method (#38695)

In 1.5.0, the in-place floor division magic method mistakingly performed the floor division out-of-place. We’ve fixed this in 1.5.1.

<p align="center"> <table align="center"> <tr><th>Version 1.5.0</th><th>Version 1.5.1</th></tr> <tr valign="top"> <td><sub><pre lang="python">

tensor = torch.ones(1) expected_data_ptr = tensor.data_ptr() tensor //= 1 tensor.data_ptr() == expected_data_ptr False <td>_{<pre lang="python"> tensor = torch.ones(1) expected_data_ptr = tensor.data_ptr() tensor //= 1 tensor.data_ptr() == expected_data_ptr True </pre>}</td> </tr> </table> </p>

Documentation: fixed link to java docs. (#39039)

Quantization: Fixed weight quantization inaccuracies for LSTM (#35961)

Weight quantization was done incorrectly for LSTMs, the statistics for all weights (across layers) were combined in the observer. This meant that weights for later layers in a LSTM would use sub-optimal scales impacting accuracy. The problem gets worse as the number of layers increases.

DistributedDataParallel: Fixed single-process multi-GPU use case (#36503)

RPC: Fixed future callbacks not capturing and restoring autograd context id (#38512)

TorchScript: Fixed support with torch.unique (#38156)

ONNX: Fix pow operator export (#39791)