TorchRL 0.13.2 is a patch release focused on regression fixes, release reliability, and CI stability. It intentionally avoids new feature backports.

Highlights

• Fixed Isaac Lab reset regressions, including Direct-env autoreset opt-in behavior, observation-key normalization, and reset_to edge cases.
• Fixed SliceSampler and EnvBase.step regressions affecting compile compatibility and batch-locked environments.
• Fixed MultiSyncCollector worker-output gathering and improved preemption throughput without busy-waiting.
• Made the vLLM FP32 override opt-in so importing TorchRL no longer changes host vLLM behavior.
• Restored and hardened nightly/release publishing checks, including PyPI wheel filtering and release-build symbol stripping.

Backported fixes and maintenance

• #3869 Fix IsaacLab reset regressions.
• #3868 Make the vLLM FP32 plugin opt-in.
• #3861 Fix FP32 override registration in the vLLM plugin.
• #3858 Fix _skip_tensordict handling in EnvBase.step.
• 4679d0a Fix SliceSampler compile compatibility.
• #3856 Fix flaky Pendulum spec and collector preemption-ordering tests.
• #3843, #3841 Fix and optimize MultiSyncCollector preemption.
• #3846, #3845, #3844 Harden nightly version checks and uploads.
• #3859, #3855, #3842, #3825, 12093ae Improve release, benchmark, auto-tag, and docs CI reliability.
• #3852 Fix tutorial code links.
• #3849, #3848, #3847 Refresh SOTA dependency pins.

No new public API exports are intended in this patch release.

TorchRL v0.13.1 is a maintenance release for the 0.13 line. It carries post-0.13.0 RNN backend fixes and performance improvements, compile-friendliness fixes, SOTA example dependency refreshes, and documentation improvements.

Merged PR inventory

Recurrent modules and RNN backends

• #3818 improves Triton RNN recurrent-matmul robustness with large-hidden tiling, 64-bit offsets, and faster autotune behavior.
• #3752 adds recompute backward support and narrow RNN canonicalization to reduce learner memory pressure when multiple recurrent modules share a batch.

Compile and conversion stability

• #3819 avoids a to_module FutureWarning graph break under torch.compile while preserving the previous state-preserving conversion behavior.

SOTA implementation dependency refreshes

• #3708 updates the GRPO SOTA implementation to vllm 0.20.0.
• #3601 updates the expert-iteration SOTA implementation to transformers 5.0.0rc3.

Documentation

• #3821 fixes non-resolving API cross-references across docs and tutorials.
• #3822 migrates the docs to pytorch_sphinx_theme2 and fixes tutorial Colab, Notebook, and GitHub links.
• #3745 adds a memory-efficient RL training tutorial and cross-references for layout and recurrent-training guidance.

Newly exported public symbols

Utilities

• torchrl.cuda_memory_profile (code, docs) — context manager/decorator for scoped CUDA memory profiling.
• torchrl.cuda_memory_stats (code, docs) — helper for reading current and peak CUDA allocation/reservation statistics.
• torchrl.reset_cuda_peak_stats (code, docs) — helper for resetting CUDA peak memory counters.

Modules

• torchrl.modules.tensordict_module.canonicalize_rnn_subset (also re-exported as torchrl.modules.canonicalize_rnn_subset; specific export, package export, docs) — canonicalizes only the recurrent keys used by selected RNN modules.

Highlights

• More robust Triton RNN recurrent matrix multiplication for large hidden sizes and backend autotuning.
• Lower-memory recurrent learner updates through recompute backward and subset canonicalization.
• Cleaner torch.compile behavior for state-preserving module conversion.
• Updated docs theme and repaired generated API and tutorial links.
• New memory-efficient RL training tutorial.
• Refreshed dependency pins for GRPO and expert-iteration SOTA examples.

Installation

pip install torchrl==0.13.1

For CUDA wheel variants, follow the install index documented in the TorchRL README for the desired CUDA runtime.

Full changelog

https://github.com/pytorch/rl/compare/v0.13.0...v0.13.1

TorchRL v0.12.0 Release Notes

Highlights

• New algorithms. Five new config-based trainers — DQN, DDPG, IQL, CQL, and TD3 — are built on a new configuration system for reproducible algorithm setups (@vmoens, @bsprenger). PILCO (Probabilistic Inference for Learning Control) is now available as a built-in algorithm (@PSXBRosa, @vmoens). For diffusion-based behavioral cloning, a new DDPMModule diffusion actor and DiffusionBCLoss are included (@theap06). Async PPO infrastructure overlaps data collection and optimization (@vmoens).

• Collector and data-flow improvements. A new high-throughput auto-batching inference server automatically batches requests from multiple environments, with pluggable transport backends (threading, multiprocessing, Ray, Monarch) and built-in weight-sync integration. Paired with the new AsyncBatchedCollector, it enables asynchronous data collection with automatic batching for maximum GPU utilization (@vmoens). The new TrajectoryBatcher and AsyncTrajectoryBatcher assemble trajectories efficiently from streaming environment transitions, including variable-length trajectories and padding (@theap06). On the parallel environment side, shared-memory done flags replace mp.Event for lower-latency step synchronization, and a fast-path device-transfer optimization reduces overhead in step_and_maybe_reset (@vmoens).

• Inference backends. This release adds full SGLang integration alongside vLLM, with an SGLangWrapper policy module, an AsyncSGLang server-based inference path, NCCL weight synchronization, and GRPO support (@vmoens).

• Replay buffer. StoreStorage is a new Redis/Dragonfly-backed storage backend that lets replay buffers share experience across processes and nodes (@vmoens).

• Evaluation. A new Evaluator class provides a unified API for synchronous and asynchronous policy evaluation during training, with a process backend, collector-based stepping, weight sync via WeightSyncScheme, multi-model support, and a RayEvalWorker for distributed evaluation (@vmoens).

• Environments and platform support. A new GenesisEnv wrapper integrates the Genesis physics simulator (@ParamThakkar123). Dreamer now supports pre-vectorized environments and ships with an IsaacLab environment factory, training script, and integration guide (@vmoens). MPS support improves through float64-to-float32 downcasting in ParallelEnv, SerialEnv, and collectors, fixing previously broken Apple Silicon GPU workflows (@bsprenger).

Installation

pip install torchrl==0.12.0

Requires PyTorch >= 2.1 and TensorDict >= 0.12.0.

---

Breaking Changes

• Remove v0.12 deprecated APIs (#3670) @vmoens
  • The local_init_rb parameter has been removed from Collector and MultiCollector. Storage-level initialization is now the only behavior.
  • TransformedEnv(env=...) now raises TypeError. Use TransformedEnv(base_env=...) instead.

---

New Features

Auto-batching Inference Server

A new inference server that automatically batches requests from multiple environments for efficient GPU inference. This is a key building block for scaling RL training with many parallel environments.

• Core server and transport protocol (#3492)
• Threading transport (#3493)
• Multiprocessing transport (#3494)
• Ray transport (#3495)
• Monarch transport (#3496)
• Weight sync integration (#3497)

AsyncBatchedCollector

A new collector that combines async environments with the auto-batching inference server for maximum throughput.

• Async envs + auto-batching inference (#3498)
• Coordinator loop and direct submission mode (#3499)
• Backend params and performance optimizations (#3511)

Trajectory Batcher

• TrajectoryBatcher for assembling trajectories from streaming transitions (#3584) @theap06
• AsyncTrajectoryBatcher for asynchronous trajectory assembly (#3592) @theap06

SGLang Backend

Full SGLang support for LLM inference, mirroring the existing vLLM integration:

• Base infrastructure (#3428)
• AsyncSGLang server-based inference service (#3429)
• SGLangWrapper policy module (#3430)
• NCCL weight synchronization (#3431)
• Module structure integration (#3432)
• SGLang backend support in GRPO

Diffusion Policies

• DDPMModule diffusion actor for denoising diffusion probabilistic models (#3596) @theap06
• DiffusionBCLoss for diffusion-based behavioral cloning (#3604) @theap06

Evaluator

• Evaluator class for sync/async evaluation (#3594)
• Process backend, lazy init, and pending property (#3611)
• Collector-based stepping backend (#3624)
• Enable loggers to run as Ray actors (#3623)
• Weight sync via WeightSyncScheme + multi-model support (#3627)
• Isaac Lab Evaluator tests + init_fn plumbing for process backend (#3663)
• RayEvalWorker for distributed async evaluation (#3474)
• Named actors and from_name for RayEvalWorker (#3488)

Async PPO

• Async PPO infrastructure for overlapping collection and optimization (#3661)

Config-based Trainers

New trainers with integrated configuration system:

• DQN Trainer (#3526)
• DDPG Trainer (#3527)
• IQL Trainer (#3528)
• CQL Trainer (#3529)
• TD3 Trainer (#3557) @bsprenger
• Hook point to log average optimization losses in trainers (#3666)

Replay Buffer

• StoreStorage for Redis/Dragonfly-backed replay buffers (#3516)
• set_at_, set_, update_ methods on ReplayBuffer (#3590) @jashshah999
• Support trajs_per_batch with replay_buffer on multi-process and distributed collectors (#3618)

LLM / GRPO

• Token-in, token-out LLM wrapper mode (#3407)
• GRPO improvements: new envs, vLLM V1 compat, log-prob fixes, training stability (#3580)
• Namespace GRPO wandb metrics for auto-grouping (#3585)
• Remove placement-group xfails and fix vLLM tokenizer compat (#3586)

Environments

• GenesisEnv: wrapper for the Genesis physics simulator (#3536) @ParamThakkar123
• FinancialRegimeEnv: a vectorized financial environment (#3384) @aneesh223
• num_workers parameter for HabitatEnv (#3383) @ParamThakkar123
• Dreamer: support pre-vectorized environments (#3483)
• Dreamer: add IsaacLab environment factory (#3484)

Transforms

• Inverse for VecNorm and VecNormV2 transforms (#3416) @ParamThakkar123
• prevent_leaking_rng utility (#3401) @ParamThakkar123

Logging

• log_metrics method for efficient batch logging (#3452)
• TensorDict support in log_metrics (#3455)

Specs

• index_select support for TensorSpec (#3406) @ParamThakkar123
• strict_shape parameter for QValueModule action shape enforcement (#3593) @Lidang-Jiang

Algorithms

• PILCO (Probabilistic Inference for Learning Control) (#3582) @PSXBRosa, @vmoens

Collectors

• Lazy-init RandomPolicy action_spec from env in collectors (#3664)

Other

• __getattr__ in _dispatch_caller_parallel for transparent attribute access (#3389) @ParamThakkar123
• scalar_output_mode for loss modules with reduction='none' (#3426)
• ObsDecoder: out_channels parameter for grayscale decoding (#3472)
• Ergonomic scalar assignment for loss buffers (#3612)
• New memmap value for the CKPT_BACKEND environment variable (#3619) @theap06

---

Performance Improvements

• GPU Image Transforms for Dreamer (~5.5x faster sampling)
• SliceSampler: GPU-accelerated trajectory computation
• Always enable prefetch for replay buffer
• ParallelEnv: fast-path device transfer in step_and_maybe_reset
• ParallelEnv: replace mp.Event with shared-memory done flags for lower latency
• Lazy stack optimization for collector-to-buffer writes (#3438)
• log_metrics usage in sota-implementations (#3454)

---

Bug Fixes

MPS (Apple Silicon)

• Downcast float64 to float32 in ParallelEnv/SerialEnv on MPS (#3551) @bsprenger
• MPS float64->float32 downcast for tensors (#3548) @bsprenger
• Fix masked_scatter shape preservation on MPS in collectors (#3473)

Collectors

• Fix stale model reference in MultiCollector weight sync after device-cast (#3587)
• Fix shared mem updater with many policies (#3442)
• Fix missing raise, incorrect __torch_function__ return, and off-by-one in RayCollector (#3530) @jashshah999

Environments

• Fix check_env_specs() when state_spec contains keys not in observation_spec (#3581) @theap06
• Fix BraxEnv rejecting camera_id and render_kwargs (#3533)
• StepCounter now tracks nested truncated and done states (#3405)
• Fix ParallelEnv shutdown hang with shared-memory done flags (#3464)

Loss / Models

• Fix broken SACLoss when there is more than one qvalue_network (#3500) @ParamThakkar123
• Fix GPT2RewardModel.compute_reward_loss (#3521)
• Fix resnet order call in ImpalaNet (#3522)
• Fix vLLM CompilationConfig compatibility and Windows CI pybind11 (#3673)

Specs / TensorDict

• Fix MultiOneHot.to_numpy() returning scalar instead of array (#3589) @jashshah999
• Fix shape mismatch in _set_index_in_td with trailing dims of 1 (#3517)
• Set batch size in Composite.encode (#3411) @tobiabir

Transforms

• LineariseRewards should not squeeze trailing dim (#3614) @mathieuorhan
• Allow gradient flow through R3MTransform in training mode (#3607) @theap06

Other

• Fix DataLoadingPrimer batch_size detection for NonTensorStack (#3532)
• Fix compiled storage access (#3547)
• Fix CUDA graph capture for Bounded spec projection (#3453)
• Fix VideoRecorder support for grayscale (1-channel) observations (#3471)
• Fix functools.partial warnings (#3465) @ParamThakkar123
• Fix none handling in pendulum.py tutorial (#3595) @theap06
• Fix StepCounter._reset should not use output_spec (#3626)
• Fix per-group WandB step logging (#3625)

---

Refactors

• Refactor optimization API for multi-phase optimization (#3468) @bsprenger
• Upgrade to torchcodec for video export (#3540)
• Refactor NoisyLinear (#3082)
• Upgrade meshgrid usage to address deprecation warning (#3412)

---

Documentation

• Use new canonical collector names across docs, tutorials, and SOTA (#3665)
• Tutorial on collector trajectory assembly internals (#3600) @coder-jayp
• IsaacLab integration guide and setup script (#3486)
• RayEvalWorker API reference docs (#3487)
• SGLang backend documentation
• TransformersWrapper/ChatEnv integration documentation (#3377)
• EGL multi-GPU limitations in containers (#3456)

---

New Contributors

Welcome to the following first-time contributors!

• @aneesh223 - FinancialRegimeEnv
• @coder-jayp - Collector trajectory assembly tutorial
• @jashshah999 - RayCollector fixes, ReplayBuffer API additions, MultiOneHot fix
• @Lidang-Jiang - QValueModule strict_shape parameter
• @mathieuorhan - LineariseRewards fix
• @theap06 - Diffusion policies, trajectory batcher, R3MTransform fix, CKPT_BACKEND memmap
• @thecaptain789 - Typo fix
• @tobiabir - Composite.encode batch size fix

---

Full Changelog

https://github.com/pytorch/rl/compare/v0.11.0...v0.12.0

Highlights

This patch release includes several important bug fixes and performance improvements:

• Fixed Composite.encode() to correctly set the batch size of the output TensorDict
• Fixed StepCounter to properly track nested truncated and done states in multi-agent environments
• Fixed shared memory weight updater to work correctly with collectors using multiple policies
• Fixed _repr_html_ dispatch in parallel environments that was causing doc CI failures
• Added scalar_output_mode to loss modules for proper handling of reduction='none'
• Fixed torch.compile configuration for Dreamer
• Performance: GPU Image Transforms for Dreamer (~5.5x faster sampling)
• Performance: SliceSampler GPU acceleration for faster trajectory computation
• Performance: Always enable prefetch for replay buffer

Breaking Changes

No breaking changes in this release.

Bug Fixes

• Fixed batch size in Composite.encode: The Composite.encode() method now correctly sets the batch_size of the output TensorDict to match the shape of the tensor spec, rather than returning an empty batch size. (#3411) - @tobiabir

  Previously, calling Composite.encode(raw_vals) would return a TensorDict with batch_size=torch.Size([]) regardless of the spec's shape. This is now fixed to return the correct batch size matching the spec shape.

• Fixed StepCounter nested done/truncated tracking in multi-agent environments: StepCounter now properly updates nested truncated and done keys for multi-agent environments. (#3405) - @vmoens

  When using StepCounter with multi-agent environments (e.g., PettingZoo), the transform now correctly propagates truncated/done signals to agent-specific keys (e.g., ("agent", "truncated")) in addition to the root-level keys.

• Fixed shared memory weight updater with multiple policies: The shared memory weight updater now correctly handles collectors that use multiple policies. (#3442) - @vmoens

• Fixed _repr_html_ dispatch in parallel environments: Parallel environments no longer incorrectly dispatch private/special attribute access (like _repr_html_) to worker processes. (#3441) - @vmoens

• Added scalar_output_mode to loss modules: Loss modules (SAC, IQL, CQL, CrossQ, REDQ, DecisionTransformer) now support scalar_output_mode parameter for proper handling of reduction='none'. (#3426) - @vmoens

• Fixed torch.compile configuration for Dreamer: Fixed compilation settings for Dreamer world model training. - @vmoens

Performance Improvements

• GPU Image Transforms for Dreamer: ~5.5x faster sampling with GPU-accelerated image transforms. - @vmoens
• SliceSampler GPU acceleration: Faster trajectory computation using GPU. - @vmoens
• Always enable prefetch for replay buffer: Improved data loading performance. - @vmoens

Cleanup

• Removed pin_memory from replay buffer: Simplified replay buffer configuration. - @vmoens

Internal / CI Improvements

• Added PyTorch version check instructions to release prompt (#3443) - @vmoens
• Added tutorials CI workflow for testing sphinx tutorials (#3441) - @vmoens
• Upgraded meshgrid usage to address PyTorch deprecation warning (#3412) - @vmoens
• Added flaky test tracking system for improved CI reliability (#3408) - @vmoens
• Added file-based auto-labeling for PR components (#3402) - @vmoens
• Improved LLM prompt for release workflow (#3399) - @vmoens

Contributors

Thanks to all contributors to this release:

• @tobiabir (Tobias Birchler) - First-time contributor!
• @vmoens (Vincent Moens)

Installation

```bash pip install torchrl==0.11.1 ```

Or with conda:

```bash conda install -c pytorch torchrl=0.11.1 ```

TorchRL v0.11.0 Release Notes

Highlights

• Dreamer overhaul - Comprehensive improvements to Dreamer world model training: async collectors with profiling, RSSM fixes (scan mode, noise injection, explicit dimensions), torch.compile compatibility for value functions and TDLambda estimator, optimized DreamerEnv to avoid CUDA syncs, and updated sota-implementation with better configs. @vmoens
• Weight synchronization schemes - New modular weight sync infrastructure (torchrl.weight_update) with SharedMem, MultiProcess, and vLLM-specific (NCCL, double-buffer) schemes. Collectors now integrate seamlessly with weight sync schemes for distributed training. @vmoens
• Major collector refactor - The collector codebase has been completely restructured. The monolithic collectors.py is now split into focused modules (_single.py, _multi_base.py, _multi_sync.py, _multi_async.py, _runner.py, base.py), with cleaner separation of concerns. (#3233) @vmoens
• LLM objectives: DAPO & CISPO - New DAPO (Direct Advantage Policy Optimization) and CISPO (Clipped Importance Sampling Policy Optimization) algorithms for LLM training. @vmoens
• Trainer infrastructure - New SAC Trainer, configuration system for algorithms, timing utilities, and async collection support within trainers. @vmoens
• Tool services - New tool service infrastructure for LLM agents with Python executor, MCP tools, and web search capabilities. @vmoens
• Deprecated APIs removed - All deprecation warnings from v0.10 have been promoted to hard errors for v0.11. (#3369) @vmoens
• New environment backends - Added Procgen environments support with a new ProcgenEnv wrapper. (#3331) @ParamThakkar123
• Multi-env execution - GymEnv, BraxEnv, and DMControlEnv now support a num_envs/num_workers parameter to run multiple environments in a single call via ParallelEnv. (#3343, #3370, #3337) @ParamThakkar123

Installation

pip install torchrl==0.11.0

---

Breaking Changes

• [v0.11] Remove deprecated features and replace warnings with errors (#3369) @vmoens

  • Removes deprecated KLRewardTransform from transforms/llm.py (use torchrl.envs.llm.KLRewardTransform)
  • Removes LogReward and Recorder classes from trainers (use LogScalar and LogValidationReward)
  • Removes unbatched_*_spec properties from VmasWrapper/VmasEnv (use full_*_spec_unbatched)
  • Deletes deprecated rlhf.py modules (data/rlhf.py, envs/transforms/rlhf.py, modules/models/rlhf.py)
  • Removes replay_buffer_chunk parameter from MultiCollector
  • Replaces minimum/maximum deprecation warnings with TypeError in Bounded spec
  • Replaces critic_coef/entropy_coef deprecation warnings with TypeError in PPO and A2C losses

• [Major] Major refactoring of collectors (#3233) @vmoens

  • Splits the 5000+ line collectors.py into focused modules for single/multi sync/async collectors
  • Creates new _constants.py, _runner.py, base.py modules
  • Introduces cleaner weight synchronization scheme integration
  • Improves test coverage for multi-device and shared-device weight updates
  • Some internal APIs have changed; external API remains compatible

---

Dreamer World Model Improvements

These changes significantly improve Dreamer training performance, torch.compile compatibility, and usability. @vmoens

• [Feature] Refactor Dreamer training with async collectors, profiling, and improved config (cc917bae)

  • Major overhaul of the Dreamer sota-implementation with async data collection
  • Adds profiling support for performance analysis
  • Improved configuration with better defaults and documentation
  • Updated README with detailed usage instructions

• [Refactor] Dreamer implementation updates (3ab4b306)

  • Refactors Dreamer training script for better maintainability
  • Updates config.yaml with improved hyperparameters
  • Enhances dreamer_utils.py with additional helper functions

• [Feature] Add noise argument and scan mode to RSSMRollout (8653b6e6)

  • Adds noise argument to control stochastic sampling during rollout
  • Implements scan mode for efficient sequential processing
  • 109 lines added to model_based.py for improved RSSM flexibility

• [Feature] Add explicit dimensions and device support to RSSM modules (f350fe06)

  • Adds explicit dimension handling for batch, time, and feature dims
  • Improves device placement for RSSM components

• [Feature] Logging & RSSM fixes (d0829794)

  • Fixes RSSM module behavior and adds logging improvements
  • Updates batched_envs.py and wandb logger

• [Refactor] Use compile-aware helpers in Dreamer objectives (d8c3887d)

  • Updates Dreamer objectives to use torch.compile-compatible helpers
  • Improves performance when using torch.compile

• [BugFix] Optimize DreamerEnv to avoid CUDA sync in done checks (d57fdecf)

  • Eliminates unnecessary CUDA synchronizations in done flag checking
  • Significant performance improvement for GPU-based Dreamer training

• [BugFix] Fix ModelBasedEnvBase for torch.compile compatibility (7ff663d6)

  • Makes ModelBasedEnvBase compatible with torch.compile

• [Feature] Add allow_done_after_reset parameter to ModelBasedEnvBase (24ae0424)

  • Adds flexibility for environments that may signal done immediately after reset

• [BugFix] Final polish for Dreamer utils and Collector tests (5aefdd5c)

  • Final cleanup and polish for Dreamer implementation

---

Weight Synchronization Schemes

New modular infrastructure for weight synchronization between training and inference workers. @vmoens

• [Feature] Weight Synchronization Schemes - Core Infrastructure (e8f6fa5a)

  • New torchrl.weight_update module with 2400+ lines of weight sync infrastructure
  • SharedMemWeightSyncScheme: Uses shared memory for fast intra-node sync
  • MultiProcessWeightSyncScheme: Uses multiprocessing queues for cross-process sync
  • Comprehensive documentation and examples in docs/source/reference/collectors.rst

• [Feature] vLLM Weight Synchronization Schemes (d0c8b7e9)

  • VllmNCCLWeightSyncScheme: NCCL-based weight sync for vLLM distributed inference
  • VllmDoubleBufferWeightSyncScheme: Double-buffered async weight updates
  • 1876 lines of vLLM-specific weight sync code

• [Feature] Collectors - Weight Sync Scheme Integration (a7707ca0)

  • Integrates weight sync schemes into collector infrastructure
  • Updates GRPO and expert-iteration implementations to use new schemes
  • Adds examples for multi-weight update patterns

• [Refactor] Weight sync schemes refactor (ae0ae064)

  • Refines weight sync API and adds additional schemes
  • Improves test coverage with 342+ new test lines

---

LLM Training: DAPO & CISPO

New policy optimization algorithms for LLM fine-tuning. @vmoens

• [Feature] DAPO (9d5c2767)

  • Implements Direct Advantage Policy Optimization for LLM training
  • Adds DAPO-specific loss computation to torchrl/objectives/llm/grpo.py

• [Feature] CISPO (ed0d8dc6)

  • Implements Clipped Importance Sampling Policy Optimization
  • Alternative to PPO/GRPO with different clipping strategy

• [Refactor] Refactor GRPO as a separate class (2bc3cb70)

  • Separates GRPO implementation for better modularity

---

Trainer Infrastructure

New trainer algorithms, configuration system, and utilities. @vmoens

• [Trainers] SAC Trainer and algorithms (02d4bfd1)

  • New SAC Trainer implementation with 786 lines of code
  • Complete sota-implementation in sota-implementations/sac_trainer/
  • Trainer configuration via YAML files

• [Feature] Trainer Algorithms - Configuration System (6bc201a5)

  • New configuration system in torchrl/trainers/algorithms/configs/
  • Configs for collectors, data, modules, objectives, transforms, weight sync schemes
  • Enables hydra-style configuration composition

• [Feature] Trainer Infrastructure - Timing and Utilities (dc215236)

  • Adds timing utilities to trainer infrastructure
  • 263 lines of enhanced trainer functionality

• [Feature] Async collection within trainers (5f1eb2c5)

  • Enables asynchronous data collection during training
  • Improves training throughput

• [Feature] PPO Trainer Updates (129f3d50)

  • Updates to PPO trainer with new features

---

Tool Services for LLM Agents

New infrastructure for tool-augmented LLM agents. @vmoens

• [Feature] Tool services (9ca0e407)

  • New torchrl/services/ module for tool execution
  • Python executor service for safe code execution
  • MCP (Model Context Protocol) tool integration
  • Web search tool example
  • 609 lines of documentation in docs/source/reference/services.rst
  • Comprehensive test coverage in test/test_services.py

• [Feature] Transform Module - ModuleTransform and Ray Service Refactor (7b85c716)

  • New ModuleTransform for applying nn.Modules as transforms
  • Refactored Ray service integration
  • Moves ray_service.py to torchrl/envs/transforms/

---

torch.compile Compatibility

Fixes to enable torch.compile with various TorchRL components. @vmoens

• [BugFix] Fix value functions for torch.compile compatibility (3bdc7b18)

  • 295 lines of new tests in test/compile/test_value.py
  • Fixes value function implementations for compile compatibility

• [BugFix] Fix TDLambdaEstimator for torch.compile compatibility (11e22ee9)

  • Updates TDLambdaEstimator to work with torch.compile

• [Feature] Add compile-aware timing and profiling helpers (764d9f71)

  • New utilities for profiling compiled code

• [Refactor] Add compile-aware timing and profiling helpers (c82447e6)

  • Performance utilities that work correctly under torch.compile

---

Features

• [Feature] Added EXP3 Scoring function (#3013) @ParamThakkar123

  • Implements the EXP3 (Exponential-weight algorithm for Exploration and Exploitation) scoring function for MCTS
  • Adds torchrl/data/map/ module with hash-based storage, query utilities, and tree structures for efficient state lookups

• [Feature] Add num_workers parameter to BraxEnv (#3370) @ParamThakkar123

  • Allows running multiple Brax environments in parallel via ParallelEnv by specifying num_workers > 1
  • Consistent API with GymEnv and DMControlEnv multi-env support

• [Feature] Added num_envs parameter in GymEnv (#3343) @ParamThakkar123

  • When num_envs > 1, GymEnv automatically returns a lazy ParallelEnv wrapping multiple environment instances
  • Simplifies multi-env setup without manually constructing ParallelEnv

• [Environments] Added Procgen environments (#3331) @ParamThakkar123

  • New ProcgenWrapper and ProcgenEnv classes to wrap OpenAI Procgen environments
  • Converts Procgen observations to TorchRL-compliant TensorDict outputs
  • Supports all 16 Procgen game environments (coinrun, starpilot, etc.)

• [Feature] Loss make_value_estimator takes a ValueEstimatorBase class (#3336) @ParamThakkar123

  • Allows passing a ValueEstimatorBase class directly to make_value_estimator() for more flexible value estimation configuration
  • Adds tests and documentation for the new API

• [Feature] Make custom_range public in ActionDiscretizer (#3333) @vmoens

  • Exposes custom_range parameter in ActionDiscretizer for user-defined discretization ranges

• [Feature] Added num_envs parameter in DMControlEnv (#3337) @ParamThakkar123

  • Similar to GymEnv, allows running multiple DMControl environments via num_workers parameter
  • Returns a lazy ParallelEnv when num_workers > 1

• [Feature] Auto-configure exploration module specs from environment (#3317) @bsprenger

  • Collectors now automatically configure exploration module specs (like AdditiveGaussianModule) from the environment's action spec
  • Adds set_exploration_modules_spec_from_env() utility function
  • Reduces boilerplate when using exploration modules with delayed spec initialization

• [Feature] Add NPU Support for Single Agent (#3229) @lowdy1

  • Adds Huawei NPU (Ascend) device support for single-agent training
  • Updates device handling logic to recognize NPU devices

• [Feature] Add support for trackio (#3196) @Xmaster6y

  • Integrates with trackio for experiment tracking and logging
  • New TrackioLogger class for seamless integration

• [Feature] Add a new Trainer hook point process_loss (#3259) @Xmaster6y

  • Adds a new hook point in the Trainer that runs after loss computation but before backward pass
  • Useful for loss modification, logging, or gradient accumulation strategies

• [Feature] Ensure MultiSyncDataCollectors returns data ordered by worker id (#3243) @LCarmi

  • Data from MultiSyncDataCollectors is now consistently ordered by worker ID
  • Makes debugging and analysis easier when working with multi-worker collectors

• [Feature] Add TanhModuleConfig (#3255) @bsprenger

  • Adds configuration class for TanhModule to support serialization and hydra-style configs

• [Feature] add TensorDictSequentialConfig (#3248) @bsprenger

  • Adds configuration class for TensorDictSequential modules

• [Feature] Weight loss outputs when using prioritized sampler (#3235) @vmoens

  • Loss modules now properly weight outputs when using prioritized replay buffers
  • Adds importance_key parameter to loss functions (DQN, DDPG, SAC, TD3, TD3+BC)
  • When importance weights are present in the sampled data, losses are automatically weighted
  • Adds comprehensive tests for prioritized replay buffer integration with all loss functions

• [Feature] Composite specs can create named tensors with 'zero' and 'rand' (#3214) @louisfaury

  • Composite specs now propagate dimension names when creating tensors via zero() and rand()

• [Feature] Enable storing rollouts on a different device (#3199) @Xmaster6y

  • Collectors can now store rollouts on a different device than the execution device
  • Useful for CPU storage while running on GPU

• [Feature] Named dims in Composite (#3174) @vmoens

  • Adds named dimension support to Composite specs via names property and refine_names() method
  • Enables better integration with PyTorch's named tensors
  • Supports ellipsis notation for partial name specification

---

Bug Fixes

• [BugFix] Fix TransformersWrapper ChatHistory.full not being set (#3375) @vmoens

• [BugFix] Fix AsyncEnvPool + LLMCollector with yield_completed_trajectories (#3373) @vmoens

  • Fixes interaction between AsyncEnvPool and LLMCollector when yielding completed trajectories

• [BugFix] Fix LLM test failures (#3360) @vmoens

  • Comprehensive fixes for LLM-related test failures across multiple modules

• [BugFix] Fixed MultiSyncCollector set_seed and split_trajs issue (#3352) @ParamThakkar123

  • Fixes seed propagation and trajectory splitting in MultiSyncDataCollector

• [BugFix] Fix VecNormV2 device gathering (#3368) @vmoens

  • Fixes device handling when gathering statistics in VecNormV2 transform

• [BugFix] Fix AsyncEnv - LLMCollector integration (#3365) @vmoens

  • Resolves integration issues between async environments and LLM collectors

• [BugFix] Fix VecNormV2 GPU device handling for stateful mode (#3364) @BY571

  • Fixes GPU device handling in VecNormV2 when using stateful normalization mode

• [BugFix] Fix Ray collector iterator bug and process group cleanup (#3363) @vmoens

  • Fixes iterator behavior in Ray-based collectors and ensures proper cleanup of process groups

• [BugFix] Fix LLM CI by replacing uvx with uv package (#3356) @vmoens

• [Bugfix] Wrong minari download first element (#3106) @marcosgalleterobbva

  • Fixes incorrect first element handling when downloading Minari datasets

• [BugFix,Test] Fix envpool failing tests (#3345) @vmoens

• [BugFix] Fixed ParallelEnv + aSyncDataCollector / MultiSyncDataCollector not working if replay_buffer is given (#3341) @ParamThakkar123

  • Fixes compatibility issue when using ParallelEnv with collectors that have a replay buffer attached

• [BugFix] treat 1-D MultiDiscrete as MultiCategorical and accept flattened masks (#3342) @ParamThakkar123

  • Improves MultiDiscrete action space handling to accept flattened action masks

• [BugFix] Fixes register_save_hook bug (#3340) @ParamThakkar123

  • Fixes save hook registration in replay buffers

• [BugFix,Test] recompiles with string failure (#3338) @vmoens

• [BugFix] Fix Safe modules annotation and doc for losses (#3334) @vmoens

• [BugFix] Fix ray modules circular import (#3319) @vmoens

• [BugFix] Fix SACLoss target_entropy="auto" ignoring action space dimensionality (#3292) @vmoens

  • Fixes automatic entropy target computation to properly account for action space dimensions

• [BugFix] Fix agent_dim in multiagent nets & account for neg dims (#3290) @vmoens

  • Fixes agent dimension handling in multi-agent networks, including support for negative dimension indices

• [BugFix] Added a missing .to(device) call in _from_transformers_generate_history (#3289) @michalgregor

• [BugFix] Fix old pytorch dependencies (#3266) @vmoens

  • Updates compatibility shims for older PyTorch versions

• [BugFix] RSSMRollout not advancing state/belief across time steps in Dreamer (#3236) @cmdout

  • Fixes critical bug where RSSM rollout was not properly advancing hidden states in Dreamer world model

• [Doc,BugFix] Fix doc and collectors (#3250) @vmoens

• [BugFix] use correct field names in InitTrackerConfig (#3245) @bsprenger

• [BugFix] Use torch.zeros for argument in torch.where (#3239) @sebimarkgraf

• [BugFix] Fix wrong assertion about collector and buffer (#3176) @vmoens

• [BugFix] AttributeError in accept_remote_rref_udf_invocation (#3168) @vmoens

Bug Fixes (ghstack commits without PR numbers)

• [BugFix] Final polish for Collector and Exploration (558396d7) @vmoens

  • Final fixes for collector and exploration module interactions

• [BugFix] Collector robustness & Async fixes (cadf23bc) @vmoens

  • Improves collector robustness and fixes async collection issues

• [BugFix] TensorStorage key filtering (0667a580) @vmoens

  • Fixes key filtering in TensorStorage

• [BugFix] Replay Buffer prefetch & SliceSampler (9d34dbe7) @vmoens

  • Fixes prefetching behavior and SliceSampler issues

• [BugFix] Fix target_entropy computation for composite action specs (06960c64) @vmoens

  • Correctly computes target entropy for composite action spaces

• [BugFix] Fix device initialization in CrossQLoss.maybe_init_target_entropy (416e4545) @vmoens

  • Fixes device placement for CrossQ loss entropy initialization

• [BugFix] Fix schemes and refactor collectors to make them readable (364e0386) @vmoens

  • Fixes weight sync schemes and improves collector code readability

• [BugFix] Fix collector devices (888095fd) @vmoens

  • Fixes device handling in collectors

• [BugFix] Fix tests (963fdd43) @vmoens

• [BugFix] Fix GRPO tests and runs (47ad9d8f) @vmoens

• [BugFix] Handle Lock/RLock types in EnvCreator (6985ca25) @vmoens

  • Properly handles threading locks in EnvCreator serialization

• [BugFix] Defer filter_warnings import to avoid module load issues (8ea954cd) @vmoens

• [BugFix] Fix CUDA sync in forked subprocess (4a2a2749) @vmoens

  • Fixes CUDA synchronization issues in forked processes

• [BugFix] Fix unique ref to lambda func (b6fe45ee) @vmoens

• [BugFix] Add pybind11 check and Windows extension pattern fix (5e89e4e7) @vmoens

---

Additional Features (ghstack commits without PR numbers)

• [Feature] Storage Shared Initialization for Multiprocessing (61c178e5) @vmoens

  • Enables shared storage initialization for multiprocessing collectors
  • Updates distributed collector examples

• [Feature] Memmap storage cleanup (7f9ea748) @vmoens

  • Improves memmap storage cleanup and resource management

• [Feature] Collector Profiling (02ed47ed) @vmoens

  • Adds profiling capabilities to collectors for performance analysis

• [Feature] auto_wrap_envs in PEnv (d781f9e9) @vmoens

  • Automatic environment wrapping in ParallelEnv

• [Feature] Auto-wrap lambda functions with EnvCreator for spawn compatibility (d250c182) @vmoens

  • Automatically wraps lambda functions with EnvCreator for spawn multiprocessing compatibility

• [Feature] Collectors' getattr_policy and getattr_env (fbdbb617) @vmoens

  • Adds attribute access methods for policy and env in collectors

• [Feature] track_policy_version in collectors.py (a089cc48) @vmoens

  • Adds policy version tracking in collectors for distributed training

• [Feature] Aggregation strategies (0e6a3565) @vmoens

  • New aggregation strategies for multi-worker data collection

• [Feature] kl_mask_threshold (7ab48a4c) @vmoens

  • Adds KL divergence masking threshold for LLM training

• [Feature] Add timing options (13434ebb) @vmoens

  • Additional timing options for performance monitoring

• [Feature] float32 patch (01d2801b) @vmoens

  • Float32 precision handling improvements

• [Feature] Support callable scale in IndependentNormal and TanhNormal distributions (d3aba7d1) @vmoens

  • Allows scale parameter to be a callable in distribution modules

---

Documentation

• [Doc] Huge doc refactoring (3d5dd1ac) @vmoens

  • Major documentation restructuring with new sections:
    • collectors_basics.rst, collectors_single.rst, collectors_distributed.rst
    • collectors_weightsync.rst, collectors_replay.rst
    • data_datasets.rst, data_replaybuffers.rst, data_samplers.rst
  • Adds pre-commit hook for Sphinx section underline checking

• [Docs] Update LLM_TEST_ISSUES.md with fix status (#3374) @vmoens

• [Docs] Enable doc builds and tutorial runs (#3335) @vmoens

  • Re-enables Sphinx-gallery tutorial execution in documentation builds
  • Adds process cleanup between tutorials to prevent resource leaks

---

CI / Infrastructure

• [CI] Speed up slow tests in tests-gpu/tests-cpu (#3395) @vmoens

  • Optimizes slow test execution with better parallelization and test isolation

• [CI] Bump version to 0.11.0 (#3392) @vmoens

• [CI] Fix auto-label workflow for fork PRs (#3388) @vmoens

• [CI] Better release workflow (#3386) @vmoens

  • Comprehensive release workflow with sanity checks, wheel collection, docs updates, and PyPI publishing
  • Adds dry-run mode for testing releases
  • Automatically updates stable docs symlink and versions.html

• [CI] Fix auto-labelling (#3387) @vmoens

• [CI] Auto-tag PRs (#3381) @vmoens

  • Adds automatic labeling for PRs based on changed files

• [CI] Add release agent prompt for LLM-assisted releases (#3380) @vmoens

  • Adds comprehensive release instructions for LLM-assisted release automation

• [CI] Add release workflow with PyPI trusted publishing (#3379) @vmoens

  • Implements PyPI trusted publishing via OIDC authentication
  • Eliminates need for PyPI API tokens in secrets

• [CI] Add granular label support for environment and data workflow jobs (#3371) @vmoens

• [CI] Fix PettingZoo CI by updating Python 3.9 to 3.10 (#3362) @vmoens

• [CI] Fix GenDGRL CI by adding missing requests dependency (#3361) @vmoens

• [CI] Fix IsaacLab tests by using explicit conda Python path (#3358) @vmoens

• [CI] Fix M1 build pip command not found (#3359) @vmoens

• [CI] Fix Windows build for free-threaded Python (3.13t, 3.14t) (#3357) @vmoens

• [CI,BugFix] Fix Habitat CI by upgrading to Python 3.10 (#3346) @vmoens

  • Upgrades Habitat CI to Python 3.10 and builds habitat-sim from source

• [CI] Fix Jumanji CI by adding missing requests dependency (#3349) @vmoens

• [CI] Fix Chess CI by correcting test file path typo (#3353) @vmoens

• [CI] Skip Windows-incompatible tests in optional deps CI (#3348) @vmoens

• [CI] Fix GPU benchmark failures (#3347) @vmoens

• [CI] Upgrade doc python version (#3222) @vmoens

• [CI] Use pip install (#3200) @vmoens

  • Migrates CI workflows from setup.py install to pip install

CI / Infrastructure (ghstack commits without PR numbers)

• [Setup] Python 3.14 in, python 3.9 out (ff86ab78) @vmoens

  • Adds Python 3.14 support and removes Python 3.9

• [CI] LLM tests integration (cccfaa69) @vmoens

  • Integrates LLM tests into CI pipeline

• [CI] Fix benchmarks for LLMs (e7ec9c31) @vmoens

• [CI] Test setup (2644b899) @vmoens

• [CI] Isolate distributed tests (5c5992d9) @vmoens

• [CI] Fix uv install with --no-deps (546a1b73) @vmoens

• [CI] Fix SOTA runs (57000fcc) @vmoens

• [CI] Fix libs (071d079f) @vmoens

• [CI] Fix missing librhash0 in doc CI (8d0f0613) @vmoens

• [CI] Add --upgrade flag for torch installs and install ffmpeg/xvfb (ce9931943) @vmoens

• [CI] Quiet coverage combine (1ac0fd0a) @vmoens

• [CI] Fix tensordict install in lib tests (8f487d40) @vmoens

---

Tests

• [Tests] Check traj_ids shape with unbatched envs (#3393) @vmoens

• [Tests] Fix test isolation in test_set_gym_environments and related tests (#3382) @vmoens

  • Improves test isolation to prevent cross-test contamination

• [Test] Use mock Llama tokenizer instead of skipping gated model test (#3376) @vmoens

• [Test] Skip Llama test when tokenizer unavailable instead of xfail (#3372) @vmoens

• [Tests] Remove _utils_internal.py in tests (#3281) @vmoens

  • Removes deprecated internal test utilities and updates tests to use public APIs

• [Test] Check LinearizeReward obs transform (#3241) @vmoens

• [Test,Benchmark] Move mocking classes file and bench for non-tensor env (#3257) @vmoens

• [Tests] Fix vmas seeding test (#3210) @matteobettini

• [Tests] Reintroduce VMAS football (#3178) @matteobettini

Tests (ghstack commits without PR numbers)

• [Test] Use spawn in older pytorch (9b0492906) @vmoens

• [Test] Fix test_num_threads that instantiate the env in the main process (852dd61b) @vmoens

• [Test] Use class references instead of lambdas in transform tests (7dd1e61d) @vmoens

• [Test] Add test for lambda wrapping in ParallelEnv (4b2b2279) @vmoens

• [Test] Ensure collector shutdown with try/finally (aa2b0311) @vmoens

• [Test] Add filterwarnings for unclosed resources (b493ab34) @vmoens

• [Test] Add retry decorator to flaky vecnorm test (6d908b69) @vmoens

• [Test] Simplify num_threads test to check threads in env factory (295cc20e) @vmoens

• [Test] Ignore script_method deprecation warning (eaaf97a4) @vmoens

• [Test] Fix failing 3.14 tests (66a48f3c) @vmoens

• [Test] Remove the forked decorator of the distributed checks (4db59660) @vmoens

• [Test] Test RB+Isaac+Ray (a0d650f9) @vmoens

• [Test] Fix flaky parallel test (4f013a81) @vmoens

• [Test] Fix vc1 (bd93f136) @vmoens

• [Quality] Fix flaky test (03ffc820) @vmoens

---

Refactors / Maintenance

• [Refactor] Updated num_envs parameter to num_workers for consistency (#3354) @ParamThakkar123

  • Renames num_envs to num_workers for consistency across environment wrappers

• [Quality] replace reduce to reduction, better error message for invalid mask (#3179) @Kayzwer

Refactors (ghstack commits without PR numbers)

• [Refactor] Move WEIGHT_SYNC_TIMEOUT to collectors._constants (3ceb6b9a) @vmoens

  • Centralizes weight sync timeout configuration

• *[Refactor] Remove TensorSpec classes (7ea8d861) @vmoens

  • Removes legacy TensorSpec class aliases

• [Refactor] Rename collectors (8779f2a3) @vmoens

  • Renames collector classes for consistency

• [Refactor] Non-daemonic processes in PEnv (aeb2e9b2) @vmoens

  • Changes ParallelEnv to use non-daemonic processes

• [Refactor] Make env creator optional for Ray (b599d9b8) @vmoens

  • Makes environment creator optional in Ray collectors

• [Refactor] Move decorate_thread_sub_func to torchrl.testing.mp_helpers (e3e9e6a7) @vmoens

  • Moves multiprocessing test helpers to proper location

• [Refactor] Use non_blocking transfers in distribution modules (5c75777f) @vmoens

  • Uses non-blocking GPU transfers in distribution modules for better performance

• [Refactor] Remove MUJOCO_EGL_DEVICE_ID auto-setting from torchrl init (8fac4c54) @vmoens

• [Refactor] Add WEIGHT_SYNC_TIMEOUT constant for collector weight synchronization (ab3768aa) @vmoens

• [BugFix,Test,Refactor] Refactor tests (eb8a885b) @vmoens

• [Refactor,Test] Move compile test to dedicated folder (253b8ddd) @vmoens

---

Dependencies

• [Dependencies] Bump ray from 2.46.0 to 2.52.1 (#3258) @dependabot

• [Environment] Fix envpool wrapper (#3339) @vmoens

  • Updates envpool wrapper for compatibility with latest envpool versions

---

Typing

• [Typing] Edit wrongfully set str type annotations (e7583b3b) @vmoens
  • Fixes incorrect string type annotations across the codebase

---

Other

• Revert "replace reduce to reduction, better error message for invalid mask" (#3182) @vmoens

• Fix (#3180) @matteobettini

• [Versioning] Fix doc versioning (#3175) @vmoens

---

Contributors

Thanks to all contributors to this release:

• @vmoens (Vincent Moens)
• @ParamThakkar123 (Param Thakkar)
• @bsprenger (Ben Sprenger)
• @Xmaster6y (Yoann Poupart)
• @BY571
• @LCarmi (Luca Carminati)
• @louisfaury (Faury Louis)
• @matteobettini (Matteo Bettini)
• @lowdy1
• @michalgregor (Michal Gregor)
• @cmdout
• @sebimarkgraf (Sebastian Moßburger)
• @Kayzwer
• @marcosgalleterobbva (Marcos Galletero Romero)
• @dependabot

Release Notes - v0.10.1

This patch release includes bug fixes, type annotation improvements, and CI enhancements cherry-picked from main.

Bug Fixes

• #3168 - @vmoens - [BugFix] AttributeError in accept_remote_rref_udf_invocation
  • Fixed AttributeError in RPC utilities when decorating classes with remote RRef invocation by handling None values in getattr calls

Features

• #3174 - @vmoens - [Feature] Named dims in Composite

  • Added support for named dimensions in Composite specs, enabling better integration with PyTorch's named tensors

• #3214 - @louisfaury - [Feature] Composite specs can create named tensors with 'zero' and 'rand'

  • Extended Composite specs to properly propagate names when creating tensors using zero() and rand() methods

Type Annotations & Documentation

• @vmoens - [Typing] Edit wrongfully set str type annotations

  • Fixed incorrect string type annotations across 19 files

• #3175 - @vmoens - [Versioning] Fix doc versioning

  • Fixed documentation versioning issues

CI/Build Improvements

• #3200 - @vmoens - [CI] Use pip install

  • Updated CI workflows to use pip install across 41 files

• @vmoens - [CI] Fix missing librhash0 in doc CI

  • Added missing librhash0 dependency in documentation CI

• @vmoens - [CI] Fix benchmarks for LLMs

  • Fixed LLM benchmark CI configurations

• #3222 - @vmoens - [CI] Upgrade doc python version

  • Upgraded Python version in documentation build workflows and added vLLM plugin entry point for FP32 overrides

TorchRL 0.10.0 Release Notes

What's New in 0.10.0

TorchRL 0.10.0 introduces significant advancements in Large Language Model (LLM) support, new algorithms, enhanced environment integrations, and numerous performance improvements and bug fixes.

Major Features

LLM Support and RLHF

• vLLM Integration Revamp: Complete overhaul of vLLM support with improved batching and performance (#3158) @vmoens
• GRPO (Generalized Reinforcement Learning from Preference Optimization): New algorithm implementation with both sync and async variants (#2970, #2997, #3006) @vmoens
• Expert Iteration and SFT: Implementation of expert iteration algorithms and supervised fine-tuning (#3017) @vmoens
• PPOTrainer: New high-level trainer class for PPO training (#3117) @vmoens
• LLM Tooling: Comprehensive tooling support for LLM environments and transformations (#2966) @vmoens
• Remote LLM Wrappers: Support for remote LLM inference with improved batching (#3116) @vmoens
• Common LLM Generation Interface: Unified kwargs for generation across vLLM and Transformers (#3107) @vmoens
• LLM Transforms:
  • AddThinkingPrompt transform for reasoning prompts (#3027) @vmoens
  • MCPToolTransform for tool integration (#2993) @vmoens
  • PythonInterpreter transform for code execution (#2988) @vmoens
  • LLMMaskedCategorical for masked categorical distributions (#3041) @vmoens
• Content Management: ContentBase system for structured content handling (#2985) @vmoens
• History Tracking: New history system for conversation management (#2965) @vmoens

New Algorithms and Training

• Async SAC: Asynchronous implementation of Soft Actor-Critic (#2946) @vmoens
• Discrete Offline CQL: SOTA implementation for discrete action spaces (#3098) @Ibinarriaga
• Multi-node Ray Support: Enhanced distributed training for GRPO (#3040) @albertbou92

Environment Support

• NPU Support: Added NPU device support for SyncDataCollector (#3155) @lowdy1
• IsaacLab Wrapper: Integration with IsaacLab simulation framework (#2937) @vmoens
• Complete PettingZoo State Support: Enhanced multi-agent environment support (#2953) @jguzzi
• Minari Integration: Support for loading datasets from local Minari cache (#3068) @Ibinarriaga

Storage and Replay Buffers

• Compressed Storage GPU: GPU acceleration for compressed replay buffers (#3062) @aorenstein68
• Packing: New data packing functionality for efficient storage (#3060) @vmoens
• Ray Replay Buffer: Enhanced distributed replay buffer support (#2949) @vmoens

🔧 Improvements and Enhancements

Performance Optimizations

• Bounded Specs Memory: Single copy optimization for bounded specifications (#2977) @vmoens
• Log-prob Computation: Avoid unnecessary log-prob calculations when retrieving distributions (#3081) @vmoens
• LLM Wrapper Queuing: Performance fixes in LLM wrapper queuing (#3125) @vmoens
• vmap Deactivation: Selective vmap deactivation in objectives for better performance (#2957) @vmoens

API Improvements

• Public SAC Methods: Exposed public methods for SAC algorithm (#3085) @vmoens
• Composite Entropy: Fixed entropy computation for nested keys (#3101) @juandelos
• Multi-head Entropy: Per-head entropy coefficients for PPO (#2972) @FelixS
• ClippedPPOLoss: Support for composite value networks (#3031) @LouisFaury
• LineariseRewards: Support for negative weights (#3064) @YoannPoupart
• GAE Typing: Improved typing with optional value networks (#3029) @LouisFaury
• Explained Variance: Optional explained variance logging (#3010) @OswaldZink
• Frame Control: Worker-level control over frames_per_batch (#3020) @AlexGhh

Developer Experience

• Colored Logger: Enhanced logging with colored output (#2967) @vmoens
• Better Error Handling: Improved error catching in env.rollout and rb.add (#3102) @vmoens
• Warning Management: Better warning control for various components (#3099, #3115) @vmoens
• Faster Tests: Optimized test suite performance (#3162) @vmoens

Bug Fixes

Core Functionality

• PRB Serialization: Fixed Prioritized Replay Buffer serialization and loading (#3151, #2963) @vmoens
• Binary Operations: Fixed Binary tensor reshaping and clone operations (#3084, #3077) @LucaCarminati @vmoens
• Categorical Spec: Fixed dtype sampling and masking issues (#2980, #2981) @LouisFaury
• ActionMask: Compatibility with composite action specifications (#3022) @LouisFaury
• GAE with LSTM: Fixed shifted value computation with LSTM networks (#2941) @vmoens
• Cross-entropy: Fixed log-prob computation for batched input (#3080) @vmoens

Environment and Wrapper Fixes

• TransformedEnv: Fixed in-place modification of specs (#3076) @vmoens
• Parallel Environments: Fixed partial and nested done states (#2959) @vmoens
• Gym Actions: Fixed single action passing when action key is not "action" (#2942) @vmoens
• Brax Memory: Fixed memory leak in Brax environments (#3052) @vmoens
• Atari Patching: Fixed patching for NonTensorData observations (#3091) @MarcosGR

Collector and Replay Buffer Fixes

• LLMCollector: Fixed trajectory collection when multiple trajectories complete (#3018) @albertbou92
• Postprocessing: Consistent postprocessing when using replay buffers in collectors (#3144) @vmoens
• Weight Updates: Fixed original weights retrieval in collectors (#2951) @vmoens
• Transform Handling: Fixed transform application and metadata preservation (#3047, #3050) @vmoens

Compatibility and Infrastructure

• PyTorch 2.1.1: Fixed compatibility issues (#3157) @vmoens
• NPU Attribute: Fixed missing NPU attribute (#3159) @vmoens
• CUDA Graph: Fixed update_policy_weights_ with CUDA graphs (#3003) @vmoens
• Stream Capturing: Robust CUDA stream capturing calls (#2950) @vmoens

Documentation and Tutorials

• DQN with RNN Tutorial: Upgraded tutorial with latest best practices (#3152) @vmoens
• LLM API Documentation: Comprehensive documentation for LLM environments and transforms (#2991) @vmoens
• Multi-head Entropy: Better documentation for multi-head entropy usage (#3109) @vmoens
• LSTM Module: Fixed import examples in documentation (#3138) @arvindcr4
• A2C Documentation: Updated AcceptedKeys documentation (#2987) @simeet-n
• History API: Added missing docstrings for History functionality (#3083) @vmoens
• Multi-agent PPO: Fixed tutorial issues (#2940) @matteobettini
• WeightUpdater: Updated documentation after renaming (#3007) @albertbou92

Infrastructure and CI

• Pre-commit Updates: Updated formatting and linting tools (#3108) @vmoens
• Benchmark CI: Fixed benchmark runs and added missing dependencies (#3092, #3163) @vmoens
• Windows CI: Fixed Windows continuous integration (#3028) @vmoens
• Old Dependencies: Fixed CI for older dependency versions (#3165) @vmoens
• C++ Linting: Fixed C++ code linting issues (#3129) @vmoens
• Build System: Improved pyproject.toml usage and versioning (#3089, #3166) @vmoens

🏆 Contributors

Special thanks to all contributors who made this release possible:

• @albertbou92 (Albert Bou) - GRPO multi-node support and LLM improvements
• @Ibinarriaga - CQL offline algorithm and Minari integration
• @aorenstein68 (Adrian Orenstein) - Compressed storage GPU support
• @LouisFaury (Louis Faury) - Categorical spec and PPO improvements
• @LucaCarminati (Luca Carminati) - Binary tensor fixes
• @jguzzi (Jérôme Guzzi) - PettingZoo state support
• @lowdy1 - NPU device support
• @FelixS (Felix Sittenauer) - Multi-head entropy coefficients
• @YoannPoupart (Yoann Poupart) - LineariseRewards improvements
• @OswaldZink (Oswald Zink) - Explained variance logging
• @AlexGhh (Alexandre Ghelfi) - Frame control improvements
• @MarcosGR (Marcos Galletero Romero) - Atari patching fixes
• @matteobettini (Matteo Bettini) - Tutorial fixes
• @simeet-n (Simeet Nayan) - Documentation improvements
• @arvindcr4 - Documentation fixes
• @felixy12 (Felix Yu) - State dict reference fixes
• @SendhilPanchadsaram (Sendhil Panchadsaram) - Documentation typo fixes
• @abhishekunique (Abhishek) - WandB logger and value estimation improvements
• @骑马小猫 - DQN module typo fix
• @ZainRizvi (Zain Rizvi) - CI improvements and meta-pytorch migration
• @mikayla-gawarecki (Mikayla Gawarecki) - Usage tracking and ConditionalPolicySwitch

🔗 Compatibility

• PyTorch: Compatible with PyTorch 2.1.1+ -- recommended >=2.8.0,<2.9.0 for full compatibility
• TensorDict: Updated to work with TensorDict 0.10+
• Python: Supports Python 3.9+

📦 Installation

pip install torchrl==0.10.0

For the latest features:

pip install git+https://github.com/pytorch/rl.git@release/0.10.0

TorchRL 0.9.2 Release Notes

This release focuses on bug fixes, performance improvements, and code quality enhancements.

🚀 New Features

• LineariseRewards: Now supports negative weights for more flexible reward shaping (#3064)

🐛 Bug Fixes

• Fixed policy reference handling in state dictionaries (#3043)
• Improved unbatched data handling in LLM wrappers (#3070)
• Fixed cross-entropy log-probability computation for batched inputs (#3080)
• Fixed Binary clone() operations (#3077)
• Fixed in-place spec modifications in TransformedEnv (#3076)

⚡ Performance Improvements

• Optimized distribution sampling by avoiding unnecessary log-probability computations (#3081)

🔧 Code Quality

• Standardized coefficient naming in A2C and PPO algorithms (#3079)

📦 Installation

pip install torchrl==0.9.2

Thanks to all contributors: @felixy12, @Xmaster6y, @louisfaury and @LCarmi

Fixes an critical issue with vLLMWrapper and TransformersWrapper, where a stack of History objects is resent to stack, resulting in a bug.

We are excited to announce the release of TorchRL 0.9.0! This release introduces a comprehensive LLM API for language model fine-tuning, extensive torch.compile compatibility across all algorithms, and numerous performance improvements.

🚀 Major Features

🤖 LLM API - Complete Framework for Language Model Fine-tuning

TorchRL now includes a comprehensive LLM API for post-training and fine-tuning of language models! This new framework provides everything you need for RLHF, supervised fine-tuning, and tool-augmented training:

• Unified LLM Wrappers torchrl.modules.llm: Seamless integration with Hugging Face models and vLLM inference engines
• Conversation Management torchrl.data.llm.history: Advanced History class for multi-turn dialogue with automatic chat template detection
• Tool Integration torchrl.envs.llm.transforms: Built-in support for Python code execution, function calling, and custom tool transforms
• Specialized Objectives torchrl.objectives.llm: GRPO (Group Relative Policy Optimization) and SFT loss functions optimized for language models
• High-Performance Collectors torchrl.collectors.llm: Async data collection with distributed training support
• Flexible Environments torchrl.envs.llm: Transform-based architecture for reward computation, data loading, and conversation augmentation

The LLM API follows TorchRL's modular design principles, allowing you to mix and match components for your specific use case. Check out the complete documentation and GRPO implementation example to get started!

Unified LLM Wrappers

• TransformersWrapper: Seamless integration with Hugging Face models
• vLLMWrapper: High-performance inference with vLLM engines
• Consistent API: Both wrappers provide unified input/output interfaces using TensorClass objects
• Multiple input modes: Support for history, text, and tokenized inputs
• Configurable outputs: Text, tokens, masks, and log probabilities

Advanced Conversation Management

• History class: Advanced bidirectional conversation management with automatic chat template detection
• Multi-model support: Automatic template detection for various model families (Qwen, DialoGPT, Falcon, DeepSeek, etc.)
• Assistant token masking: Identify which tokens were generated by the assistant for RL applications
• Tool calling support: Handle function calls and tool responses in conversations
• Batch operations: Efficient tensor operations for processing multiple conversations

🛠️ Tool Integration

• PythonInterpreter transform: Built-in Python code execution capabilities
• MCPToolTransform: General tool calling support
• Extensible architecture: Easy to add custom tool transforms
• Safe execution: Controlled environment for tool execution

🎯 Specialized Objectives

• GRPOLoss: Group Relative Policy Optimization loss function optimized for language models
• SFTLoss: Supervised fine-tuning loss with assistant token masking support
• MCAdvantage: Monte-Carlo advantage estimation for LLM training
• KL divergence rewards: Built-in KL penalty computation

⚡ High-Performance Collectors

• LLMCollector: Async data collection with distributed training support
• RayLLMCollector: Multi-node distributed collection using Ray
• Weight synchronization: Automatic model weight updates across distributed setups
• Trajectory management: Efficient handling of variable-length conversations

🔄 Flexible Environments

• ChatEnv: Transform-based architecture for conversation management
• Transform-based rewards: Modular reward computation and data loading
• Dataset integration: Built-in support for loading prompts from datasets
• Thinking prompts: Chain-of-thought reasoning support

📚 Complete Implementation Example

A full GRPO implementation is provided in sota-implementations/grpo/ with:

• Multi-GPU support with efficient device management
• Mixed precision training
• Gradient accumulation
• Automatic checkpointing
• Comprehensive logging with Weights & Biases
• Hydra configuration system
• Asynchronous training support with Ray

🆕 New Features

LLM API Components

• LLMMaskedCategorical (#3041) - Categorical distribution with masking for LLM token selection
• AddThinkingPrompt transform (#3027) - Add chain-of-thought reasoning prompts
• MCPToolTransform (#2993) - Model Context Protocol tool integration
• PythonInterpreter transform (#2988) - Python code execution in LLM environments
• ContentBase (#2985) - Base class for structured content in LLM workflows
• LLM Tooling (#2966) - Comprehensive tool integration framework
• History API (#2965) - Advanced conversation management system
• LLM collector (#2879) - Specialized data collection for language models
• vLLM wrapper (#2830) - High-performance vLLM integration
• Transformers policy (#2825) - Hugging Face transformers integration

Environment Enhancements

• IsaacLab wrapper (#2937) - NVIDIA Isaac Lab environment support
• Complete PettingZooWrapper state support (#2953) - Full state management for multi-agent environments
• ConditionalPolicySwitch transform (#2711) - Dynamic policy switching based on conditions
• Async environments (#2864) - Asynchronous environment execution
• VecNormV2 (#2867) - Improved vector normalization with batched environment support

Algorithm Improvements

• Async GRPO (#2997) - Asynchronous Group Relative Policy Optimization
• Expert Iteration and SFT (#3017) - Expert iteration and supervised fine-tuning algorithms
• Async SAC (#2946) - Asynchronous Soft Actor-Critic implementation
• Multi-node Ray support for GRPO (#3040) - Distributed GRPO training

Data Management

• RayReplayBuffer (#2835) - Distributed replay buffer using Ray
• RayReplayBuffer usage examples (#2949) - Comprehensive usage examples
• Policy factory for collectors (#2841) - Flexible policy creation in collectors
• Local and Remote WeightUpdaters (#2848) - Distributed weight synchronization

Performance Optimizations

• Deactivate vmap in objectives (#2957) - Improved performance by disabling vectorized operations
• Hold a single copy of low/high in bounded specs (#2977) - Memory optimization for bounded specifications
• Use TensorDict._new_unsafe in step (#2905) - Performance improvement in environment steps
• Memoize calls to encode and related methods (#2907) - Caching for improved performance

Utility Features

• Compose.pop (#3026) - Remove transforms from composition
• Add optional Explained Variance logging (#3010) - Enhanced logging capabilities
• Enabling worker level control on frames_per_batch (#3020) - Granular control over data collection
• collector.start() (#2935) - Explicit collector lifecycle management
• Timer transform (#2806) - Timing capabilities for environments
• MultiAction transform (#2779) - Multi-action environment support
• Transform for partial steps (#2777) - Partial step execution support

🔧 Performance Improvements

• VecNormV2: Improved vector normalization with better bias correction timing (#2900, #2901)
• MaskedCategorical cross_entropy: Faster loss computation (#2882)
• Avoid padding in transformer wrapper: Memory and performance optimization (#2881)
• Set padded token log-prob to 0.0: Improved numerical stability (#2857)
• Better device checks: Enhanced device management (#2909)
• Local dtype maps: Optimized dtype handling (#2936)

🐛 Bug Fixes

LLM API Fixes

• Variable length vllm wrapper answer stacking (#3049) - Fixed stacking issues with variable-length responses
• LLMCollector trajectory collection methods (#3018) - Fixed trajectory collection when multiple trajectories complete simultaneously
• Fix IFEval GRPO runs (#3012) - Resolved issues with IFEval dataset runs
• Fix cuda cache empty in GRPO scripts (#3016) - Memory management improvements
• Right log-prob size in transformer wrapper (#2856) - Fixed log probability tensor sizing
• Fix gc import (#2862) - Import error resolution

Environment Fixes

• Brax memory leak fix (#3052) - Resolved memory leaks in Brax environments
• Fix behavior of partial, nested dones in PEnv and TEnv (#2959) - Improved done state handling
• Fix shifted value computation with an LSTM (#2941) - LSTM value computation fixes
• Fix single action pass to gym when action key is not "action" (#2942) - Action key handling improvements
• Fix PEnv device copies (#2840) - Device management in parallel environments

Data Management Fixes

• Fix minari dataloading (#3054) - Resolved Minari dataset loading issues
• RB.add unsqueezes tds when applying the transform (#3047) - Replay buffer transform handling
• Fix PRB serialization (#2963) - Prioritized replay buffer serialization
• Fix lazy-stack in RBs (#2880) - Lazy stacking in replay buffers
• Keep original class in LazyStackStorage through lazy_stack (#2873) - Class preservation in lazy stacking

Algorithm Fixes

• Fix deprecated list index (#3005) - Updated deprecated list indexing
• update_policy_weights_() with cudagraph (#3003) - CUDA graph compatibility
• Fix compile compatibility of PPO losses (#2889) - Compilation compatibility
• Fix .item() warning on tensors that require grad (#2885) - Gradient tensor handling
• Fix KL penalty (#2908) - KL divergence computation fixes

Specification and Type Fixes

• Fixes the Categorical is_in with non-long integer (#2981) - Type compatibility improvements
• Categorical spec samples the right dtype when masked (#2980) - Masked categorical sampling
• Binary can have empty shape (#2979) - Empty shape handling
• ActionMask is compatible with composite action specs (#3022) - Composite action specification support
• Fix composite setitem (#2778) - Composite specification item setting

General Fixes

• Fix various test failures (#2994) - Test suite improvements
• Fix wrong split_trajectories import (#3023) - Import error resolution
• Fix typo (#2969) - Documentation typo fixes
• Fix device in PPO tests (#2971) - Device handling in tests
• Fix device in args of PPO losses (#2969) - PPO loss device arguments

📚 Documentation

• Document the LLM env and transform API (#2991) - Comprehensive LLM API documentation
• Update documentation for _AcceptedKeys in a2c.py (#2987) - A2C documentation improvements
• WeightUpdaterBase docs update after renaming (#3007) - Updated documentation for renamed components
• Fix doc pipeline (#2992) - Documentation build improvements
• Fix Doc (#2919) - General documentation fixes
• Fix doc setup (#2922) - Documentation setup improvements
• Better doc for Transform class (#2797) - Transform class documentation
• Add docstring for MCTSForest.extend (#2795) - MCTS documentation
• Fix tutorials (#2772, #2768) - Tutorial fixes and improvements

🧪 Testing and Quality

• Fix wrong import (#3033) - Import error fixes
• Fix error catches (#2982) - Error handling improvements
• Fix warnings in tests (#2886) - Warning suppression in tests
• Test and fix life cycle of env with dynamic non-tensor spec (#2812) - Environment lifecycle testing
• Capture deprec warnings (#2799) - Deprecation warning handling
• Fix old deps tests (#2500) - Dependency testing improvements

🔄 Refactoring and Code Quality

• Refactor the weight update logic (#2914) - Improved weight update architecture
• Refactor LLM data structures (#2834) - LLM data structure improvements
• Rename RLHF files to LLM (#2833) - File organization improvements
• Refactor TransformersWrapper class (#2871) - Transformers wrapper improvements
• Refactor vLLMWrapper class (#2870) - vLLM wrapper improvements
• Remove from_vllm and from_hf_transformers (#2874) - Cleanup of deprecated methods
• Simplify LLMEnv (#2897) - LLM environment simplification

🚀 CI and Infrastructure

• Fix win CI (#3028) - Windows CI improvements
• Fix SDL install (#2978) - SDL installation fixes
• Build wheels on osx 15 (#2934) - macOS 15 compatibility
• Fix tensordict upper version to 0.9 (#2933) - Dependency version management
• Fix nightly and benchmark CIs (#2930) - CI pipeline improvements
• Fix envnames in SOTA tests (#2921) - Test environment naming
• egl for all (#2915) - EGL support improvements
• Fix LLM tests (#2918) - LLM test suite fixes
• Fix old deps (#2916) - Dependency management
• Upgrade to cuda 12.8 (#2820) - CUDA version upgrade
• Fix libs workflows (#2800) - Library workflow improvements

📦 Dependencies and Setup

• Remove distutils imports (#2836) - Modern Python compatibility
• Fix no_python_abi_suffix error (#2863) - Python ABI suffix handling
• Upgrade to v0.7 (#2745) - Dependency version updates
• Fix Cairo-2 Chess import error (#2743) - Chess environment dependencies

🗑️ Deprecations and Removals

• Enact deprecations (#2917) - Implementation of planned deprecations
• Remove LLM features for release (#2912) - Temporary removal for release stability
• Softly change default behavior of auto_unwrap (#2793) - Default behavior changes
• *Gracing old Spec with v0.8 versioning (#2751) - Specification versioning
• Remove InPlaceSampler (#2750) - Deprecated sampler removal
• Remove OrnsteinUhlenbeckProcessWrapper (#2749) - Deprecated wrapper removal
• Remove AdditiveGaussianWrapper (#2748) - Deprecated wrapper removal
• Remove NormalParamWrapper (#2747) - Deprecated wrapper removal
• Change the default MLP depth (#2746) - Default configuration changes

🔧 Minor Improvements

• Fix sota runs (#3042) - SOTA implementation improvements
• remove unused variables in GRPO scripts (#3038) - Code cleanup
• Fix deprecated list index (#3005) - Deprecation warning fixes
• gitignore ipynb (#2954) - Git ignore improvements
• Quick edits to .md files (#2931) - Documentation improvements
• Fix typos in advantages.py (#2492) - Documentation typo fixes
• Remove redundant return (#2925) - Code cleanup
• Fix some typos (#2811) - Documentation improvements

📊 Migration Guide

LLM API Usage

The new LLM API provides a complete framework for language model fine-tuning. Key components include:

from torchrl.envs.llm import ChatEnv
from torchrl.modules.llm import TransformersWrapper
from torchrl.objectives.llm import GRPOLoss
from torchrl.collectors.llm import LLMCollector

# Create environment with Python tool execution
env = ChatEnv(
    tokenizer=tokenizer,
    system_prompt="You are an assistant that can execute Python code.",
    batch_size=[1]
).append_transform(PythonInterpreter())

# Wrap your language model
llm = TransformersWrapper(
    model=model,
    tokenizer=tokenizer,
    input_mode="history"
)

# Set up GRPO training
loss_fn = GRPOLoss(llm, critic, gamma=0.99)
collector = LLMCollector(env, llm, frames_per_batch=100)

# Training loop
for data in collector:
    loss = loss_fn(data)
    loss.backward()
    optimizer.step()

Breaking Changes

• Some deprecated wrappers have been removed (NormalParamWrapper, AdditiveGaussianWrapper, etc.)
• Default MLP depth has been changed
• Default behavior of auto_unwrap has been modified

Performance Recommendations

• Use the new VecNormV2 for improved normalization performance. Can be used through a keyword arg in regular VecNorm transform.
• Leverage async environments and collectors for better throughput.
• Consider using RayReplayBuffer for distributed training scenarios.

🙏 Acknowledgments

We would like to thank all contributors who made this release possible, especially those who contributed to the LLM API framework and the comprehensive testing and documentation improvements.

---

For detailed usage examples and tutorials, please refer to the TorchRL documentation and the LLM API reference.

Async Collector execution

This release major upgrades is a patch to collector.start() to allow collectors (single or multi-proc) to run asynchronously. #2935

An example is provided in the async SAC example. #2946

Single-agent reset

Fixes #2958 where partial resets are not handled correctly when a BatchedEnv is transformed - as the "done" checks were inconsistent. We now enforce that root "_reset" entries always precede their respective leaves.

Fix shifted values in GAE using LSTMs

Using an LSTM within GAE is facilitated by ensuring that shifted=True and shifted=False work properly (with appropriate warnings/errors if other hyperparameters need to be set). #2941

Full Changelog: https://github.com/pytorch/rl/compare/v0.8.0...v0.8.1

TorchRL v0.8.0: Async envs and better weight update API

• Async environments: #2864 introduces asynchronous environments, which can be built using different backends (currently "threading" or "multiprocessing"). Instantiating an async env is roughly the same as a parallel one:

  from torchrl.envs import AsyncEnvPool
  env = AsyncEnvPool([partial(GymEnv, "Pendulum-v1"), partial(GymEnv, "Pendulum-v1")], backend="threading")
  

  These environments support the regular environment methods (reset, step or rollout) but their main advantage lies in their new async methods:

  s0 = env.rand_action(env.reset())
  env.async_step_send(s0)
  # receive
  result = env.async_step_recv()
  

  In this example, result will contain the results of the call to step for one or two environments. The environment indices can be found in the result['env_index'] entry (the name of that key is stored in env._env_idx_key).
• Support for environments with tensorclass attributes (#2788)
• Distributed RayReplayBuffer (#2835)
• Gymnasium 1.1 compatibility (#2898): we managed to make TorchRL compatible with Gymnasium 1.1 as this version lets users choose how to handle partial resets, which facilitates integration in the library.
• VecNormV2, a new version of vecnorm which is more numerically stable and easier to handle. This can be created directly through the usual VecNorm by passing the new_api keyword argument.
• policy factory for collectors: you can now pass a factory for your policy instead of passing the real object. Given that the collector will update the weights of the policy when asked to, this will in most cases not cause any synchronization problem with the copy that is used by the training pipeline.
• An Update API for policy weights in collector: we have isolated the weight update API in a torchrl.collectors.WeightUpdaterBase abstract class. This should the entry point for any user wanting to implement their own weight update strategy, alleviating the need to subclass or patch the collector or the policy directly.

Packaging

We relaxed TorchRL dependency to make it compatible with any pytorch version. The current status is:

• tensordict dependency will from now on be enforced (>=0.8.1,<0.9.0 for this release)
• For PyTorch prior to 2.7.0, backward compatibility is guaranteed to some extend (most classes should work, unless new features are used) but C++ binaries (for prioritized replay buffers) will not work.
• For PyTorch >= 2.7.0, C++ binaries should work across versions. In other words, torchrl binaries for 0.8.0 will work with PyTorch 2.7.0, 2.8.0 etc., and the same goes for the future TorchRL 0.9.0... A big thanks to @janeyx99 for enabling this!

New features

[Feature] Add EnvBase.all_actions (#2780) (67c3e9a4a) by @kurtamohler ghstack-source-id: 7abf9d469f740be5f14daffa2330811f7572dad9 [Feature] Add MCTSForest/Tree.to_string (#2794) (f8626690c) by @kurtamohler ghstack-source-id: 2127bf24d66e44fb310d12ff5f72e92aa0371cd7 [Feature] Add include_hash_inv arg to ChessEnv (#2766) (3be85c691) by @kurtamohler ghstack-source-id: f6920d781835902a6db02f74c5e5a3041243c5e3 [Feature] Add option for auto-resetting envs in GAE (#2851) (f5f3ae4e2) by @lin-erica Co-authored-by: Erica Lin elin@theaiinstitute.com [Feature] Async environments (#2864) (4f00025af) by @vmoens ghstack-source-id: 0a70ce0129d2ee6f85bb22adda3c332ff65e7501 [Feature] Capture wrong spec transforms (1/N) (#2805) (d3dca73f3) by @vmoens ghstack-source-id: f2d938b3dfe88af66622099f60cd7e3026289a02 [Feature] Collectors for async envs (#2893) (4ba50667e) by @vmoens ghstack-source-id: 764c21d0f2c3b217440e1a6f12ee797b17820c1d [Feature] DensifyReward postproc (#2823) (53065cf56) by @vmoens ghstack-source-id: ef6a0f52601642c8944f63f9e3ac9e963425734e [Feature] Dynamic specs for make_composite_from_td (#2829) (413571b8d) by @vmoens ghstack-source-id: 79e31e737c9f67ff20ce9fe32081e5b0a83de947 [Feature] Enable Hash.inv (#2757) (32c4623b3) by @kurtamohler ghstack-source-id: 956708121067855e519382a37764f06f53b16aa7 [Feature] Env with tensorclass attributes (#2788) (ab76027c1) by @vmoens ghstack-source-id: dc00ea3d23e015756974cd5c2ce638b55e5f6f92 [Feature] Gymnasium 1.1 compatibility (#2898) (78cd7550d) by @vmoens ghstack-source-id: e0891867f4318380f01c15449f9f26070b78536d [Feature] History API (#2890) (fd10fe213) by @vmoens ghstack-source-id: 5b9723f6e1c327625e1a9be6f6eac68b91ed8492 [Feature] History.default_spec (#2894) (8ce11a859) by @vmoens ghstack-source-id: 40b8a492765a85adaccb591f1bc173754bacc313 [Feature] Local and Remote WeightUpdaters (#2848) (27d3680e7) by @vmoens ghstack-source-id: 2962530f87b596d038e3a13a934ea09064af2964 [Feature] Make PPO ready for text-based data (#2857) (595ddb4fa) by @vmoens ghstack-source-id: eeda5e2355e573e74cf7c080994cd47520ecd45b [Feature] MultiAction transform (#2779) (621776a21) by @vmoens ghstack-source-id: 0a6f7f916ee6f9c6d450c511385bdfdb1d911da0 [Feature] NonTensor batched arg (#2816) (b97bdb5e6) by @vmoens ghstack-source-id: c6de1bd1f1475b8d02df2ff3eb7438a50f2ae450 [Feature] Pass lists of policy_factory (#2888) (82f8ec26d) by @vmoens ghstack-source-id: e42b100096c6e38365f8a80681473746f51d8a77 [Feature] RayReplayBuffer (#2835) (50af98432) by @vmoens ghstack-source-id: 32eff06494037a1a30e532539794035c035f1e81 [Feature] Set padded token log-prob to 0.0 (#2856) (b9ddfa967) by @vmoens ghstack-source-id: 2b2993e0b15afae17326e6583390d57068712d4f [Feature] Support lazy tensordict inputs in ppo loss (#2883) (c9caf3d9c) by @vmoens ghstack-source-id: 89098ba3ca61b1524aeddc68f54c377f29c8dc8b [Feature] TensorDictPrimer with single default_value callable (#2732) (59e85458b) by @vmoens ghstack-source-id: a9a677f24fc1e6a47312d0a96ab60daae543ff78 [Feature] Timer transform (#2806) (104b88092) by @vmoens ghstack-source-id: e42f2aece15f90afc457e1fb3e41a1f7be1a6a85 [Feature] Transform for partial steps (#2777) (7c034e331) by @vmoens ghstack-source-id: 587f91e33dfe1d59b73c4b2f2f1c21760ee79d2e [Feature] VecNormV2 (#2867) (40fcdb6bb) by @vmoens ghstack-source-id: 639d07ff54be200d54621c2c4619ebd0d3d7d79e [Feature] VecNormV2: Usage with batched envs (#2901) (b08e7ace3) by @vmoens ghstack-source-id: 5e14ed982b71b0e5192b0687c5259a3b49a81157 [Feature] pass policy-factory in mp data collectors (#2859) (31af2c529) by @vmoens ghstack-source-id: bce8abe9853d5ec187f91ffbcd8b940fa18ec8ab [Feature] policy factory for collectors (#2841) (49a8a42a0) by @vmoens ghstack-source-id: 96b928e938b8b07fc7de23483358202737571f8e [Feature] reset_time in Timer (#2807) (5a4637969) by @vmoens ghstack-source-id: 36a74fd20b78e1cdde6bca19b4f95c3d9062d761 [Feature] transformers policy (#2825) (eea932c3c) by @vmoens ghstack-source-id: 870c221b4ebae132a44944f0be0ee78da540d115

Fixes

[BugFix] Apply inverse transform to input of TransformedEnv._reset (#2787) (1ed5d293a) by @kurtamohler ghstack-source-id: 5f7c1fbd19b716f2b1602c34cf2ae1362f7bc7f6 [BugFix] Avoid calling reset during env init (#2770) (09e93c19c) by @vmoens ghstack-source-id: 5ab8281c34aacfd7dbbfc0e285d88bcae0aededf [BugFix] Ensure that Composite.set returns self as TensorDict does (#2784) (e084c02dd) by @vmoens ghstack-source-id: 23fe46b61dc2c9548fd9de7e4100431918fd0370 [BugFix] Fix .item() warning on tensors that require grad (#2885) (b66fcd4cf) by @vmoens ghstack-source-id: 502bdda3f5700dc900cf5c748839c965b1d67c1b [BugFix] Fix KL penalty (#2908) (96c300322) by @vmoens ghstack-source-id: 475dccb0bcddbfe3bd2d826c5389834fb95e1ab8 [BugFix] Fix MultiAction reset (#2789) (76aa9bc0c) by @kurtamohler ghstack-source-id: a2f7bfdd7522a214430182dac65687a977b1a10d [BugFix] Fix PEnv device copies (#2840) (6e40548ab) by @vmoens ghstack-source-id: df39fd2e4cd72f24c645b0ac32b46ab3e8d847fc [BugFix] Fix batch_locked check in check_env_specs + error message callable (#2817) (9c98b82c3) by @vmoens ghstack-source-id: c722b164133c27c05dd21add3e7f3158189dd515 [BugFix] Fix calls to _reset_env_preprocess (#2798) (ea76ffb62) by @vmoens ghstack-source-id: 59925635a87b196a5bcb0fb251afe4cc7b8b103e [BugFix] Fix collector timeouts (#2774) (f6084b657) by @vmoens ghstack-source-id: cb71d95143beb22db1fe1752e72f70c19f43be79 [BugFix] Fix collector with no buffers and devices (#2809) (d4f88460a) by @vmoens ghstack-source-id: 5367df9fcfdf549108be852476b049a0b978e348 [BugFix] Fix compile compatibility of PPO losses (#2889) (9bc85f4f0) by @vmoens ghstack-source-id: b346033641e5d27560fbfa011a006446e56a4e31 [BugFix] Fix composite setitem (#2778) (c2a149d66) by @vmoens ghstack-source-id: f33b49beb4cf8c0c8b156559b1abbee8ac77db20 [BugFix] Fix env.full_done_spec~s~ (#2815) (f5c0666c2) by @vmoens ghstack-source-id: ba0d371d10b3f46ec1172fbec639ccc4d5559659 [BugFix] Fix forced batch-size in _skip_tensordict (#2808) (3acf49101) by @vmoens ghstack-source-id: dac84e8b8835e870bce1772d7893c30b6f9af59c [BugFix] Fix gc import (#2862) (a183f02a5) by @vmoens ghstack-source-id: b732d4f805d98ceaaa45326d619fce623c10482f [BugFix] Fix lazy-stack in RBs (#2880) (e80732ede) by @vmoens ghstack-source-id: 38399ee991bc065445f4eb1c84b71e7d844d794c [BugFix] Fix property getter in RayReplayBuffer (#2869) (04d70c1e4) by @vmoens [BugFix] Fix slow and flaky non-tensor parallel env test (#2926) by @vmoens ghstack-source-id: fcb5caa56e05176958b3468a7d6f69e363cfe558 [BugFix] Fix update shape mismatch in _skip_tensordict (#2792) (3e42e7a2f) by @vmoens ghstack-source-id: 27e7d444c126e48fdb70d951a0cc7beaee1db3a8 [BugFix] Fixed VideoRecorder crash when passing fps (#2827) (5ec9bc56e) by Alexandre Brown [BugFix] GAE warning when gamma/lmbda are tensors (#2838) (d56111599) by @louisfaury Co-authored-by: Louis Faury louis.faury@helsing.ai [BugFix] Keep original class in LazyStackStorage through lazy_stack (#2873) (70f5c0649) by @vmoens ghstack-source-id: 661cd65c86648ffb2ee6ead40110ac3d57477514 [BugFix] NonTensor should not convert anything to numpy (#2771) (3da27506c) by @vmoens ghstack-source-id: 7644f6c695490f34d6455703418c59cfa718a9f0 [BugFix] PPOs with composite distribution (#2791) (edfa25d96) by @louisfaury Co-authored-by: Louis Faury louis.faury@helsing.ai [BugFix] Refactor _skip_tensordict to avoid update calls (#2802)" (#2802) (e0d3eee3b) by @vmoens ghstack-source-id: 0f31b879f1e4643080530db8f7c7091e281b560f [BugFix] Remove neg dim checks in expand for all specs (#2906) (c5afe3c90) by @vmoens ghstack-source-id: f718328527275e0be591c5d12c334add8f65f7a4 [BugFix] Right log-prob size in transformer wrapper (#2854) (f81deacd1) by @vmoens ghstack-source-id: 98baa635ca07d5bf7e69a9e3bc43012ae2d91bf0 [BugFix] Test and fix life cycle of env with dynamic non-tensor spec (#2812) (b538c66c5) by @vmoens ghstack-source-id: 77da3a6baf0cb42525dd3a564b36ac03a531d17a [BugFix] Tree make node fix (#2839) (ba8be9c44) by Rolo [BugFix] Use brackets to get non-tensor data in gym envs (#2769) (84f6b0483) by @vmoens ghstack-source-id: 3101141eb5b7435c7a4047f5ee84b66c1d74af13 [BugFix] correct dim for resolving dtype in _split_and_pad_sequence (#2801) (21c4d87c7) by KubaMichalczyk Co-authored-by: Jakub Michalczyk jakub.michalczyk@sportec-solutions.com

Refactors

[Refactor] Avoid padding in transformer wrapper (#2881) (9c4c086a5) by @vmoens ghstack-source-id: de28bab17fc3d59889ea9f2fd152de5001b92320 [Refactor] Fix repeats order (#2887) (93ba865c2) by @vmoens ghstack-source-id: 0bedd5c756f92d23083905ae8a6ddd992ba0b415 [Refactor] MaskedCategorical cross_entropy usage for faster loss (#2882) (3e1f4ff1c) by @vmoens ghstack-source-id: 84330cf08ad8798e2cd4f6a8f3ec146a9de8e1e4 [Refactor] Refactor the weight update logic (#2914) (0da904425) by @vmoens ghstack-source-id: 72b710ab1788090364c068c59b28a21e09221236 [Refactor] Rename weight updaters (#2892) (efe938956) by @vmoens ghstack-source-id: 8889046277b94db0076fa72787295fd9419ab183 [Refactor] TransformersWrapper class (#2871) (5d7256122) by @vmoens ghstack-source-id: 8d5442611e9f1cf499cd59ed3e61a0602459c94d [Refactor] VecNormV2: update before norm, bias_correction at the right time (#2900) (c3310b87f) by @vmoens ghstack-source-id: a90aeb268a83dc2e45735f7b6b19b4e63e572ba7

Miscellaneous

[BE] Ensure abstractmethods are implemented for specs (#2790) (bd78913fe) by @vmoens ghstack-source-id: 7b943aa84bc497e7e8195f633cb15105de137f04 [BE] Fix some typos (#2811) (0ae140568) by @antoinebrl [BE] Make better logits in cost tests (#2775) (42ed42c71) by @vmoens ghstack-source-id: be9ea92b3f3d2592e426eaeaff7b81e50472cf16 [BE] Remove deprec specs from tests (#2767) (27a8ecc29) by @vmoens ghstack-source-id: 717bb31b1773c5c8b180c456f1bbad8a022dc55a [BE] _set_seed returns None + type annotations (#2903) (3a9f244de) by @antoinebrl [CI] Fix envnames in SOTA tests (#2921) by @vmoens ghstack-source-id: 3b518e2a81e9d988db2fbd12883eabbe486d32db [CI] Fix libs workflows (#2800) (8dd1be7c3) by @vmoens [CI] Fix nightly and benchmark CIs (#2930) by @vmoens ghstack-source-id: f39b2573ba58e7808389af3782aed8809759fa2b [CI] Fix old deps (#2916) (4162db690) by @vmoens ghstack-source-id: 109de71c622760679d449d906b0f33b3f1866975 [CI] Fix wheels (#2876) (e1d3fd488) by @vmoens ghstack-source-id: 0f2602146c4371d2fc6ac33f139b1eebb0829559 [CI] Upgrade to cuda 12.8 (#2820) (8c9dc050f) by @vmoens ghstack-source-id: e0ad7d6c00d53b74b23022836535c453a37df238 [CI] egl for all (#2915) (425952b96) by @vmoens ghstack-source-id: 1b5e13c44f5dff1a55f9c78a174f7164f37d76a1 [Deprecation] Enact deprecations (#2917) (b247526a1) by @vmoens ghstack-source-id: 690a9f62e274e9f14a89532dd7d07176188560e9 [Deprecation] Softly change default behavior of auto_unwrap (#2793) (2046bc536) by @vmoens ghstack-source-id: c28c11ecf68fba0ffde652205ea8e46f8da07cf1 [Doc] Add docstring for MCTSForest.extend (#2795) (a3a1ebefe) by @kurtamohler ghstack-source-id: 7fa8834376a1afd9187d7f1d43a97f70d713a160 [Doc] Better doc for Transform class (#2797) (dd59290d9) by @vmoens ghstack-source-id: 16e563bc810586d31772b58f9923439b632985c7 [Doc] Fix (and deactivate) tutorials (#2785) (f1c42e083) by @vmoens ghstack-source-id: 56c7757c36a2d609688ce0777a49d54763d3e691 [Doc] Fix Doc (#2919) by @vmoens ghstack-source-id: dacd1e7467994c73b22b5f111ac6c486d43d7b58 [Doc] Fix EnvCreator's doc (#2868) (586a5413d) by @louisfaury Co-authored-by: Louis Faury louis.faury@helsing.ai [Doc] Fix doc CI (#2932) by @vmoens ghstack-source-id: 5a27f9153c77d659ee691117c6af60a2d4b022bf [Doc] Fix formatting errors (#2786) (03d658630) by @vmoens ghstack-source-id: ac1f3da66c1374d3d19fed88e80f8ed5407b3459 [Doc] Fix tutorials (#2768) (75f113ff5) by @vmoens [Doc] Fix tutos (#2772) (b27ee6d8d) by @vmoens [Doc] Solve ref issues in docstrings (#2776) (f5445a4bd) by @vmoens ghstack-source-id: 09823fa85a94115291e7434478776fb0834f9b39 [Doc] Update discrete.py (#2850) (619fec69c) by oswald [Docs] Fix doc setup (#2922) by @vmoens ghstack-source-id: 91d359ad591a7f8062191825d04ebda112f2cf7d [Environment] Fix lib CI failures (#2923) by @vmoens ghstack-source-id: c046f5ded86a3a07d66eaddcaef24b69c2d77c01 [Environment] Fix lib CI failures (#2929) by @vmoens ghstack-source-id: febe20b0915025cc253f6aa0404258f3a020e1e6 [Lint] pyupgrade (#2819) (40b147e81) by @vmoens ghstack-source-id: dcdf51db31b8f6bcfad7fd4dc53f5b5ad8098c5d [Minor] Quick edits to .md files (#2931) by @vmoens ghstack-source-id: d0fbd3da72d41e6cdbfc4761990cb939993eb816 [Performance] Memoize calls to encode and related methods within step (#2907) (0475cbf64) by @vmoens ghstack-source-id: 8acd4839d4ba5f45373d0a0fcb52b15c149d37f1 [Performance] Use TensorDict._new_unsafe in step (#2905) (e5cba04df) by @vmoens ghstack-source-id: 8a117fb7f9c5b24173408d217f59ec23da7db33c [Quality] Better device checks (#2909) (382430db3) by @vmoens ghstack-source-id: 7174415de2b4221c6c5fca4a31525ed26bc8d6f9 [Quality] Limit warning filter to torchrl (#2762) (85d1e70d3) by @antoinebrl [Quality] Remove redundant return (#2925) (21ef7253c) by @b10902118 [Setup] Fix no_python_abi_suffix error (#2863) (7df831753) by @vmoens ghstack-source-id: 55c845efd936558116f8fdc356f22aca88943f99 [Setup] Remove distutils imports (#2836) (4c55b65b9) by @antoinebrl [Test] Capture deprec warnings (#2799) (fb641de13) by @vmoens ghstack-source-id: bcbf41c245c979d0f21524889ad2be8ef4c10c40 [Test] Fix warnings in tests (#2886) (6f634c6fb) by @vmoens ghstack-source-id: d4ed75d4dae2f0d62adff567d5dcc5fd2f98ce3a [Versioning] Bump 0.8.0 (#2920) by @vmoens ghstack-source-id: 74b3ef75b2b911c097c1d985068958b697a00134

A big thanks to the community supporting this project! There would be no TorchRL if it wasn't for its users.

We are releasing TorchRL 0.7.2, a minor update that addresses several important bug fixes to improve the stability and reliability of our library.

This release is particularly crucial as it resolves a critical issue (#2840) where, under certain conditions, the device setting of the parallel environment would prevent the tensors in the buffers from being properly cloned. This resulted in rollouts returning the same tensor instances across steps, potentially leading to incorrect behavior and results.

Due to the severity of this bug, we strongly recommend that all users upgrade to TorchRL 0.7.2 to ensure the accuracy and reliability of their experiments.

The full list of changes can be found below:

• [Doc] Fix formatting errors by @vmoens (#2786)
• [BugFix] correct dim for resolving dtype in _split_and_pad_sequence by @KubaMichalczyk and vmoens (#2801)
• [BugFix] Fix collector with no buffers and devices by @vmoens (#2809)
• [BE] Fix some typos by antoinebrl and @vmoens (#2811)
• [Doc] Add docstring for MCTSForest.extend by @kurtamohler and @vmoens (#2795)
• [CI] Fix libs workflows by @vmoens (#2800)
• [BugFix] Fix env.full_done_spec~s~ by @vmoens (#2815)
• [BugFix] Fix batch_locked check in check_env_specs + error message ca… by @vmoens (#2817)
• [BugFix] GAE warning when gamma/lmbda are tensors by louisfaury and @vmoens (#2838)
• [BugFix] Tree make node fix by rolo and @vmoens (#2839)
• [BugFix] Fix PEnv device copies by @vmoens (#2840)

Full Changelog: https://github.com/pytorch/rl/compare/v0.7.1...v0.7.2

We are pleased to announce the release of torchrl v0.7.1, which includes several bug fixes, documentation updates, and backend improvements.

Bug Fixes

• Fixed collector timeouts (#2774)
• Fixed composite setitem (#2778)
• Ensured that Composite.set returns self as TensorDict does (#2784)
• Fixed PPOs with composite distribution (#2791)
• Used brackets to get non-tensor data in gym envs (#2769)
• Avoided calling reset during env init (#2770)
• NonTensor should not convert anything to numpy (#2771)

Documentation Updates:

• Fixed tutorials (#2768)
• Solved ref issues in docstrings (#2776)
• Fixed formatting errors (#2786)

Backend Improvements:

• Made better logits in cost tests (#2775)
• Ensured abstractmethods are implemented for specs (#2790)
• Removed deprec specs from tests (#2767)

Thank you to @antoinebrl, and @louisfaury for contributing to this release!

Full Changelog: https://github.com/pytorch/rl/compare/v0.7.0...v0.7.1

As always, we want to warmly thank the RL community who's supporting this project. A special thanks to our first time contributors:

• @priba made their first contribution in https://github.com/pytorch/rl/pull/2543
• @carschandler made their first contribution in https://github.com/pytorch/rl/pull/2545
• @4d616e61 made their first contribution in https://github.com/pytorch/rl/pull/2624
• @valterschutz made their first contribution in https://github.com/pytorch/rl/pull/2626
• @raresdan made their first contribution in https://github.com/pytorch/rl/pull/2616
• @oslumbers made their first contribution in https://github.com/pytorch/rl/pull/2609
• @codingWhale13 made their first contribution in https://github.com/pytorch/rl/pull/2682

as well as all the users who wrote issues, suggestions, started discussions here, on discord, on the pytorch forum or elsewhere! We value your feedback!

BC-Breaking changes and Deprecated behaviors

Removed classes

As announced, we removed the following classes:

• AdditiveGaussianWrapper
• InPlaceSampler
• NormalParamWrapper
• OrnsteinUhlenbeckProcessWrapper

Default MLP config

The default MLP depth has passed from 3 to 0 (i.e., now MLP(in_features=3, out_features=4) is equivalent to a regular nn.Linear layer).

Locking envs

Environments specs are now carefully locked by default (#2729, #2730). This means that

env.observation_spec = spec

is allowed (specs will be unlocked/re-locked automatically) but

env.observation_spec["value"] = spec

will not work. The core idea here is that we want to cache as much info as we can, such as action keys or whether the env has dynamic specs. We can only do that if we can guarantee that the env has not been modified. Locking the specs provides us such guarantee. Note that a version of this already existed but it was not as robust as the new one.

Changes to composite distributions

TL;DR: We're changing the way log-probs and entropies are collected and written in ProbabilisticTensorDictModule and in CompositeDistribution. The "sample_log_prob" default key will soon be "<value>_log_prob (or ("path", "to", "<value>_log_prob") for nested keys). For CompositeDistribution, a different log-prob will be written for each leaf tensor in the distribution. This new behavior is controlled by the tensordict.nn.set_composite_lp_aggregate(mode: bool) function or by the COMPOSITE_LP_AGGREGATE environment variable. We strongly encourage users to adopt the new behavior by setting tensordict.nn.set_composite_lp_aggregate(False).set() at the beginning of their training script.

The behavior of CompositeDistribution and its interaction with on-policy losses such as PPO has changed. The PPO documentation now includes a section about multi-head policies and the examples also give such information.

See the tensordict v0.7.0 release notes or #2707 to know more.

[Deprecation] Change the default MLP depth (#2746) (12e6bce60) by @vmoens ghstack-source-id: bd34b8e9112c4fc3a30bd095e3ac073a7d0b5469 [Deprecation] Gracing old *Spec with v0.8 versioning (#2751) (fa697fe59) by @vmoens ghstack-source-id: e7c6e0a4b8520da887fe7e602a351c3c72a08c4c [Deprecation] Remove AdditiveGaussianWrapper (#2748) (6c7f4fbda) by @vmoens ghstack-source-id: 78f248e1239a04fc5213aa4418a158f741679593 [Deprecation] Remove InPlaceSampler (#2750) (0feef11f9) by @vmoens ghstack-source-id: eeae1bf0611a5d293f533767eee7b9700e720cc8 [Deprecation] Remove NormalParamWrapper (#2747) (a38604e47) by @vmoens ghstack-source-id: 4a70178f54f9e25d602c86a0b61248d66f3e39bd [Deprecation] Remove OrnsteinUhlenbeckProcessWrapper (#2749) (0111a8795) by @vmoens ghstack-source-id: 401fdfaca2e27122d5a67fc7177e1015047f0098

New features

Compile compatibility

We gave a strong focus on a better compatibility with torch.compile across the SOTA training scripts which now all accept a compile=1 argument. The overall speedups range from 1 to 4x

<img width="566" alt="Screenshot 2025-02-05 at 21 20 54" src="https://github.com/user-attachments/assets/bd36ce4d-426e-4d7d-a4da-ff62eee78240" />

Loss module speedups are displayed in the README.md page.

Replay buffers are also mostly compatible with compile now (with the notable exception of distributed and memmaped ones).

Specs: auto_spec_, <attr>_spec_unbatched

You can now use env.auto_spec_ to set the specs automatically based on a dummy rollout.

For batched environments, the unbatched spec can now be accessed via env.<attr>_spec_unbatched. This is useful to create random policies, for example.

New transforms

We added TrajCounter (#2532), Hash and Tokenizer (#2648, #2700) and LineariseReward (#2681).

LazyStackStorage

We provide a new ListStorage-based storage (LazyStackStorage) that automatically represents samples as a LazyStackedTensorDict which makes it easy to store ragged tensors (although not contiguously in memory) #2723.

ChessEnv

A new torchrl.envs.ChessEnv allows users to train agents to play chess!

Tutorials on exporting torchrl modules

We also opensourced a tutorial to export TorchRL modules on hardware: #2557

Full list of features

[Feature, Test] Adding tests for envs that have no specs (#2621) (c72583f75) by @vmoens ghstack-source-id: 4c75691baa1e70f417e518df15c4208cff189950 [Feature,Refactor] Chess improvements: fen, pgn, pixels, san, action mask (#2702) (d425777b8) by @vmoens ghstack-source-id: f294a2bc99a17911c9b62558d530b148d3c0350f [Feature] A2C compatibility with compile (#2464) (507766a88) by @vmoens ghstack-source-id: 66a7f0d1dd82d6463d61c1671e8e0a14ac9a55e7 [Feature] ActionDiscretizer custom sampling (#2609) (3da76f006) @oslumbers Co-authored-by: Oliver Slumbers oliver.slumbers@helsing.ai [Feature] Add Hash transform (#2648) (50011dcf1) @kurtamohler ghstack-source-id: dccf63fe4f9d5f76947ddb7d5dedcff87ff8cdc5 [Feature] Add Choice spec (#2713) (9368ca68e) @kurtamohler ghstack-source-id: afa315a311845ab39ade3e75046f32757f9d94f1 [Feature] Add LossModule.reset_parameters_recursive (#2546) (218d5bf70) by @kurtamohler [Feature] Add Stack transform (#2567) (594462d6b) by @kurtamohler [Feature] Add deterministic_sample to masked categorical (#2708) (49d9897af) by @vmoens ghstack-source-id: d34fcf9b44d7a7c60dbde80b0835189f990ef226 [Feature] Adds ordinal distributions (#2520) (c851e1698) by @louisfaury Co-authored-by: @louisfaury [Feature] Avoid some recompiles of ReplayBuffer.extend/sample (#2504) (0f29c7e93) @kurtamohler [Feature] CQL compatibility with compile (#2553) (e2be42e82) by @vmoens ghstack-source-id: d362d6c17faa0eb609009bce004bb4766e345d5e [Feature] CROSSQ compatibility with compile (#2554) (01a421e76) by @vmoens ghstack-source-id: 98a2b30e8f6a1b0bc583a9f3c51adc2634eb8028 [Feature] CatFrames.make_rb_transform_and_sampler (#2643) (9ee1ae7ee) by @vmoens ghstack-source-id: 7ecf952ec9f102a831aefdba533027ff8c4c29cc [Feature] ChessEnv (#2641) (17983d43e) by @vmoens ghstack-source-id: 087c3b12cd621ea11a252b34c4896133697bce1a [Feature] Composite.batch_size (#2597) (2e82cab19) by @vmoens ghstack-source-id: 621884a559a71e80a4be36c7ba984fd08be47952 [Feature] Composite.pop (#2598) (8d16c12bd) by @vmoens ghstack-source-id: 64d5bd736657ef56e37d57726dfcfd25b16b699f [Feature] Composite.separates (#2599) (83e0b0568) by @vmoens ghstack-source-id: fbfc4308a81cd96ecc61723df8c0eb870c442def [Feature] Custom conversion tool for gym specs (#2726) (dbc8e2ee0) by @vmoens ghstack-source-id: d38bb02f15267a9b1637b3ed25fb44ef013e2456 [Feature] DDPG compatibility with compile (#2555) (7d7cd9538) by @vmoens ghstack-source-id: f18928a419f81794d6870fd4e9fe1205c1b137e1 [Feature] DQN compatibility with compile (#2571) (f149811da) by @vmoens ghstack-source-id: 113dc8c4a5562d217ed867ace1942b2f6b8a39f9 [Feature] DT compatibility with compile (#2556) (fbfe10488) by @vmoens ghstack-source-id: 362b6e88bad4397f35036391729e58f4f7e4a25d [Feature] Discrete SAC compatibility with compile (#2569) (9e2d214fa) by @vmoens ghstack-source-id: ddc131acedbbe451b28758e757a8c240ebd72b80 [Feature] Ensure out-place policy compatibility in rollout and collectors (#2717) (ec370c6b6) by @vmoens ghstack-source-id: 41a6aa56e0a045a20224b96f9537a7ae3ae14494 [Feature] EnvBase.auto_specs_ (#2601) (d537dcb63) by @vmoens ghstack-source-id: 329679238c5172d7ff13097ceaa189479d4f4145 [Feature] EnvBase.check_env_specs (#2600) (00d3199ec) by @vmoens ghstack-source-id: 332dbf92db496c71c5ce6aba340ad123eac0f5d6 [Feature] GAIL compatibility with compile (#2573) (6482766b8) by @vmoens ghstack-source-id: 98c7602ec0343d7a83cb19bddeb579484c42e77e [Feature] IQL compatibility with compile (#2649) (2cfc2abd6) by @vmoens ghstack-source-id: 77bca166701d28dd69ef3964f55ab4f3e4b17fed [Feature] LLMHashingEnv (#2635) (30d21e599) by @vmoens ghstack-source-id: d1a20ecd023008683cf18cf9e694340cfdbdac8a [Feature] LazyStackStorage (#2723) (fe3f00c6c) by @vmoens ghstack-source-id: e9c031470aa0bdafbb2b26c73c06b25685a128e5 [Feature] Linearise reward transform (#2681) (ff1ff7e9c) by @louisfaury Co-authored-by: @louisfaury [Feature] Log each entropy for composite distributions in PPO (#2707) (319bb68f0) by @louisfaury Co-authored-by: @louisfaury [Feature] Log pbar rate in SOTA implementations (#2662) (1ce25f19a) by @vmoens ghstack-source-id: 283cc1bb4ad2d60281296d2cfb78ec41c77f4129 [Feature] MCTSForest (#2307) (e9d167711) by @vmoens ghstack-source-id: 9ac5cd3de39a4dbe1c7c33cb71ff6f45a886ae65 [Feature] Make PPO compatible with composite actions and log-probs (#2665) (256a7002c) by @vmoens ghstack-source-id: c41718e697f9b6edda17d4ddb5bd6d41402b7c30 [Feature] PPO compatibility with compile (#2652) (f5a187d7d) by @vmoens ghstack-source-id: 0ed29f352fcd85f0dc0683d90e95bdbecf6c14f9 [Feature] Re-enable cache for specs (#2730) (4262ab91e) by @vmoens ghstack-source-id: 797132312bfd9749f8926a2dd0b03eff65b8f51c [Feature] SAC compatibility with compile (#2655) (87a59fb30) by @vmoens ghstack-source-id: b57caeaf6e2d3690fb3311f4c9b8cca8575d3974 [Feature] Send info dict to the storage device in RBs (#2527) (d524d0d6b) by @vmoens ghstack-source-id: 4ed60d649b17f96b49f90d234e679937c60a3c32 [Feature] TD3 compatibility with compile (#2658) (1b7eda199) by @vmoens ghstack-source-id: fb94307557f2b8604403b48211e3da6fb2139e28 [Feature] TD3-bc compatibility with compile (#2657) (91064bc27) by @vmoens ghstack-source-id: 8a33e39829f620c1e1a579a0255162ba93eaca91 [Feature] TensorSpec.enumerate() (#2354) (14b63e4f0) by @vmoens ghstack-source-id: 9db2f5ee47a197eb0403cb4622266fb03b99360f [Feature] TrajCounter transform (#2532) (05aeb8975) by @vmoens ghstack-source-id: 62a3091e5c9072f26266143319f30de1729c0d4e [Feature] UnaryTransform for input entries (#2700) (093a1599f) by @vmoens ghstack-source-id: bb0ea97f47bdad6ba5e73692969fece4e2efbfb4 [Feature] example_data for NonTensor spec (#2698) (80690d221) by @vmoens ghstack-source-id: 6fe5d82763dfcc9044d6debe88f0f34bb739c987 [Feature] automatically determine return_contiguous (#2724) (cac93eb0e) by @vmoens ghstack-source-id: 6d1fc31d87cb021e6286cdb07db2d9b0e2302f7d [Feature] env.step_mdp (#2636) (4bc40a808) by @vmoens ghstack-source-id: 145e37cd772fdd74e35e5ffe6accc5c81ad689f3 [Feature] flexible batch_locked for jumanji (#2382) (35a78139b) by @vmoens ghstack-source-id: e356b6511ff3da8a6c583747214cfa90f42c9083 [Feature] lock_ / unlock_ graphs (#2729) (601483e71) by @vmoens ghstack-source-id: 01e375e636b97b26a89f9bbab2e955db6c85978a [Feature] multiagent data standardization: PPO advantages (#2677) (b7a0d11e5) by @matteobettini Co-authored-by: Vincent Moens vmoens@meta.com [Feature] no_cuda_sync arg in collectors (#2727) (280297aee) by @vmoens ghstack-source-id: 9baba31b3ee844882fd4b6a6f69874946caf3b3e [Feature] single_<attr>_spec (#2549) (58c384713) by @vmoens ghstack-source-id: 27e247ea1775e455999a114dd6d95fac748376c4 [Feature] spec.cardinality (#2638) (dd26ae79f) by @vmoens ghstack-source-id: 1160900f8a81dd51dc72436e1af69c8248bff162 [Feature] spec.is_empty(recurse) (#2596) (097d8ad98) by @vmoens ghstack-source-id: faa3b1df5133c77462d6dd013d3854d684cc7e94 [Feature] timeit.printevery (#2653) (187de7c8b) by @vmoens ghstack-source-id: 19165bbfbea5cdc0a6b159493fb02571bab872f3 [Minor,Feature] Add prefix arg to timeit.todict (#2576) (7bc84d15d) by @vmoens ghstack-source-id: f1ff685caf6e8950d02dfc44ad2c1eb496495ad1 [Minor,Feature] group_optimizers (#2577) (7829bd3f3) by @vmoens ghstack-source-id: 81a94ed641544a420bb1c455921ca6a17ecd6a22

Doc

[Doc] Add AOTInductor back (#2564) (9f8f77cdb) by @vmoens ghstack-source-id: 774eb5973045861f284fdc67f74945b1eecdeaf2 [Doc] Add Tokenizer and auto-reset doc link (#2754) (ee4006a6b) by @vmoens ghstack-source-id: 90f55b568e85ae151bea4370025144c19e74602b [Doc] Add Stack transform link in docs (#2689) (c5f1565de) by @kurtamohler [Doc] Adding recurrent policies to export tutorial (#2559) (705123870) by @vmoens ghstack-source-id: 1f1af399b120db8bbb1789748641f44fd3b1bd5e [Doc] Better doc for SliceSampler (#2607) (90572ac11) by @vmoens ghstack-source-id: 7d79ef7d37c4dc2ffbdff5b422cf5da24d93c0da [Doc] Fix broken links and formatting issues in doc (#2574) (5a2d9e205) by @vmoens ghstack-source-id: 4e3f84fe436de6a6e9696894cd06318a98e4a23b [Doc] Fix modules doc (#2531) (edbf3dee3) by @vmoens [Doc] Fix tutorials (#2560) (2f3b4cd4d) by @vmoens ghstack-source-id: 6c9114384015e76e96b3bbd0c8893cc42344537a [Doc] Fix typo in torchrl/modules/distributions/continuous.py (#2624) (b2e9f291a) by @Mana [Doc] Fix typos (#2682) (f672c708f) by Nils Kiele Co-authored-by: Vincent Moens vincentmoens@gmail.com [Doc] MADDPG bug fix of buffer device and improve explaination (#2519) (3e4b2928e) by @matteobettini [Doc] Minor fixes to the docs and type hints (#2548) (50a35f69b) by @thomasbbrunner [Doc] Tutorial on exporting TorchRL models (#2557) (c0187a93e) by @vmoens ghstack-source-id: b93146e22d8376563e7ac302b5cff95f09ae50d4 [Doc] Typo in docs for actors.py (#2545) (19dbeebf0) by @carschandler [Doc] Update docstring for TruncatedNormal with correct parameter names (#2625) (d22266d05) by @valterschutz Co-authored-by: Valter Schutz valterschutz@proton.me [Doc] actor docstrings (#2626) (825779935) by @valterschutz Co-authored-by: Valter Schutz valterschutz@proton.me [Doc] fix several typos (#2603) (de153bf45) by @carschandler [Doc] torchrl_demo.py revamp (#2561) (304e707ef) by @vmoens ghstack-source-id: 2f0087850e4a7d4d4393f0662156af9bfca8e3e1 [Example] Efficient Trajectory Sampling with CompletedTrajRepertoire (#2642) (b840a772c) by @vmoens ghstack-source-id: 4d5c587c69230aa8f3a1b9b6fe19f52fa683d703 [Example] RNN-based policy example (#2675) (d009835b4) by @vmoens ghstack-source-id: ef0087e9b5cba40be428f57ef70ecd2f63483d03 [Example] Using Collector's device args (#2705) (539c2158d) by @vmoens ghstack-source-id: 9aec8daa53000bdfd6091be706c7bc46778d5983

Performance

[Performance] Accelerate slice sampler on GPU (#2672) (84c3ec322) by @vmoens ghstack-source-id: a4dc1515d8b51f5ec150b2fae4e1a84254f2af09 [Performance] Avoid cloning trajs in SliceSampler (#2671) (4fd54fef4) by @vmoens ghstack-source-id: 2e133fcea716b202694cfa84df3f6e4ba3507bbc [Performance] Improve performance of compiled ReplayBuffer (#2529) (2a07f4c0f) by @kurtamohler [Benchmark] Add benchmark for compiled ReplayBuffer.extend/sample (#2514) (5e03a5518) @kurtamohler ghstack-source-id: d4562697e2c1a8392cf5bdcadb50f8b7b6939e41

Better engineering

[BE] Add trailing spaces when necessary (#2581) (600760f5b) by @vmoens ghstack-source-id: 198b5b5668cce8336d44206c10dacb8a9b1a9785 [BE] Add type annotation for tensor_keys to facilitate auto-complete (#2696) (4b3279a3f) by @vmoens ghstack-source-id: b4a8fe38e7c6b028759eef082f65f26036bc0250 [Refactor,CI] Refactor SOTA tests (#2583) (c0ba3ff54) by @vmoens ghstack-source-id: b14c59bb1ca7bf056bde05fa0abd01fa7e9b3710 [Refactor] Allow safe-tanh for torch >= 2.6.0 (#2580) (1474f8517) by @vmoens ghstack-source-id: 92df1954451453ee051bbde499f6db5ebaafed49 [Refactor] Deprecate recurrent_mode API to use decorators/CMs instead (#2584) (14b277513) by @vmoens ghstack-source-id: 80f705e022abc111df3960fc09576d5e266ed4dd [Refactor] Refactor trees (#2634) (57dc25a44) by @vmoens ghstack-source-id: 368ba4c4402b6db0bc8b0688802ce161db9776b7 [Refactor] Rename Recorder and LogReward (#2616) (607ebc52d) by Goia Rares Dan Tiago [Refactor] Use <spec>_unbatched in VMAS (#2593) (a126a6f94) by @vmoens ghstack-source-id: 2190278de44ba59a3bc8d38398fddae9ecc42a84 [Refactor] Use default device instead of CPU in losses (#2687) (c3b9d1dc7) by @vmoens ghstack-source-id: 8b98062c3ae88d8780ef7428fdfa07e305c790b9 [Refactor] compile compatibility improvements (#2578) (db7f08d76) by @vmoens ghstack-source-id: 95f8241b56e42b80e828485cb5f377288bff6f5e [Quality,BE] Better doc for step_mdp (#2639) (ef5a37d8a) by @vmoens ghstack-source-id: 1f5aed6fb2e97ead9d379f9545ae742f7728c585 [Quality] Better TD construction in codebase (#2565) (a4c1ee3b3) by @vmoens ghstack-source-id: 9e280d9d7d4a735e5055beb0450d933547530e55 [Quality] Better warning when c++ binaries failed to be imported (#2541) (0a13cbd5e) by @vmoens [Quality] IMPALA auto-device (#2654) (526b38d5c) by @vmoens ghstack-source-id: abbb3048f33c9f7f6a623e32e139871093ea74fa [Minor] Fix doc and MARL tests (#2759) (ad7d2a10b) by @vmoens ghstack-source-id: 9308be3ebc7fac30b5bde321792eb97069d55996 [Minor] Fix fbcode imports of mocking classes (#2526) (da0bf1897) by @vmoens ghstack-source-id: 74f9f3bedf8f48988a1956084548f6cd2f720934 [Minor] Make fbcode happy with imports (#2517) (a70b258cd) by @vmoens ghstack-source-id: d4bfce9d51269bc0ab6154ee4c2d1e1ff7af0895

Bug fixes

[BugFix, BE] Document and fix fps passing in recorder and loggers (#2694) (61e05b3d9) by @vmoens ghstack-source-id: b3996a9a27643eb5da8a78135f6b9fcef3685f17 [BugFix,Doc] Fix BATCHED_PIPE_TIMEOUT refs and doc (#2695) (dc25a55a7) by @vmoens ghstack-source-id: 6e43c4ff1c319545cf0952abf6f35f3e7ed473e0 [BugFix,Doc] Revert dynamic shape in export tutorial (#2563) (9d292a007) by @vmoens ghstack-source-id: fc856218e840469a5bb0143241d100e9cc612538 [BugFix,Test,Benchmark] Fix graph breaks induced by device context manager (#2602) (152bc81b7) by @vmoens ghstack-source-id: 0df2728928280a43de4abd30afed20826b0af091 [BugFix,Test] test chess rendering (#2721) (ddbb6fdd5) by @vmoens ghstack-source-id: 59b37e6fa2f8c11f600eea334da0bd8257ed382c [BugFix] Account for composite actions in gym (#2718) (1246db197) by @vmoens ghstack-source-id: c09b59904a89d45fa24a61a5e8a24fe307320794 [BugFix] Account for terminating data in SAC losses (#2606) (c8676f4a8) by @vmoens ghstack-source-id: dc1870292786c262b4ab6a221b3afb551e0efb9b [BugFix] ActionDiscretizer scalar integration (#2619) (830f2f26c) by @vmoens ghstack-source-id: b22102f3730914b125ef0f813f4d2f22dec0b26e [BugFix] Allow expanding TensorDictPrimer transforms shape with parent batch size (#2552) (83a7a57da) by Albert Bou Co-authored-by: Vincent Moens vmoens@meta.com [BugFix] Avoid KeyError in slice sampler (for compile) (#2670) (21eeca42c) by @vmoens ghstack-source-id: 6e2a3036f0e50d365387cced50a761b97a47317d [BugFix] Better account of composite distributions in PPO (#2622) (90c8e40f6) by @vmoens ghstack-source-id: 3d86f99bc5b20a53e4092d786e96a5f7e83405ac [BugFix] Compatibility of tensordict primers with batched envs (specifically for LSTM and GRU) (#2668) (f4709c143) by @vmoens ghstack-source-id: e1da58ecfd36ca01b8a11fe90e5f3c5fe77f064c [BugFix] Fix MARL PPO tutorial action_spec call (#2628) (1ca134cc3) by @vmoens ghstack-source-id: 1d9058c45b28c0f0279e4243a2a0f96c622a51d8 [BugFix] Fix batching envs with non tensor data (#2674) (ab4250ec7) by @vmoens ghstack-source-id: daba8a95459cfa978da09291757b6380fab4f308 [BugFix] Fix call to tree.plot in tests (#2547) (09d6866e0) by @vmoens ghstack-source-id: 4a5babbf46294ab6ed4a791e26cfacaf3a41a2e0 [BugFix] Fix collector length with non-empty batch size (#2575) (b87597922) by @vmoens ghstack-source-id: 0c6a7a49f0570fad083340a64dd89c0f4c220c06 [BugFix] Fix compile weakrefs errors (#2742) (ffa99b2a2) by @vmoens ghstack-source-id: 3cb4c62f465a3c0581064b3ff89290b9d225eb3f [BugFix] Fix device transfer for collectors with init_random_frames mixed devices (#2704) (1d45117ba) by @vmoens ghstack-source-id: 1684399a7c84dd19b396db6c903fbf68c971c73d [BugFix] Fix export aoti_compile_and_package API change (#2629) (1cffffee9) by @vmoens ghstack-source-id: 07a0f063f8955815157c2a3eac02c6460a82f672 [BugFix] Fix failing tests (#2582) (863121a27) by @vmoens ghstack-source-id: a43a2e3dbf76cd63c57ae00028df04b41a4e2f2b [BugFix] Fix get_default_device calls in older PT versions (#2586) (705ecc2bb) by @vmoens ghstack-source-id: fd3a739d38feba075073801dda362be598822a94 [BugFix] Fix imports (#2605) (d90b9e3d1) by @vmoens ghstack-source-id: db85f2611c1c0b22e9179b4fdd6c2dcea78ac8dd [BugFix] Fix init_random_frames=0 (#2645) (19dfefc84) by @vmoens ghstack-source-id: 38a544ea15631f9affb4c385c09e7c4df94af55d [BugFix] Fix missing min/max alpha clamps in losses (#2684) (ed656a15f) by @vmoens [BugFix] Fix output of SipHash(as_tensor=False) (#2664) (1fc9577c4) by @kurtamohler [BugFix] Fix partial device transfers in collector (#2703) (afb81de51) by @vmoens ghstack-source-id: 2cd74c2d6fceaf079122ae801b67bdbfc29cddaf [BugFix] Fix pendulum device (#2516) (6799a7f5d) by @vmoens ghstack-source-id: bcaf20de6e317d4bda0e1511e0b1e46653a6f352 [BugFix] Fix safe probabilistic backward by removing in-place modif (#2755) (2f8c118e3) by @vmoens ghstack-source-id: 574eb1f9b662c1eb5be25e97020e11b3fadf625e [BugFix] Fix tests failing because of https://github.com/pytorch/pytorch/pull/137602 (165163abe) by @vmoens [BugFix] Fix typing for python 3.9 (#2631) (e7062a1d6) by @vmoens ghstack-source-id: 663da84096214611804a726e2d38d27a6f21c958 [BugFix] Fix typing in chess env (#2646) (cb8e241b2) by @vmoens ghstack-source-id: ad6086bbb7d1ee528ca24ec1d1232da47372e2b5 [BugFix] Fix typing in llm env (#2647) (e3c304733) by @vmoens ghstack-source-id: b5608f91756b5a81141941903158417a111e0710 [BugFix] Fix version parsing in extensions (#2542) (997d90e1b) by @vmoens ghstack-source-id: 903f2b01b508b81b1b4f92c4297d390da79fe8a2 [BugFix] PettingZoo dict action spaces (#2692) (1a6c9e2d0) by @matteobettini [BugFix] Remove erroneous python 3.8 compatibility classifier (#2540) (528875a9f) by @vmoens [BugFix] Remove raisers in specs (#2651) (bb6f87adb) by @vmoens ghstack-source-id: a005a62847aa2ff1d286f2c4ad13fd14f9e631d3 [BugFix] Rename RayCollector example file to avoid ImportError (#2525) (8eac84ad2) by Albert Bou [BugFix] Support for tensor collection in the PPOLoss (#2543) (0eabb7897) by Pau Riba Co-authored-by: Pau Riba pau.riba@helsing.ai [BugFix] Temporarily remove unsafe caching in envs (#2728) (dc63e820d) by @vmoens ghstack-source-id: a139cf6dc9fcfcfa525a6aa6375163d379593550 [BugFix] Wrong spec returned (#2604) (a1e21f598) by @matteobettini [BugFix] action_spec_unbatched whenever necessary (#2592) (d30599ec0) by @vmoens ghstack-source-id: ec87794dabaf5023dac85cfc898a7c000e93331d [BugFix] adapt log-prob TD batch-size to advantage shape in PPO (#2756) (cb37521e1) by @vmoens ghstack-source-id: 8ccd12f65f4a74a42356a630e0e5a1f015337d4a [BugFix] make buffers zero-dim in exploration modules (#2591) (a47b32c07) by @vmoens ghstack-source-id: fd2705eb9132169da4871b27b354f7895c644061 [BugFix] patch rand_action in TransformedEnv to read the base_env method (#2699) (2c19fcc70) by @vmoens ghstack-source-id: 04e2e85e2675cf34c349ebadb8fa85a5aff2e532 [BugFix] requested_frames_per_batch in distributed collectors (#2579) (408cf7d04) by @vmoens ghstack-source-id: 49289de6956460d9aed13d982eb8003eafc35118 [BugFix] skip_done_states in SAC (#2613) (de61e4d5e) by @vmoens ghstack-source-id: 39d97360e3b0e45dd8c327487eac50ddafe2254d

CI and Tests

[Test] Add tests and a few fixes for ChessEnv (#2661) (7bbd7e3b6) @kurtamohler ghstack-source-id: d0fbb520e35c74305041340722a7560ac2f958f2 [Test] Add tests for CatFrames with PermuteTransform (#2715) (d4e401993) @kurtamohler ghstack-source-id: e554d1cda8d7e4458c9397f1f93345c855e68e5c [Test] Add tests for Tree (#2738) (bb9440b40) @kurtamohler ghstack-source-id: 8f7aa07a4d36aa3664eaa19cc35bd66fb9e61c24 [Test] Fix warnings in SOTA tests (#2710) (a90106475) by @vmoens ghstack-source-id: c79223b5d6548a6c5a6ef649f6eb8e1703258815 [Test] More comprehensive tests for auto_spec (#2640) (6c7d233a4) by @vmoens ghstack-source-id: 75352490436fd706af3d36f9b8016e80a8a3f46a [Test] Skip tokenizer tests if transformers is not in workspace (#2744) (20a19fe2a) by @vmoens ghstack-source-id: b92facfd14cba62511e7888567c94d3986419ab5 [Test] Str2StrEnv test (#2725) (5fd509232) by @vmoens ghstack-source-id: 45a0e5f4b33c4624758171b9fe31f1e3932ff5e4 [CI, BugFix] Py3.8 for old deps (#2568) (f3275dab0) by @vmoens ghstack-source-id: 13c7923c0e5c8725c12c3bacc6c21b250d9f7457 [CI] Change doc image (#2632) (2511c04a5) by @vmoens ghstack-source-id: eceab242294ec55135d79f29e848345a5d5d455e [CI] Cuda 12.4 (#2733) (37a514d6c) by @vmoens ghstack-source-id: 2f3842a17d03e530add9608ee4525347a7c6a0e5 [CI] Fix Cairo-2 Chess import error (#2743) (10f015e0c) by @vmoens ghstack-source-id: c2bcbfc4522bd1b4f1fea3dbb006dc9552b09cb4 [CI] Fix docs upload (#2587) (0f592266f) by @vmoens ghstack-source-id: 49d7df06340fc432c29cd9f2d0ed2ae3d5619a38 [CI] Fix dreamer run in SOTA tests (#2627) (aed03fda4) by @vmoens ghstack-source-id: dfe3ab6fe0d29fcdcaf57f31f84d04e07e36bad3 [CI] Fix nightly build (#2666) (133d70936) by @vmoens ghstack-source-id: 5502fa94b6abcc154e020dcb165093fdc30ca025 [CI] Fix olddeps dm_control (#2734) (3ac61270f) by @vmoens ghstack-source-id: 750edcb8cd6b17167f77fb7c9ebd538608cfbde6 [CI] Fix windows build (#2760) (03f56ffb0) by @vmoens [CI] Install stable torch/tv in docs when on release branch (#2761) (57bdc6aec) by @vmoens ghstack-source-id: 7c39c049c7cff0ee112be2d07597f2e291d2fafd [CI] Local import of PIL (#2720) (d628a507f) by @vmoens ghstack-source-id: 6eb4ace11022632e902a7277dd51344bb9fe1f65 [CI] Longer timeout for windows (#2765) (4c06ce2b8) by @vmoens ghstack-source-id: 381e7e39d650e0178178a78076321a2210237b39 [CI] Make MAX_IDLE_COUNT a feature of tests (#2752) (963f3cdf6) by @vmoens ghstack-source-id: 2bf31dfff3d7862a54abeea86c8c5cc47a0f302d [CI] Remove gym import in test_libs.py (#2719) (f2cf5e044) by @vmoens ghstack-source-id: b0474588cfc81ed135d70efb58203c0b503f4ff0 [CI] Revert upgrade of upload image in docs (#2585) (236d38f8a) by @vmoens ghstack-source-id: f323dd2667a073b6c763ed17a793ecd0eec6b7be [CI] Upgrade GHA versions (#2740) (cd4f359ef) by @vmoens ghstack-source-id: 1876f1f0c18cb11c74edc9d96c17fdc985bc7b1a [CI] Upgrade cu121 to cu124 (#2764) (5da1f6522) by @vmoens ghstack-source-id: 4b3c9c0c31a60a5e151ff13b21e54853dc426416 [CI] Upgrade to v0.7 (#2745) (0ecfbe36e) by @vmoens ghstack-source-id: e548bbbb4578d44a8eee000ab0a40c89713afc27 [CI] linux_job_v2.yml (#2570) (527a26a27) by @vmoens ghstack-source-id: ae13b53bd2885263e80019c087171421f5f7d0d5 [CI] minari[hf] (#2722) (dda0df165) by @vmoens ghstack-source-id: 6eb84d906dfbc66839706f328e214014aef7b65f [CI] workflow permissions (#2706) (b000685f3) by @vmoens ghstack-source-id: f520a1b1e7697b1147cb29e66e2ecb1d07cb4cbc

What's Changed

We introduce wrappers for ML-Agents and OpenSpiel. See the doc here for OpenSpiel and here for MLAgents.

We introduce support for [partial steps](#2377, #2381), allowing you to run rollouts that ends only when all envs are done without resetting those who have reached a termination point.

We add the capability of passing replay buffers directly to data collectors, to avoid inter-process synced communications - thereby drastically speeding up data collection. See the doc of the collectors for more info.

The GAIL algorithm has also been integrated in the library (#2273).

We ensure that all loss modules are compatible with torch.compile without graph breaks (for a typical built). Execution of compiled losses is usually in the range of 2x faster than its eager counterpart.

Finally, we have sadly decided not to support Gymnasium v1.0 and future releases as the new autoreset API is fundamentally incompatible with TorchRL. Furthermore, it does not guarantee the same level of reproducibility as previous releases. See this discussion for more information.

We provide wheels for aarch64 machines, but not being able to upload them to PyPI we provide them attached to these release notes.

Deprecations

• [Deprecation] Deprecate default num_cells in MLP (#2395) by @vmoens
• [Deprecations] Deprecate in view of v0.6 release #2446 by @vmoens

New environments

• [Feature] Add OpenSpielWrapper and OpenSpielEnv (#2345) by @kurtamohler
• [Feature] Add env wrapper for Unity MLAgents (#2469) by @kurtamohler

New features

• [Feature] Add group_map support to MLAgents wrappers (#2491) by @kurtamohler
• [Feature] Add scheduler for alpha/beta parameters of PrioritizedSampler (#2452) Co-authored-by: Vincent Moens by @LTluttmann
• [Feature] Check number of kwargs matches num_workers (#2465) Co-authored-by: Vincent Moens by @antoine.broyelle
• [Feature] Compiled and cudagraph for policies #2478 by @vmoens
• [Feature] Consistent Dropout (#2399) Co-authored-by: Vincent Moens by @depictiger
• [Feature] Deterministic sample for Masked one-hot #2440 by @vmoens
• [Feature] Dict specs in vmas (#2415) Co-authored-by: Vincent Moens by @55539777+matteobettini
• [Feature] Ensure transformation keys have the same number of elements (#2466) by @f.broyelle
• [Feature] Make benchmarked losses compatible with torch.compile #2405 by @vmoens
• [Feature] Partial steps in batched envs #2377 by @vmoens
• [Feature] Pass replay buffers to MultiaSyncDataCollector #2387 by @vmoens
• [Feature] Pass replay buffers to SyncDataCollector #2384 by @vmoens
• [Feature] Prevent loading existing mmap files in storages if they already exist #2438 by @vmoens
• [Feature] RNG for RBs (#2379) by @vmoens
• [Feature] Randint on device for buffers #2470 by @vmoens
• [Feature] SAC compatibility with composite distributions. (#2447) by @albertbou92
• [Feature] Store MARL parameters in module (#2351) by @vmoens
• [Feature] Support wrapping IsaacLab environments with GymEnv (#2380) by @yu-fz
• [Feature] TensorDictMap #2306 by @vmoens
• [Feature] TensorDictMap Query module #2305 by @vmoens
• [Feature] TensorDictMap hashing functions #2304 by @vmoens
• [Feature] break_when_all_done in rollout #2381 by @vmoens
• [Feature] inline hold_out_net #2499 by @vmoens
• [Feature] replay_buffer_chunk #2388 by @vmoens

New Algorithms

• [Algorithm] GAIL (#2273) Co-authored-by: Vincent Moens by @Sebastian.dittert

Fixes

• [BugFix, CI] Set TD_GET_DEFAULTS_TO_NONE=1 in all CIs (#2363) by @vmoens
• [BugFix] Add MultiCategorical support in PettingZoo action masks (#2485) Co-authored-by: Vincent Moens by @matteobettini
• [BugFix] Allow for composite action distributions in PPO/A2C losses (#2391) by @albertbou92
• [BugFix] Avoid reshape(-1) for inputs to DreamerActorLoss (#2496) by @kurtamohler
• [BugFix] Avoid reshape(-1) for inputs to objectives modules (#2494) Co-authored-by: Vincent Moens by @kurtamohler
• [BugFix] Better dumps/loads (#2343) by @vmoens
• [BugFix] Extend RB with lazy stack #2453 by @vmoens
• [BugFix] Extend RB with lazy stack (revamp) #2454 by @vmoens
• [BugFix] Fix Compose input spec transform (#2463) Co-authored-by: Louis Faury @louisfaury
• [BugFix] Fix DeviceCastTransform #2471 by @vmoens
• [BugFix] Fix LSTM in GAE with vmap (#2376) by @vmoens
• [BugFix] Fix MARL-DDPG tutorial and other MODE usages (#2373) by @vmoens
• [BugFix] Fix displaying of tensor sizes in buffers #2456 by @vmoens
• [BugFix] Fix dumps for SamplerWithoutReplacement (#2506) by @vmoens
• [BugFix] Fix get-related errors (#2361) by @vmoens
• [BugFix] Fix invalid CUDA ID error when loading Bounded variables across devices (#2421) by @cbhua
• [BugFix] Fix listing of updated keys in collectors (#2460) by @vmoens
• [BugFix] Fix old deps tests #2500 by @vmoens
• [BugFix] Fix support for MiniGrid envs (#2416) by @kurtamohler
• [BugFix] Fix tictactoeenv.py #2417 by @vmoens
• [BugFix] Fixes to RenameTransform (#2442) Co-authored-by: Vincent Moens by @thomasbbrunner
• [BugFix] Make sure keys are exclusive in envs (#1912) by @vmoens
• [BugFix] TensorDictPrimer updates spec instead of overwriting (#2332) Co-authored-by: Vincent Moens by @matteobettini
• [BugFix] Use a RL-specific NO_DEFAULT instead of TD's one (#2367) by @vmoens
• [BugFix] compatibility to new Composite dist log_prob/entropy APIs #2435 by @vmoens
• [BugFix] torch 2.0 compatibility fix #2475 by @vmoens

Performance

• [Performance] Faster CatFrames.unfolding with padding="same" (#2407) by @kurtamohler
• [Performance] Faster PrioritizedSliceSampler._padded_indices (#2433) by @kurtamohler
• [Performance] Faster SliceSampler._tensor_slices_from_startend (#2423) by @kurtamohler
• [Performance] Faster target update using foreach (#2046) by @vmoens

Documentation

• [Doc] Better doc for inverse transform semantic #2459 by @vmoens
• [Doc] Correct minor erratum in knowledge_base entry (#2383) by @depictiger
• [Doc] Document losses in README.md #2408 by @vmoens
• [Doc] Fix README example (#2398) by @vmoens
• [Doc] Fix links to tutos (#2409) by @vmoens
• [Doc] Fix pip3install typos in Readme (#2342) by @43245438+TheRisenPhoenix
• [Doc] Fix policy in getting started (#2429) by @vmoens
• [Doc] Fix tutorials for release #2476 by @vmoens
• [Doc] Fix wrong default value for flatten_tensordicts in ReplayBufferTrainer (#2502) by @vmoens
• [Doc] Minor fixes to comments and docstrings (#2443) by @thomasbbrunner
• [Doc] Refactor README (#2352) by @vmoens
• [Docs] Use more appropriate ActorValueOperator in PPOLoss documentation (#2350) by @GaetanLepage
• [Documentation] README rewrite and broken links (#2023) by @vmoens

Not user facing

• [CI, BugFix] Fix CI (#2489) by @vmoens
• [CI] Add aarch64-linux wheels (#2434) by @vmoens
• [CI] Disable compile tests on windows #2510 by @vmoens
• [CI] Fix 3.12 gymnasium installation #2474 by @vmoens
• [CI] Fix CI errors (#2394) by @vmoens
• [CI] Fix GPU benchmark upload (#2508) by @vmoens
• [CI] Fix Minari tests (#2419) Co-authored-by: Vincent Moens by @42100908+younik
• [CI] Fix benchmark workflows (#2488) by @vmoens
• [CI] Fix broken workflows (#2418) by @vmoens
• [CI] Fix broken workflows (#2428) by @vmoens
• [CI] Fix gymnasium version in minari #2512 by @vmoens
• [CI] Fix h5py dependency in olddeps #2513 by @vmoens
• [CI] Fix windows build legacy #2450 by @vmoens
• [CI] Fix winndows compile tests #2511 by @vmoens
• [CI] Remove 3.8 jobs #2412 by @vmoens
• [CI] Resolve DMC and mujoco pinned versions (#2396) by @vmoens
• [CI] Run docs on all PRs (#2413) by @vmoens
• [CI] pin DMC and mujoco (#2374) by @vmoens
• [Minor] Fix test_compose_action_spec (#2493) Co-authored-by: Louis Faury by @louisfaury
• [Minor] Fix typos in advantages.py (#2492) Co-authored-by: Louis Faury by @louisfaury
• [Quality] Split utils.h and utils.cpp (#2348) by @vmoens
• [Refactor] Limit the deepcopies in collectors #2451 by @vmoens
• [Refactor] Refactor calls to get without default that raise KeyError (#2353) by @vmoens
• [Refactor] Rename specs to simpler names (#2368) by @vmoens
• [Refactor] Use empty_like in storage construction #2455 by @vmoens
• [Versioning] Fix torch deps (#2340) by @vmoens
• [Versioning] Gymnasium 1.0 incompatibility errors #2484 by @vmoens
• [Versioning] Versions for 0.6 (#2509) by @vmoens

New Contributors

As always, we want to show how appreciative we are of the vibrant open-source community that keeps TorchRL alive.

• @yu-fz made their first contribution in https://github.com/pytorch/rl/pull/2380
• @cbhua made their first contribution in https://github.com/pytorch/rl/pull/2421
• @younik made their first contribution in https://github.com/pytorch/rl/pull/2419
• @thomasbbrunner made their first contribution in https://github.com/pytorch/rl/pull/2442
• @LTluttmann made their first contribution in https://github.com/pytorch/rl/pull/2452
• @louisfaury made their first contribution in https://github.com/pytorch/rl/pull/2463
• @antoinebrl made their first contribution in https://github.com/pytorch/rl/pull/2466

Full Changelog: https://github.com/pytorch/rl/compare/v0.5.0...v0.6.0

What's Changed

This new release makes it possible to run environments that output non-tensor data. #1944

We also introduce dynamic specs, allowing environments to change the size of the observations / actions during the course of a rollout. This feature is compatible with parallel environment and collectors! #2143

Additionally, it is now possible to update a Replay Buffer in-place by assigning values at a given index. #2224

Finally, TorchRL is now compatible with Python 3.12 (#2282, #2281).

As always, a huge thanks to the vibrant OSS community that helps us developt this library!

New algorithms

• [Algorithm] CrossQ by @BY571 in https://github.com/pytorch/rl/pull/2033
• [Algorithm] TD3+BC by @BY571 in https://github.com/pytorch/rl/pull/2249

Features

• [Feature] ActionDiscretizer by @vmoens in https://github.com/pytorch/rl/pull/2247
• [Feature] Add KL approximation in PPO loss metadata by @albertbou92 in https://github.com/pytorch/rl/pull/2166
• [Feature] Add modules.AdditiveGaussianModule by @kurtamohler in https://github.com/pytorch/rl/pull/2296
• [Feature] Add modules.OrnsteinUhlenbeckProcessModule by @kurtamohler in https://github.com/pytorch/rl/pull/2297
• [Feature] Autocomplete for losses by @vmoens in https://github.com/pytorch/rl/pull/2148
• [Feature] Crop Transform by @albertbou92 in https://github.com/pytorch/rl/pull/2336
• [Feature] Dynamic specs by @vmoens in https://github.com/pytorch/rl/pull/2143
• [Feature] Extract primers from modules that contain RNNs by @albertbou92 in https://github.com/pytorch/rl/pull/2127
• [Feature] Jumanji from_pixels=True by @vmoens in https://github.com/pytorch/rl/pull/2129
• [Feature] Make ProbabilisticActor compatible with Composite distributions by @vmoens in https://github.com/pytorch/rl/pull/2220
• [Feature] Replay buffer checkpointers by @vmoens in https://github.com/pytorch/rl/pull/2137
• [Feature] Some improvements to VecNorm by @vmoens in https://github.com/pytorch/rl/pull/2251
• [Feature] Split-trajectories and represent as nested tensor by @vmoens in https://github.com/pytorch/rl/pull/2043
• [Feature] _make_ordinal_device by @vmoens in https://github.com/pytorch/rl/pull/2237
• [Feature] assigning values to RB storage by @vmoens in https://github.com/pytorch/rl/pull/2224

Bug fixes

• [BugFix,Feature] Allow non-tensor data in envs by @vmoens in https://github.com/pytorch/rl/pull/1944
• [BugFix] Allow zero alpha value for PrioritizedSampler by @albertbou92 in https://github.com/pytorch/rl/pull/2164
• [BugFix] Expose MARL modules by @vmoens in https://github.com/pytorch/rl/pull/2321
• [BugFix] Fit vecnorm out_keys by @vmoens in https://github.com/pytorch/rl/pull/2157
• [BugFix] Fix Brax by @vmoens in https://github.com/pytorch/rl/pull/2233
• [BugFix] Fix OOB sampling in PrioritizedSliceSampler by @vmoens in https://github.com/pytorch/rl/pull/2239
• [BugFix] Fix VecNorm test in test_collectors.py by @vmoens in https://github.com/pytorch/rl/pull/2162
• [BugFix] Fix to in MultiDiscreteTensorSpec by @Quinticx in https://github.com/pytorch/rl/pull/2204
• [BugFix] Fix and test PRB priority update across dims and rb types by @vmoens in https://github.com/pytorch/rl/pull/2244
• [BugFix] Fix another ctx test by @vmoens in https://github.com/pytorch/rl/pull/2284
• [BugFix] Fix async gym env with non-sync resets by @vmoens in https://github.com/pytorch/rl/pull/2170
• [BugFix] Fix async gym when all reset by @vmoens in https://github.com/pytorch/rl/pull/2144
• [BugFix] Fix brax wrapping by @vmoens in https://github.com/pytorch/rl/pull/2190
• [BugFix] Fix collector tests where device ordinal is needed by @vmoens in https://github.com/pytorch/rl/pull/2240
• [BugFix] Fix collectors with non tensors by @vmoens in https://github.com/pytorch/rl/pull/2232
• [BugFix] Fix done/terminated computation in slice samplers by @vmoens in https://github.com/pytorch/rl/pull/2213
• [BugFix] Fix info reading with async gym by @vmoens in https://github.com/pytorch/rl/pull/2150
• [BugFix] Fix isaac - bis by @vmoens in https://github.com/pytorch/rl/pull/2119
• [BugFix] Fix lib tests by @vmoens in https://github.com/pytorch/rl/pull/2218
• [BugFix] Fix max value within buffer during update priority by @vmoens in https://github.com/pytorch/rl/pull/2242
• [BugFix] Fix max-priority update by @vmoens in https://github.com/pytorch/rl/pull/2215
• [BugFix] Fix non-tensor passage in _StepMDP by @vmoens in https://github.com/pytorch/rl/pull/2260
• [BugFix] Fix non-tensor passage in _StepMDP by @vmoens in https://github.com/pytorch/rl/pull/2262
• [BugFix] Fix prefetch in samples without replacement - .sample() compatibility issues by @vmoens in https://github.com/pytorch/rl/pull/2226
• [BugFix] Fix sampling in NonTensorSpec by @vmoens in https://github.com/pytorch/rl/pull/2172
• [BugFix] Fix sampling of values from NonTensorSpec by @vmoens in https://github.com/pytorch/rl/pull/2169
• [BugFix] Fix slice sampler end computation at the cursor place by @vmoens in https://github.com/pytorch/rl/pull/2225
• [BugFix] Fix sliced PRB when only traj is provided by @vmoens in https://github.com/pytorch/rl/pull/2228
• [BugFix] Fix strict length in PRB+SliceSampler by @vmoens in https://github.com/pytorch/rl/pull/2202
• [BugFix] Fix strict_length in prioritized slice sampler by @vmoens in https://github.com/pytorch/rl/pull/2194
• [BugFix] Fix tanh normal mode by @vmoens in https://github.com/pytorch/rl/pull/2198
• [BugFix] Fix tensordict private imports by @vmoens in https://github.com/pytorch/rl/pull/2275
• [BugFix] Fix test_specs.py by @vmoens in https://github.com/pytorch/rl/pull/2214
• [BugFix] Fix torch 2.3 compatibility of padding indices by @vmoens in https://github.com/pytorch/rl/pull/2216
• [BugFix] Fix truncated normal by @vmoens in https://github.com/pytorch/rl/pull/2147
• [BugFix] Fix typo in weight assignment in PRB by @vmoens in https://github.com/pytorch/rl/pull/2241
• [BugFix] Fix update_priority generic signature for Samplers by @vmoens in https://github.com/pytorch/rl/pull/2252
• [BugFix] Fix vecnorm state-dicts by @vmoens in https://github.com/pytorch/rl/pull/2158
• [BugFix] Global import of optional library by @matteobettini in https://github.com/pytorch/rl/pull/2217
• [BugFix] Gym async with _reset full of True by @vmoens in https://github.com/pytorch/rl/pull/2145
• [BugFix] MLFlow logger by @GJBoth in https://github.com/pytorch/rl/pull/2152
• [BugFix] Make DMControlEnv aware of truncated signals by @vmoens in https://github.com/pytorch/rl/pull/2196
• [BugFix] Make _reset follow done shape by @matteobettini in https://github.com/pytorch/rl/pull/2189
• [BugFix] EnvBase._complete_done to complete "terminated" key properly by @kurtamohler in https://github.com/pytorch/rl/pull/2294
• [BugFix] LazyTensorStorage only allocates data on the given device by @matteobettini in https://github.com/pytorch/rl/pull/2188
• [BugFix] done = done | truncated in collector by @vmoens in https://github.com/pytorch/rl/pull/2333
• [BugFix] buffer iter for samplers without replacement + prefetch by @JulianKu in https://github.com/pytorch/rl/pull/2185
• [BugFix] buffer __iter__ for samplers without replacement + prefetch by @JulianKu in https://github.com/pytorch/rl/pull/2178
• [BugFix] missing deprecated kwargs by @fedebotu in https://github.com/pytorch/rl/pull/2125

Docs

• [Doc] Add Custom Options for VideoRecorder by @N00bcak in https://github.com/pytorch/rl/pull/2259
• [Doc] Add documentation for masks in tensor specs by @kurtamohler in https://github.com/pytorch/rl/pull/2289
• [Doc] Better doc for make_tensordict_primer by @vmoens in https://github.com/pytorch/rl/pull/2324
• [Doc] Dynamic envs by @vmoens in https://github.com/pytorch/rl/pull/2191
• [Doc] Edit README for local installs by @vmoens in https://github.com/pytorch/rl/pull/2255
• [Doc] Fix algorithms references in tutos by @vmoens in https://github.com/pytorch/rl/pull/2320
• [Doc] Fix documentation mismatch for default argument by @TheRisenPhoenix in https://github.com/pytorch/rl/pull/2149
• [Doc] Fix links in doc by @vmoens in https://github.com/pytorch/rl/pull/2151
• [Doc] Fix mistakes in docs for Trainer checkpointing backends by @kurtamohler in https://github.com/pytorch/rl/pull/2285
• [Doc] Indicate necessary context to run multiprocessed collectors in doc by @GJBoth in https://github.com/pytorch/rl/pull/2126
• [Doc] Restore colab links by @vmoens in https://github.com/pytorch/rl/pull/2197
• [Doc] Update README.md by @KPCOFGS in https://github.com/pytorch/rl/pull/2155
• [Doc] default_interaction_type doc by @vmoens in https://github.com/pytorch/rl/pull/2177
• [Docs] InitTracker cleanup by @matteobettini in https://github.com/pytorch/rl/pull/2330
• [Docs] Reintroduce BenchMARL pointers in MARL tutos by @matteobettini in https://github.com/pytorch/rl/pull/2159

Performance

• [Performance, Refactor, BugFix] Faster loading of uninitialized storages by @vmoens in https://github.com/pytorch/rl/pull/2221
• [Performance] consolidate TDs in ParallelEnv without buffers by @vmoens in https://github.com/pytorch/rl/pull/2231

Others

• Fix "Run in Colab" and "Download Notebook" links in tutorials by @kurtamohler in https://github.com/pytorch/rl/pull/2268
• Fix brax examples by @Jendker in https://github.com/pytorch/rl/pull/2318
• Fixed several broken links in readme.md by @drMJ in https://github.com/pytorch/rl/pull/2156
• Revert "[BugFix] Fix non-tensor passage in _StepMDP" by @vmoens in https://github.com/pytorch/rl/pull/2261
• Revert "[BugFix] Fix tensordict private imports" by @vmoens in https://github.com/pytorch/rl/pull/2276
• Revert "[BugFix] buffer __iter__ for samplers without replacement + prefetch" by @vmoens in https://github.com/pytorch/rl/pull/2182
• [CI, Tests] Fix windows tests by @vmoens in https://github.com/pytorch/rl/pull/2337
• [CI] Bump jinja2 from 3.1.3 to 3.1.4 in /docs by @dependabot in https://github.com/pytorch/rl/pull/2250
• [CI] Fix CI by @vmoens in https://github.com/pytorch/rl/pull/2245
• [CI] Fix nightly by @vmoens in https://github.com/pytorch/rl/pull/2279
• [CI] Fix wheels by @vmoens in https://github.com/pytorch/rl/pull/2274
• [CI] Pin transformers version to < 4.42.0 to make vmap happy by @vmoens in https://github.com/pytorch/rl/pull/2278
• [CI] Upgrade SDL to install pygame 2.6 by @vmoens in https://github.com/pytorch/rl/pull/2248
• [CI] Windows build fix by @vmoens in https://github.com/pytorch/rl/pull/2335
• [CI] python 3.12 nightlies by @vmoens in https://github.com/pytorch/rl/pull/2281
• [Example,BugFix] Add a Async gym env example by @vmoens in https://github.com/pytorch/rl/pull/2139
• [MINOR] Fix unclear language by @software-samurai in https://github.com/pytorch/rl/pull/2165
• [Minor] Code quality improvements by @vmoens in https://github.com/pytorch/rl/pull/2140
• [Quality] Fix low/high in SOTA implementations by @vmoens in https://github.com/pytorch/rl/pull/2266
• [Quality] Fix repr of MARL modules by @vmoens in https://github.com/pytorch/rl/pull/2192
• [Quality] Remove global seeding in set_seed by @vmoens in https://github.com/pytorch/rl/pull/2195
• [Quality] Warn if the sampler is not prioritized but update_priority is called by @vmoens in https://github.com/pytorch/rl/pull/2253
• [Quality] better error message for CompositeSpec shape mismatch by @vmoens in https://github.com/pytorch/rl/pull/2223
• [Refactor] Deprecate NormalParamWrapper by @vmoens in https://github.com/pytorch/rl/pull/2308
• [Refactor] Remove _run_checks from TensorDict.__init__ by @vmoens in https://github.com/pytorch/rl/pull/2256
• [Refactor] Update all instances of exploration *Wrapper to *Module by @kurtamohler in https://github.com/pytorch/rl/pull/2298
• [Refactor] Use td.transpose in multi-step transform by @vmoens in https://github.com/pytorch/rl/pull/2288
• [Refactor] tensordict._tensordict -> tensordict._C by @vmoens in https://github.com/pytorch/rl/pull/2286
• [Tests] Fix VMAS tests by @matteobettini in https://github.com/pytorch/rl/pull/2287
• [Tests] Fix windows tests by @vmoens in https://github.com/pytorch/rl/pull/2219
• [Versioning] Add python 3.12 to setup.py by @vmoens in https://github.com/pytorch/rl/pull/2282
• [Versioning] Allow any torch version for local builds by @vmoens in https://github.com/pytorch/rl/pull/2130
• [Versioning] Bump torch 2.0 as minimal version by @vmoens in https://github.com/pytorch/rl/pull/2200
• [Versioning] v0.5 bump by @vmoens in https://github.com/pytorch/rl/pull/2267
• [Versioning] windows build - add legacy back and .bat env-script by @vmoens in https://github.com/pytorch/rl/pull/2339
• init by @vmoens in https://github.com/pytorch/rl/pull/2322

New Contributors

• @GJBoth made their first contribution in https://github.com/pytorch/rl/pull/2126
• @TheRisenPhoenix made their first contribution in https://github.com/pytorch/rl/pull/2149
• @drMJ made their first contribution in https://github.com/pytorch/rl/pull/2156
• @KPCOFGS made their first contribution in https://github.com/pytorch/rl/pull/2155
• @software-samurai made their first contribution in https://github.com/pytorch/rl/pull/2165
• @JulianKu made their first contribution in https://github.com/pytorch/rl/pull/2178
• @Quinticx made their first contribution in https://github.com/pytorch/rl/pull/2204
• @kurtamohler made their first contribution in https://github.com/pytorch/rl/pull/2268
• @N00bcak made their first contribution in https://github.com/pytorch/rl/pull/2259
• @Jendker made their first contribution in https://github.com/pytorch/rl/pull/2318

Full Changelog: https://github.com/pytorch/rl/compare/v0.4.0...v0.5.0

New Features:

• Better video rendering
  • [Feature] A PixelRenderTransform by @vmoens in https://github.com/pytorch/rl/pull/2099
  • [Feature] Video recording in SOTA examples by @vmoens in https://github.com/pytorch/rl/pull/2070
  • [Feature] VideoRecorder for datasets and replay buffers by @vmoens in https://github.com/pytorch/rl/pull/2069
• Replay buffer: sampling trajectories is now much easier, cleaner and faster
  • [Benchmark] Benchmark slice sampler by @vmoens in https://github.com/pytorch/rl/pull/1992
  • [Feature] Add PrioritizedSliceSampler by @Cadene in https://github.com/pytorch/rl/pull/1875
  • [Feature] Span slice indices on the left and on the right by @vmoens in https://github.com/pytorch/rl/pull/2107
  • [Feature] batched trajectories - SliceSampler compatibility by @vmoens in https://github.com/pytorch/rl/pull/1775
  • [Performance] Faster slice sampler by @vmoens in https://github.com/pytorch/rl/pull/2031
• Datasets: allow preprocessing datasets after download
  • [Feature] Preproc for datasets by @vmoens in https://github.com/pytorch/rl/pull/1989
• Losses: reduction parameters and non-functional execution
  • [Feature] Add reduction parameter to On-Policy losses. by @albertbou92 in https://github.com/pytorch/rl/pull/1890
  • [Feature] Adds value clipping in ClipPPOLoss loss by @albertbou92 in https://github.com/pytorch/rl/pull/2005
  • [Feature] Offline objectives reduction parameter by @albertbou92 in https://github.com/pytorch/rl/pull/1984
• Environment API: support "fork" start method in ParallelEnv, better handling of auto-resetting envs.
  • [Feature] Use non-default mp start method in ParallelEnv by @vmoens in https://github.com/pytorch/rl/pull/1966
  • [Feature] Auto-resetting envs by @vmoens in https://github.com/pytorch/rl/pull/2073
• Transforms
  • [Feature] Allow any callable to be used as transform by @vmoens in https://github.com/pytorch/rl/pull/2027
  • [Feature] invert transforms appended to a RB by @vmoens in https://github.com/pytorch/rl/pull/2111
  • [Feature] Extend TensorDictPrimer default_value options by @albertbou92 in https://github.com/pytorch/rl/pull/2071
  • [Feature] Fine grained DeviceCastTransform by @vmoens in https://github.com/pytorch/rl/pull/2041
  • [Feature] BatchSizeTransform by @vmoens in https://github.com/pytorch/rl/pull/2030
  • [Feature] Allow non-sorted keys in CatFrames by @vmoens in https://github.com/pytorch/rl/pull/1913
  • [Feature] env.append_transform by @vmoens in https://github.com/pytorch/rl/pull/2040
• New environment and improvements:
  • [Environment] Meltingpot by @matteobettini in https://github.com/pytorch/rl/pull/2054
  • [Feature] Return depth from RoboHiveEnv by @sriramsk1999 in https://github.com/pytorch/rl/pull/2058
  • [Feature] PettingZoo possibility to choose reset strategy by @matteobettini in https://github.com/pytorch/rl/pull/2048

Other features

• [Feature] Add time_dim arg in value modules by @vmoens in https://github.com/pytorch/rl/pull/1946
• [Feature] Batched actions wrapper by @vmoens in https://github.com/pytorch/rl/pull/2018
• [Feature] Better repr of RBs by @vmoens in https://github.com/pytorch/rl/pull/1991
• [Feature] Execute rollouts with regular nn.Module instances by @vmoens in https://github.com/pytorch/rl/pull/1947
• [Feature] Logger by @vmoens in https://github.com/pytorch/rl/pull/1858
• [Feature] Passing lists of keyword arguments in reset for batched envs by @vmoens in https://github.com/pytorch/rl/pull/2076
• [Feature] RB MultiStep transform by @vmoens in https://github.com/pytorch/rl/pull/2008
• [Feature] Replace RewardClipping with SignTransform in Atari examples by @albertbou92 in https://github.com/pytorch/rl/pull/1870
• [Feature] reset_parameters for multiagent nets by @matteobettini in https://github.com/pytorch/rl/pull/1970
• [Feature] optionally set truncated = True at the end of rollouts by @vmoens in https://github.com/pytorch/rl/pull/2042

Miscellaneous

• Fix onw typo by @kit1980 in https://github.com/pytorch/rl/pull/1917
• Rename SOTA-IMPLEMENTATIONS.md to README.md by @matteobettini in https://github.com/pytorch/rl/pull/2093
• Revert "[BugFix] Fix Isaac" by @vmoens in https://github.com/pytorch/rl/pull/2118
• Update getting-started-5.py by @vmoens in https://github.com/pytorch/rl/pull/1894
• [BugFix, Performance] Fewer imports at root by @vmoens in https://github.com/pytorch/rl/pull/1930
• [BugFix,CI] Fix Windows CI by @vmoens in https://github.com/pytorch/rl/pull/1983
• [BugFix,CI] Fix sporadically failing tests in CI by @vmoens in https://github.com/pytorch/rl/pull/2098
• [BugFix,Refactor] Dreamer refactor by @BY571 in https://github.com/pytorch/rl/pull/1918
• [BugFix] Adaptable non-blocking for mps and non cuda device in batched-envs by @vmoens in https://github.com/pytorch/rl/pull/1900
• [BugFix] Call contiguous on rollout results in TestMultiStepTransform by @vmoens in https://github.com/pytorch/rl/pull/2025
• [BugFix] Dedicated tests for on policy losses reduction parameter by @albertbou92 in https://github.com/pytorch/rl/pull/1974
• [BugFix] Extend with a list of tensordicts by @vmoens in https://github.com/pytorch/rl/pull/2032
• [BugFix] Fix Atari DQN ensembling by @vmoens in https://github.com/pytorch/rl/pull/1981
• [BugFix] Fix CQL/IQL pbar update by @vmoens in https://github.com/pytorch/rl/pull/2020
• [BugFix] Fix Exclude / Double2Float transforms by @vmoens in https://github.com/pytorch/rl/pull/2101
• [BugFix] Fix Isaac by @vmoens in https://github.com/pytorch/rl/pull/2072
• [BugFix] Fix KLPENPPOLoss KL computation by @vmoens in https://github.com/pytorch/rl/pull/1922
• [BugFix] Fix MPS sync in device transform by @vmoens in https://github.com/pytorch/rl/pull/2061
• [BugFix] Fix OOB TruncatedNormal LP by @vmoens in https://github.com/pytorch/rl/pull/1924
• [BugFix] Fix R2Go once more by @vmoens in https://github.com/pytorch/rl/pull/2089
• [BugFix] Fix Ray collector example error by @albertbou92 in https://github.com/pytorch/rl/pull/1908
• [BugFix] Fix Ray collector on Python > 3.8 by @albertbou92 in https://github.com/pytorch/rl/pull/2015
• [BugFix] Fix RoboHiveEnv tests by @sriramsk1999 in https://github.com/pytorch/rl/pull/2062
• [BugFix] Fix _reset data passing in parallel env by @vmoens in https://github.com/pytorch/rl/pull/1880
• [BugFix] Fix a bug in SliceSampler, indexes outside sampler lengths were produced by @vladisai in https://github.com/pytorch/rl/pull/1874
• [BugFix] Fix args/kwargs passing in advantages by @vmoens in https://github.com/pytorch/rl/pull/2001
• [BugFix] Fix batch-size expansion in functionalization by @vmoens in https://github.com/pytorch/rl/pull/1959
• [BugFix] Fix broken gym tests by @vmoens in https://github.com/pytorch/rl/pull/1980
• [BugFix] Fix clip_fraction in PO losses by @vmoens in https://github.com/pytorch/rl/pull/2021
• [BugFix] Fix colab in tutos by @vmoens in https://github.com/pytorch/rl/pull/2113
• [BugFix] Fix env.shape regex matches by @vmoens in https://github.com/pytorch/rl/pull/1940
• [BugFix] Fix examples by @vmoens in https://github.com/pytorch/rl/pull/1945
• [BugFix] Fix exploration in losses by @vmoens in https://github.com/pytorch/rl/pull/1898
• [BugFix] Fix flaky rb tests by @vmoens in https://github.com/pytorch/rl/pull/1901
• [BugFix] Fix habitat by @vmoens in https://github.com/pytorch/rl/pull/1941
• [BugFix] Fix jumanji by @vmoens in https://github.com/pytorch/rl/pull/2064
• [BugFix] Fix load_state_dict and is_empty td bugfix impact by @vmoens in https://github.com/pytorch/rl/pull/1869
• [BugFix] Fix mp_start_method for ParallelEnv with single_for_serial by @vmoens in https://github.com/pytorch/rl/pull/2007
• [BugFix] Fix multiple context syntax in multiagent examples by @matteobettini in https://github.com/pytorch/rl/pull/1943
• [BugFix] Fix offline CatFrames by @vmoens in https://github.com/pytorch/rl/pull/1953
• [BugFix] Fix offline CatFrames for pixels by @vmoens in https://github.com/pytorch/rl/pull/1964
• [BugFix] Fix prints of size error when no file is associated with memmap by @vmoens in https://github.com/pytorch/rl/pull/2090
• [BugFix] Fix replay buffer extension with lists by @vmoens in https://github.com/pytorch/rl/pull/1937
• [BugFix] Fix reward2go for nd tensors by @vmoens in https://github.com/pytorch/rl/pull/2087
• [BugFix] Fix robohive by @vmoens in https://github.com/pytorch/rl/pull/2080
• [BugFix] Fix sampling without replacement with ndim storages by @vmoens in https://github.com/pytorch/rl/pull/1999
• [BugFix] Fix slice sampler compatibility with split_trajs and MultiStep by @vmoens in https://github.com/pytorch/rl/pull/1961
• [BugFix] Fix slicesampler terminated/truncated signaling by @vmoens in https://github.com/pytorch/rl/pull/2044
• [BugFix] Fix strict-length for spanning trajectories by @vmoens in https://github.com/pytorch/rl/pull/1982
• [BugFix] Fix strict_length=True in SliceSampler by @vmoens in https://github.com/pytorch/rl/pull/2037
• [BugFix] Fix unwanted lazy stacks by @vmoens in https://github.com/pytorch/rl/pull/2102
• [BugFix] Fix update in serial / parallel env by @vmoens in https://github.com/pytorch/rl/pull/1866
• [BugFix] Fix vmas stacks by @vmoens in https://github.com/pytorch/rl/pull/2105
• [BugFix] Fixed import for importlib by @DanilBaibak in https://github.com/pytorch/rl/pull/1914
• [BugFix] Make KL-controllers independent of the model by @vmoens in https://github.com/pytorch/rl/pull/1903
• [BugFix] Make sure ParallelEnv does not overflow mem when policy requires grad by @vmoens in https://github.com/pytorch/rl/pull/1909
• [BugFix] More robust _StepMDP and multi-purpose envs by @vmoens in https://github.com/pytorch/rl/pull/2038
• [BugFix] No grad on collector reset by @matteobettini in https://github.com/pytorch/rl/pull/1927
• [BugFix] Non exclusive terminated and truncated by @vmoens in https://github.com/pytorch/rl/pull/1911
• [BugFix] Refactor reductions by @vmoens in https://github.com/pytorch/rl/pull/1968
• [BugFix] Remove split_trajectories's reference to ("next", "done"). by @initmaks in https://github.com/pytorch/rl/pull/2094
• [BugFix] Remove reset on last step of a rollout by @matteobettini in https://github.com/pytorch/rl/pull/1936
• [BugFix] Robust sync for non_blocking=True by @vmoens in https://github.com/pytorch/rl/pull/2034
• [BugFix] Set default value for normalize_advantage to False. by @DobromirM in https://github.com/pytorch/rl/pull/2050
• [BugFix] Set strict=False in tensordict.select() calls for objective classes by @albertbou92 in https://github.com/pytorch/rl/pull/2004
• [BugFix] SliceSampler device and index mesh by @vmoens in https://github.com/pytorch/rl/pull/1996
• [BugFix] Solve recursion issue in losses hook by @vmoens in https://github.com/pytorch/rl/pull/1897
• [BugFix] Update cql docstring example by @BY571 in https://github.com/pytorch/rl/pull/1951
• [BugFix] Update iql docstring example by @BY571 in https://github.com/pytorch/rl/pull/1950
• [BugFix] Use same signature for append_transform in all cases by @vmoens in https://github.com/pytorch/rl/pull/2091
• [BugFix] Use setdefault in _cache_values by @vmoens in https://github.com/pytorch/rl/pull/1910
• [BugFix] Use traj_terminated in SliceSampler by @Cadene in https://github.com/pytorch/rl/pull/1884
• [BugFix] Vmap randomness for value estimator by @BY571 in https://github.com/pytorch/rl/pull/1942
• [BugFix] better device consistency in EGreedy by @vmoens in https://github.com/pytorch/rl/pull/1867
• [BugFix] check_env_specs seeding logic by @vmoens in https://github.com/pytorch/rl/pull/1872
• [BugFix] fix formatting for VideoRecorder docstring by @sriramsk1999 in https://github.com/pytorch/rl/pull/1985
• [BugFix] fix trunc normal device by @vmoens in https://github.com/pytorch/rl/pull/1931
• [BugFix] missing annotations import by @vmoens in https://github.com/pytorch/rl/pull/2074
• [BugFix] state typo in RNG control module by @vmoens in https://github.com/pytorch/rl/pull/1878
• [BugFix] to_observation_norm now works with keys which are not strings by @maxweissenbacher in https://github.com/pytorch/rl/pull/2045
• [BugFix] union -> intersection in _StepMDP check by @vmoens in https://github.com/pytorch/rl/pull/2039
• [CI,Doc] Sanitize version by @vmoens in https://github.com/pytorch/rl/pull/2120
• [CI] Doc on release tag by @vmoens in https://github.com/pytorch/rl/pull/2116
• [CI] Fix CI issues by @vmoens in https://github.com/pytorch/rl/pull/2084
• [CI] Fix Doc CI by @matteobettini in https://github.com/pytorch/rl/pull/2106
• [CI] Fixes sympy error by fixing mpmath version by @vmoens in https://github.com/pytorch/rl/pull/1988
• [CI] Install ffmpeg in Robohive tests by @vmoens in https://github.com/pytorch/rl/pull/2063
• [CI] Install stable torch and tensordict for release tests by @vmoens in https://github.com/pytorch/rl/pull/1978
• [CI] Remove all macos x86 jobs by @vmoens in https://github.com/pytorch/rl/pull/2117
• [CI] Remove x86 OSX jobs by @vmoens in https://github.com/pytorch/rl/pull/2112
• [CI] Schedule workflows for releases by @vmoens in https://github.com/pytorch/rl/pull/2114
• [CI] Temporarily remove snapshot from CI by @vmoens in https://github.com/pytorch/rl/pull/2000
• [CI] Unpin mpmath by @vmoens in https://github.com/pytorch/rl/pull/1997
• [CI] Upgrade 3.8 to 3.10 GPU jobs by @vmoens in https://github.com/pytorch/rl/pull/2013
• [Deprecation] Deprecate in prep for release by @vmoens in https://github.com/pytorch/rl/pull/1820
• [Doc,Feature] Better doc for modules and list of kwargs when possible by @vmoens in https://github.com/pytorch/rl/pull/1990
• [Doc] Fix tutos by @vmoens in https://github.com/pytorch/rl/pull/1863
• [Doc] Getting started tutos by @vmoens in https://github.com/pytorch/rl/pull/1886
• [Doc] Improve PrioritizedSampler doc and get rid of np dependency as much as possible by @vmoens in https://github.com/pytorch/rl/pull/1881
• [Doc] Installation instructions in API ref by @vmoens in https://github.com/pytorch/rl/pull/1871
• [Doc] Per-release doc by @vmoens in https://github.com/pytorch/rl/pull/2108
• [Documentation] Correct MaskedEnv Example in ActionMask Transform Documentation by @Jonathanace in https://github.com/pytorch/rl/pull/2060
• [Examples] Move examples to sota-implementations by @vmoens in https://github.com/pytorch/rl/pull/2016
• [Minor] Add env.shape attribute by @vmoens in https://github.com/pytorch/rl/pull/1938
• [Minor] Lint by @vmoens in https://github.com/pytorch/rl/pull/2096
• [Minor] Move distributed examples to examples by @vmoens in https://github.com/pytorch/rl/pull/2097
• [Minor] Remove duplicate if statement in storages by @vmoens in https://github.com/pytorch/rl/pull/2066
• [Minor] Remove warnings in test_cost by @vmoens in https://github.com/pytorch/rl/pull/1902
• [Minor] Support init lazy storages with add by @vmoens in https://github.com/pytorch/rl/pull/2028
• [Minor] Use the main branch for the M1 build wheels by @DanilBaibak in https://github.com/pytorch/rl/pull/1965
• [Performance] Faster DMC by @vmoens in https://github.com/pytorch/rl/pull/2002
• [Quality] Capture errors in specs transforms by @vmoens in https://github.com/pytorch/rl/pull/2092
• [Quality] Make sure deprec warnings are displayed by @vmoens in https://github.com/pytorch/rl/pull/2088
• [Refactor,Feature] Refactor collector shapes and stack_result in sync collector by @vmoens in https://github.com/pytorch/rl/pull/1994
• [Refactor] Clearer separation between single_task and share_individual_td by @vmoens in https://github.com/pytorch/rl/pull/2026
• [Refactor] Faster and more generic multi-agent nets by @vmoens in https://github.com/pytorch/rl/pull/1921
• [Refactor] Refactor split_trajectories by @vmoens in https://github.com/pytorch/rl/pull/1955
• [Refactor] Remove remnant legacy functional calls by @vmoens in https://github.com/pytorch/rl/pull/1973
• [Refactor] Use filter_empty=False in apply for params by @vmoens in https://github.com/pytorch/rl/pull/1882
• [Refactor] Use filter_empty=True in apply by @vmoens in https://github.com/pytorch/rl/pull/1879
• [Tutorial] PettingZoo Parallel competitive tutorial by @matteobettini in https://github.com/pytorch/rl/pull/2047
• [Versioning] Deprecations for 0.4 by @vmoens in https://github.com/pytorch/rl/pull/2109
• [Versioning] New torch version by @vmoens in https://github.com/pytorch/rl/pull/2110
• [Versioning] v0.4.0 by @vmoens in https://github.com/pytorch/rl/pull/1860

New Contributors

• @vladisai made their first contribution in https://github.com/pytorch/rl/pull/1874
• @Cadene made their first contribution in https://github.com/pytorch/rl/pull/1884
• @sriramsk1999 made their first contribution in https://github.com/pytorch/rl/pull/1985
• @DobromirM made their first contribution in https://github.com/pytorch/rl/pull/2050
• @Jonathanace made their first contribution in https://github.com/pytorch/rl/pull/2060
• @maxweissenbacher made their first contribution in https://github.com/pytorch/rl/pull/2045
• @initmaks made their first contribution in https://github.com/pytorch/rl/pull/2094

A big thanks to our dear contributors as well as the entire user base for helping with this lib!

Full Changelog: https://github.com/pytorch/rl/compare/v0.3.0...v0.4.0

This release provides a bunch of bug fixes and speedups.

What's Changed

[BugFix] Fix broken gym tests (#1980) [BugFix,CI] Fix Windows CI (#1983) [Minor] Cleanup [CI] Install stable torch and tensordict for release tests (#1978) [Refactor] Remove remnant legacy functional calls (#1973) [Minor] Use the main branch for the M1 build wheels (#1965) [BugFix] Fixed import for importlib (#1914) [BugFix] Fix offline CatFrames for pixels (#1964) [BugFix] Fix offline CatFrames (#1953) [BugFix] Fix batch-size expansion in functionalization (#1959) [BugFix] Update iql docstring example (#1950) [BugFix] Update cql docstring example (#1951) [BugFix] Fix examples (#1945) [BugFix] Remove reset on last step of a rollout (#1936) [BugFix] Vmap randomness for value estimator (#1942) [BugFix] Fix multiple context syntax in multiagent examples (#1943) [BugFix] Fix habitat (#1941) [BugFix] Fix env.shape regex matches (#1940) [Minor] Add env.shape attribute (#1938) [BugFix] Fix replay buffer extension with lists (#1937) [BugFix] No grad on collector reset (#1927) [BugFix] fix trunc normal device (#1931) [BugFix, Performance] Fewer imports at root (#1930) [BugFix] Fix OOB TruncatedNormal LP (#1924) [BugFix] Fix KLPENPPOLoss KL computation (#1922) [Doc] Fix onw typo (#1917) [BugFix] Make sure ParallelEnv does not overflow mem when policy requires grad (#1909) [BugFix] Non exclusive terminated and truncated (#1911) [BugFix] Use setdefault in _cache_values (#1910) [BugFix] Fix Ray collector example error (#1908) [BugFix] Make KL-controllers independent of the model (#1903) [Minor] Remove warnings in test_cost (#1902) [BugFix] Adaptable non-blocking for mps and non cuda device in batched-envs (#1900) [BugFix] Fix flaky rb tests (#1901) [BugFix] Fix exploration in losses (#1898) [BugFix] Solve recursion issue in losses hook (#1897) [Doc] Update getting-started-5.py (#1894) [Doc] Getting started tutos (#1886) [BugFix] Use traj_terminated in SliceSampler (#1884) [Doc] Improve PrioritizedSampler doc and get rid of np dependency as much as possible (#1881) [BugFix] Fix _reset data passing in parallel env (#1880) [BugFix] state typo in RNG control module (#1878) [BugFix] Fix a bug in SliceSampler, indexes outside sampler lengths were produced (#1874) [BugFix] check_env_specs seeding logic (#1872) [BugFix] Fix update in serial / parallel env (#1866) [Doc] Installation instructions in API ref (#1871) [BugFix] better device consistency in EGreedy (#1867) [BugFix] Fix load_state_dict and is_empty td bugfix impact (#1869) [Doc] Fix tutos (#1863)

Full Changelog: https://github.com/pytorch/rl/compare/v0.3.0...v0.3.1

In this release, we focused on building a Data Hub for offline RL, providing a universal <lib>2gym conversion tool (#1795) and improving the doc.

TorchRL Data Hub

TorchRL now offers many offline datasets in robotics and control or gaming, all under a single data format (TED for TorchRL Episode Data Format). All datasets are one step away of being downloaded: dataset = <Name>ExperienceReplay(dataset_id, root="/path/to/storage", download=True) is all you need to get started. This means that you can now download OpenX #1751 or Roboset #1743 datasets and combine them in a single replay buffer #1768 or swap one another in no time and with no extra code. We allow many new sampling techniques, like sampling slices of trajectories with or without repetition etc. As always you can append your favourite transform to these transforms.

TorchRL2Gym universal converter

#1795 introduces a new universal converter for simulation libraries to gym. As RL practitioner, it's sometimes difficult to accommodate for the many different environment APIs that exist. TorchRL now provides a way of registering any env in gym(nasium). This allows users to build their dataset in torchrl and integrate them in their code base with no effort if they are already using gym as a backend. It also allows to transform DMControl or Brax envs (among others) to gym without the need for an extra library.

PPO and A2C compatibility with distributed models

Functional calls can now be turned off for PPO and A2C loss modules, allowing users to run RLHF training loops at scale! #1804

## TensorDict-free replay buffers

You can now use TorchRL's replay buffer with ANY tensor-based structure, whether it involves dict, tuples or lists. In principle, storing data contiguously on disk given any gym environment is as simple as

rb = ReplayBuffer(storage=LazyMemmapStorage(capacity))
obs_, reward, terminal, truncated, info = env.step(action)
rb.add((obs, obs_, reward, terminal, truncated, info, action))

# sampling a tuple obs, reward, terminal, truncated, info
obs, obs_, reward, terminal, truncated, info = rb.sample()

This is independent of TensorDict and it supports many components of our replay buffers as well as transforms. Check the doc here.

## Multiprocessed replay buffers

TorchRL's replay buffers can now be shared across processes. Multiprocessed RBs can not only be read from but also extended on different workers. #1724

SOTA checks

We introduce a list of scripts to check that our training scripts work ok before each release: #1822

Throughput of Gym and DMControl

We removed loads of checks in GymLikeEnv if some basic conditions are met, which improves the throughput significantly for simple envs. #1803

## Algorithms

We introduce discrete CQL #1666 , discrete IQL #1793 and Impala #1506.

What's Changed: PR description

• [BugFix] Fix incorrect deprecation warning by @mikemykhaylov in https://github.com/pytorch/rl/pull/1655
• [Bug] TensorDictMaxValueWriter raises error when no sample in a batch is accepted by @albertbou92 in https://github.com/pytorch/rl/pull/1664
• [BugFix] Fix "done" instead of "terminated" mistakes by @MarCnu in https://github.com/pytorch/rl/pull/1661
• [Feature] CatFrames constant padding by @albertbou92 in https://github.com/pytorch/rl/pull/1663
• doc(README): remove typo by @Deep145757 in https://github.com/pytorch/rl/pull/1665
• [Docs] Update README.md by @vaibhav-009 in https://github.com/pytorch/rl/pull/1667
• [Minor] Update dreamer example tests by @vmoens in https://github.com/pytorch/rl/pull/1668
• [Feature] Introduce grouping in VMAS by @matteobettini in https://github.com/pytorch/rl/pull/1658
• [BugFix] assertion error message, envs/util.py by @laszloKopits in https://github.com/pytorch/rl/pull/1669
• [Doc] Set action_spec instead of input_spec by @FrankTianTT in https://github.com/pytorch/rl/pull/1657
• [BugFix] Fix submitit IP address/node name retrieval by @vmoens in https://github.com/pytorch/rl/pull/1672
• [Doc] Document (and test) compound actor by @vmoens in https://github.com/pytorch/rl/pull/1673
• [Doc] Update rollout_recurrent.png to account for terminal by @vmoens in https://github.com/pytorch/rl/pull/1677
• [Doc] Add EGreedyWrapper back in the doc by @vmoens in https://github.com/pytorch/rl/pull/1680
• [Doc] Fix TanhDelta docstring by @matteobettini in https://github.com/pytorch/rl/pull/1683
• [Doc] Add discord badge on README by @vmoens in https://github.com/pytorch/rl/pull/1686
• [CI] Downgrade RAY to fix CI by @vmoens in https://github.com/pytorch/rl/pull/1687
• [BugFix] MaxValueWriter cuda compatibility by @albertbou92 in https://github.com/pytorch/rl/pull/1689
• Upload docs for preview on HUD by @DanilBaibak in https://github.com/pytorch/rl/pull/1682
• [Doc] Update pendulum and rnn tutos by @vmoens in https://github.com/pytorch/rl/pull/1691
• [Algorithm] Discrete CQL by @BY571 in https://github.com/pytorch/rl/pull/1666
• [BugFix] Minor fix in the logging of PPO and A2C examples by @albertbou92 in https://github.com/pytorch/rl/pull/1693
• [CI] Enable retry mechanism by @DanilBaibak in https://github.com/pytorch/rl/pull/1681
• [Refactor] Minor changes in prep of https://github.com/pytorch/tensordict/pull/541 by @vmoens in https://github.com/pytorch/rl/pull/1696
• [BugFix] fix dreamer actor by @FrankTianTT in https://github.com/pytorch/rl/pull/1697
• [Refactor] Deprecate direct usage of memmap tensors by @vmoens in https://github.com/pytorch/rl/pull/1684
• Revert "[Refactor] Deprecate direct usage of memmap tensors" by @vmoens in https://github.com/pytorch/rl/pull/1698
• [Refactor] Deprecate direct usage of memmap tensors by @vmoens in https://github.com/pytorch/rl/pull/1699
• [Doc] Fix discord link by @vmoens in https://github.com/pytorch/rl/pull/1701
• [BugFix] make sure the params of exploration-wrapper is float by @FrankTianTT in https://github.com/pytorch/rl/pull/1700
• [Fix] EndOfLifeTransform fix in end of life detection by @albertbou92 in https://github.com/pytorch/rl/pull/1705
• [CI] Fix benchmark on gpu by @vmoens in https://github.com/pytorch/rl/pull/1706
• [Algorithm] IMPALA and VTrace module by @albertbou92 in https://github.com/pytorch/rl/pull/1506
• [Doc] Fix discord link by @vmoens in https://github.com/pytorch/rl/pull/1712
• [Refactor] Refactor functional calls in losses by @vmoens in https://github.com/pytorch/rl/pull/1707
• [CI] Fix CI by @vmoens in https://github.com/pytorch/rl/pull/1711
• [BugFix] Make casting to 'meta' device uniform across cost modules by @vmoens in https://github.com/pytorch/rl/pull/1715
• [BugFix] Change ppo mujoco example to match paper results by @albertbou92 in https://github.com/pytorch/rl/pull/1714
• [Minor] Hide params in ddpg actor-critic by @vmoens in https://github.com/pytorch/rl/pull/1716
• [BugFix] Fix hold_out_net by @vmoens in https://github.com/pytorch/rl/pull/1719
• [BugFix] RewardSum key check by @matteobettini in https://github.com/pytorch/rl/pull/1718
• [Feature] Allow usage of a different device on main and sub-envs in ParallelEnv and SerialEnv by @vmoens in https://github.com/pytorch/rl/pull/1626
• [Refactor] Better weight update in collectors by @vmoens in https://github.com/pytorch/rl/pull/1723
• [Feature] Shared replay buffers by @vmoens in https://github.com/pytorch/rl/pull/1724
• [CI] FIx nightly builds on osx by @vmoens in https://github.com/pytorch/rl/pull/1726
• [BugFix] _call_actor_net does not handle multiple inputs by @albertbou92 in https://github.com/pytorch/rl/pull/1728
• [Feature] Python-based RNN Modules by @albertbou92 in https://github.com/pytorch/rl/pull/1720
• [BugFix, Test] Fix flaky gym vecenvs tests by @vmoens in https://github.com/pytorch/rl/pull/1727
• [BugFix] Fix non-full TensorStorage indexing by @vmoens in https://github.com/pytorch/rl/pull/1730
• [Feature] Minari datasets by @vmoens in https://github.com/pytorch/rl/pull/1721
• [Feature] All VMAS scenarios available by @matteobettini in https://github.com/pytorch/rl/pull/1731
• [Feature] pickle-free RB checkpointing by @vmoens in https://github.com/pytorch/rl/pull/1733
• [CI] Fix doc upload by @vmoens in https://github.com/pytorch/rl/pull/1738
• [BugFix] Fix RNNs trajectory split in VMAP calls by @vmoens in https://github.com/pytorch/rl/pull/1736
• [CI] Fix doc upload by @vmoens in https://github.com/pytorch/rl/pull/1739
• [BugFix, Feature] Fix DDQN implementation by @vmoens in https://github.com/pytorch/rl/pull/1737
• [Algorithm] Update DQN example by @albertbou92 in https://github.com/pytorch/rl/pull/1512
• [BugFix] Use rsync in doc workflow by @vmoens in https://github.com/pytorch/rl/pull/1741
• [BugFix] Fix compat with new memmap API by @vmoens in https://github.com/pytorch/rl/pull/1744
• [Feature] Roboset datasets by @vmoens in https://github.com/pytorch/rl/pull/1743
• [Algorithm] Simpler IQL example by @BY571 in https://github.com/pytorch/rl/pull/998
• [Performance] Faster RNNs by @vmoens in https://github.com/pytorch/rl/pull/1732
• [BugFix, Test] Fix torch.vmap call in RNN tests by @vmoens in https://github.com/pytorch/rl/pull/1749
• [BugFix] Fix discrete SAC log-prob by @vmoens in https://github.com/pytorch/rl/pull/1750
• [Minor] Remove dead code in RolloutFromModel by @ianbarber in https://github.com/pytorch/rl/pull/1752
• [Minor] Fix runnability of RLHF example in examples/rlhf by @ianbarber in https://github.com/pytorch/rl/pull/1753
• [Feature] SliceSampler by @vmoens in https://github.com/pytorch/rl/pull/1748
• [CI] Fix windows CI by @vmoens in https://github.com/pytorch/rl/pull/1746
• [CI] Fix CI for optional dependencies by @vmoens in https://github.com/pytorch/rl/pull/1754
• [Feature] V-D4RL by @vmoens in https://github.com/pytorch/rl/pull/1756
• [Benchmark] Fix RB benchmarks by @vmoens in https://github.com/pytorch/rl/pull/1760
• [BugFix] Fix RLHF by @vmoens in https://github.com/pytorch/rl/pull/1757
• [BugFix] Fix slice sampler by @vmoens in https://github.com/pytorch/rl/pull/1762
• [Feature] BurnInTransform by @albertbou92 in https://github.com/pytorch/rl/pull/1765
• [Bug] Minor change burnin transform by @albertbou92 in https://github.com/pytorch/rl/pull/1770
• [BugFix] Fix sampling of last item in SliceSampler by @vmoens in https://github.com/pytorch/rl/pull/1774
• [Feature] Open-X Embodiement datasets by @vmoens in https://github.com/pytorch/rl/pull/1751
• [BugFix] Fix documentation of threads for batched envs. by @skandermoalla in https://github.com/pytorch/rl/pull/1776
• [BugFix, CI] Fix OpenML datasets runs by @vmoens in https://github.com/pytorch/rl/pull/1779
• [Versioning] Bump v0.3.0 and fix m1-wheels by @vmoens in https://github.com/pytorch/rl/pull/1780
• [Feature] Composite replay buffers by @vmoens in https://github.com/pytorch/rl/pull/1768
• [BugFix, Feature] Vmap randomness in losses by @BY571 in https://github.com/pytorch/rl/pull/1740
• [Algorithm] Update discrete SAC example by @BY571 in https://github.com/pytorch/rl/pull/1745
• [Docs] Pointers to BenchMARL by @matteobettini in https://github.com/pytorch/rl/pull/1710
• [Feature] Immutable writer for datasets by @vmoens in https://github.com/pytorch/rl/pull/1781
• [Feature] Remove and check for prints in codebase using flake8-print by @vmoens in https://github.com/pytorch/rl/pull/1758
• [BUG] Missing import for some Samplers in Data module by @albertbou92 in https://github.com/pytorch/rl/pull/1784
• [BugFix] Ensure that infos and samples have the same batch-size in SamplerEnsemble by @vmoens in https://github.com/pytorch/rl/pull/1786
• [BugFix] Writers extend() method should always return indices in data.device by @albertbou92 in https://github.com/pytorch/rl/pull/1785
• [Doc] Revamp envs doc by @vmoens in https://github.com/pytorch/rl/pull/1787
• [BugFix] Less flaky gym vecenv test by @vmoens in https://github.com/pytorch/rl/pull/1790
• [CI] Regroup tests by @vmoens in https://github.com/pytorch/rl/pull/1791
• [CI] Remove stable GPU tests from CI by @vmoens in https://github.com/pytorch/rl/pull/1792
• Update README.md to fix CI banner by @vmoens in https://github.com/pytorch/rl/pull/1794
• [Feature] SamplerWithoutReplacement state dictionary by @matteobettini in https://github.com/pytorch/rl/pull/1788
• [BugFix] Higher time threshold for PEnv by @vmoens in https://github.com/pytorch/rl/pull/1799
• [Feature] SignTransform by @albertbou92 in https://github.com/pytorch/rl/pull/1798
• [Feature] Extend MaxValueWriter with reduce parameter for the rank_key by @albertbou92 in https://github.com/pytorch/rl/pull/1796
• [BugFix] Fixes bug in MaxValueWriter tests by @albertbou92 in https://github.com/pytorch/rl/pull/1801
• [Performance] faster gym-like class by @vmoens in https://github.com/pytorch/rl/pull/1803
• [Feature] GenDGRL by @vmoens in https://github.com/pytorch/rl/pull/1773
• [Performance] Minor improvements to step_and_maybe_reset in batched envs by @vmoens in https://github.com/pytorch/rl/pull/1807
• [Algorithm] Discrete IQL by @BY571 in https://github.com/pytorch/rl/pull/1793
• [Doc] More depth in VMAS docs by @matteobettini in https://github.com/pytorch/rl/pull/1802
• [BugFix] Remove select() in favor of empty() by @vmoens in https://github.com/pytorch/rl/pull/1811
• Bump jinja2 from 3.1.2 to 3.1.3 in /docs by @dependabot in https://github.com/pytorch/rl/pull/1812
• [BugFix] Make TransformedEnv mirror allow_done_after_reset property of base env by @matteobettini in https://github.com/pytorch/rl/pull/1810
• [Doc] Update StepCounter doc by @skandermoalla in https://github.com/pytorch/rl/pull/1813
• [Feature] Improve info_dict reader by @vmoens in https://github.com/pytorch/rl/pull/1809
• [CI, Minor] Regroup Gen-DGRL CI with other libs by @vmoens in https://github.com/pytorch/rl/pull/1814
• [Versioning] Housekeeping in setup.py by @vmoens in https://github.com/pytorch/rl/pull/1816
• [Feature] TorchRL2Gym conversion by @vmoens in https://github.com/pytorch/rl/pull/1795
• [BugFix, CI] Fix snapshop imports in stable CI by @vmoens in https://github.com/pytorch/rl/pull/1821
• [Feature] More flexibility in loading PettingZoo by @matteobettini in https://github.com/pytorch/rl/pull/1817
• [Docs] Fix doc of ToTensorImage transforms.py by @skandermoalla in https://github.com/pytorch/rl/pull/1824
• [BugFix] Fix device of container generated values in transforms by @vmoens in https://github.com/pytorch/rl/pull/1827
• [Feature] Atari DQN dataset by @vmoens in https://github.com/pytorch/rl/pull/1815
• [Feature] Non-functional objectives (PPO, A2C, Reinforce) by @vmoens in https://github.com/pytorch/rl/pull/1804
• [Refactor] change default CKPT_BACKEND to torch by @vmoens in https://github.com/pytorch/rl/pull/1830
• pyproject.toml: remove unknown properties by @GaetanLepage in https://github.com/pytorch/rl/pull/1828
• [Doc, Feature] Doc improvements for video recording and CSV video formats by @vmoens in https://github.com/pytorch/rl/pull/1829
• [Feature] PyTrees in replay buffers by @vmoens in https://github.com/pytorch/rl/pull/1831
• [BugFix] Fix sequential step counts by @vmoens in https://github.com/pytorch/rl/pull/1838
• [Doc] TED format by @vmoens in https://github.com/pytorch/rl/pull/1836
• [Doc] References to TED by @vmoens in https://github.com/pytorch/rl/pull/1839
• [BugFix] Temporarily set lazy legacy to True by @vmoens in https://github.com/pytorch/rl/pull/1840
• [BugFix] Fix gym info scalar infos by @vmoens in https://github.com/pytorch/rl/pull/1842
• [Refactor] LAZY_LEGACY_OP=False by @vmoens in https://github.com/pytorch/rl/pull/1832
• [Feature] serial_for_single arg in batched envs by @vmoens in https://github.com/pytorch/rl/pull/1846
• [BugFix] Fix VD4RL by @vmoens in https://github.com/pytorch/rl/pull/1834
• [Doc] Make tutos runnable without colab by @vmoens in https://github.com/pytorch/rl/pull/1826
• [Feature] Fine control over devices in collectors by @vmoens in https://github.com/pytorch/rl/pull/1835
• [Feature, BugFix] Better thread control in penv and collectors by @vmoens in https://github.com/pytorch/rl/pull/1848
• [CI] Update macos image by @vmoens in https://github.com/pytorch/rl/pull/1849
• [BugFix] thread setting bug by @vmoens in https://github.com/pytorch/rl/pull/1852
• Remove unused completed_keys property from StepCounter. by @skandermoalla in https://github.com/pytorch/rl/pull/1854
• [Feature] Submitit run script by @albertbou92 in https://github.com/pytorch/rl/pull/1822
• [BugFix] Fix flaky gym penv test by @vmoens in https://github.com/pytorch/rl/pull/1853
• [CI] Fix macos build by @vmoens in https://github.com/pytorch/rl/pull/1856

New Contributors

• @mikemykhaylov made their first contribution in https://github.com/pytorch/rl/pull/1655
• @MarCnu made their first contribution in https://github.com/pytorch/rl/pull/1661
• @Deep145757 made their first contribution in https://github.com/pytorch/rl/pull/1665
• @vaibhav-009 made their first contribution in https://github.com/pytorch/rl/pull/1667
• @laszloKopits made their first contribution in https://github.com/pytorch/rl/pull/1669
• @ianbarber made their first contribution in https://github.com/pytorch/rl/pull/1752
• @dependabot made their first contribution in https://github.com/pytorch/rl/pull/1812
• @GaetanLepage made their first contribution in https://github.com/pytorch/rl/pull/1828

Full Changelog: https://github.com/pytorch/rl/compare/v0.2.1...v0.3.0

What's Changed

• [Feature] Warning for init_random_frames rounding in collectors by @matteobettini in https://github.com/pytorch/rl/pull/1616
• [Feature] Add support of non-pickable gym env by @duburcqa in https://github.com/pytorch/rl/pull/1615
• [BugFix] Add keys to GAE in PPO/A2C by @vmoens in https://github.com/pytorch/rl/pull/1618
• [BugFix] Fix gym benchmark by @vmoens in https://github.com/pytorch/rl/pull/1619
• [BugFix] Fix shape setting in CompositeSpec by @vmoens in https://github.com/pytorch/rl/pull/1620
• [Deprecation] Deprecate ambiguous device for memmap replay buffer by @vmoens in https://github.com/pytorch/rl/pull/1624
• [CI] Fix CI (python and cuda versions) by @vmoens in https://github.com/pytorch/rl/pull/1621
• [Feature] Max Value Writer by @albertbou92 in https://github.com/pytorch/rl/pull/1622
• [CI] Cython<3 for d4rl by @vmoens in https://github.com/pytorch/rl/pull/1634
• [BugFix] make cursor a torch.long tensor by @vmoens in https://github.com/pytorch/rl/pull/1639
• [BugFix] Gracefully handle C++ import error in TorchRL by @vmoens in https://github.com/pytorch/rl/pull/1640
• [Feature] step_and_maybe_reset in env by @vmoens in https://github.com/pytorch/rl/pull/1611
• [BugFix] Avoid overlapping temporary dirs during training by @vmoens in https://github.com/pytorch/rl/pull/1635
• [Feature] Exclude all private keys in collectors by @vmoens in https://github.com/pytorch/rl/pull/1644
• [BugFix] Fix tutos by @vmoens in https://github.com/pytorch/rl/pull/1648
• [Feature] Lazy imports for implement_for during torchrl import by @vmoens in https://github.com/pytorch/rl/pull/1646
• [Refactor] Put all buffers on CPU in examples by @vmoens in https://github.com/pytorch/rl/pull/1645
• [BugFix] Fix storage device by @vmoens in https://github.com/pytorch/rl/pull/1650
• [BugFix] Fix EXAMPLES.md by @vmoens in https://github.com/pytorch/rl/pull/1649
• [Release] 0.2.1 by @vmoens in https://github.com/pytorch/rl/pull/1642

New Contributors

• @duburcqa made their first contribution in https://github.com/pytorch/rl/pull/1615

Full Changelog: https://github.com/pytorch/rl/compare/v0.2.0...v0.2.1

TorchRL 0.2.0

This release provides many new features and bug fixes.

TorchRL now publishes Apple Silicon compatible wheels. We drop coverage of python 3.7 in favour of 3.11.

New and updated algorithms

Most algorithms have been cleaned and designed to reach (at least) SOTA results.

Compatibility with MARL settings has been drastically improved, and we provide a good amount of MARL examples within the library:

A prototype RLHF training script is also proposed (#1597)

A whole new category of offline RL algorithms have been integrated: Decision transformers.

• [Algorithm] Update offpolicy examples by @BY571 in https://github.com/pytorch/rl/pull/1206
• [Algorithm] Online Decision transformer by @BY571 in https://github.com/pytorch/rl/pull/1149
• [Algorithm] QMixer loss and multiagent models by @matteobettini in https://github.com/pytorch/rl/pull/1378
• [Algorithm] RLHF end-to-end, clean by @vmoens in https://github.com/pytorch/rl/pull/1597
• [Algorithm] Update A2C examples by @albertbou92 in https://github.com/pytorch/rl/pull/1521
• [Algorithm] Update DDPG Example by @BY571 in https://github.com/pytorch/rl/pull/1525
• [Algorithm] Update DT by @BY571 in https://github.com/pytorch/rl/pull/1560
• [Algorithm] Update PPO examples by @albertbou92 in https://github.com/pytorch/rl/pull/1495
• [Algorithm] Update SAC Example by @BY571 in https://github.com/pytorch/rl/pull/1524
• [Algorithm] Update TD3 Example by @BY571 in https://github.com/pytorch/rl/pull/1523

New features

One of the major new features of the library is the introduction of the terminated / truncated / done distinction at no cost within the library. All third-party and primary environments are now compatible with this, as well as losses and data collection primitives (collector etc). This feature is also compatible with complex data structures, such as those found in MARL training pipelines.

All losses are now compatible with tensordict-free inputs, for a more generic deployment.

New transforms

Atari games can now benefit from a EndOfLifeTransform that allows to use the end-of-life as a done state in the loss (#1605)

We provide a KL transform to add a KL factor to the reward in RLHF settings.

Action masking is made possible through the ActionMask transform (#1421)

VC1 is also integrated for better image embedding.

• [Feature] Allow sequential transforms to work offline by @vmoens in https://github.com/pytorch/rl/pull/1136
• [Feature] ClipTransform + rename min/maximum -> low/high by @vmoens in https://github.com/pytorch/rl/pull/1500
• [Feature] End-of-life transform by @vmoens in https://github.com/pytorch/rl/pull/1605
• [Feature] KL Transform for RLHF by @vmoens in https://github.com/pytorch/rl/pull/1196
• [Features] Conv3dNet and PermuteTransform by @xmaples in https://github.com/pytorch/rl/pull/1398
• [Feature, Refactor] Scale in ToTensorImage based on the dtype and new from_int parameter by @hyerra in https://github.com/pytorch/rl/pull/1208
• [Feature] CatFrames used as inverse by @BY571 in https://github.com/pytorch/rl/pull/1321
• [Feature] Masking actions by @vmoens in https://github.com/pytorch/rl/pull/1421
• [Feature] VC1 integration by @vmoens in https://github.com/pytorch/rl/pull/1211

New models

We provide GRU alongside LSTM for POMDP training.

MARL model coverage is now richer of a MultiAgentMLP and MultiAgentCNN! Other improvments for MARL include coverage for nested keys in most places of the library (losses, data collection, environments...)/

• [Feature] Support for GRU by @vmoens in https://github.com/pytorch/rl/pull/1586
• [Feature] TanhModule by @vmoens in https://github.com/pytorch/rl/pull/1213
• [Features] Conv3dNet and PermuteTransform by @xmaples in https://github.com/pytorch/rl/pull/1398
• [Feature] CNN version of MultiAgentMLP by @MarkHaoxiang in https://github.com/pytorch/rl/pull/1479

Other features (misc)

• [Feature] RLHF Rollouts (reopened) by @vmoens in https://github.com/pytorch/rl/pull/1329
• [Feature] Add CQL by @BY571 in https://github.com/pytorch/rl/pull/1239
• [Feature] Allow multiple (nested) action, reward, done keys in env,vec_env and collectors by @matteobettini in https://github.com/pytorch/rl/pull/1462
• [Feature] Auto-DoubleToFloat by @vmoens in https://github.com/pytorch/rl/pull/1442
• [Feature] CompositeSpec.lock by @vmoens in https://github.com/pytorch/rl/pull/1143
• [Feature] Device transform by @vmoens in https://github.com/pytorch/rl/pull/1472
• [Feature] Dispatch DiscreteSAC loss module by @Blonck in https://github.com/pytorch/rl/pull/1248
• [Feature] Dispatch PPO loss module by @Blonck in https://github.com/pytorch/rl/pull/1249
• [Feature] Dispatch REDQ loss module by @Blonck in https://github.com/pytorch/rl/pull/1251
• [Feature] Dispatch SAC loss module by @Blonck in https://github.com/pytorch/rl/pull/1244
• [Feature] Dispatch TD3 loss module by @Blonck in https://github.com/pytorch/rl/pull/1254
• [Feature] Dispatch for DDPG loss module by @Blonck in https://github.com/pytorch/rl/pull/1215
• [Feature] Dispatch for SAC loss module by @Blonck in https://github.com/pytorch/rl/pull/1223
• [Feature] Dispatch reinforce loss module by @Blonck in https://github.com/pytorch/rl/pull/1252
• [Feature] Distpatch IQL loss module by @Blonck in https://github.com/pytorch/rl/pull/1230
• [Feature] Fix DType casting lazy init by @vmoens in https://github.com/pytorch/rl/pull/1589
• [Feature] Heterogeneous Environments compatibility by @matteobettini in https://github.com/pytorch/rl/pull/1411
• [Feature] Log hparams from python dict by @matteobettini in https://github.com/pytorch/rl/pull/1517
• [Feature] MARL exploration e-greedy compatibility by @matteobettini in https://github.com/pytorch/rl/pull/1277
• [Feature] Make advantages compatible with Terminated, Truncated, Done by @vmoens in https://github.com/pytorch/rl/pull/1581
• [Feature] Make losses inherit from TDMBase by @vmoens in https://github.com/pytorch/rl/pull/1246
• [Feature] Making action masks compatible with q value modules and e-greedy by @matteobettini in https://github.com/pytorch/rl/pull/1499
• [Feature] Nested keys in OrnsteinUhlenbeckProcess by @matteobettini in https://github.com/pytorch/rl/pull/1305
• [Feature] Optional mapping of "state" in gym specs by @matteobettini in https://github.com/pytorch/rl/pull/1431
• [Feature] Parallel environments lazy heterogenous data compatibility by @matteobettini in https://github.com/pytorch/rl/pull/1436
• [Feature] Pettingzoo: add multiagent dimension to single agent groups by @matteobettini in https://github.com/pytorch/rl/pull/1550
• [Feature] RLHF Reward Model (reopened) by @vmoens in https://github.com/pytorch/rl/pull/1328
• [Feature] RLHF dataloading by @vmoens in https://github.com/pytorch/rl/pull/1309
• [Feature] RLHF networks by @apbard in https://github.com/pytorch/rl/pull/1319
• [Feature] Refactor categorical dists: Masked one-hot and pass-through gradients by @vmoens in https://github.com/pytorch/rl/pull/1488
• [Feature] ReplayBuffer.empty by @vmoens in https://github.com/pytorch/rl/pull/1238
• [Feature] Separate losses by @MateuszGuzek in https://github.com/pytorch/rl/pull/1240
• [Feature] Single call to value network in advantages [bis] by @vmoens in https://github.com/pytorch/rl/pull/1263
• [Feature] Single call to value network in advantages by @vmoens in https://github.com/pytorch/rl/pull/1256
• [Feature] TensorStorage by @vmoens in https://github.com/pytorch/rl/pull/1310
• [Feature] Threaded collection and parallel envs by @vmoens in https://github.com/pytorch/rl/pull/1559
• [Feature] Unbind specs by @vmoens in https://github.com/pytorch/rl/pull/1555
• [Feature] VMAS obs dict by @matteobettini in https://github.com/pytorch/rl/pull/1419
• [Feature] VMAS: choose between categorical or one-hot actions by @matteobettini in https://github.com/pytorch/rl/pull/1484
• [Feature] dispatch for DQNLoss by @vmoens in https://github.com/pytorch/rl/pull/1194
• [Feature] log histograms by @vmoens in https://github.com/pytorch/rl/pull/1306
• [Feature] make csv logger exist_ok on logging folder by @matteobettini in https://github.com/pytorch/rl/pull/1561
• [Feature] shifted for all adv by @vmoens in https://github.com/pytorch/rl/pull/1276

New environments and third-party improvements

We now cover SMAC-v2, PettingZoo, IsaacGymEnvs (prototype) and RoboHive. The D4RL dataset can now be used without the eponym library, which permit training with more recent or older versions of gym.

• [Environment, Docs] SMACv2 and docs on action masking by @matteobettini in https://github.com/pytorch/rl/pull/1466
• [Environment] Petting zoo by @matteobettini in https://github.com/pytorch/rl/pull/1471
• [Feature] D4rl direct download by @MateuszGuzek in https://github.com/pytorch/rl/pull/1430
• [Feature] Gym 'vectorized' envs compatibility by @vmoens in https://github.com/pytorch/rl/pull/1519
• [Feature] Gym compatibility: Terminal and truncated by @vmoens in https://github.com/pytorch/rl/pull/1539
• [Feature] IsaacGymEnvs integration by @vmoens in https://github.com/pytorch/rl/pull/1443
• [Feature] RoboHive integration by @vmoens in https://github.com/pytorch/rl/pull/1119

Performance improvements

We provide several speed improvements, in particular for data collection.

• [Performance] Accelerate GAE by @Blonck in https://github.com/pytorch/rl/pull/1142
• [Performance] Accelerate TD lambda return estimate by @Blonck in https://github.com/pytorch/rl/pull/1158
• [Performance] Accelerate _split_and_pad_sequence by @Blonck in https://github.com/pytorch/rl/pull/1147
• [Performance] Faster GAE by @vmoens in https://github.com/pytorch/rl/pull/1153
• [Performance] Faster losses by @vmoens in https://github.com/pytorch/rl/pull/1272
• [Performance] Improve performance and streamline the generating of the gammalambda tensor by @Blonck in https://github.com/pytorch/rl/pull/1171
• [Performance] Miscellaneous efficiency improvements by @vmoens in https://github.com/pytorch/rl/pull/1513
• [Performance] Reduce key accessing in transforms by @matteobettini in https://github.com/pytorch/rl/pull/1590
• [Performance] Some efficiency improvements by @vmoens in https://github.com/pytorch/rl/pull/1250
• [Performance] Vmas vectorized reset by @matteobettini in https://github.com/pytorch/rl/pull/1146

Bug fixes

• [BugFIx] Fix entropy signature in truncated normal by @vmoens in https://github.com/pytorch/rl/pull/1536
• [BugFix,CI] Fix virtualenv not found by @vmoens in https://github.com/pytorch/rl/pull/1280
• [BugFix] Add torch.no_grad() for rendering in multiagent PPO tutorial by @matteobettini in https://github.com/pytorch/rl/pull/1511
• [BugFix] Batched envs compatibility with custom keys by @matteobettini in https://github.com/pytorch/rl/pull/1348
• [BugFix] C++17 by @vmoens in https://github.com/pytorch/rl/pull/1169
• [BugFix] Check env specs for nested envs by @matteobettini in https://github.com/pytorch/rl/pull/1332
• [BugFix] CompositeSpec.unsqueeze by @btx0424 in https://github.com/pytorch/rl/pull/1464
• [BugFix] DDPG select also critic input for actor loss by @matteobettini in https://github.com/pytorch/rl/pull/1563
• [BugFix] DQN loss dispatch respect configured tensordict keys by @Blonck in https://github.com/pytorch/rl/pull/1285
• [BugFix] Discrete SAC rewrite by @matteobettini in https://github.com/pytorch/rl/pull/1461
• [BugFix] Empty-spec tolerance by @vmoens in https://github.com/pytorch/rl/pull/1501
• [BugFix] Fix Brax reset by @vmoens in https://github.com/pytorch/rl/pull/1195
• [BugFix] Fix CatFrames by @vmoens in https://github.com/pytorch/rl/pull/1336
• [BugFix] Fix ClipTransform device by @vmoens in https://github.com/pytorch/rl/pull/1508
• [BugFix] Fix Cython for D4RL by @vmoens in https://github.com/pytorch/rl/pull/1429
• [BugFix] Fix DDPG by @vmoens in https://github.com/pytorch/rl/pull/1183
• [BugFix] Fix DDPG squeezing by @matteobettini in https://github.com/pytorch/rl/pull/1487
• [BugFix] Fix Dreamer test error by @vmoens in https://github.com/pytorch/rl/pull/1558
• [BugFix] Fix Gym Categorical/One-hot issues by @vmoens in https://github.com/pytorch/rl/pull/1482
• [BugFix] Fix KL import errors by @vmoens in https://github.com/pytorch/rl/pull/1207
• [BugFix] Fix KLTransform execution with LSTM by @vmoens in https://github.com/pytorch/rl/pull/1426
• [BugFix] Fix KeyError in inverse transform replay buffer by @BY571 in https://github.com/pytorch/rl/pull/1165
• [BugFix] Fix LSTM - VecEnv compatibility by @vmoens in https://github.com/pytorch/rl/pull/1427
• [BugFix] Fix LSTM use with padded/masked segments by @smorad in https://github.com/pytorch/rl/pull/1399
• [BugFix] Fix NoopResetEnv behavior when trials exceeded. by @skandermoalla in https://github.com/pytorch/rl/pull/1477
• [BugFix] Fix QValueModule multi_one_hot by @smorad in https://github.com/pytorch/rl/pull/1439
• [BugFix] Fix RLHF tests - transformers v4.34 by @vmoens in https://github.com/pytorch/rl/pull/1601
• [BugFix] Fix RewardSum spec transform to mimic reward spec by @matteobettini in https://github.com/pytorch/rl/pull/1478
• [BugFix] Fix SAC alpha optim by @vmoens in https://github.com/pytorch/rl/pull/1192
• [BugFix] Fix SAC by @vmoens in https://github.com/pytorch/rl/pull/1189
• [BugFix] Fix SAC by @vmoens in https://github.com/pytorch/rl/pull/1190
• [BugFix] Fix SACv2 by @vmoens in https://github.com/pytorch/rl/pull/1191
• [BugFix] Fix SMAC-v2 by @vmoens in https://github.com/pytorch/rl/pull/1538
• [BugFix] Fix TD3 and compat with https://github.com/pytorch-labs/tensordict/pull/482 by @vmoens in https://github.com/pytorch/rl/pull/1375
• [BugFix] Fix TD3 inplace updates by @vmoens in https://github.com/pytorch/rl/pull/1219
• [BugFix] Fix TD3 target net by @vmoens in https://github.com/pytorch/rl/pull/1186
• [BugFix] Fix LazyStackedCompositeSpec and introducing consolidate_spec by @matteobettini in https://github.com/pytorch/rl/pull/1392
• [BugFix] Fix step_mdp() by @matteobettini in https://github.com/pytorch/rl/pull/1334
• [BugFix] Fix action mask test by @vmoens in https://github.com/pytorch/rl/pull/1492
• [BugFix] Fix brax by @vmoens in https://github.com/pytorch/rl/pull/1346
• [BugFix] Fix bug in ppo example config by @degensean in https://github.com/pytorch/rl/pull/1396
• [BugFix] Fix envpool by @vmoens in https://github.com/pytorch/rl/pull/1530
• [BugFix] Fix error message of .set_keys() in advantage modules by @Blonck in https://github.com/pytorch/rl/pull/1218
• [BugFix] Fix examples by @vmoens in https://github.com/pytorch/rl/pull/1173
• [BugFix] Fix locked params modif by @vmoens in https://github.com/pytorch/rl/pull/1307
• [BugFix] Fix max length by @vmoens in https://github.com/pytorch/rl/pull/1233
• [BugFix] Fix missing ("next", "observation") key in dispatch of losses by @Blonck in https://github.com/pytorch/rl/pull/1235
• [BugFix] Fix nested CompositeSpec creation by @vmoens in https://github.com/pytorch/rl/pull/1261
• [BugFix] Fix nightly tensordict dependency by @skandermoalla in https://github.com/pytorch/rl/pull/1302
• [BugFix] Fix ppo example by @vmoens in https://github.com/pytorch/rl/pull/1225
• [BugFix] Fix ppo training NaN occurences by @vmoens in https://github.com/pytorch/rl/pull/1403
• [BugFix] Fix reward sum within parallel envs by @vmoens in https://github.com/pytorch/rl/pull/1454
• [BugFix] Fix run_type_checks by @vmoens in https://github.com/pytorch/rl/pull/1570
• [BugFix] Fix safe tanh for older torch versions by @vmoens in https://github.com/pytorch/rl/pull/1220
• [BugFix] Fix serialization of parallel envs by @vmoens in https://github.com/pytorch/rl/pull/1197
• [BugFix] Fix split_trajs by @vmoens in https://github.com/pytorch/rl/pull/1444
• [BugFix] Fix tanh/atanh vmap compatibility by @vmoens in https://github.com/pytorch/rl/pull/1217
• [BugFix] Fix the bug of RoundRobinWriter.extend(data) by @xmaples in https://github.com/pytorch/rl/pull/1295
• [BugFix] Fix tutorials by @vmoens in https://github.com/pytorch/rl/pull/1382
• [BugFix] Fix typo in CatFrames Transform error message. by @skandermoalla in https://github.com/pytorch/rl/pull/1491
• [BugFix] Fix vmap in VmapModule (torch 1.13 compat) by @vmoens in https://github.com/pytorch/rl/pull/1350
• [BugFix] Improve collector buffer initialisation when policy spec is unavailable by @matteobettini in https://github.com/pytorch/rl/pull/1547
• [BugFix] Instantiate 2 losses with different keys by @matteobettini in https://github.com/pytorch/rl/pull/1553
• [BugFix] KL module integration by @vmoens in https://github.com/pytorch/rl/pull/1212
• [BugFix] Key selection in batched envs by @vmoens in https://github.com/pytorch/rl/pull/1253
• [BugFix] Load collector frames and iter by @matteobettini in https://github.com/pytorch/rl/pull/1557
• [BugFix] Make VecNorm Transform pickable by @albertbou92 in https://github.com/pytorch/rl/pull/1596
• [BugFix] Minor fixes PPO / A2C examples by @albertbou92 in https://github.com/pytorch/rl/pull/1591
• [BugFix] Multiagent "auto" entropy fix in SAC by @matteobettini in https://github.com/pytorch/rl/pull/1494
• [BugFix] Nested envs compatibility by @matteobettini in https://github.com/pytorch/rl/pull/1347
• [BugFix] Nested key in replay buffer by @matteobettini in https://github.com/pytorch/rl/pull/1485
• [BugFix] Nested keys in transforms by @matteobettini in https://github.com/pytorch/rl/pull/1355
• [BugFix] Nested keys to probabilistic modules by @matteobettini in https://github.com/pytorch/rl/pull/1363
• [BugFix] Parametric rand_action() in BaseEnv by @matteobettini in https://github.com/pytorch/rl/pull/1267
• [BugFix] Parametric collectors by @matteobettini in https://github.com/pytorch/rl/pull/1303
• [BugFix] Patch SAC to allow state_dict manipulation before exec by @vmoens in https://github.com/pytorch/rl/pull/1607
• [BugFix] PettingZoo seeding by @matteobettini in https://github.com/pytorch/rl/pull/1554
• [BugFix] Pickable buffer by @albertbou92 in https://github.com/pytorch/rl/pull/1410
• [BugFix] QValue modules and nested action by @matteobettini in https://github.com/pytorch/rl/pull/1351
• [BugFix] Reward sum custom key by @matteobettini in https://github.com/pytorch/rl/pull/1413
• [BugFix] SafeModule not safely handling specs by @matteobettini in https://github.com/pytorch/rl/pull/1352
• [BugFix] Small patches to SMAC by @matteobettini in https://github.com/pytorch/rl/pull/1533
• [BugFix] Sparse info in SMACv2 by @matteobettini in https://github.com/pytorch/rl/pull/1546
• [BugFix] ToTensorImage unsqueeze would not update the observation spec by @hyerra in https://github.com/pytorch/rl/pull/1161
• [BugFix] Torch 1.13 compat by @vmoens in https://github.com/pytorch/rl/pull/1294
• [BugFix] Unbreak tensordict import by @vmoens in https://github.com/pytorch/rl/pull/1231
• [BugFix] Vectorized priority update in replay buffers by @matteobettini in https://github.com/pytorch/rl/pull/1598
• [BugFix] _transpose_time with single dim by @vmoens in https://github.com/pytorch/rl/pull/1155
• [BugFix] RewardSum transform for multiple reward keys by @matteobettini in https://github.com/pytorch/rl/pull/1544
• [BugFix] step_mdp nested keys by @matteobettini in https://github.com/pytorch/rl/pull/1339
• [BugFix] include buffers in policy_weights by @vmoens in https://github.com/pytorch/rl/pull/1185
• [BugFix] load_state_dict in param updates for collectors by @vmoens in https://github.com/pytorch/rl/pull/1145
• [BugFix] make value estimator with value_key from the PPOLoss init arg by @xmaples in https://github.com/pytorch/rl/pull/1144
• [BugFix] unlock in tensordictmodules tests by @vmoens in https://github.com/pytorch/rl/pull/1417
• [BugFix] valid_size not saved as attribute by @tcbegley in https://github.com/pytorch/rl/pull/1337

Miscellaneous

• Envpool Tests to Nova by @osalpekar in https://github.com/pytorch/rl/pull/1283
• Fix CI by @matteobettini in https://github.com/pytorch/rl/pull/1368
• Fix MacOS Mujoco Failure by @osalpekar in https://github.com/pytorch/rl/pull/1450
• Linux GPU Brax Unittests by @osalpekar in https://github.com/pytorch/rl/pull/1133
• Linux Gym Unittests to GHA by @osalpekar in https://github.com/pytorch/rl/pull/1139
• Linux Olddeps tests to Nova by @osalpekar in https://github.com/pytorch/rl/pull/1289
• Move to More Efficient Windows Runner by @osalpekar in https://github.com/pytorch/rl/pull/1476
• OptDeps Tests to Nova by @osalpekar in https://github.com/pytorch/rl/pull/1290
• Remove Distributed CCI job by @osalpekar in https://github.com/pytorch/rl/pull/1374
• Remove Envpool from CCI by @osalpekar in https://github.com/pytorch/rl/pull/1390
• Remove old CircleCI Lint by @osalpekar in https://github.com/pytorch/rl/pull/1134
• Removing Migrated and Unused CCI jobs by @osalpekar in https://github.com/pytorch/rl/pull/1288
• Revert "[Feature] Single call to value network in advantages" by @vmoens in https://github.com/pytorch/rl/pull/1262
• Revert "[Refactor,Performance] Faster collectors" by @vmoens in https://github.com/pytorch/rl/pull/1330
• Sklearn test to Nova by @osalpekar in https://github.com/pytorch/rl/pull/1291
• Windows Unittests on GHA by @osalpekar in https://github.com/pytorch/rl/pull/1086
• [Benchmark,CI] Benchmarks in PR (pre) by @vmoens in https://github.com/pytorch/rl/pull/1342
• [Benchmark,CI] Benchmarks in PR by @vmoens in https://github.com/pytorch/rl/pull/1341
• [Benchmark] Benchmark Gym vs TorchRL by @vmoens in https://github.com/pytorch/rl/pull/1602
• [Benchmark] Benchmark losses by @vmoens in https://github.com/pytorch/rl/pull/1287
• [Benchmark] Benchmark number GPU vectorised environments in VMAS (TorchRL vs RLlib) by @matteobettini in https://github.com/pytorch/rl/pull/1446
• [Benchmark] Improve benchmark precision + step_mdp + fix GPU by @vmoens in https://github.com/pytorch/rl/pull/1340
• [CI] Add macOS M1 binaries Wheels by @DanilBaibak in https://github.com/pytorch/rl/pull/1504
• [CI] Add ninja for MacOS builts by @vmoens in https://github.com/pytorch/rl/pull/1564
• [CI] Concurrency on gha by @vmoens in https://github.com/pytorch/rl/pull/1152
• [CI] Deprecate Windows GPU CCI by @osalpekar in https://github.com/pytorch/rl/pull/1387
• [CI] Doc CI fix by @matteobettini in https://github.com/pytorch/rl/pull/1384
• [CI] Fix CI PettingZoo by @matteobettini in https://github.com/pytorch/rl/pull/1528
• [CI] Fix CI by @vmoens in https://github.com/pytorch/rl/pull/1529
• [CI] Fix GHA gpu tests by @vmoens in https://github.com/pytorch/rl/pull/1356
• [CI] Fix Jax version in Jumanji by @vmoens in https://github.com/pytorch/rl/pull/1242
• [CI] Fix Mujoco version by @vmoens in https://github.com/pytorch/rl/pull/1475
• [CI] Fix RoboHive CI by @vmoens in https://github.com/pytorch/rl/pull/1541
• [CI] Fix brax and habitat by @vmoens in https://github.com/pytorch/rl/pull/1353
• [CI] Fix examples CI by @matteobettini in https://github.com/pytorch/rl/pull/1489
• [CI] Fix failing jobs by @vmoens in https://github.com/pytorch/rl/pull/1318
• [CI] Fix failing jobs by @vmoens in https://github.com/pytorch/rl/pull/1335
• [CI] Fix habitat CI by @vmoens in https://github.com/pytorch/rl/pull/1537
• [CI] Fix jumanji by @vmoens in https://github.com/pytorch/rl/pull/1566
• [CI] Fix nightly build dependency on tensordict by @vmoens in https://github.com/pytorch/rl/pull/1300
• [CI] Fix opt deps machine and docker by @vmoens in https://github.com/pytorch/rl/pull/1362
• [CI] Fix tuto deps by @matteobettini in https://github.com/pytorch/rl/pull/1416
• [CI] Fix wheels by @vmoens in https://github.com/pytorch/rl/pull/1301
• [CI] Less old deps by @vmoens in https://github.com/pytorch/rl/pull/1255
• [CI] Less warnings in CI (costs) by @vmoens in https://github.com/pytorch/rl/pull/1349
• [CI] Merge Distributed and Linux GPU job by @osalpekar in https://github.com/pytorch/rl/pull/1182
• [CI] Migrate examples by @vmoens in https://github.com/pytorch/rl/pull/1364
• [CI] Move linux stable to GHA by @vmoens in https://github.com/pytorch/rl/pull/1503
• [CI] Reduce CI time by @vmoens in https://github.com/pytorch/rl/pull/1226
• [CI] Remove CCI Config by @osalpekar in https://github.com/pytorch/rl/pull/1456
• [CI] Remove examples from CCI by @vmoens in https://github.com/pytorch/rl/pull/1367
• [CI] Update cuda version by @vmoens in https://github.com/pytorch/rl/pull/1380
• [CI] Windows GPU Tests by @osalpekar in https://github.com/pytorch/rl/pull/1386
• [Doc] Add link to paper in readme by @giadefa in https://github.com/pytorch/rl/pull/1298
• [Doc] Add paper refs in doc and KB by @vmoens in https://github.com/pytorch/rl/pull/1241
• [Doc] CITATION.cff by @vmoens in https://github.com/pytorch/rl/pull/1229
• [Doc] Do not clean gh-pages by @vmoens in https://github.com/pytorch/rl/pull/1150
• [Doc] Fix GPU benchmark by @vmoens in https://github.com/pytorch/rl/pull/1151
• [Doc] Fix advantage examples by @vmoens in https://github.com/pytorch/rl/pull/1600
• [Doc] Fix default value of tanh_loc in the documentation of TruncatedNormal. by @skandermoalla in https://github.com/pytorch/rl/pull/1205
• [Doc] Fix doctest examples by @degensean in https://github.com/pytorch/rl/pull/1393
• [Doc] Fix exploration modules docstrings by @vmoens in https://github.com/pytorch/rl/pull/1326
• [Doc] Fix tanh_loc in docstrings by @vmoens in https://github.com/pytorch/rl/pull/1203
• [Doc] TorchRL Logo by @vmoens in https://github.com/pytorch/rl/pull/1234
• [Doc] Update citation by @vmoens in https://github.com/pytorch/rl/pull/1228
• [Doc] Update coding_ppo.py by @kushaangupta in https://github.com/pytorch/rl/pull/1483
• [Doc] correct typos in pendulum tutorial by @kushaangupta in https://github.com/pytorch/rl/pull/1502
• [Doc] fixed typos in ppo tutorial by @MatteoGaetzner in https://github.com/pytorch/rl/pull/1314
• [Docs] Fix multi-agent tutorial by @matteobettini in https://github.com/pytorch/rl/pull/1599
• [Docs] Multi-agent environments by @matteobettini in https://github.com/pytorch/rl/pull/1383
• [Example] Multiagent examples: MAPPO-IPPO-MADDPG-IDDPG-IQL-QMIX-VDN by @matteobettini in https://github.com/pytorch/rl/pull/1027
• [Fix] Remove loss device by @matteobettini in https://github.com/pytorch/rl/pull/1395
• [Lint] Add TorchFix linter by @kit1980 in https://github.com/pytorch/rl/pull/1580
• [Minor] Capture error in CatFrame edit by @vmoens in https://github.com/pytorch/rl/pull/1498
• [Minor] Fix prints by @vmoens in https://github.com/pytorch/rl/pull/1257
• [Minor] Fix typo by @vmoens in https://github.com/pytorch/rl/pull/1193
• [Minor] Missing commit from #1488 by @vmoens in https://github.com/pytorch/rl/pull/1490
• [Minor] Missing lint by @vmoens in https://github.com/pytorch/rl/pull/1556
• [Minor] More efficient SAC v1 by @vmoens in https://github.com/pytorch/rl/pull/1507
• [Minor] Remove ya gymnasium deprecation warning in vectorized envs by @vmoens in https://github.com/pytorch/rl/pull/1573
• [Minor] small fixes by @vmoens in https://github.com/pytorch/rl/pull/1237
• [Nova] Jumanji Tests to GHA by @osalpekar in https://github.com/pytorch/rl/pull/1282
• [Nova] Remove windows Unittests from CCI by @osalpekar in https://github.com/pytorch/rl/pull/1159
• [Nova] Removing CircleCI Gym Unittests by @osalpekar in https://github.com/pytorch/rl/pull/1179
• [Nova] Vmas Tests to GHA by @osalpekar in https://github.com/pytorch/rl/pull/1284
• [Quality] Filter out warnings in subprocs by @vmoens in https://github.com/pytorch/rl/pull/1552
• [Refacto] Migration due to tensordict 473 and 474 by @vmoens in https://github.com/pytorch/rl/pull/1354
• [Refactor,Performance] Faster collectors (bis) by @vmoens in https://github.com/pytorch/rl/pull/1331
• [Refactor,Performance] Faster collectors by @vmoens in https://github.com/pytorch/rl/pull/1327
• [Refactor] Better GymLikeEnv by @vmoens in https://github.com/pytorch/rl/pull/1168
• [Refactor] Better batch-size handling by RBs by @vmoens in https://github.com/pytorch/rl/pull/1311
• [Refactor] Better updaters by @vmoens in https://github.com/pytorch/rl/pull/1184
• [Refactor] Change objectives parameter/buffer/target logic by @vmoens in https://github.com/pytorch/rl/pull/1424
• [Refactor] Edit ppo params by @vmoens in https://github.com/pytorch/rl/pull/1322
• [Refactor] Expose all wrappers in torchrl.envs by @vmoens in https://github.com/pytorch/rl/pull/1532
• [Refactor] Faster envs (2) by @vmoens in https://github.com/pytorch/rl/pull/1457
• [Refactor] Fix imports by @vmoens in https://github.com/pytorch/rl/pull/1551
• [Refactor] Follow-up on tensordict PR 473 by @vmoens in https://github.com/pytorch/rl/pull/1361
• [Refactor] More unravel fixes by @vmoens in https://github.com/pytorch/rl/pull/1357
• [Refactor] Nested reward and done specs by @vmoens in https://github.com/pytorch/rl/pull/1115
• [Refactor] Refactor DDPG loss in standalone methods by @vmoens in https://github.com/pytorch/rl/pull/1603
• [Refactor] Refactor _reset in ParallelEnv by @vmoens in https://github.com/pytorch/rl/pull/1172
• [Refactor] Refactor losses for generalization by @vmoens in https://github.com/pytorch/rl/pull/1286
• [Refactor] Remove pkg_resources import by @vmoens in https://github.com/pytorch/rl/pull/1379
• [Refactor] Remove private calls to _set by @vmoens in https://github.com/pytorch/rl/pull/1370
• [Refactor] Shape ops in LSTM based on tensor shape, not tensordict by @vmoens in https://github.com/pytorch/rl/pull/1170
• [Refactor] Use _set_tuple for faster set by @vmoens in https://github.com/pytorch/rl/pull/1372
• [Refactor] Use wait instead of is_set to get results in ParallelEnv by @vmoens in https://github.com/pytorch/rl/pull/1562
• [Refactor] Use masking in collectors by @vmoens in https://github.com/pytorch/rl/pull/1412
• [Refactor] Vmas nested by @matteobettini in https://github.com/pytorch/rl/pull/1366
• [Refactor] the usage of tensordict keys in loss modules by @Blonck in https://github.com/pytorch/rl/pull/1175
• [Setup] Update setup.py python versions by @vmoens in https://github.com/pytorch/rl/pull/1496
• [Test,BugFix] Fix Jax backend tests by @vmoens in https://github.com/pytorch/rl/pull/1162
• [Test,CI,Feature] Total time per test by @vmoens in https://github.com/pytorch/rl/pull/1232
• [Test] Remove import of test class by @matteobettini in https://github.com/pytorch/rl/pull/1549
• [Test] Skip tests in python 3.11 by @vmoens in https://github.com/pytorch/rl/pull/1535
• [Test] Skip threading tests in OSX by @vmoens in https://github.com/pytorch/rl/pull/1571
• [Test] Test split trajs by @vmoens in https://github.com/pytorch/rl/pull/1445
• [Test] Test state_dict and loss modules by @vmoens in https://github.com/pytorch/rl/pull/1527
• [Tests] Collector compatibility for heterogeneous environments by @matteobettini in https://github.com/pytorch/rl/pull/1414
• [Tests] DDPG extra critic input tests by @matteobettini in https://github.com/pytorch/rl/pull/1568
• [Tutorial] Multiagent PPO tutorial by @matteobettini in https://github.com/pytorch/rl/pull/1385
• [Versioning] Python 3.11 by @vmoens in https://github.com/pytorch/rl/pull/1433
• [Versioning] Use python 3.8 for GPU tests by @vmoens in https://github.com/pytorch/rl/pull/1577
• [Versioning] Write version all cases in setup.py by @vmoens in https://github.com/pytorch/rl/pull/1579
• d4rl Test to Nova by @osalpekar in https://github.com/pytorch/rl/pull/1293
• python 3.11 in README by @vmoens in https://github.com/pytorch/rl/pull/1434

New Contributors

• @Blonck made their first contribution in https://github.com/pytorch/rl/pull/1142
• @hyerra made their first contribution in https://github.com/pytorch/rl/pull/1161
• @skandermoalla made their first contribution in https://github.com/pytorch/rl/pull/1205
• @giadefa made their first contribution in https://github.com/pytorch/rl/pull/1298
• @MatteoGaetzner made their first contribution in https://github.com/pytorch/rl/pull/1314
• @MateuszGuzek made their first contribution in https://github.com/pytorch/rl/pull/1240
• @degensean made their first contribution in https://github.com/pytorch/rl/pull/1393
• @smorad made their first contribution in https://github.com/pytorch/rl/pull/1399
• @kushaangupta made their first contribution in https://github.com/pytorch/rl/pull/1483
• @kit1980 made their first contribution in https://github.com/pytorch/rl/pull/1580
• @MarkHaoxiang made their first contribution in https://github.com/pytorch/rl/pull/1479
• @DanilBaibak made their first contribution in https://github.com/pytorch/rl/pull/1504

A great THANKS to our contributors, in particular (but not in any particular order) @skandermoalla, @matteobettini, @BY571 and @albertbou92 for their tremendous dedication.

Full Changelog: https://github.com/pytorch/rl/compare/v0.1.1...v0.2.0

What's Changed

• [Feature] Stacking specs by @vmoens in https://github.com/pytorch/rl/pull/892
• [Feature] Multicollector interruptor by @albertbou92 in https://github.com/pytorch/rl/pull/963
• [BugFix] VMAS api fix by @matteobettini in https://github.com/pytorch/rl/pull/978
• [CI] Fix D4RL tests in CI by @vmoens in https://github.com/pytorch/rl/pull/976
• [CI] Fix CI by @vmoens in https://github.com/pytorch/rl/pull/982
• [Refactor] Binary spec inherits from discrete spec by @matteobettini in https://github.com/pytorch/rl/pull/984
• [Feature] _DataCollector -> DataCollectorBase by @vmoens in https://github.com/pytorch/rl/pull/985
• [Feature] Discrete SAC by @BY571 in https://github.com/pytorch/rl/pull/882
• [Refactor, Doc] Refactor refs to SafeModule to TensorDictModule unless necessary by @vmoens in https://github.com/pytorch/rl/pull/986
• [BugFix] Quickfix by @vmoens in https://github.com/pytorch/rl/pull/991
• [Feature] Add Dropout to MLP module by @BY571 in https://github.com/pytorch/rl/pull/988
• [Feature] Warn when collectors collect more frames than requested by @matteobettini in https://github.com/pytorch/rl/pull/989
• [BugFix] make "_reset", "step_count", and other done_based keys follow done_spec by @matteobettini in https://github.com/pytorch/rl/pull/981
• [Feature] Bandit datasets by @vmoens in https://github.com/pytorch/rl/pull/912
• [BugFix] Fix sampling in PPO tutorial by @vmoens in https://github.com/pytorch/rl/pull/996
• [Refactor] Refactor losses (value function, doc, input batch size) by @vmoens in https://github.com/pytorch/rl/pull/987
• [BugFix,Feature,Doc] Fix replay buffers sampling info, docstrings and iteration by @vmoens in https://github.com/pytorch/rl/pull/1003
• [Feature] Replace ValueError by warning in collectors when total_frames is not an exact multiple of frames_per_batch by @albertbou92 in https://github.com/pytorch/rl/pull/999
• [BugFix] Only call replay buffer transforms when there are by @vmoens in https://github.com/pytorch/rl/pull/1008
• [BugFix] Patch tests in 1008 by @vmoens in https://github.com/pytorch/rl/pull/1009
• [Feature] Multidim value functions by @vmoens in https://github.com/pytorch/rl/pull/1007
• [BugFix] Fix exploration (OU and Gaussian) by @vmoens in https://github.com/pytorch/rl/pull/1006
• [CI] Fix python version in habitat by @vmoens in https://github.com/pytorch/rl/pull/1010
• Advantages pass time_dimand docfix by @matteobettini in https://github.com/pytorch/rl/pull/1014
• [Refactor] Faster transformed distributions by @vmoens in https://github.com/pytorch/rl/pull/1017
• [WIP, CI] Upgrade cuda channel by @vmoens in https://github.com/pytorch/rl/pull/1019
• [BugFix] Fix collector reset with truncation by @vmoens in https://github.com/pytorch/rl/pull/1021
• [Refactor] Improve collector performance by @matteobettini in https://github.com/pytorch/rl/pull/1020
• [BugFix] Fix params and buffer casting for policies by @vmoens in https://github.com/pytorch/rl/pull/1022
• [Feature] PPO allow entropy logging when entropy_coeff is 0 by @matteobettini in https://github.com/pytorch/rl/pull/1025
• [Feature] Distributed data collector (ray) by @albertbou92 in https://github.com/pytorch/rl/pull/930
• [Refactor] Minor changes in tensordict construction by @vmoens in https://github.com/pytorch/rl/pull/1029
• [CI] Fix Brax 0.9.0 by @vmoens in https://github.com/pytorch/rl/pull/1011
• [Feature] Multiagent API in vmas by @matteobettini in https://github.com/pytorch/rl/pull/983
• [Feature] Benchmarking worflow by @vmoens in https://github.com/pytorch/rl/pull/1028
• [Benchmark] Fix adv benchmark by @vmoens in https://github.com/pytorch/rl/pull/1030
• [Doc] Refactor DDPG and DQN tutos to narrow the scope by @vmoens in https://github.com/pytorch/rl/pull/979
• Revert "[Doc] Refactor DDPG and DQN tutos to narrow the scope" by @vmoens in https://github.com/pytorch/rl/pull/1032
• [BugFix] Advantage normalisation in ClipPPOLoss is done after computing gain1 by @albertbou92 in https://github.com/pytorch/rl/pull/1033
• [BugFix] Codecov SHA error by @vmoens in https://github.com/pytorch/rl/pull/1035
• [Doc] DDPG and DQN refactoring -- Doc cleaning by @vmoens in https://github.com/pytorch/rl/pull/1036
• [BugFix,CI] Fix macos codecov install by @vmoens in https://github.com/pytorch/rl/pull/1039
• [BugFix] kwargs update in distributed collectors by @vmoens in https://github.com/pytorch/rl/pull/1040
• [Feature] make_composite_from_td by @vmoens in https://github.com/pytorch/rl/pull/1042
• [Refactor] Import envpool locally to avoid importing gym at root level by @vmoens in https://github.com/pytorch/rl/pull/1041
• [Minor] Fix a typo by @FrankTianTT in https://github.com/pytorch/rl/pull/1046
• [BugFix] Fix param tying in loss modules by @vmoens in https://github.com/pytorch/rl/pull/1037
• [Refactor] less ad-hoc disable_env_checker check by @vmoens in https://github.com/pytorch/rl/pull/1047
• [Refactor] Improve distributed collectors by @vmoens in https://github.com/pytorch/rl/pull/1044
• [Doc] Document tensordict modules by @vmoens in https://github.com/pytorch/rl/pull/1053
• [Doc] Minor changes to contributing.md by @vmoens in https://github.com/pytorch/rl/pull/1054
• [Doc] A bit more doc on modules by @vmoens in https://github.com/pytorch/rl/pull/1056
• [Refactor] Import enum and interaction_type utils by @Goldspear in https://github.com/pytorch/rl/pull/1055
• [Feature] Deduplicate calls to common layers in PPO by @vmoens in https://github.com/pytorch/rl/pull/1057
• [BugFix] CompositeSpec nested key deletion by @btx0424 in https://github.com/pytorch/rl/pull/1059
• [Feature] Add MaskedCategorical distribution by @xiaomengy in https://github.com/pytorch/rl/pull/1012
• [Refactor] resetting envs in collectors always passes the _reset entry by @vmoens in https://github.com/pytorch/rl/pull/1061
• [Refactor] Better integration of QValue tools by @vmoens in https://github.com/pytorch/rl/pull/1063
• MUJOCO_INSTALLATION.md: Fix typo by @traversaro in https://github.com/pytorch/rl/pull/1064
• [Refactor] Removes "reward" from root tensordicts by @vmoens in https://github.com/pytorch/rl/pull/1065
• [Test] Fix tests for older pytorch versions by @vmoens in https://github.com/pytorch/rl/pull/1066
• [Feature] Reward2go Transform by @BY571 in https://github.com/pytorch/rl/pull/1038
• [CI] Reduce tests by @vmoens in https://github.com/pytorch/rl/pull/1071
• [Feature] Skip existing for advantage modules by @vmoens in https://github.com/pytorch/rl/pull/1070
• [BugFix] Fix parallel env data passing on cuda by @vmoens in https://github.com/pytorch/rl/pull/1024
• [Refactor] Deprecate interaction_mode by @vmoens in https://github.com/pytorch/rl/pull/1067
• [Doc] Update KB: cannot find -lGL by @vmoens in https://github.com/pytorch/rl/pull/1073
• [Doc] fix figures display issues in documentation of actors.py by @DamienAllonsius in https://github.com/pytorch/rl/pull/1074
• [Example] PPO simplified example by @albertbou92 in https://github.com/pytorch/rl/pull/1004
• [Feature] Update td in step (not overwrite) by @vmoens in https://github.com/pytorch/rl/pull/1075
• [CI] Remove migrated CircleCI macOS jobs by @seemethere in https://github.com/pytorch/rl/pull/1069
• [Feature] Target Return Transform by @BY571 in https://github.com/pytorch/rl/pull/1045
• [Test] Fix tensorboard tests with ImageIO 2.26 by @vmoens in https://github.com/pytorch/rl/pull/1083
• [Feature] LSTMModule by @vmoens in https://github.com/pytorch/rl/pull/1084
• [BugFix] Change default of skip_existing to None by @tcbegley in https://github.com/pytorch/rl/pull/1082
• [Example] A2C simplified example by @albertbou92 in https://github.com/pytorch/rl/pull/1076
• [BugFix] Fix output_spec transform calls by @vmoens in https://github.com/pytorch/rl/pull/1091
• [Feature] Indexing Discrete and OneHot specs by @remidomingues in https://github.com/pytorch/rl/pull/1081
• [Refactor] Refactor DQN by @vmoens in https://github.com/pytorch/rl/pull/1085
• [Feature] Auto-init updaters and raise a warning if not present by @vmoens in https://github.com/pytorch/rl/pull/1092
• [BugFix] Remove false warnings in losses by @vmoens in https://github.com/pytorch/rl/pull/1096
• [CI, BugFix] Fix CI warnings and errors by @vmoens in https://github.com/pytorch/rl/pull/1100
• [Refactor] Update vmap imports to torch by @vmoens in https://github.com/pytorch/rl/pull/1102
• [Refactor] Make advantages non-differentiable by default (except in losses) by @vmoens in https://github.com/pytorch/rl/pull/1104
• [Feature] Indexing specs by @remidomingues in https://github.com/pytorch/rl/pull/1105
• [BugFix] Fix EnvPoool by @vmoens in https://github.com/pytorch/rl/pull/1106
• [Feature,Doc] QValue refactoring and QNet + RNN tuto by @vmoens in https://github.com/pytorch/rl/pull/1060
• [BugFix] Fix Gym imports by @vmoens in https://github.com/pytorch/rl/pull/1023
• [CI] pytest should not skip tests for dependencies by @rohitnig in https://github.com/pytorch/rl/pull/1048
• [BugFix, Doc] Fix tutos by @vmoens in https://github.com/pytorch/rl/pull/1107
• [CI] Fix tutos (2) by @vmoens in https://github.com/pytorch/rl/pull/1109
• [Doc] Fix doc rendering by @vmoens in https://github.com/pytorch/rl/pull/1112
• Added the entry for skip-tests in the environment.yml by @rohitnig in https://github.com/pytorch/rl/pull/1113
• [CI] Upgrade ubuntu version in GHA by @vmoens in https://github.com/pytorch/rl/pull/1116
• Fix in windows unit test by @mischab in https://github.com/pytorch/rl/pull/1099
• Revert "Fix in windows unit test" by @mischab in https://github.com/pytorch/rl/pull/1117
• [Nova] Lint job on GHA by @osalpekar in https://github.com/pytorch/rl/pull/1114
• [Nova] Remove CircleCI Wheels Builds by @osalpekar in https://github.com/pytorch/rl/pull/1121
• [BugFix] Set exploration mode to MODE in all losses by default by @vmoens in https://github.com/pytorch/rl/pull/1123
• [BugFix] Instruct the value key to PPOLoss by @vmoens in https://github.com/pytorch/rl/pull/1124
• [Feature] CatFrames for offline data by @vmoens in https://github.com/pytorch/rl/pull/1122
• [CI] Fix windows CI by @vmoens in https://github.com/pytorch/rl/pull/1128
• [Refactor] Buffers tensorclass compat and tutorial by @vmoens in https://github.com/pytorch/rl/pull/1101
• [Feature] Marking the time dimension by @vmoens in https://github.com/pytorch/rl/pull/1095
• [Doc] Add tuto and time dim info in docs by @vmoens in https://github.com/pytorch/rl/pull/1130
• [Doc] Fix locked samples from RBs and ccl of tuto by @vmoens in https://github.com/pytorch/rl/pull/1132
• [BugFix] Fix unlock in RB by @vmoens in https://github.com/pytorch/rl/pull/1135
• [BugFix] extract the info dict from a list by @xmaples in https://github.com/pytorch/rl/pull/1131
• [Feature] Added support for vector-based rewards from environments in MO-Gymnasium by @dennismalmgren in https://github.com/pytorch/rl/pull/992
• [Versioning] v0.1.1 by @vmoens in https://github.com/pytorch/rl/pull/1137

New Contributors

• @FrankTianTT made their first contribution in https://github.com/pytorch/rl/pull/1046
• @Goldspear made their first contribution in https://github.com/pytorch/rl/pull/1055
• @btx0424 made their first contribution in https://github.com/pytorch/rl/pull/1059
• @traversaro made their first contribution in https://github.com/pytorch/rl/pull/1064
• @DamienAllonsius made their first contribution in https://github.com/pytorch/rl/pull/1074
• @seemethere made their first contribution in https://github.com/pytorch/rl/pull/1069
• @remidomingues made their first contribution in https://github.com/pytorch/rl/pull/1081
• @rohitnig made their first contribution in https://github.com/pytorch/rl/pull/1048
• @mischab made their first contribution in https://github.com/pytorch/rl/pull/1099
• @osalpekar made their first contribution in https://github.com/pytorch/rl/pull/1114
• @xmaples made their first contribution in https://github.com/pytorch/rl/pull/1131
• @dennismalmgren made their first contribution in https://github.com/pytorch/rl/pull/992

Full Changelog: https://github.com/pytorch/rl/compare/v0.1.0...v0.1.1

First official beta release of the library!

What's Changed

• QuickFix Versioning by @fedebotu in https://github.com/pytorch/rl/pull/958
• Version 0.0.5 by @vmoens in https://github.com/pytorch/rl/pull/957
• [Minor] Warning when loading memmap storage on uninitialized td by @vmoens in https://github.com/pytorch/rl/pull/961
• [Refactor] Defaults split_trajs to False by @vmoens in https://github.com/pytorch/rl/pull/947
• [Feature] InitTracker transform by @vmoens in https://github.com/pytorch/rl/pull/962
• [Feature] RenameTransform by @vmoens in https://github.com/pytorch/rl/pull/964
• [Feature] Implicit Q-Learning (IQL) by @BY571 in https://github.com/pytorch/rl/pull/933
• [Refactor] Refactor data collectors constructors by @vmoens in https://github.com/pytorch/rl/pull/970
• [Feature, Refactor] Iterable replay buffers by @vmoens in https://github.com/pytorch/rl/pull/968
• [Doc] README rewrite by @vmoens in https://github.com/pytorch/rl/pull/971
• [Refactor] A less verbose torchrl by @vmoens in https://github.com/pytorch/rl/pull/973
• [Feature] torch.distributed collectors by @vmoens in https://github.com/pytorch/rl/pull/934
• [Feature] Offline datasets: D4RL by @vmoens in https://github.com/pytorch/rl/pull/928

Full Changelog: https://github.com/pytorch/rl/compare/v0.0.5...v0.1.0

We change the env.step API, see https://github.com/pytorch/rl/pull/941 for more info.

What's Changed

• [BugFix] Fix dreamer training loop by @vmoens in https://github.com/pytorch/rl/pull/915
• [Doc] PPO Tutorial by @vmoens in https://github.com/pytorch/rl/pull/913
• [Doc] Create your pendulum tutorial by @vmoens in https://github.com/pytorch/rl/pull/911
• [BugFix] Deploy doc by @vmoens in https://github.com/pytorch/rl/pull/920
• [BugFix] Nvidia not found fix by @vmoens in https://github.com/pytorch/rl/pull/922
• [Feature] Rework to_one_hot and to_categorical to take a tensor as parameter by @riiswa in https://github.com/pytorch/rl/pull/816
• [Doc] Tutorial revamp by @vmoens in https://github.com/pytorch/rl/pull/926
• [BugFix] Fix EnvPool spec shapes by @vmoens in https://github.com/pytorch/rl/pull/932
• [BugFix] Fix CompositeSpec.to_numpy method by @riiswa in https://github.com/pytorch/rl/pull/931
• [CI] Do not run nightly workflows on forked repos by @XuehaiPan in https://github.com/pytorch/rl/pull/936
• [Refactor] set_default -> setdefault by @tcbegley in https://github.com/pytorch/rl/pull/935
• [BugFix] Step and maybe reset by @vmoens in https://github.com/pytorch/rl/pull/938
• [Doc] Minor doc improvements by @vmoens in https://github.com/pytorch/rl/pull/907
• [Doc] Add debug doc by @acohen13 in https://github.com/pytorch/rl/pull/940
• [BugFix] Propagate args to TransformedEnv's state_dict by @fedebotu in https://github.com/pytorch/rl/pull/944
• [BugFix] Vmas expanded specs by @matteobettini in https://github.com/pytorch/rl/pull/942
• [Quality] RB constuctors cleanup by @vmoens in https://github.com/pytorch/rl/pull/945
• [Doc] Refactor KB by @vmoens in https://github.com/pytorch/rl/pull/946
• [BugFix] Upgrade vision's functional import by @vmoens in https://github.com/pytorch/rl/pull/948
• [BugFix] Deprecate tensordict.set check skips in transforms by @vmoens in https://github.com/pytorch/rl/pull/951
• [BugFix] Upgrade tensordict deps by @vmoens in https://github.com/pytorch/rl/pull/953
• [CI] Fix windows CI by @vmoens in https://github.com/pytorch/rl/pull/954
• [Refactor] Refactor composite spec keys to match tensordict by @vmoens in https://github.com/pytorch/rl/pull/956
• [Refactor] Refactor the step to include reward and done in the 'next' tensordict by @vmoens in https://github.com/pytorch/rl/pull/941

New Contributors

• @XuehaiPan made their first contribution in https://github.com/pytorch/rl/pull/936
• @acohen13 made their first contribution in https://github.com/pytorch/rl/pull/940
• @fedebotu made their first contribution in https://github.com/pytorch/rl/pull/944

Full Changelog: https://github.com/pytorch/rl/compare/v0.0.4...v0.0.5

What's Changed

• [CI, Doc] Update functorch source installation command by @zou3519 in https://github.com/pytorch/rl/pull/446
• [BugFix] TransformedEnv attributes inheritance by @vmoens in https://github.com/pytorch/rl/pull/467
• [Feature] Cleanup mocking envs init and new by @vmoens in https://github.com/pytorch/rl/pull/469
• [Tests] Adding tensordict __repr__ tests by @sladebot in https://github.com/pytorch/rl/pull/435
• [Logging]: implement MLFlow logging integration by @rayanht in https://github.com/pytorch/rl/pull/432
• [BugFix] MLFlow import fix by @vmoens in https://github.com/pytorch/rl/pull/473
• [BugFix] Fixed pip install by @brandonsj in https://github.com/pytorch/rl/pull/475
• [Features]: Changed _inplace_update cls parameter passing in __new__ by @nicolas-dufour in https://github.com/pytorch/rl/pull/464
• [Feature]: ModelBased Envs by @nicolas-dufour in https://github.com/pytorch/rl/pull/333
• [Feature] make ReplayBufferTrainer compatible with storing trajectories by @vmoens in https://github.com/pytorch/rl/pull/476
• [Tutorial] DQN tutorial by @vmoens in https://github.com/pytorch/rl/pull/474
• [Feature] reader hooks for GymLike by @vmoens in https://github.com/pytorch/rl/pull/478
• [BugFix] TensorSpec.zero(None) failure fix by @vmoens in https://github.com/pytorch/rl/pull/483
• [Feature]: Support for planners and CEM by @nicolas-dufour in https://github.com/pytorch/rl/pull/384
• [Feature] Replaced device_safe() with device by @ordinskiy in https://github.com/pytorch/rl/pull/485
• [Feature]: TensorDictPrimer transform by @nicolas-dufour in https://github.com/pytorch/rl/pull/456
• [Feature]: erase() method for torchrl.timeit by @nicolas-dufour in https://github.com/pytorch/rl/pull/480
• [Feature] Added support for single collector in sync_async_collector by @nicolas-dufour in https://github.com/pytorch/rl/pull/482
• [BugFix] removing unwanted device_safe() by @vmoens in https://github.com/pytorch/rl/pull/486
• [Refactoring] Refactored get_stats_random_rollout by @nicolas-dufour in https://github.com/pytorch/rl/pull/481
• [Feature] VIP Integration by @JasonMa2016 in https://github.com/pytorch/rl/pull/487
• [Refactoring] Minor tweaks to recorder and logger by @nicolas-dufour in https://github.com/pytorch/rl/pull/489
• [Feature]: Deactivate typechecks in envs by @nicolas-dufour in https://github.com/pytorch/rl/pull/490
• [BugFix] Vectorized td_lambda with gamma tensor does not match the serial version by @vmoens in https://github.com/pytorch/rl/pull/400
• [BugFix] Fix TensorDictPrimer init by @vmoens in https://github.com/pytorch/rl/pull/491
• [Feature] Optional auto-reset when done for collectors and batched envs by @vmoens in https://github.com/pytorch/rl/pull/492
• [BugFix] Defaulting passing_devices to None by @himjohntang in https://github.com/pytorch/rl/pull/477
• Revert "[BugFix] Defaulting passing_devices to None" by @vmoens in https://github.com/pytorch/rl/pull/494
• [BugFix] Multi-agent fixes by @vmoens in https://github.com/pytorch/rl/pull/488
• [BugFix] Defaulting passing_devices to None by @vmoens in https://github.com/pytorch/rl/pull/495
• [Feature] Lazy initialization of CatTensors by @vmoens in https://github.com/pytorch/rl/pull/497
• [Cleanup] Removing cuda 10.2 references by @vmoens in https://github.com/pytorch/rl/pull/498
• [BugFix] Migration to pytorch org by @vmoens in https://github.com/pytorch/rl/pull/499
• [Refactoring] Import at root to enable vmap monkey-patching by @vmoens in https://github.com/pytorch/rl/pull/500
• [BugFix] python version for linting checks by @vmoens in https://github.com/pytorch/rl/pull/502
• [Feature] Replay Buffers refactor by @bamaxw in https://github.com/pytorch/rl/pull/330
• [Feature] Rename step_tensordict in step_mdp by @romainjln in https://github.com/pytorch/rl/pull/512
• [Lint] re-instantiate F821 by @vmoens in https://github.com/pytorch/rl/pull/516
• [BugFix] run_type_checks for TransformedEnvs by @vmoens in https://github.com/pytorch/rl/pull/513
• [BugFix] making first_dim and last_dim negative in FlattenObservation when a parent is set by @vmoens in https://github.com/pytorch/rl/pull/511
• [Feature] Add info dict key-spec pairs to observation_spec by @tcbegley in https://github.com/pytorch/rl/pull/504
• [BugFix] Changing the dm_control import to fail if not installed by @zeenolife in https://github.com/pytorch/rl/pull/515
• [CI] Add coverage with codecov by @silvestrebahi in https://github.com/pytorch/rl/pull/523
• Revert "[CI] Add coverage with codecov" by @vmoens in https://github.com/pytorch/rl/pull/525
• [Quality] Use relative imports for local c++ deps by @apbard in https://github.com/pytorch/rl/pull/526
• [Feature] Nightly release by @vmoens in https://github.com/pytorch/rl/pull/519
• [Feature] Add make_tensordict() function by @sicong-huang in https://github.com/pytorch/rl/pull/522
• [Doc] Misc readme fixes by @GavinPHR in https://github.com/pytorch/rl/pull/532
• [BugFix] Replacing inference_mode decorator with no_grad to fix state_dict loading error by @GavinPHR in https://github.com/pytorch/rl/pull/530
• [BugFix] Transformed ParallelEnv meta data are broken when passing to device by @vmoens in https://github.com/pytorch/rl/pull/531
• [Doc] Add coverage banner by @vmoens in https://github.com/pytorch/rl/pull/533
• [BugFix] Fix colab link of coding_dqn.ipynb by @Benjamin-eecs in https://github.com/pytorch/rl/pull/543
• [BugFix] Fix optional imports by @vmoens in https://github.com/pytorch/rl/pull/535
• [BugFix] Restore missing keys in data collector output by @tcbegley in https://github.com/pytorch/rl/pull/521
• [Lint] reorganize imports by @apbard in https://github.com/pytorch/rl/pull/545
• [BugFix] Single-cpu compatibility by @vmoens in https://github.com/pytorch/rl/pull/548
• [BugFix] vision install and other deps in optdeps by @vmoens in https://github.com/pytorch/rl/pull/552
• [Feature] Implemented device argument for modules.models by @yushiyangk in https://github.com/pytorch/rl/pull/524
• [BugFix] Fix ellipsis indexing of 2d TensorDicts by @vmoens in https://github.com/pytorch/rl/pull/559
• [BugFix] Additive gaussian exploration spec fix by @vmoens in https://github.com/pytorch/rl/pull/560
• [BugFix] Disabling video step for wandb by @vmoens in https://github.com/pytorch/rl/pull/561
• [BugFix] Various device fix by @vmoens in https://github.com/pytorch/rl/pull/558
• [Feature] Allow collectors to accept regular modules as policies by @tcbegley in https://github.com/pytorch/rl/pull/546
• [BugFix] Fix push binary nightly action by @psolikov in https://github.com/pytorch/rl/pull/566
• [BugFix] TensorDict comparison by @vmoens in https://github.com/pytorch/rl/pull/567
• [BugFix] Fix SyncDataCollector reset by @jrobine in https://github.com/pytorch/rl/pull/571
• [Doc] Banners on README.md by @vmoens in https://github.com/pytorch/rl/pull/572
• [Feature] Log printing in alphabetical order when creating a replay buffer by @nikhlrao in https://github.com/pytorch/rl/pull/573
• [BugFix] Add eps to reward normalization by @vmoens in https://github.com/pytorch/rl/pull/574
• [BugFix] Fix argument for PPOLoss.get_entropy_bonus() by @vmoens in https://github.com/pytorch/rl/pull/578
• [Feature] Restructure torchrl/objectives by @sgrigory in https://github.com/pytorch/rl/pull/580
• [Docs] Documentation revamp by @vmoens in https://github.com/pytorch/rl/pull/581
• [Doc] Publishing on pytorch.org by @vmoens in https://github.com/pytorch/rl/pull/582
• Revert "[Doc] Publishing on pytorch.org" by @vmoens in https://github.com/pytorch/rl/pull/584
• [Doc] Publishing on pytorch.org by @vmoens in https://github.com/pytorch/rl/pull/585
• Revert "[Doc] Publishing on pytorch.org" by @vmoens in https://github.com/pytorch/rl/pull/586
• [Doc] Publishing on pytorch.org by @vmoens in https://github.com/pytorch/rl/pull/587
• [Feature] More restrictive tests on docstrings by @vmoens in https://github.com/pytorch/rl/pull/457
• [BugFix] Wrong stack import in tests by @vmoens in https://github.com/pytorch/rl/pull/590
• [Feature] Exclude "_" out_keys in tensordictmodel by @jlesuffleur in https://github.com/pytorch/rl/pull/589
• [Feature]: Dreamer support by @nicolas-dufour in https://github.com/pytorch/rl/pull/341
• [Doc] Missing doc for prototype RB by @vmoens in https://github.com/pytorch/rl/pull/595
• [Feature] Update list of supported libraries by @vmoens in https://github.com/pytorch/rl/pull/594
• [BugFix] Fix timeit count registration by @vmoens in https://github.com/pytorch/rl/pull/598
• [Naming] Renaming ProbabilisticTensorDictModule keys by @vmoens in https://github.com/pytorch/rl/pull/603
• [Feature] Categorical encoding for action space by @artkorenev in https://github.com/pytorch/rl/pull/593
• [BugFix] ReplayBuffer's storage now signal back when changes happen by @paulomarciano in https://github.com/pytorch/rl/pull/614
• [Doc] Typos in tensordict tutorial by @PaLeroy in https://github.com/pytorch/rl/pull/621
• [Doc] Integrate knowledge base in docs by @hatala91 in https://github.com/pytorch/rl/pull/622
• [Doc] Updating docs requirements by @vmoens in https://github.com/pytorch/rl/pull/624
• [Feature] Make torchrl runnable without functorch and with gym==0.13 by @vmoens in https://github.com/pytorch/rl/pull/386
• [Feature] Habitat integration by @vmoens in https://github.com/pytorch/rl/pull/514
• [Feature] Checkpointing by @vmoens in https://github.com/pytorch/rl/pull/549
• Add support for null dim argument in TensorDict.squeeze by @jgonik in https://github.com/pytorch/rl/pull/608
• [Version] Updating to torch 1.13 by @vmoens in https://github.com/pytorch/rl/pull/627
• [Feature] Sub-memmap tensors by @vmoens in https://github.com/pytorch/rl/pull/626
• [BugFix] copy_ changes the index if the dest and source memmap tensors share the same file location by @vmoens in https://github.com/pytorch/rl/pull/631
• [Feature] Unfold transforms for folded TransformedEnv by @alexanderlobov in https://github.com/pytorch/rl/pull/630
• [BugFix] make TensorDictReplayBuffer.extend call super().extend with stacked_td by @vmoens in https://github.com/pytorch/rl/pull/634
• [BugFix] correct the use of step_mdp method in data collector by @adityagandhamal in https://github.com/pytorch/rl/pull/637
• [Feature] Added implement_for decorator by @ordinskiy in https://github.com/pytorch/rl/pull/618
• [Feature] Make DQN compatible with nn.Module by @svarolgunes in https://github.com/pytorch/rl/pull/632
• [Example] Distributed Replay Buffer Prototype Example Implementation by @adityagoel4512 in https://github.com/pytorch/rl/pull/615
• [Feature] Benchmark storage types by @adityagoel4512 in https://github.com/pytorch/rl/pull/633
• [Feature] Remove wild imports in the library by @sosmond in https://github.com/pytorch/rl/pull/642
• [BugFix] Prevent transform parent from being reassigned by @jasonfkut in https://github.com/pytorch/rl/pull/641
• [Feature] Too many deepcopy in transforms.py by @romainjln in https://github.com/pytorch/rl/pull/625
• [Naming] Rename keys_in to in_keys in transforms.py and related modules by @sardaankita in https://github.com/pytorch/rl/pull/656
• [Refactoring] Refactor dreamer helper in smaller pieces by @vmoens in https://github.com/pytorch/rl/pull/662
• [Feature] VIPRewardTransform by @vmoens in https://github.com/pytorch/rl/pull/658
• [BugFix] make_trainer possible bug for on-policy cases by @albertbou92 in https://github.com/pytorch/rl/pull/655
• [Naming] Fixing key names by @vmoens in https://github.com/pytorch/rl/pull/668
• [Test] Check dtypes of envs by @vmoens in https://github.com/pytorch/rl/pull/666
• [Refactor] Relying on the standalone tensordict -- phase 1 by @vmoens in https://github.com/pytorch/rl/pull/650
• [Doc] More doc on trainers by @vmoens in https://github.com/pytorch/rl/pull/663
• [BugFix] PPO example GAE import by @albertbou92 in https://github.com/pytorch/rl/pull/671
• [BugFix] Use GitHub for flake8 pre-commit hook by @vmoens in https://github.com/pytorch/rl/pull/679
• [BugFix] Update to strict select by @vmoens in https://github.com/pytorch/rl/pull/675
• [Feature] Auto-compute stats for ObservationNorm by @romainjln in https://github.com/pytorch/rl/pull/669
• [Doc] _make_collector helper function by @albertbou92 in https://github.com/pytorch/rl/pull/678
• [Doc] BatchSubSampler class docstrings example by @albertbou92 in https://github.com/pytorch/rl/pull/677
• [BugFix] PPO objective crashes if advantage_module is None by @albertbou92 in https://github.com/pytorch/rl/pull/676
• [Refactor] Refactor 'next_' into nested tensordicts by @vmoens in https://github.com/pytorch/rl/pull/649
• [Doc] More doc about environments by @vmoens in https://github.com/pytorch/rl/pull/683
• [Doc] Fix missing tensordict install for doc by @vmoens in https://github.com/pytorch/rl/pull/685
• [CI] Added CircleCI pipeline to test compatibility across supported gym versions by @ordinskiy in https://github.com/pytorch/rl/pull/645
• [BugFix] ConvNet forward method with tensors of more than 4 dimensions by @albertbou92 in https://github.com/pytorch/rl/pull/686
• [Feature] add standard_normal for RewardScaling by @adityagandhamal in https://github.com/pytorch/rl/pull/682
• [Feature] Jumanji envs by @yingchenlin in https://github.com/pytorch/rl/pull/674
• [Feature] Default collate_fn by @vmoens in https://github.com/pytorch/rl/pull/688
• [BugFix] Fix Examples by @vmoens in https://github.com/pytorch/rl/pull/687
• [Refactoring] Replace direct gym version checks with decorated functions (#) by @ordinskiy in https://github.com/pytorch/rl/pull/691
• Version 0.0.3 by @vmoens in https://github.com/pytorch/rl/pull/696
• [Docs] Host TensorDict docs inside TorchRL docs by @tcbegley in https://github.com/pytorch/rl/pull/693
• [BugFix] Fix docs build by @tcbegley in https://github.com/pytorch/rl/pull/698
• [BugFix] Proper error messages for orphan transform creation by @vmoens in https://github.com/pytorch/rl/pull/697
• [Feature] Append, init and insert transforms in ReplayBuffer by @altre in https://github.com/pytorch/rl/pull/695
• [Feature] A2C objective class and train example by @albertbou92 in https://github.com/pytorch/rl/pull/680
• [Doc, Test] Add A2C script test and doc by @vmoens in https://github.com/pytorch/rl/pull/702
• [BugFix] Initialising the classes LazyTensorStorage with a nested TensorDict raises error by @albertbou92 in https://github.com/pytorch/rl/pull/703
• [BugFix] Fix init_random_frames in A2C example test by @vmoens in https://github.com/pytorch/rl/pull/706
• [Formatting] Upgrade formatting libs by @vmoens in https://github.com/pytorch/rl/pull/705
• [Doc] Document undefined symbol error with torch version < 1.13 by @nickspell in https://github.com/pytorch/rl/pull/707
• [Doc] Tuto integration by @vmoens in https://github.com/pytorch/rl/pull/681
• [Quality] Deprecate .ipynb tutos by @vmoens in https://github.com/pytorch/rl/pull/710
• [Test] Fix wrong skip message when functorch is installed by @vmoens in https://github.com/pytorch/rl/pull/711
• [BugFix, Doc] Clone TensorDict docs into _local_build by @tcbegley in https://github.com/pytorch/rl/pull/712
• [Feature] Migrate to tensordict.nn.TensorDictModule by @tcbegley in https://github.com/pytorch/rl/pull/700
• [Doc] Fix Tutos TODOs by @vmoens in https://github.com/pytorch/rl/pull/713
• [BugFix] RoundRobinWriter, possible duplicated code in the extend method by @albertbou92 in https://github.com/pytorch/rl/pull/709
• [Feature] Add OptimizerHook by @aakhundov in https://github.com/pytorch/rl/pull/716
• [Feature] Support for in-place functionalization by @tcbegley in https://github.com/pytorch/rl/pull/714
• [BugFix] Fix TorchRL demo tutorial by @vmoens in https://github.com/pytorch/rl/pull/721
• [Docs] Update tutorial links in readme by @tcbegley in https://github.com/pytorch/rl/pull/724
• [Feature] Extend PPO loss helper to allow for more customisation by @albertbou92 in https://github.com/pytorch/rl/pull/718
• [BugFix] Model maker functions for A2C and PPO fail for discrete action space envs by @albertbou92 in https://github.com/pytorch/rl/pull/717
• [Minor] docstrings and setup fixes by @vmoens in https://github.com/pytorch/rl/pull/726
• [BugFix] Avoid wrongfully erasing observation keys from specs in CatTensors by @vmoens in https://github.com/pytorch/rl/pull/727
• [BugFix] Avoid wrongfully erasing observation keys from tensordict in CatTensors by @vmoens in https://github.com/pytorch/rl/pull/729
• [Doc] More doc for data collectors by @vmoens in https://github.com/pytorch/rl/pull/732
• [Feature] Port test_fake_tensordict to torchrl by @vmoens in https://github.com/pytorch/rl/pull/731
• [Feature] Use ObservationNorm.init_stats for stats computation in example scripts by @romainjln in https://github.com/pytorch/rl/pull/715
• [BugFix] init_stats over multiple dimensions by @vmoens in https://github.com/pytorch/rl/pull/735
• [Refactor] logger creation in examples by @acforvs in https://github.com/pytorch/rl/pull/733
• [Feature] Brax envs by @yingchenlin in https://github.com/pytorch/rl/pull/722
• [Refactor] Adopt prototype ProbabilisticTensorDictModule and ProbabilisticTensorDictSequential by @tcbegley in https://github.com/pytorch/rl/pull/728
• [Doc] Link to doc in README by @vmoens in https://github.com/pytorch/rl/pull/740
• [Feature] Make GAE return a 'value_target' entry by @vmoens in https://github.com/pytorch/rl/pull/741
• [Feature] SamplerWithoutReplacement by @vmoens in https://github.com/pytorch/rl/pull/742
• [Doc, CI] Update doc workflow to run on PR and only publishes doc on main. by @EmGarr in https://github.com/pytorch/rl/pull/745
• [Feature] Better advantage API for higher order derivatives by @vmoens in https://github.com/pytorch/rl/pull/744
• [Refactor] Cosmetic improvements to advantage modules by @vmoens in https://github.com/pytorch/rl/pull/746
• [BugFix] Fix NoopReset in parallel settings by @vmoens in https://github.com/pytorch/rl/pull/747
• [Refactor] Remove env.is_done attribute by @vmoens in https://github.com/pytorch/rl/pull/748
• [Refactor] Drop prototype imports by @tcbegley in https://github.com/pytorch/rl/pull/738
• [BugFix] Fixes for speed branch merge on tensordict by @vmoens in https://github.com/pytorch/rl/pull/755
• [BugFix] Fix size-match unsqueeze deprecation by @vmoens in https://github.com/pytorch/rl/pull/750
• [Feature] FrameSkipTransform by @vmoens in https://github.com/pytorch/rl/pull/749
• [BugFix] Better memory management for collectors by @vmoens in https://github.com/pytorch/rl/pull/763
• Minor cleaning in BaseEnv classes by @matteobettini in https://github.com/pytorch/rl/pull/767
• Revert "Minor cleaning in BaseEnv classes" by @vmoens in https://github.com/pytorch/rl/pull/768
• Cleaning in envs common.py by @matteobettini in https://github.com/pytorch/rl/pull/769
• Making _set_seed abstract by @matteobettini in https://github.com/pytorch/rl/pull/770
• [Feature] Remove the Nd*TensorSpec classes by @riiswa in https://github.com/pytorch/rl/pull/772
• [BugFix] Reinstantiate custom value key for multioutput value networks by @vmoens in https://github.com/pytorch/rl/pull/754
• [Feature] Add Step Counter transform by @riiswa in https://github.com/pytorch/rl/pull/756
• [BugFix] Batched environments with non empty batch size by @matteobettini in https://github.com/pytorch/rl/pull/774
• Allow undounded boxes creation from gym spaces by @matteobettini in https://github.com/pytorch/rl/pull/778
• [BugFix] Doc built cmake error by @vmoens in https://github.com/pytorch/rl/pull/780
• [Feature] Lazy TensorClass storage by @tcbegley in https://github.com/pytorch/rl/pull/752
• [BugFix] SyncDataCollector init when device and env_device are different by @albertbou92 in https://github.com/pytorch/rl/pull/765
• [Feature] RewardSum transform by @albertbou92 in https://github.com/pytorch/rl/pull/751
• [BugFix] Fix PPO clip by @vmoens in https://github.com/pytorch/rl/pull/786
• [Feature] MultiDiscreteTensorSpec by @riiswa in https://github.com/pytorch/rl/pull/783
• [Doc] Doc revamp by @vmoens in https://github.com/pytorch/rl/pull/782
• [BugFix] ParallelEnv handling of done flag by @matteobettini in https://github.com/pytorch/rl/pull/788
• [BugFix] Sorting nested keys by @matteobettini in https://github.com/pytorch/rl/pull/787
• [Doc] README index by @vmoens in https://github.com/pytorch/rl/pull/791
• Add windows wheel build to CircleCI by @yohann-benchetrit in https://github.com/pytorch/rl/pull/759
• [Algorithm] MPPI planner by @vmoens in https://github.com/pytorch/rl/pull/701
• [Doc] Better doc links by @vmoens in https://github.com/pytorch/rl/pull/795
• [Doc] Missing headers by @vmoens in https://github.com/pytorch/rl/pull/796
• [Doc] Knowledge base section by @vmoens in https://github.com/pytorch/rl/pull/797
• [Feature] Vmas library wrapper by @matteobettini in https://github.com/pytorch/rl/pull/785
• [Doc] Duplicate HabitatEnv entry in docs by @matteobettini in https://github.com/pytorch/rl/pull/798
• [Feature] MultiDiscreteTensorSpec nvec with several axes by @riiswa in https://github.com/pytorch/rl/pull/789
• [Refactor] Graduate Replay Buffer prototype by @KamilPiechowiak in https://github.com/pytorch/rl/pull/794
• [BugFix] Solve R3MTransform init problem by @vmoens in https://github.com/pytorch/rl/pull/803
• [Refactor] Simplify FlattenObservation default kwargs by @vmoens in https://github.com/pytorch/rl/pull/805
• [Format] Fix lint by @vmoens in https://github.com/pytorch/rl/pull/811
• [Doc, BugFix] Fix tutos errors by @vmoens in https://github.com/pytorch/rl/pull/817
• [Doc] Pretrained models tutorial by @vmoens in https://github.com/pytorch/rl/pull/814
• [Doc, BugFix] Fix tensordictmodule tutorial by @vmoens in https://github.com/pytorch/rl/pull/819
• [BugFix] Fix MultOneHotDiscreteTensorSpec.is_in by @riiswa in https://github.com/pytorch/rl/pull/818
• [Doc] Using R3M with a replay buffer by @vmoens in https://github.com/pytorch/rl/pull/820
• [CodeQuality] call all() without making a list by @riiswa in https://github.com/pytorch/rl/pull/821
• [BugFix] [Feature] "_reset" flag for env reset by @matteobettini in https://github.com/pytorch/rl/pull/800
• [CI] Add unit test workflows for Windows by @yohann-benchetrit in https://github.com/pytorch/rl/pull/804
• [BugFix] Fix habitat integration and doc by @vmoens in https://github.com/pytorch/rl/pull/812
• [Minor] Better error reporting by @vmoens in https://github.com/pytorch/rl/pull/822
• [Minor] Add ninja to deps in toml file by @vmoens in https://github.com/pytorch/rl/pull/823
• [BugFix] Device of info specs by @vmoens in https://github.com/pytorch/rl/pull/824
• [BugFix] Fix envs specs and info reading by @vmoens in https://github.com/pytorch/rl/pull/825
• [Feature] Dtype in vmas tests by @matteobettini in https://github.com/pytorch/rl/pull/827
• [BugFix] Fix R3M observation spec transform by @vmoens in https://github.com/pytorch/rl/pull/830
• small change to make @robandpdx a contributor by @robandpdx in https://github.com/pytorch/rl/pull/831
• [Feature] Exclude and select transforms by @vmoens in https://github.com/pytorch/rl/pull/832
• [BugFix] Updating Recorder to accomodate "solved" key by @ShahRutav in https://github.com/pytorch/rl/pull/833
• [BugFIx] Changed "set_count" set in collectors by @matteobettini in https://github.com/pytorch/rl/pull/835
• [Algorithm] Td3 by @BY571 in https://github.com/pytorch/rl/pull/684
• [Doc] A Succinct Summary of Reinforcement Learning by @vmoens in https://github.com/pytorch/rl/pull/840
• [Feature, BugFix] ObservationNorm keep_dims and RewardSum init by @vmoens in https://github.com/pytorch/rl/pull/839
• [BugFix] Improve done checking of collectors by @matteobettini in https://github.com/pytorch/rl/pull/838
• [BugFix] Sync with tensordict (meta-tensor deprecation) by @vmoens in https://github.com/pytorch/rl/pull/842
• [Feature] Refactor CatFrames using a proper preallocated buffer by @vmoens in https://github.com/pytorch/rl/pull/847
• [CI] Add Github-Actions workflows for Windows wheels & nightly-build by @yohann-benchetrit in https://github.com/pytorch/rl/pull/837
• [Doc] Fix broken link Dreamer by @atonkamanda in https://github.com/pytorch/rl/pull/853
• [BugFix] Loading state_dict on uninitialized CatFrames by @vmoens in https://github.com/pytorch/rl/pull/855
• [Refactor] Move loggers to torchrl.record by @vmoens in https://github.com/pytorch/rl/pull/854
• [Refactor] specs batch size refactoring by @vmoens in https://github.com/pytorch/rl/pull/829
• [Feature] Max pool Transform by @albertbou92 in https://github.com/pytorch/rl/pull/841
• [Feature] Refactor advantages for continuous batches by @vmoens in https://github.com/pytorch/rl/pull/848
• [BugFix, Doc] Minor fix in doc by @vmoens in https://github.com/pytorch/rl/pull/858
• [Versioning] Version 0.0.4a by @vmoens in https://github.com/pytorch/rl/pull/859
• [Feature] Vmas to device by @matteobettini in https://github.com/pytorch/rl/pull/850
• [BugFix] Fix zero-ing from specs in RewardSum by @vmoens in https://github.com/pytorch/rl/pull/860
• [Feature] Loading R3M and VIP from ResNet by @vmoens in https://github.com/pytorch/rl/pull/863
• [Feature] SAC V2 by @vmoens in https://github.com/pytorch/rl/pull/864
• [BugFix] Avoid collision of "step_count" key from transform and collector by @vmoens in https://github.com/pytorch/rl/pull/868
• [Refactor] Better init for CatFrames buffers + removing default init values by @vmoens in https://github.com/pytorch/rl/pull/874
• [Refactor] Minor refactorings to envs by @vmoens in https://github.com/pytorch/rl/pull/872
• [Refactor] Removing inplace transform attribute by @vmoens in https://github.com/pytorch/rl/pull/871
• [BugFix] Run checks when creating fake_td by @vmoens in https://github.com/pytorch/rl/pull/877
• [Refactor] Box device by @vmoens in https://github.com/pytorch/rl/pull/881
• [Feature] Multithreaded env by @sgrigory in https://github.com/pytorch/rl/pull/734
• [Refactor] Turn off default advantage normalization in PPO by @vmoens in https://github.com/pytorch/rl/pull/887
• [CI] Fix habitat-gym imports by @vmoens in https://github.com/pytorch/rl/pull/890
• [CI] Fix cuda versions by @vmoens in https://github.com/pytorch/rl/pull/889
• [CI] Fix windows install by @vmoens in https://github.com/pytorch/rl/pull/888
• MacOS CPU unit test workflow using GitHub Actions by @robandpdx in https://github.com/pytorch/rl/pull/886
• Linux CPU unit test workflow using GitHub Actions by @robandpdx in https://github.com/pytorch/rl/pull/826
• [Major, BugFix, Test] Refactor Transforms tests by @vmoens in https://github.com/pytorch/rl/pull/878
• [Bugfix] Codecov does not cover multiprocessed tests #879 by @kadeng in https://github.com/pytorch/rl/pull/893
• [CI, BugFix] Fix gym related errors by @vmoens in https://github.com/pytorch/rl/pull/895
• [WIP] Linux GPU unit test workflow using GitHub Actions by @robandpdx in https://github.com/pytorch/rl/pull/885
• [BugFix] Compose cloning fix by @vmoens in https://github.com/pytorch/rl/pull/899
• [Feature] Simplifying collector envs by @vmoens in https://github.com/pytorch/rl/pull/870
• [CI,Feature] Upgrade to gymnasium by @vmoens in https://github.com/pytorch/rl/pull/898
• [Doc] Add record utils to doc by @vmoens in https://github.com/pytorch/rl/pull/904
• [Test] Improve exception message match by @apbard in https://github.com/pytorch/rl/pull/906
• [BugFix] Dreamer helpers are broken with batched envs by @vmoens in https://github.com/pytorch/rl/pull/903
• [Feature] RandomCropTensorDict transform by @vmoens in https://github.com/pytorch/rl/pull/908
• [Versioning] Version 0.0.4b by @vmoens in https://github.com/pytorch/rl/pull/909

New Contributors

• @sladebot made their first contribution in https://github.com/pytorch/rl/pull/435
• @rayanht made their first contribution in https://github.com/pytorch/rl/pull/432
• @brandonsj made their first contribution in https://github.com/pytorch/rl/pull/475
• @ordinskiy made their first contribution in https://github.com/pytorch/rl/pull/485
• @JasonMa2016 made their first contribution in https://github.com/pytorch/rl/pull/487
• @himjohntang made their first contribution in https://github.com/pytorch/rl/pull/477
• @romainjln made their first contribution in https://github.com/pytorch/rl/pull/512
• @apbard made their first contribution in https://github.com/pytorch/rl/pull/526
• @sicong-huang made their first contribution in https://github.com/pytorch/rl/pull/522
• @psolikov made their first contribution in https://github.com/pytorch/rl/pull/566
• @jrobine made their first contribution in https://github.com/pytorch/rl/pull/571
• @nikhlrao made their first contribution in https://github.com/pytorch/rl/pull/573
• @sgrigory made their first contribution in https://github.com/pytorch/rl/pull/580
• @jlesuffleur made their first contribution in https://github.com/pytorch/rl/pull/589
• @artkorenev made their first contribution in https://github.com/pytorch/rl/pull/593
• @paulomarciano made their first contribution in https://github.com/pytorch/rl/pull/614
• @hatala91 made their first contribution in https://github.com/pytorch/rl/pull/622
• @jgonik made their first contribution in https://github.com/pytorch/rl/pull/608
• @adityagandhamal made their first contribution in https://github.com/pytorch/rl/pull/637
• @svarolgunes made their first contribution in https://github.com/pytorch/rl/pull/632
• @adityagoel4512 made their first contribution in https://github.com/pytorch/rl/pull/615
• @jasonfkut made their first contribution in https://github.com/pytorch/rl/pull/641
• @sardaankita made their first contribution in https://github.com/pytorch/rl/pull/656
• @albertbou92 made their first contribution in https://github.com/pytorch/rl/pull/655
• @yingchenlin made their first contribution in https://github.com/pytorch/rl/pull/674
• @altre made their first contribution in https://github.com/pytorch/rl/pull/695
• @nickspell made their first contribution in https://github.com/pytorch/rl/pull/707
• @aakhundov made their first contribution in https://github.com/pytorch/rl/pull/716
• @acforvs made their first contribution in https://github.com/pytorch/rl/pull/733
• @EmGarr made their first contribution in https://github.com/pytorch/rl/pull/745
• @matteobettini made their first contribution in https://github.com/pytorch/rl/pull/767
• @riiswa made their first contribution in https://github.com/pytorch/rl/pull/772
• @yohann-benchetrit made their first contribution in https://github.com/pytorch/rl/pull/759
• @KamilPiechowiak made their first contribution in https://github.com/pytorch/rl/pull/794
• @robandpdx made their first contribution in https://github.com/pytorch/rl/pull/831
• @ShahRutav made their first contribution in https://github.com/pytorch/rl/pull/833
• @BY571 made their first contribution in https://github.com/pytorch/rl/pull/684
• @atonkamanda made their first contribution in https://github.com/pytorch/rl/pull/853
• @kadeng made their first contribution in https://github.com/pytorch/rl/pull/893

Full Changelog: https://github.com/pytorch/rl/compare/v0.0.2a...v0.0.4b

What's Changed

• [CI, Doc] Update functorch source installation command by @zou3519 in https://github.com/pytorch/rl/pull/446
• [BugFix] TransformedEnv attributes inheritance by @vmoens in https://github.com/pytorch/rl/pull/467
• [Feature] Cleanup mocking envs init and new by @vmoens in https://github.com/pytorch/rl/pull/469
• [Tests] Adding tensordict __repr__ tests by @sladebot in https://github.com/pytorch/rl/pull/435
• [Logging]: implement MLFlow logging integration by @rayanht in https://github.com/pytorch/rl/pull/432
• [BugFix] MLFlow import fix by @vmoens in https://github.com/pytorch/rl/pull/473
• [BugFix] Fixed pip install by @brandonsj in https://github.com/pytorch/rl/pull/475
• [Features]: Changed _inplace_update cls parameter passing in __new__ by @nicolas-dufour in https://github.com/pytorch/rl/pull/464
• [Feature]: ModelBased Envs by @nicolas-dufour in https://github.com/pytorch/rl/pull/333
• [Feature] make ReplayBufferTrainer compatible with storing trajectories by @vmoens in https://github.com/pytorch/rl/pull/476
• [Tutorial] DQN tutorial by @vmoens in https://github.com/pytorch/rl/pull/474
• [Feature] reader hooks for GymLike by @vmoens in https://github.com/pytorch/rl/pull/478
• [BugFix] TensorSpec.zero(None) failure fix by @vmoens in https://github.com/pytorch/rl/pull/483
• [Feature]: Support for planners and CEM by @nicolas-dufour in https://github.com/pytorch/rl/pull/384
• [Feature] Replaced device_safe() with device by @ordinskiy in https://github.com/pytorch/rl/pull/485
• [Feature]: TensorDictPrimer transform by @nicolas-dufour in https://github.com/pytorch/rl/pull/456
• [Feature]: erase() method for torchrl.timeit by @nicolas-dufour in https://github.com/pytorch/rl/pull/480
• [Feature] Added support for single collector in sync_async_collector by @nicolas-dufour in https://github.com/pytorch/rl/pull/482
• [BugFix] removing unwanted device_safe() by @vmoens in https://github.com/pytorch/rl/pull/486
• [Refactoring] Refactored get_stats_random_rollout by @nicolas-dufour in https://github.com/pytorch/rl/pull/481
• [Feature] VIP Integration by @JasonMa2016 in https://github.com/pytorch/rl/pull/487
• [Refactoring] Minor tweaks to recorder and logger by @nicolas-dufour in https://github.com/pytorch/rl/pull/489
• [Feature]: Deactivate typechecks in envs by @nicolas-dufour in https://github.com/pytorch/rl/pull/490
• [BugFix] Vectorized td_lambda with gamma tensor does not match the serial version by @vmoens in https://github.com/pytorch/rl/pull/400
• [BugFix] Fix TensorDictPrimer init by @vmoens in https://github.com/pytorch/rl/pull/491
• [Feature] Optional auto-reset when done for collectors and batched envs by @vmoens in https://github.com/pytorch/rl/pull/492
• [BugFix] Defaulting passing_devices to None by @himjohntang in https://github.com/pytorch/rl/pull/477
• Revert "[BugFix] Defaulting passing_devices to None" by @vmoens in https://github.com/pytorch/rl/pull/494
• [BugFix] Multi-agent fixes by @vmoens in https://github.com/pytorch/rl/pull/488
• [BugFix] Defaulting passing_devices to None by @vmoens in https://github.com/pytorch/rl/pull/495
• [Feature] Lazy initialization of CatTensors by @vmoens in https://github.com/pytorch/rl/pull/497
• [Cleanup] Removing cuda 10.2 references by @vmoens in https://github.com/pytorch/rl/pull/498
• [BugFix] Migration to pytorch org by @vmoens in https://github.com/pytorch/rl/pull/499
• [Refactoring] Import at root to enable vmap monkey-patching by @vmoens in https://github.com/pytorch/rl/pull/500
• [BugFix] python version for linting checks by @vmoens in https://github.com/pytorch/rl/pull/502
• [Feature] Replay Buffers refactor by @bamaxw in https://github.com/pytorch/rl/pull/330
• [Feature] Rename step_tensordict in step_mdp by @romainjln in https://github.com/pytorch/rl/pull/512
• [Lint] re-instantiate F821 by @vmoens in https://github.com/pytorch/rl/pull/516
• [BugFix] run_type_checks for TransformedEnvs by @vmoens in https://github.com/pytorch/rl/pull/513
• [BugFix] making first_dim and last_dim negative in FlattenObservation when a parent is set by @vmoens in https://github.com/pytorch/rl/pull/511
• [Feature] Add info dict key-spec pairs to observation_spec by @tcbegley in https://github.com/pytorch/rl/pull/504
• [BugFix] Changing the dm_control import to fail if not installed by @zeenolife in https://github.com/pytorch/rl/pull/515
• [CI] Add coverage with codecov by @silvestrebahi in https://github.com/pytorch/rl/pull/523
• Revert "[CI] Add coverage with codecov" by @vmoens in https://github.com/pytorch/rl/pull/525
• [Quality] Use relative imports for local c++ deps by @apbard in https://github.com/pytorch/rl/pull/526
• [Feature] Nightly release by @vmoens in https://github.com/pytorch/rl/pull/519
• [Feature] Add make_tensordict() function by @sicong-huang in https://github.com/pytorch/rl/pull/522
• [Doc] Misc readme fixes by @GavinPHR in https://github.com/pytorch/rl/pull/532
• [BugFix] Replacing inference_mode decorator with no_grad to fix state_dict loading error by @GavinPHR in https://github.com/pytorch/rl/pull/530
• [BugFix] Transformed ParallelEnv meta data are broken when passing to device by @vmoens in https://github.com/pytorch/rl/pull/531
• [Doc] Add coverage banner by @vmoens in https://github.com/pytorch/rl/pull/533
• [BugFix] Fix colab link of coding_dqn.ipynb by @Benjamin-eecs in https://github.com/pytorch/rl/pull/543
• [BugFix] Fix optional imports by @vmoens in https://github.com/pytorch/rl/pull/535
• [BugFix] Restore missing keys in data collector output by @tcbegley in https://github.com/pytorch/rl/pull/521
• [Lint] reorganize imports by @apbard in https://github.com/pytorch/rl/pull/545
• [BugFix] Single-cpu compatibility by @vmoens in https://github.com/pytorch/rl/pull/548
• [BugFix] vision install and other deps in optdeps by @vmoens in https://github.com/pytorch/rl/pull/552
• [Feature] Implemented device argument for modules.models by @yushiyangk in https://github.com/pytorch/rl/pull/524
• [BugFix] Fix ellipsis indexing of 2d TensorDicts by @vmoens in https://github.com/pytorch/rl/pull/559
• [BugFix] Additive gaussian exploration spec fix by @vmoens in https://github.com/pytorch/rl/pull/560
• [BugFix] Disabling video step for wandb by @vmoens in https://github.com/pytorch/rl/pull/561
• [BugFix] Various device fix by @vmoens in https://github.com/pytorch/rl/pull/558
• [Feature] Allow collectors to accept regular modules as policies by @tcbegley in https://github.com/pytorch/rl/pull/546
• [BugFix] Fix push binary nightly action by @psolikov in https://github.com/pytorch/rl/pull/566
• [BugFix] TensorDict comparison by @vmoens in https://github.com/pytorch/rl/pull/567
• [BugFix] Fix SyncDataCollector reset by @jrobine in https://github.com/pytorch/rl/pull/571
• [Doc] Banners on README.md by @vmoens in https://github.com/pytorch/rl/pull/572
• [Feature] Log printing in alphabetical order when creating a replay buffer by @nikhlrao in https://github.com/pytorch/rl/pull/573
• [BugFix] Add eps to reward normalization by @vmoens in https://github.com/pytorch/rl/pull/574
• [BugFix] Fix argument for PPOLoss.get_entropy_bonus() by @vmoens in https://github.com/pytorch/rl/pull/578
• [Feature] Restructure torchrl/objectives by @sgrigory in https://github.com/pytorch/rl/pull/580
• [Docs] Documentation revamp by @vmoens in https://github.com/pytorch/rl/pull/581
• [Doc] Publishing on pytorch.org by @vmoens in https://github.com/pytorch/rl/pull/582
• Revert "[Doc] Publishing on pytorch.org" by @vmoens in https://github.com/pytorch/rl/pull/584
• [Doc] Publishing on pytorch.org by @vmoens in https://github.com/pytorch/rl/pull/585
• Revert "[Doc] Publishing on pytorch.org" by @vmoens in https://github.com/pytorch/rl/pull/586
• [Doc] Publishing on pytorch.org by @vmoens in https://github.com/pytorch/rl/pull/587
• [Feature] More restrictive tests on docstrings by @vmoens in https://github.com/pytorch/rl/pull/457
• [BugFix] Wrong stack import in tests by @vmoens in https://github.com/pytorch/rl/pull/590
• [Feature] Exclude "_" out_keys in tensordictmodel by @jlesuffleur in https://github.com/pytorch/rl/pull/589
• [Feature]: Dreamer support by @nicolas-dufour in https://github.com/pytorch/rl/pull/341
• [Doc] Missing doc for prototype RB by @vmoens in https://github.com/pytorch/rl/pull/595
• [Feature] Update list of supported libraries by @vmoens in https://github.com/pytorch/rl/pull/594
• [BugFix] Fix timeit count registration by @vmoens in https://github.com/pytorch/rl/pull/598
• [Naming] Renaming ProbabilisticTensorDictModule keys by @vmoens in https://github.com/pytorch/rl/pull/603
• [Feature] Categorical encoding for action space by @artkorenev in https://github.com/pytorch/rl/pull/593
• [BugFix] ReplayBuffer's storage now signal back when changes happen by @paulomarciano in https://github.com/pytorch/rl/pull/614
• [Doc] Typos in tensordict tutorial by @PaLeroy in https://github.com/pytorch/rl/pull/621
• [Doc] Integrate knowledge base in docs by @hatala91 in https://github.com/pytorch/rl/pull/622
• [Doc] Updating docs requirements by @vmoens in https://github.com/pytorch/rl/pull/624
• [Feature] Make torchrl runnable without functorch and with gym==0.13 by @vmoens in https://github.com/pytorch/rl/pull/386
• [Feature] Habitat integration by @vmoens in https://github.com/pytorch/rl/pull/514
• [Feature] Checkpointing by @vmoens in https://github.com/pytorch/rl/pull/549
• Add support for null dim argument in TensorDict.squeeze by @jgonik in https://github.com/pytorch/rl/pull/608
• [Version] Updating to torch 1.13 by @vmoens in https://github.com/pytorch/rl/pull/627
• [Feature] Sub-memmap tensors by @vmoens in https://github.com/pytorch/rl/pull/626
• [BugFix] copy_ changes the index if the dest and source memmap tensors share the same file location by @vmoens in https://github.com/pytorch/rl/pull/631
• [Feature] Unfold transforms for folded TransformedEnv by @alexanderlobov in https://github.com/pytorch/rl/pull/630
• [BugFix] make TensorDictReplayBuffer.extend call super().extend with stacked_td by @vmoens in https://github.com/pytorch/rl/pull/634
• [BugFix] correct the use of step_mdp method in data collector by @adityagandhamal in https://github.com/pytorch/rl/pull/637
• [Feature] Added implement_for decorator by @ordinskiy in https://github.com/pytorch/rl/pull/618
• [Feature] Make DQN compatible with nn.Module by @svarolgunes in https://github.com/pytorch/rl/pull/632
• [Example] Distributed Replay Buffer Prototype Example Implementation by @adityagoel4512 in https://github.com/pytorch/rl/pull/615
• [Feature] Benchmark storage types by @adityagoel4512 in https://github.com/pytorch/rl/pull/633
• [Feature] Remove wild imports in the library by @sosmond in https://github.com/pytorch/rl/pull/642
• [BugFix] Prevent transform parent from being reassigned by @jasonfkut in https://github.com/pytorch/rl/pull/641
• [Feature] Too many deepcopy in transforms.py by @romainjln in https://github.com/pytorch/rl/pull/625
• [Naming] Rename keys_in to in_keys in transforms.py and related modules by @sardaankita in https://github.com/pytorch/rl/pull/656
• [Refactoring] Refactor dreamer helper in smaller pieces by @vmoens in https://github.com/pytorch/rl/pull/662
• [Feature] VIPRewardTransform by @vmoens in https://github.com/pytorch/rl/pull/658
• [BugFix] make_trainer possible bug for on-policy cases by @albertbou92 in https://github.com/pytorch/rl/pull/655
• [Naming] Fixing key names by @vmoens in https://github.com/pytorch/rl/pull/668
• [Test] Check dtypes of envs by @vmoens in https://github.com/pytorch/rl/pull/666
• [Refactor] Relying on the standalone tensordict -- phase 1 by @vmoens in https://github.com/pytorch/rl/pull/650
• [Doc] More doc on trainers by @vmoens in https://github.com/pytorch/rl/pull/663
• [BugFix] PPO example GAE import by @albertbou92 in https://github.com/pytorch/rl/pull/671
• [BugFix] Use GitHub for flake8 pre-commit hook by @vmoens in https://github.com/pytorch/rl/pull/679
• [BugFix] Update to strict select by @vmoens in https://github.com/pytorch/rl/pull/675
• [Feature] Auto-compute stats for ObservationNorm by @romainjln in https://github.com/pytorch/rl/pull/669
• [Doc] _make_collector helper function by @albertbou92 in https://github.com/pytorch/rl/pull/678
• [Doc] BatchSubSampler class docstrings example by @albertbou92 in https://github.com/pytorch/rl/pull/677
• [BugFix] PPO objective crashes if advantage_module is None by @albertbou92 in https://github.com/pytorch/rl/pull/676
• [Refactor] Refactor 'next_' into nested tensordicts by @vmoens in https://github.com/pytorch/rl/pull/649
• [Doc] More doc about environments by @vmoens in https://github.com/pytorch/rl/pull/683
• [Doc] Fix missing tensordict install for doc by @vmoens in https://github.com/pytorch/rl/pull/685
• [CI] Added CircleCI pipeline to test compatibility across supported gym versions by @ordinskiy in https://github.com/pytorch/rl/pull/645
• [BugFix] ConvNet forward method with tensors of more than 4 dimensions by @albertbou92 in https://github.com/pytorch/rl/pull/686
• [Feature] add standard_normal for RewardScaling by @adityagandhamal in https://github.com/pytorch/rl/pull/682
• [Feature] Jumanji envs by @yingchenlin in https://github.com/pytorch/rl/pull/674
• [Feature] Default collate_fn by @vmoens in https://github.com/pytorch/rl/pull/688
• [BugFix] Fix Examples by @vmoens in https://github.com/pytorch/rl/pull/687
• [Refactoring] Replace direct gym version checks with decorated functions (#) by @ordinskiy in https://github.com/pytorch/rl/pull/691
• Version 0.0.3 by @vmoens in https://github.com/pytorch/rl/pull/696
• [Docs] Host TensorDict docs inside TorchRL docs by @tcbegley in https://github.com/pytorch/rl/pull/693
• [BugFix] Fix docs build by @tcbegley in https://github.com/pytorch/rl/pull/698
• [BugFix] Proper error messages for orphan transform creation by @vmoens in https://github.com/pytorch/rl/pull/697
• [Feature] Append, init and insert transforms in ReplayBuffer by @altre in https://github.com/pytorch/rl/pull/695
• [Feature] A2C objective class and train example by @albertbou92 in https://github.com/pytorch/rl/pull/680
• [Doc, Test] Add A2C script test and doc by @vmoens in https://github.com/pytorch/rl/pull/702
• [BugFix] Initialising the classes LazyTensorStorage with a nested TensorDict raises error by @albertbou92 in https://github.com/pytorch/rl/pull/703
• [BugFix] Fix init_random_frames in A2C example test by @vmoens in https://github.com/pytorch/rl/pull/706
• [Formatting] Upgrade formatting libs by @vmoens in https://github.com/pytorch/rl/pull/705
• [Doc] Document undefined symbol error with torch version < 1.13 by @nickspell in https://github.com/pytorch/rl/pull/707
• [Doc] Tuto integration by @vmoens in https://github.com/pytorch/rl/pull/681
• [Quality] Deprecate .ipynb tutos by @vmoens in https://github.com/pytorch/rl/pull/710
• [Test] Fix wrong skip message when functorch is installed by @vmoens in https://github.com/pytorch/rl/pull/711
• [BugFix, Doc] Clone TensorDict docs into _local_build by @tcbegley in https://github.com/pytorch/rl/pull/712
• [Feature] Migrate to tensordict.nn.TensorDictModule by @tcbegley in https://github.com/pytorch/rl/pull/700
• [Doc] Fix Tutos TODOs by @vmoens in https://github.com/pytorch/rl/pull/713
• [BugFix] RoundRobinWriter, possible duplicated code in the extend method by @albertbou92 in https://github.com/pytorch/rl/pull/709
• [Feature] Add OptimizerHook by @aakhundov in https://github.com/pytorch/rl/pull/716
• [Feature] Support for in-place functionalization by @tcbegley in https://github.com/pytorch/rl/pull/714
• [BugFix] Fix TorchRL demo tutorial by @vmoens in https://github.com/pytorch/rl/pull/721
• [Docs] Update tutorial links in readme by @tcbegley in https://github.com/pytorch/rl/pull/724
• [Feature] Extend PPO loss helper to allow for more customisation by @albertbou92 in https://github.com/pytorch/rl/pull/718
• [BugFix] Model maker functions for A2C and PPO fail for discrete action space envs by @albertbou92 in https://github.com/pytorch/rl/pull/717
• [Minor] docstrings and setup fixes by @vmoens in https://github.com/pytorch/rl/pull/726
• [BugFix] Avoid wrongfully erasing observation keys from specs in CatTensors by @vmoens in https://github.com/pytorch/rl/pull/727
• [BugFix] Avoid wrongfully erasing observation keys from tensordict in CatTensors by @vmoens in https://github.com/pytorch/rl/pull/729
• [Doc] More doc for data collectors by @vmoens in https://github.com/pytorch/rl/pull/732
• [Feature] Port test_fake_tensordict to torchrl by @vmoens in https://github.com/pytorch/rl/pull/731
• [Feature] Use ObservationNorm.init_stats for stats computation in example scripts by @romainjln in https://github.com/pytorch/rl/pull/715
• [BugFix] init_stats over multiple dimensions by @vmoens in https://github.com/pytorch/rl/pull/735
• [Refactor] logger creation in examples by @acforvs in https://github.com/pytorch/rl/pull/733
• [Feature] Brax envs by @yingchenlin in https://github.com/pytorch/rl/pull/722
• [Refactor] Adopt prototype ProbabilisticTensorDictModule and ProbabilisticTensorDictSequential by @tcbegley in https://github.com/pytorch/rl/pull/728
• [Doc] Link to doc in README by @vmoens in https://github.com/pytorch/rl/pull/740
• [Feature] Make GAE return a 'value_target' entry by @vmoens in https://github.com/pytorch/rl/pull/741
• [Feature] SamplerWithoutReplacement by @vmoens in https://github.com/pytorch/rl/pull/742
• [Doc, CI] Update doc workflow to run on PR and only publishes doc on main. by @EmGarr in https://github.com/pytorch/rl/pull/745
• [Feature] Better advantage API for higher order derivatives by @vmoens in https://github.com/pytorch/rl/pull/744
• [Refactor] Cosmetic improvements to advantage modules by @vmoens in https://github.com/pytorch/rl/pull/746
• [BugFix] Fix NoopReset in parallel settings by @vmoens in https://github.com/pytorch/rl/pull/747
• [Refactor] Remove env.is_done attribute by @vmoens in https://github.com/pytorch/rl/pull/748
• [Refactor] Drop prototype imports by @tcbegley in https://github.com/pytorch/rl/pull/738
• [BugFix] Fixes for speed branch merge on tensordict by @vmoens in https://github.com/pytorch/rl/pull/755
• [BugFix] Fix size-match unsqueeze deprecation by @vmoens in https://github.com/pytorch/rl/pull/750
• [Feature] FrameSkipTransform by @vmoens in https://github.com/pytorch/rl/pull/749
• [BugFix] Better memory management for collectors by @vmoens in https://github.com/pytorch/rl/pull/763
• Minor cleaning in BaseEnv classes by @matteobettini in https://github.com/pytorch/rl/pull/767
• Revert "Minor cleaning in BaseEnv classes" by @vmoens in https://github.com/pytorch/rl/pull/768
• Cleaning in envs common.py by @matteobettini in https://github.com/pytorch/rl/pull/769
• Making _set_seed abstract by @matteobettini in https://github.com/pytorch/rl/pull/770
• [Feature] Remove the Nd*TensorSpec classes by @riiswa in https://github.com/pytorch/rl/pull/772
• [BugFix] Reinstantiate custom value key for multioutput value networks by @vmoens in https://github.com/pytorch/rl/pull/754
• [Feature] Add Step Counter transform by @riiswa in https://github.com/pytorch/rl/pull/756
• [BugFix] Batched environments with non empty batch size by @matteobettini in https://github.com/pytorch/rl/pull/774
• Allow undounded boxes creation from gym spaces by @matteobettini in https://github.com/pytorch/rl/pull/778
• [BugFix] Doc built cmake error by @vmoens in https://github.com/pytorch/rl/pull/780
• [Feature] Lazy TensorClass storage by @tcbegley in https://github.com/pytorch/rl/pull/752
• [BugFix] SyncDataCollector init when device and env_device are different by @albertbou92 in https://github.com/pytorch/rl/pull/765
• [Feature] RewardSum transform by @albertbou92 in https://github.com/pytorch/rl/pull/751
• [BugFix] Fix PPO clip by @vmoens in https://github.com/pytorch/rl/pull/786
• [Feature] MultiDiscreteTensorSpec by @riiswa in https://github.com/pytorch/rl/pull/783
• [Doc] Doc revamp by @vmoens in https://github.com/pytorch/rl/pull/782
• [BugFix] ParallelEnv handling of done flag by @matteobettini in https://github.com/pytorch/rl/pull/788
• [BugFix] Sorting nested keys by @matteobettini in https://github.com/pytorch/rl/pull/787
• [Doc] README index by @vmoens in https://github.com/pytorch/rl/pull/791
• Add windows wheel build to CircleCI by @yohann-benchetrit in https://github.com/pytorch/rl/pull/759
• [Algorithm] MPPI planner by @vmoens in https://github.com/pytorch/rl/pull/701
• [Doc] Better doc links by @vmoens in https://github.com/pytorch/rl/pull/795
• [Doc] Missing headers by @vmoens in https://github.com/pytorch/rl/pull/796
• [Doc] Knowledge base section by @vmoens in https://github.com/pytorch/rl/pull/797
• [Feature] Vmas library wrapper by @matteobettini in https://github.com/pytorch/rl/pull/785
• [Doc] Duplicate HabitatEnv entry in docs by @matteobettini in https://github.com/pytorch/rl/pull/798
• [Feature] MultiDiscreteTensorSpec nvec with several axes by @riiswa in https://github.com/pytorch/rl/pull/789
• [Refactor] Graduate Replay Buffer prototype by @KamilPiechowiak in https://github.com/pytorch/rl/pull/794
• [BugFix] Solve R3MTransform init problem by @vmoens in https://github.com/pytorch/rl/pull/803
• [Refactor] Simplify FlattenObservation default kwargs by @vmoens in https://github.com/pytorch/rl/pull/805
• [Format] Fix lint by @vmoens in https://github.com/pytorch/rl/pull/811
• [Doc, BugFix] Fix tutos errors by @vmoens in https://github.com/pytorch/rl/pull/817
• [Doc] Pretrained models tutorial by @vmoens in https://github.com/pytorch/rl/pull/814
• [Doc, BugFix] Fix tensordictmodule tutorial by @vmoens in https://github.com/pytorch/rl/pull/819
• [BugFix] Fix MultOneHotDiscreteTensorSpec.is_in by @riiswa in https://github.com/pytorch/rl/pull/818
• [Doc] Using R3M with a replay buffer by @vmoens in https://github.com/pytorch/rl/pull/820
• [CodeQuality] call all() without making a list by @riiswa in https://github.com/pytorch/rl/pull/821
• [BugFix] [Feature] "_reset" flag for env reset by @matteobettini in https://github.com/pytorch/rl/pull/800
• [CI] Add unit test workflows for Windows by @yohann-benchetrit in https://github.com/pytorch/rl/pull/804
• [BugFix] Fix habitat integration and doc by @vmoens in https://github.com/pytorch/rl/pull/812
• [Minor] Better error reporting by @vmoens in https://github.com/pytorch/rl/pull/822
• [Minor] Add ninja to deps in toml file by @vmoens in https://github.com/pytorch/rl/pull/823
• [BugFix] Device of info specs by @vmoens in https://github.com/pytorch/rl/pull/824
• [BugFix] Fix envs specs and info reading by @vmoens in https://github.com/pytorch/rl/pull/825
• [Feature] Dtype in vmas tests by @matteobettini in https://github.com/pytorch/rl/pull/827
• [BugFix] Fix R3M observation spec transform by @vmoens in https://github.com/pytorch/rl/pull/830
• small change to make @robandpdx a contributor by @robandpdx in https://github.com/pytorch/rl/pull/831
• [Feature] Exclude and select transforms by @vmoens in https://github.com/pytorch/rl/pull/832
• [BugFix] Updating Recorder to accomodate "solved" key by @ShahRutav in https://github.com/pytorch/rl/pull/833
• [BugFIx] Changed "set_count" set in collectors by @matteobettini in https://github.com/pytorch/rl/pull/835
• [Algorithm] Td3 by @BY571 in https://github.com/pytorch/rl/pull/684
• [Doc] A Succinct Summary of Reinforcement Learning by @vmoens in https://github.com/pytorch/rl/pull/840
• [Feature, BugFix] ObservationNorm keep_dims and RewardSum init by @vmoens in https://github.com/pytorch/rl/pull/839
• [BugFix] Improve done checking of collectors by @matteobettini in https://github.com/pytorch/rl/pull/838
• [BugFix] Sync with tensordict (meta-tensor deprecation) by @vmoens in https://github.com/pytorch/rl/pull/842
• [Feature] Refactor CatFrames using a proper preallocated buffer by @vmoens in https://github.com/pytorch/rl/pull/847
• [CI] Add Github-Actions workflows for Windows wheels & nightly-build by @yohann-benchetrit in https://github.com/pytorch/rl/pull/837
• [Doc] Fix broken link Dreamer by @atonkamanda in https://github.com/pytorch/rl/pull/853
• [BugFix] Loading state_dict on uninitialized CatFrames by @vmoens in https://github.com/pytorch/rl/pull/855
• [Refactor] Move loggers to torchrl.record by @vmoens in https://github.com/pytorch/rl/pull/854
• [Refactor] specs batch size refactoring by @vmoens in https://github.com/pytorch/rl/pull/829
• [Feature] Max pool Transform by @albertbou92 in https://github.com/pytorch/rl/pull/841
• [Feature] Refactor advantages for continuous batches by @vmoens in https://github.com/pytorch/rl/pull/848
• [BugFix, Doc] Minor fix in doc by @vmoens in https://github.com/pytorch/rl/pull/858
• [Versioning] Version 0.0.4a by @vmoens in https://github.com/pytorch/rl/pull/859
• [Feature] Vmas to device by @matteobettini in https://github.com/pytorch/rl/pull/850
• [BugFix] Fix zero-ing from specs in RewardSum by @vmoens in https://github.com/pytorch/rl/pull/860
• [Feature] Loading R3M and VIP from ResNet by @vmoens in https://github.com/pytorch/rl/pull/863
• [Feature] SAC V2 by @vmoens in https://github.com/pytorch/rl/pull/864
• [BugFix] Avoid collision of "step_count" key from transform and collector by @vmoens in https://github.com/pytorch/rl/pull/868
• [Refactor] Better init for CatFrames buffers + removing default init values by @vmoens in https://github.com/pytorch/rl/pull/874
• [Refactor] Minor refactorings to envs by @vmoens in https://github.com/pytorch/rl/pull/872
• [Refactor] Removing inplace transform attribute by @vmoens in https://github.com/pytorch/rl/pull/871
• [BugFix] Run checks when creating fake_td by @vmoens in https://github.com/pytorch/rl/pull/877
• [Refactor] Box device by @vmoens in https://github.com/pytorch/rl/pull/881
• [Feature] Multithreaded env by @sgrigory in https://github.com/pytorch/rl/pull/734
• [Refactor] Turn off default advantage normalization in PPO by @vmoens in https://github.com/pytorch/rl/pull/887
• [CI] Fix habitat-gym imports by @vmoens in https://github.com/pytorch/rl/pull/890
• [CI] Fix cuda versions by @vmoens in https://github.com/pytorch/rl/pull/889
• [CI] Fix windows install by @vmoens in https://github.com/pytorch/rl/pull/888
• MacOS CPU unit test workflow using GitHub Actions by @robandpdx in https://github.com/pytorch/rl/pull/886
• Linux CPU unit test workflow using GitHub Actions by @robandpdx in https://github.com/pytorch/rl/pull/826
• [Major, BugFix, Test] Refactor Transforms tests by @vmoens in https://github.com/pytorch/rl/pull/878
• [Bugfix] Codecov does not cover multiprocessed tests #879 by @kadeng in https://github.com/pytorch/rl/pull/893
• [CI, BugFix] Fix gym related errors by @vmoens in https://github.com/pytorch/rl/pull/895
• [WIP] Linux GPU unit test workflow using GitHub Actions by @robandpdx in https://github.com/pytorch/rl/pull/885
• [BugFix] Compose cloning fix by @vmoens in https://github.com/pytorch/rl/pull/899
• [Feature] Simplifying collector envs by @vmoens in https://github.com/pytorch/rl/pull/870
• [CI,Feature] Upgrade to gymnasium by @vmoens in https://github.com/pytorch/rl/pull/898
• [Doc] Add record utils to doc by @vmoens in https://github.com/pytorch/rl/pull/904
• [Test] Improve exception message match by @apbard in https://github.com/pytorch/rl/pull/906
• [BugFix] Dreamer helpers are broken with batched envs by @vmoens in https://github.com/pytorch/rl/pull/903
• [Feature] RandomCropTensorDict transform by @vmoens in https://github.com/pytorch/rl/pull/908
• [Versioning] Version 0.0.4b by @vmoens in https://github.com/pytorch/rl/pull/909

New Contributors

• @sladebot made their first contribution in https://github.com/pytorch/rl/pull/435
• @rayanht made their first contribution in https://github.com/pytorch/rl/pull/432
• @brandonsj made their first contribution in https://github.com/pytorch/rl/pull/475
• @ordinskiy made their first contribution in https://github.com/pytorch/rl/pull/485
• @JasonMa2016 made their first contribution in https://github.com/pytorch/rl/pull/487
• @himjohntang made their first contribution in https://github.com/pytorch/rl/pull/477
• @romainjln made their first contribution in https://github.com/pytorch/rl/pull/512
• @apbard made their first contribution in https://github.com/pytorch/rl/pull/526
• @sicong-huang made their first contribution in https://github.com/pytorch/rl/pull/522
• @psolikov made their first contribution in https://github.com/pytorch/rl/pull/566
• @jrobine made their first contribution in https://github.com/pytorch/rl/pull/571
• @nikhlrao made their first contribution in https://github.com/pytorch/rl/pull/573
• @sgrigory made their first contribution in https://github.com/pytorch/rl/pull/580
• @jlesuffleur made their first contribution in https://github.com/pytorch/rl/pull/589
• @artkorenev made their first contribution in https://github.com/pytorch/rl/pull/593
• @paulomarciano made their first contribution in https://github.com/pytorch/rl/pull/614
• @hatala91 made their first contribution in https://github.com/pytorch/rl/pull/622
• @jgonik made their first contribution in https://github.com/pytorch/rl/pull/608
• @adityagandhamal made their first contribution in https://github.com/pytorch/rl/pull/637
• @svarolgunes made their first contribution in https://github.com/pytorch/rl/pull/632
• @adityagoel4512 made their first contribution in https://github.com/pytorch/rl/pull/615
• @jasonfkut made their first contribution in https://github.com/pytorch/rl/pull/641
• @sardaankita made their first contribution in https://github.com/pytorch/rl/pull/656
• @albertbou92 made their first contribution in https://github.com/pytorch/rl/pull/655
• @yingchenlin made their first contribution in https://github.com/pytorch/rl/pull/674
• @altre made their first contribution in https://github.com/pytorch/rl/pull/695
• @nickspell made their first contribution in https://github.com/pytorch/rl/pull/707
• @aakhundov made their first contribution in https://github.com/pytorch/rl/pull/716
• @acforvs made their first contribution in https://github.com/pytorch/rl/pull/733
• @EmGarr made their first contribution in https://github.com/pytorch/rl/pull/745
• @matteobettini made their first contribution in https://github.com/pytorch/rl/pull/767
• @riiswa made their first contribution in https://github.com/pytorch/rl/pull/772
• @yohann-benchetrit made their first contribution in https://github.com/pytorch/rl/pull/759
• @KamilPiechowiak made their first contribution in https://github.com/pytorch/rl/pull/794
• @robandpdx made their first contribution in https://github.com/pytorch/rl/pull/831
• @ShahRutav made their first contribution in https://github.com/pytorch/rl/pull/833
• @BY571 made their first contribution in https://github.com/pytorch/rl/pull/684
• @atonkamanda made their first contribution in https://github.com/pytorch/rl/pull/853
• @kadeng made their first contribution in https://github.com/pytorch/rl/pull/893

Full Changelog: https://github.com/pytorch/rl/compare/v0.0.2a...v0.0.4b