NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
PyPI
PyPI
A modular, primitive-first, python-first PyTorch library for Reinforcement Learning
Latest stable
0.15.0
Last release 12 days ago
10 Nov 2026
Ships fairly regularly
a new release about every 6 weeks
Nearly every release is documented
notes for 31 of 31 stable releases
Nothing withdrawn
no release was ever pulled
4 years old
37 releases · first in 2022
Drag across the ruler to compare, or tap a release to open it.
PyTorch 2.11 and 2.12 emit a DeprecationWarning; v0.17 will require PyTorch 2.13. #4595 by @theap06.
TorchRL 0.15 introduces powerful new capabilities for robot learning, offline RL, data interoperability, and training at scale. New primitives include ReplayBuffer.export with Hugging Face and LeRobot adapters, export_onnx_policy, MenagerieEnv, LowLevelController, TransformerModule, RewardModelLoss, StopOnSignal, and the TD-MPC2, Flow Q-Learning and Transitive RL objectives.
ReplayBuffer.export writes stored data in storage order or by trajectory and step through exporter adapters, with HuggingFaceDatasetAdapter for Hugging Face datasets and LeRobotDatasetAdapter for the LeRobot format. ReplayBuffer.as_dataset() lets a torch.utils.data.DataLoader read any replay buffer, and a tutorial walks through exporting MicroDuck rollouts to LeRobot. #4592, #4611 and #4628 by @bsprenger, #4387 by @theap06.TdMpc2Trainer, each with a Hydra config and usable on its own, plus a runnable recipe under sota-implementations/tdmpc2_trainer. #4457 and the stack it builds on by @bsprenger.export_onnx_policy writes feed-forward and recurrent TensorDict policies to ONNX with inputs and outputs named after their keys, MicroDuckEnv(policy_observation=True) matches the observation the robot runtime feeds its policy, MenagerieEnv loads any MuJoCo Menagerie robot by name, and LowLevelController lets a high-level policy drive frozen skills. #4626, #4625 and #4386 by @theap06, #4329 by @vmoens.FQLLoss, FQLTrainer, flow-matching and one-step policies, and a recipe under sota-implementations/fql. OfflineToOnlineTrainer is generalized so FQL can pretrain offline before training online. 0bd9892b and 5e402a4e by @neuriv.tensordict>=0.15.2,<0.16.0. #4595 and #4600 by @theap06.pip install torchrl on Python 3.10 still installs 0.15, with PyTorch 2.14. #4706 by @theap06.EnvBase.reset and EnvBase.rollout now default to set_state=False, completing the change announced in v0.14. Pass set_state=True if you relied on state entries in the input being applied. #4501 by @coder-jayp.auto_register_policy_transforms=True, the AsyncBatchedCollector and AsyncEnvPool backend defaults, and the unified Trainer checkpoint format. Their FutureWarnings stay until then. #4698 by @theap06.SharedMemWeightSyncScheme(atomic=True) gives workers private weight copies that switch only on update, so they never act on half-written weights. It becomes the default in v0.16; until then the default warns. #4695 by @theap06.DeprecationWarning; v0.17 will require PyTorch 2.13. #4595 by @theap06.PairwiseDataset and GPT2RewardModel are deprecated in favor of RewardModelLoss and will be removed in v0.17. #4434 and #4442 by @theap06.TdMpc2QEnsemble, TdMpc2Loss, TdMpc2Planner, TdMpc2OptimizationStepper and TdMpc2Trainer, following arXiv:2310.16828. Trainer.optim_steps() accepts per-call overrides for schedules such as pretraining. #4449, #4451, #4452, #4453, #4454, #4455, #4456, #4457 and #4424 by @bsprenger.FlowMatchingPolicy, OneStepPolicy, FQLLoss, FQLTrainer and a recipe under sota-implementations/fql. 0bd9892b and 5e402a4e by @neuriv.TransitiveQLoss (arXiv:2510.22512), matched numerically against the authors' JAX code, the SubgoalTriplet sample unit and an OGBench recipe. #4596, #4597 and #4598 by @theap06.RewardModelLoss, a Bradley-Terry pairwise loss for any sequence-scoring module, with a reward-model training recipe. #3922 and #4434 by @theap06.PPOTrainer.from_env builds a standard PPO setup from an env and actor/critic networks; GAE accepts a validity mask and GAE.for_recurrent(...) builds a recurrent-safe estimator. #4391, #4404 and #4413 by @vmoens.TransformerModule runs a causal transformer over [B, T] windows for training or one step at a time against a key/value cache for collection, with a reference CausalTransformer backbone. #4193 and #4432 by @theap06.export_onnx_policy exports feed-forward, LSTMModule and GRUModule policies to ONNX with inputs and outputs named after their keys (pip install torchrl[onnx]), and MicroDuckEnv(policy_observation=True) matches the robot runtime's observation. #4626 and #4625 by @theap06.LowLevelController, ClosedLoopMultiAction, MicroDuckSkillEnv and a skill zoo with MicroDuckSkills.from_pretrained, plus new MicroDuck head-level reward, turning and jumping tasks. #4329, #4361, #4392, #4393, #4394, #4395 and #4396 by @vmoens.MenagerieEnv loads any MuJoCo Menagerie robot by name, with a task config and a zero default reward that a transform can replace. #4386 by @theap06.LastAction and a public DoneTransform for group-level done flags. #4360 and #4356 by @YeonwooSung.Storage.as_dataset() and ReplayBuffer.as_dataset() for torch.utils.data.DataLoader, with tensordict_collate. #4387 by @theap06.ReplayBuffer.export with HuggingFaceDatasetAdapter and LeRobotDatasetAdapter. #4592 and #4611 by @bsprenger.RateLimitedReplayBuffer(samples_per_insert=...), ReplayBuffer.wait_until_sampleable() and sample(wait=True), and SliceSampler(output_layout="batch_time"). #4418, #4408 and #4414 by @vmoens.replay_write_mode="rollout" or "trajectory", report in-flight statistics, and expose worker_policy_versions(). #4412, #4416 and #4410 by @vmoens.Evaluator adds per-env episode sampling, seeded paired episodes and bootstrap confidence intervals; EvaluatorHook runs it on a schedule from a Trainer; LogValidationReward can record complete episodes. #4500 by @theap06, #4415 by @vmoens and #4482 by @quinnarnold.StopOnSignal turns SIGINT/SIGTERM into a stop and a final checkpoint; Trainer checkpoints include RNG state and load the newest file from a rotation directory; recipes resume with resume=<dir>. #4380, #4381, #4384 and #4385 by @vmoens.Logger.with_prefix("name") places metrics under a namespace, and the SOTA recipes log under training/, evaluation/ and timing/. #4417, #4440, #4403 and #4420 by @vmoens.ParallelEnv workers that report their own metadata now use shared buffers instead of pipes. #4341 by @vmoens.fullgraph=True. #4645 and #4648 by @bsprenger.CQLLoss computes the CQL(H) regularizer and Lagrange update of the paper and reference code; CQLLoss, DiscreteCQLLoss, IQLLoss, DiscreteIQLLoss, REDQLoss, CrossQLoss and QMixerLoss apply prioritized-replay weights; HardUpdate copies every value_network_update_interval steps instead of every interval + 1; and PrioritizedSliceSampler respects episode boundaries after a multi-env storage wraps and sets is_init at slice starts. #4693, #4694 and #4700 by @theap06.MultiProcessWeightSyncScheme workers no longer read the trainer's live CPU weights and see new weights only after update_policy_weights_(), Collector(AsyncEnvPool(...)) no longer crashes on the first batch, AsyncEnvPool(exchange="shm") works with outer transforms and partial resets, syncing a non-policy model no longer hangs, and multi-process collectors with a replay buffer and a batched env no longer crash. #4701 by @theap06. Ray collector.stats() responds while collecting. #4400 by @coder-jayp.RewardSum sums along time (#4539 by @aalquwayfili), StepCounter with several done groups (#4587 by @mottopanikeiku), RandomCropTensorDict last window (#4578 by @jackiectl2), spec fixes (#4548 by @coder-jayp, #4556 by @YeonwooSung, #4591 by @mottopanikeiku).SliceSampler span edge cases (#4700 by @theap06), compact checkpoints of partially filled buffers (#4644 by @theap06), memory-mapped checkpoints under TensorDict 0.16+ (#4551 by @peterdsharpe).MultiAgentMLP export with shared parameters (#4700 by @theap06), MCAdvantage on constant groups (#4610 by @catherinelee274), compiled DreamerV3 reward decoding on CUDA (#4657 by @arun-elr), TD3 with CUDA graphs (#4636 by @peterdsharpe), detached optimizer metrics (#4561 by @bsprenger), IncrementalTokenizer prompts after tool calls (#4674 by @arun-elr).pip install "torchrl==0.15.0" "tensordict>=0.15.2,<0.16.0".pip install "https://github.com/pytorch/rl/releases/download/v0.15.0/torchrl-0.15.0+cu132-cp312-cp312-manylinux_2_28_x86_64.whl".loss_cql values. Users of the CQL, IQL, REDQ, CrossQ and QMixer losses with prioritized replay can pass use_prioritized_weights=False to reproduce v0.14 numbers.r_training, done_percentage, train/, eval/ or time/ need the new metric names, or pass telemetry="minimal" to the on-policy trainers.SharedMemWeightSyncScheme(atomic=True) to opt into the v0.16 behavior now, or atomic=False to keep the current one without a warning.For the complete commit-by-commit history, see v0.14.0...v0.15.0.
Thanks to @vmoens, @theap06, @bsprenger, @peterdsharpe, @coder-jayp, @huydhn, @YeonwooSung, @neuriv, @arun-elr, @mottopanikeiku, @sramireddy2, @aalquwayfili, @catherinelee274, @itwasabhi, @jackiectl2 and @quinnarnold for the changes included in this release.
We especially welcome first-time contributors @peterdsharpe, @neuriv, @arun-elr, @mottopanikeiku, @sramireddy2, @aalquwayfili, @catherinelee274 and @jackiectl2.
Your coding agent can read these notes before it upgrades. Set up the MCP server →
It also corrects transform device handling and introduces the v0.17 device deprecations described below.
TorchRL 0.14.1 is a maintenance release covering environment resets, collectors, replay buffers, planning, training, logging, and LLM workflows. It also corrects transform device handling and introduces the v0.17 device deprecations described below.
An upcoming TorchRL release will require torch>=2.13 (discussed in #4523). The existing lower bound on the torch version is 2.1.0, so TorchRL can be installed next to torch releases that lack APIs it calls. Source builds and NGC containers of torch 2.13, which report versions such as 2.13.0a0+<sha>, will also meet the requirement.
TorchRL 0.14.1 and earlier releases keep working with older torch. Once the requirement is released:
Release builds are configured for the latest stable PyTorch, 2.14.1. TorchRL 0.14.1 requires TensorDict >=0.14.3,<0.15.0.
Install the release with:
pip install "torch==2.14.1" "torchrl==0.14.1" "tensordict>=0.14.3,<0.15.0"TransformedEnv without an additional transform now preserves its inner transforms and constructs successfully. #4336 by @vmoens.ParallelEnv(use_buffers=False) preserves the observations and done flags of workers left untouched by a partial reset. Consolidation also avoids locking the caller's reset data. #4339 by @vmoens.RandomTruncationTransform reads nested step counters and writes the corresponding nested truncation and done flags. Optional step_count_key, truncated_key, and done_key arguments support customized keys, and partial resets preserve other workers' horizons. #4350 by @YeonwooSung.TicTacToeEnv computes wins independently across batched boards, handles mixed finished and active single-player games, and accepts rand_action(None). See the return-type migration note below for direct calls to win. #4349 by @YeonwooSung.GymEnv imports ale_py before resolving Atari NoFrameskip IDs and ale_py:-prefixed IDs, as well as the existing ALE/ namespace. This fixes environment registration where plugins are not loaded automatically. #4358 by @YeonwooSung.SerialEnv and ParallelEnv preserve worker batch dimensions when partial-step masks select whole workers. A mask selecting only part of a worker now raises ValueError; place the partial-step transform inside each worker for that case. BatchSubSampler treats leading batch dimensions as independent trajectories. #4389 by @vmoens.ActionDiscretizer constructs exactly the requested number of bins, including for counts where floating-point step accumulation previously produced an extra bin. #4443 by @Nicholas022400701.Choice.squeeze(dim) also squeezes its constituent choices, keeping their shapes consistent with the returned spec. #4462 by @yupengtang.Bounded.rand can sample its inclusive upper bound and avoids overflow when bounds span a wide range. #4497 by @jayzuccarelli and #4498 by @theap06.MultiCategorical.to_one_hot handles batched specs and scalar samples from a shape-[1] spec. #4504 by @Nicholas022400701.DirectRLEnvWarp. #4194 by @theap06.ParallelEnv shutdown errors now name the closed environment class, and DiscreteCQLLoss missing-key errors include the actual Q-value key. #4496 by @David-Wu1119.TensorDictMaxValueWriter restores checkpointed heap entries with the types required for subsequent writes. Restored replay buffers can continue adding data and replacing their lowest-ranked entries without comparison errors. #4370 by @yupengtang.CEMPlanner and MPPIPlanner retain the full planning horizon while excluding rewards after the first termination or truncation from trajectory scores. The reward on the done step is retained, and nested done groups follow environment precedence. #4359 by @YeonwooSung.SACLossConfig forwards only the fields accepted by the selected continuous or discrete SAC loss, preventing constructor errors from variant-specific arguments. #4338 by @yupengtang.CrossQLoss.set_keys works after a value estimator has been created. Renamed reward, done, and terminated keys propagate to the estimator without accessing a nonexistent value key. #4372 by @yupengtang.env attribute when the loss partial supplies action_spec or bounds. Explicit action domains are preserved; collectors with an environment still provide inferred bounds when needed. #4368 by @aswanth-07.ReplayBuffer.can_sample() holds the replay and write locks while checking sampler readiness, avoiding races with concurrent writers. #4439 by @vmoens.ParameterScheduler honors explicit zero minimum and maximum values and omits its backend module from state_dict, making saved state serializable. #4444 and #4445 by @Nicholas022400701.CatTensorsConfig uses the configured concatenation key, and InitTrackerConfig defaults to the same is_init key as InitTracker. #4426 and #4428 by @bsprenger.GRPOLoss handles reference log-probability padding and custom reference keys correctly, while DAPO loss construction no longer fails on an invalid initializer. #4363 by @coder-jayp.GAEConfig.average_gae now matches the GAE constructor default. See the migration note below if you relied on the previous config default. #4472 by @YeonwooSung.ReinforceLoss scores actions stored in the input tensordict under the actor's current distribution, including composite actions, instead of sampling replacement actions. #4525 by @theap06.MultiAsyncCollector checks worker liveness after every batch and honors update_at_each_batch during policy updates. #4505 by @theap06.MultiSyncCollector and MultiAsyncCollector assign fresh trajectory IDs on iteration resets while partial resets keep unaffected trajectories. #4515 by @theap06.RayCollector releases completed object references without calling the removed ray.internal.free API. #4519 by @coder-jayp.WandbLogger creates its save directory before initializing a run, so logging can start when that directory does not yet exist. #4405 by @vmoens.CSVLogger counts text and video steps independently. Loading legacy checkpoints also preserves existing text and hyperparameter files by continuing their numbering instead of overwriting them. #4374 by @yupengtang.log_dir=None and store configured directories as absolute paths, so resuming after a working-directory change uses the original location. WandbLogger rejects loading state from a different live run instead of silently changing its recorded identity. Logger fixes from #4382 by @vmoens.vLLMWrapper no longer fills missing prompt log-probabilities with invented zero scores or exposes response-only scores as a full sequence. Full-sequence alignment and GRPO/KL masking remain correct when vLLM generation omits prompt scores. #4255 by @YeonwooSung.LLMCollector(yield_only_last_steps=True) counts intermediate dialog turns toward total_dialog_turns, even though it returns only final steps. Asynchronous prefetch also accounts for pending and in-progress turns when deciding whether to launch more work. #4354 by @YeonwooSung.ChatEnv.reset preserves roles and conversation structure when its query contains a History or chat-message list. Existing conversations, including batched non-tensor wrappers, receive the configured system prompt correctly. #4355 by @YeonwooSung.IncrementalTokenizer derives the reusable full-token key from the complete configured NestedKey, preserving deep prefixes and handling string and single-component keys consistently. #4351 by @YeonwooSung.Tokenizer follows its parent environment's current device for token tensors and attention masks instead of caching an earlier device. The output destination is exposed through out_device. #4352 by @YeonwooSung.ModuleTransform, KLRewardTransform, and RetrieveLogProb place wrapped models on the constructor device and update an explicit input/output placement policy when .to(device) is called. KLRewardTransform registers its reference model as a submodule so device moves reach it. #4353 by @YeonwooSung.BrowserTransform starts a usable event loop when called from a thread without one. #4485 by @David-Wu1119.GAEConfig.average_gae now defaults to False instead of True, matching direct GAE construction. Set average_gae=True explicitly to retain the previous Hydra-configured behavior. #4472 by @YeonwooSung.InitTrackerConfig.init_key now defaults to "is_init" instead of None, matching InitTracker. Set init_key=None explicitly if your configuration relied on the old value. #4428 by @bsprenger.WandbLogger.load_state_dict raises RuntimeError when the saved run ID differs from the active run ID. Construct the logger with the saved ID and resume="must" before loading, or use get_logger(..., state_dict=saved_state) to resume the saved run. #4382 by @vmoens.LogProbs.prompt remains unset (None) and LogProbs.full contains NaN at unavailable prompt positions while retaining prompt-plus-response length. Custom consumers should mask these alignment entries; GRPOLoss and KLComputation exclude unavailable prompt positions from their computations. Use generate=False when prompt scores are required. #4255 by @YeonwooSung.TicTacToeEnv.win now returns a boolean tensor with shape [..., 1] instead of a Python boolean. Direct callers should handle the batch-shaped result; use .item() only when a scalar result is required. #4349 by @YeonwooSung.Tokenizer.device is deprecated and will be removed in v0.17. Use Tokenizer.out_device, which follows the parent environment; None leaves outputs on the tokenizer's chosen device. #4352 by @YeonwooSung.device attributes of ModuleTransform, KLRewardTransform, and RetrieveLogProb are deprecated and will be removed in v0.17. They describe an explicit input/output placement policy, not every parameter or buffer's location. Place models with the constructor device= argument or .to(device), inspect individual tensors when checking placement, and let wrapped modules own device routing. Until v0.17, constructor device= retains the I/O policy and later .to(device) updates it; dtype-only moves leave it unchanged. #4353 by @YeonwooSung.release/2.14; the package version files are set to 0.14.1. d844b7b8f by @vmoens._step() methods, and action, reward, and done keys across multiple groups, with a two-team example. #4232 by @YeonwooSung.linux_job_v3 job. #4524 by @huydhn.NoisyLinear, NoisyLazyLinear, and reset_noise documentation clarifies that forward passes reuse existing noise. Call module.apply(reset_noise) when a fresh noise sample is needed, including after optimization steps. #4357 by @YeonwooSung.Thanks to @aswanth-07, @bsprenger, @coder-jayp, @David-Wu1119, @huydhn, @jayzuccarelli, @Nicholas022400701, @theap06, @vmoens, @YeonwooSung, and @yupengtang for the changes included in this release.
The deprecated MultiCollector.postprocs alias has been removed. Use MultiCollector.postproc. #4108 by @vmoens.
TorchRL 0.14 is a major feature release spanning model-based reinforcement learning, distributed training and inference, replay-buffer composition, checkpointing, on-policy optimization, vision-language-action workflows, and simulation tooling. It also tightens several contracts announced in earlier releases and requires TensorDict 0.14.2 or later in the 0.14 release line.
DreamerV3BlockGRU modules, Triton and CUDA-graph acceleration, corrected losses and continuation semantics, reusable percentile return normalization, Hydra-compatible model configuration, and a reproducible DMC Walker benchmark. See #4072, #4074, #4075, #4125, #4126, #4132, #4133, #4158, #4160, #4174, #4187, #4196, and #4216 by @vmoens, @theap06, and @bsprenger, with correctness fixes in #4119 by @gtnv and #4118 and #4128 by @unography.GRPOTrainer, PPO-EWMA with a decoupled proximal policy, reusable percentile value normalization, and group-wise value-estimator standardization. See #4140 by @coder-jayp and #4126, #4127, #4212, #4213, and #4214 by @vmoens.MicroDuckEnv multitask locomotion environment and training examples, and richer torchrl.render notebooks and MuJoCo WASM tooling. See #4107, #4183, #4203, #4204, #4205, #4212, and #4218 by @vmoens.A single multitask policy handles forward walking, backward walking, sidestepping, and hopping. This checkpoint was filmed after 27.95 million transitions with one native simulator per tile; the top row walks forward and backward, while the bottom row sidesteps and hops.
Watch or download the MP4 (0.8 MB). Training details and learning curves are in #4212.
The tables below reproduce measurements reported in the linked pull requests. Hardware, tensor shapes, dependency versions, and benchmark scope differ between tables; comparisons are meaningful within each table, not across them.
On an NVIDIA GB200 with BF16 activations, FP32 parameters, batch 16, hidden and projection width 512, eight blocks, and one layer, the fused Triton block-GRU substantially reduces recurrent execution time relative to the scan backend. #4160
| Sequence length | Backend | Forward | Forward + backward | Peak memory |
|---|---|---|---|---|
| 64 | scan | 65.426 ms | 115.099 ms | 0.160 GB |
| 64 | Triton | 4.223 ms | 7.443 ms | 0.149 GB |
| 512 | scan | 540.584 ms | 855.082 ms | 0.782 GB |
| 512 | Triton | 28.273 ms | 62.608 ms | 0.690 GB |
A recompute-based Triton backward then reduced saved-state materialization and peak memory on the same hardware; forward timings remained unchanged within noise. #4174
| Shape | Earlier forward + backward | Recompute backward | Peak memory |
|---|---|---|---|
| B=256, T=512, H=128, P=64 | 26.943 ms | 19.589 ms | 1.612 -> 0.807 GB |
| B=2048, T=64, H=128, P=64 | 14.307 ms | 6.883 ms | 1.614 -> 0.809 GB |
| B=2048, T=512, H=128, P=64 | 103.401 ms | 43.963 ms | 12.453 -> 5.578 GB |
| B=16, T=512, H=P=512 | 63.405 ms | 62.694 ms | 0.690 -> 0.355 GB |
For batch 16, sequence length 64, and belief width 512 on an NVIDIA A10G, compiled higher-order scan improves RSSM inference and training throughput. Unrolling the scan by eight retains that speed while sharply reducing its training allocation. #4132
| RSSM execution | Inference | Forward + backward | Training peak allocation |
|---|---|---|---|
| eager loop | 90.1 ms | 216.2 ms | 43.24 MiB |
| compiled loop step | 49.9 ms | 134.6 ms | 36.35 MiB |
| compiled scan, unroll 1 | 22.78 ms | 101.00 ms | 210.18 MiB |
| compiled scan, unroll 8 | 18.01 ms | 93.04 ms | 58.89 MiB |
At the complete-learner level on an NVIDIA GB200, CUDA-graph capture removes thousands of per-update launches and turns the full model, actor, value, replay-value, optimizer, and slow-target update into the fastest measured path. Compilation and capture warm-up, replay sampling, and environment collection are excluded. #4216
| Learner implementation | Median per update | Transitions/s | Relative result |
|---|---|---|---|
| pinned upstream DreamerV3 JAX | 22.04 ms | 46,468 | 1.00x |
| TorchRL compiled scan | 358.51 ms | 2,856 | 16.3x slower |
| TorchRL compiled scan + CUDA graph | 17.83 ms | 57,415 | 1.24x faster |
On an NVIDIA GB200 with 16 environments, 5 ms environment latency, and a two-layer 1024-unit policy, consolidated direct process slots nearly double collection throughput relative to the shared-memory coordinator. Grouping four environments per worker instead targets host-memory efficiency. #4272
| Transport | Frames/s | Median batch latency | Host RSS |
|---|---|---|---|
| shared-memory coordinator, one env/worker | 1,009.81-1,061.53 | 240.45-242.73 ms | 15,312-15,375 MiB |
| shared-memory coordinator, four envs/worker | 1,072.05-1,077.66 | 237.55-238.14 ms | 5,459-5,969 MiB |
| direct process slots, consolidated transfer | 1,940.39-2,024.66 | 115.61-122.17 ms | 14,763-15,781 MiB |
For a 170.8-million-parameter policy on the same GPU class, static CUDA-graph batches raise inference-server request throughput across batch sizes. These are server-only measurements: the integrated A10G collection workload in the same PR remained about 9-10% slower than eager end to end, which identifies collation and transport as separate optimization targets. #4269
| Batch | Eager requests/s | CUDA-graph requests/s | Eager median round trip | Graph median round trip |
|---|---|---|---|---|
| 1 | 171.58-183.98 | 538.48-577.29 | 5.36-5.75 ms | 1.72-1.79 ms |
| 16 | 1,248.55-1,303.08 | 3,465.92-3,587.45 | 11.44-12.03 ms | 4.32-4.49 ms |
| 64 | 1,647.64-1,692.65 | 5,291.55-5,431.54 | 32.35-34.10 ms | 11.25-11.58 ms |
The combined process-slot, pinned-staging, replay write-back, and setup-heap stack was measured on the same 64-environment pixel workload and 4-GPU node for both runs. The train ratio couples environment and learner rates, so both improve together. This is a stack-level result and is not attributed to any one PR. #4305, #4306, #4307, and #4308
| Configuration | Environment steps/s | Learner updates/s |
|---|---|---|
| before the stack | 361 | 0.70 |
| with the stack | 1,249 | 2.44 |
| measured speedup | 3.46x | 3.46x |
Finally, paired 9.1-hour runs show the effect of sending at least 64 consecutive transitions per process-worker message. The gap remained between 17% and 20% in every half-hour window. #4315
| Process-slot chunking | Whole-run steps/s | Whole-run updates/s | Steady-window steps/s | Steps in 9.1 hours |
|---|---|---|---|---|
| 16 transitions/message | 1,018 | 1.94 | 1,000-1,033 | 33.3M |
| 64 transitions/message | 1,219 | 2.33 | 1,198-1,245 | 39.9M |
QValueModule and QValueActor now default to strict action-shape validation. A mismatch with the action spec raises RuntimeError; pass strict_shape="auto" to reshape compatible outputs or strict_shape=False to disable validation. Explicit legacy None follows the new strict behavior. #4179 by @vmoens.MultiCollector.postprocs alias has been removed. Use MultiCollector.postproc. #4108 by @vmoens.tensordict>=0.14.2,<0.15.0; upgrade TorchRL and TensorDict together. #4108 and #4212 by @vmoens; the updated dependency floor is tracked in #4319.use_ray_service switch is deprecated in favor of explicit service_backend selection and is scheduled for removal in v0.16. ca3261f52 by @vmoens.Trainer checkpoint format will move from the legacy backend to torchrl.checkpoint in v0.15, and the compatibility checkpoint argument is scheduled for removal in v0.16. #3967 by @vmoens.AsyncEnvPool without an explicit exchange now warns that the default will change from "queue" to "auto" in v0.15. Select "queue", "shm", or "auto" explicitly to pin the desired behavior. #4200 by @vmoens.WorldModel, WorldModelWrapper, and WorldModelLoss provide a general TensorDict-native abstraction for model-based environments, rollouts, and joint objectives. #3783 by @theap06.DreamerV3BlockGRU and DreamerV3BlockGRUCell expose sequence-first block-GRU modules with a Python implementation and an optional Triton backend whose backward pass avoids materializing block-expanded activations. #4158, #4160, and #4174 by @vmoens.DreamerV3MLPConfig exposes DreamerV3's specialized MLP through the trainer Hydra configuration system. #4187 by @bsprenger.PolicyClientModule, ProcessInferenceServer, structured inference-server configs, behavior-policy version tracking, and max-inflight controls form a complete remote-policy client path. 307eb5a64, 938d0abd1, cd1406e38, and 539aa07d8 by @vmoens.WeightSyncScheme publication. 311c1b13f, 6e60c472d, 12bef231d, and dc6dcb8a5 by @vmoens.AsyncEnvPool gains shared-memory exchange and automatic parallel environments expose worker metadata. #4022 and #4093 by @vmoens and #4109 by @theap06.AsyncEnvPool(exchange="auto") selects shared-memory transport for compatible multiprocessing environments and falls back to queues for dynamic, heterogeneous, or non-tensor schemas; resolved_exchange reports the selected transport. #4200 by @vmoens.ReplayBuffer.stats(), collector statistics across local, multiprocessing, and Ray collectors, plus LoggerMonitor and Every. #4031, #4032, and #4033 by @theap06.traj, Trajectory, predicates, filters, and iterators for selecting replay data through composable trajectory expressions. #3947 by @theap06.SampleUnit, Transition, and Sequence separate sampling semantics from storage. Sequence sampling supports exact boundary policies, burn-in, bootstrap context, dilation, per-anchor priorities, and multidimensional storage. #4045, #4051, and #4085 by @theap06 and #4050 by @coder-jayp.SliceSampler(fragmented=True, step_key=...) reconstructs fixed-length logical trajectory slices when trajectories are interleaved or non-contiguous in one-dimensional TensorDict-backed storage. #4180 by @vmoens.OfflineToOnlineReplayBuffer, prefill_replay_buffer, and OfflineToOnlineTrainer support staged transitions from static datasets to online collection, with a runnable SOTA implementation. #3900 and #3904 by @theap06.torchrl.checkpoint package handles structured state, adapters, strictness policies, RNG state, best-checkpoint selection, rotation, retention, and dependency-version diagnostics. #3967 by @vmoens and #4064 and #4098 by @theap06.ActionChunkTransform, VLAWrapperBase, TinyVLA, and LeRobotPolicyWrapper cover data preparation through policy execution. b6048dc43, e9a26b5be, a777bc4bc, and #3884 by @vmoens.CraftGroundEnv and CraftGroundWrapper integrate CraftGround's Minecraft environments with explicit pixel and action specs and composable TorchRL rewards and termination. #4107 by @vmoens.MujocoEnv.geom_contacts() and site_positions() expose backend-independent simulation signals, and patch_xml=False preserves local model paths so native MuJoCo, MJX, and MuJoCo-Torch can resolve relative assets. #4190 and #4204 by @vmoens.MicroDuckEnv provides backend-independent biped locomotion with tensorclass task libraries, weighted reward registration, task sampling, standing, tracking, sidestepping, and jumping presets. Recurrent PPO, MJLab PPO, closed-form gait, checkpoint-backed rendering, and evaluation-video examples cover the training workflow. #4183, #4204, #4205, #4212, and #4218 by @vmoens.VideoClipRef adds lazy, compact, device-aware video decoding, while torchrl.render can produce videos, checkpoint-backed artifacts, live-parameter notebooks, and MuJoCo WASM viewers with an optional bundled Node runtime. #3824, #3826, #3839, a38ea665c, 54f6d9ccd, and #4203 by @vmoens.RNDLoss, the RND transform, and a PPO+RND MuJoCo implementation add intrinsic-reward training with a complete runnable example. #3889 and #3905 by @theap06.TQCLoss adds truncated quantile critics, list-valued critic updates, and numerical-contract coverage. #4077, #4078, and #4079 by @gtnv.DistillationLoss supports token-level LLM knowledge distillation, and TRL interoperability adapters translate policies, rollouts, and training data between TRL and TorchRL. #4038 by @theap06 and #4070 by @coder-jayp.GRPOTrainer, GRPOOptimizationStepper, and WeightSyncHook provide a modular GRPO training loop with gradient accumulation, mixed precision, clipping, and asynchronous synchronization to distributed LLM inference engines; synchronous and asynchronous SOTA pipelines exercise the complete path. #4140 by @coder-jayp.PercentileValueNorm tracks an exponential moving average of configurable quantiles, while ValueNorm.scale() exposes scale-only normalization. PPO and A2C losses accept advantage_norm for robust, opt-in advantage scaling. #4126 and #4127 by @vmoens.delay_actor and capped behavior-policy ratios through max_importance_ratio. Trainer and Hydra integration can update the proximal policy with SoftUpdate or HardUpdate, including a runnable PPO-EWMA configuration. #4213 and #4214 by @vmoens.GAE, VTrace, TD0Estimator, TD1Estimator, and TDLambdaEstimator accept group_key to standardize advantages or rewards independently within groups such as tasks or agents. #4212 by @vmoens.AsyncBatchedCollector supports compile-safe pausing, background replay collection, direct replay routing, CPU affinity, multi-environment workers, process-hosted inference, and fast asynchronous defaults. #4263, #4266, #4270, #4271, #4272, #4275, and #4313 by @vmoens.LBForagingEnv integrates Level-Based Foraging through the standard TorchRL environment interface. #4301 by @Iliamsou.The feature sections above describe the complete workflows. This index collects the principal public entry points for readers upgrading code or exploring the API:
WorldModel, WorldModelWrapper, WorldModelLoss, DreamerV3BlockGRU, DreamerV3BlockGRUCell, and DreamerV3MLPConfig. #3783, #4158, #4160, #4174, and #4187.traj, Trajectory, SampleUnit, Transition, Sequence, OfflineToOnlineReplayBuffer, prefill_replay_buffer, and the torchrl.checkpoint package. #3947, #4045, #3900, and #3967.PolicyClientModule, ProcessInferenceServer, WeightSyncScheme, ReplayBuffer.stats(), LoggerMonitor, and Every.AsyncBatchedCollector, process-hosted inference, background replay collection, native stream replay, and AsyncEnvPool.resolved_exchange. #4263, #4270, #4274, and #4200.RNDLoss, TQCLoss, DistillationLoss, GRPOTrainer, GRPOOptimizationStepper, WeightSyncHook, and PercentileValueNorm. #3889, #4077, #4038, #4140, and #4126.CraftGroundEnv, CraftGroundWrapper, MicroDuckEnv, LBForagingEnv, MujocoEnv.geom_contacts(), and MujocoEnv.site_positions(). #4107, #4183, #4301, and #4190.VideoClipRef, ActionChunkTransform, VLAWrapperBase, TinyVLA, and LeRobotPolicyWrapper, together with the torchrl.render artifact and notebook workflows. #3824 and #3884.Recurrent kernels gain tiled large-hidden support, 64-bit offsets, recompute-based backward passes, narrower canonicalization, specialized GRU backward scans, and hardened Triton execution. #3818, #3752, #4154, and #4175 by @vmoens.
DreamerV3 RSSM rollout avoids unnecessary Python and tensor work, and the block-GRU scan adds optimized recurrence plus a Triton backward pass that reduces runtime and peak activation memory. #4132, #4159, #4160, and #4174 by @vmoens.
The DreamerV3 DMC reproduction supports an opt-in CUDA-graph learner path for fixed-shape updates. On the reference GB200 benchmark, complete learner updates improve from 358.51 ms with the compiled scan to 17.83 ms with CUDA graphs, excluding compilation and capture warm-up. #4216 by @vmoens.
MuJoCo-Torch batched physics steps compile as one full graph by default when compile_step=True, preventing silent eager fallback after graph breaks or recompilation limits. #4202 by @vmoens.
MultiSyncCollector preemption now uses a shared-memory interruptor instead of busy-waiting, with macOS support, and stops gathering as soon as all worker outputs arrive. #3843 and #3841 by @vmoens.
Replay-boundary queries are cached and vectorized, TransformedEnv specs lock by default, and parent-side LIBERO construction is eliminated. #4084 by @theap06, #4121 by @mathieuorhan, and #4028 by @vmoens.
Async environment and collector execution caches setup metadata, bounds shared mappings, batches shared-memory coordination, and reduces process-slot transfer overhead. #4258, #4261, #4264, #4305, and #4306 by @vmoens.
DreamerV3 uses native replay collection, compiles the complete learner step, selects its compile strategy automatically, keeps replay write-back off the learner stream, and freezes the setup heap before training. #4265, #4268, #4302, #4307, and #4308 by @vmoens.
Process-slot collectors send at least 64 consecutive transitions per worker message by default, improving driver throughput while preserving an explicit low-latency override. #4315 by @vmoens.
TanhNormal scoring, multidimensional DiffusionActor training, sparse masked actions, singleton IQL gradients, PPO execution, lazy target initialization, TruncatedNormal dtype preservation, DreamerV3 reparameterized return normalization, shared MLP output bias, V-trace bootstrapping at in-batch truncations, compile-friendly analytic entropy and KL selection, and device-safe percentile quantiles. #4080, #4148, #4149, and #4150 by @gtnv, #4114 by @theap06, #4113 and #4137 by @aswanth-07, #4125 by @vmoens, #4169 by @quinnarnold, #4210 by @yurekami, #4234 by @YeonwooSung, and #4241 by @coder-jayp.ParallelEnv transport. Compiled stochastic policies keep fresh random samples under CUDA graphs, async receive deadlines bound the complete call without losing partial results, queue-backed pools shut down without deadlocking, stacked composite specs propagate lock state, and batched native MuJoCo workers receive distinct reproducible seeds. #4147 and #4146 by @gtnv, #4027 by @theap06, #4143 by @coder-jayp, #3867 by @discobot, and #3873, #4184, #4197, #4198, and #4208 by @vmoens.GradScaler location on PyTorch releases older than 2.3. #4189 by @bsprenger and #4244 by @coder-jayp.KLRewardTransformConfig and StorageEnsembleWriterConfig now resolve to importable implementations with matching public fields. #4236 by @YeonwooSung.ChessEnv.rand_action(tensordict) now samples from the action mask represented by the supplied mask or FEN/PGN state, including through the public transformed environment. #4239 by @YeonwooSung.WandbLogger forwards its configured base URL during authentication, and runtime diagnostics report CUDA timing and synchronization context more precisely. #4259 and #4260 by @vmoens.TensorDictPrimer state lifecycle, and the MicroDuck tutorial covers task libraries, reward registration, task sampling, per-task advantages, backends, gait control, and training. #4181 and #4219 by @vmoens, #4123 by @coder-jayp, and #4220 by @YeonwooSung.DQNLoss with ValueEstimators.TDLambda, including greedy next-action bootstrapping. #4233 by @YeonwooSung.Examples have systematic CI execution, AsyncEnvPool has continuous dispatch and throughput benchmarks, and regression coverage was strengthened for noisy layers, SFT loss, reward-to-go values, tensor lambdas, chat-template round trips, optional RSSM scan dependencies, and repository-independent render test paths. #4152 and #4182 by @vmoens, #4163, #4168, #4170, and #4171 by @quinnarnold, #4235 by @YeonwooSung, and #4242 and #4243 by @coder-jayp.
Release maintenance refreshes the expert-iteration NLTK and Transformers dependencies, records the repository contribution contracts, and exercises MuJoCo-Torch from main alongside nightly PyTorch on macOS. #4199, #4201, #4207, and #4211.
Continuous benchmarks track async collection across merges, direct process-slot collection, benchmarked commit identities, and baseline collector and DreamerV3 measurements. Performance alerts no longer prevent result publication, and labelled merges can run the complete benchmark suite. #4287, #4289, #4292, #4293, #4298, and #4312 by @vmoens.
The DreamerV3 process-inference benchmark selects the transport-owned environment exchange so both thread and process series remain runnable. #4320 by @vmoens.
Regression coverage initializes MicroDuck fixtures deterministically, exercises prioritized-replay weights in DDPG, and preserves metadata in static inference batches. #4279 by @coder-jayp and #4288 and #4303 by @vmoens.
Contributor guidance records target-device tensor and module construction as a repository contract. #4314 by @vmoens.
pip install "torchrl==0.14.0" "tensordict>=0.14.2,<0.15.0".QValueModule or QValueActor with shaped categorical specs. Choose strict validation, automatic reshaping, or explicit opt-out rather than relying on the former warning-only default.MultiCollector.postprocs with MultiCollector.postproc, and review the v0.15 checkpoint and collector-default warnings before the next release.exchange to multiprocessing AsyncEnvPool instances to keep queue transport or opt into automatic shared-memory selection before the default changes.For the complete commit-by-commit history, see v0.13.3...v0.14.0.
We are delighted to announce that @theap06 is joining @vmoens as a maintainer of TorchRL. His exceptional work across model-based reinforcement learning, replay buffers, checkpointing, and the library as a whole has made TorchRL substantially stronger. Thank you, @theap06, for the care, energy, and extraordinary amount of work you have brought to the project. We are looking forward to building and shipping what comes next together, alongside the entire TorchRL community.
Thanks to @Agade09, @aswanth-07, @bsprenger, @coder-jayp, @discobot, @fallintoplace, @gtnv, @harryfrzz, @Iliamsou, @itwasabhi, @lin-erica, @mathieuorhan, @ParamThakkar123, @quinnarnold, @saputkin, @theap06, @unography, @vmoens, @xyf5432, @YeonwooSung, @younik, and @yurekami for the changes included in this release.
We especially welcome first-time contributors @Agade09, @aswanth-07, @discobot, @fallintoplace, @gtnv, @harryfrzz, @Iliamsou, @quinnarnold, @saputkin, @unography, @xyf5432, @YeonwooSung, and @yurekami.
No breaking changes are intended in this release.
TorchRL 0.13.3 is a patch release focused on correctness, state restoration, device metadata, and environment reliability. It also includes the backward-compatible value-estimator chunk-dimension option from #4003.
value_chunk_dim, allowing value estimators and their Hydra configs to chunk along a selected batch dimension while preserving the existing default.TensorDict device metadata before collector writes into replay-buffer storage, without moving incompatible tensor leaves.PrioritizedSampler.load_state_dict, preventing live state from aliasing caller-owned or memory-mapped tensors.SliceSamplerWithoutReplacement and PrioritizedSliceSampler despite their multiple-inheritance method resolution order.ParallelEnv default to pipe transport for MPS-backed environments and stages pipe data through CPU memory.MINARI_DATASETS_PATH around minari.load_dataset, allowing datasets downloaded into TorchRL's cache to load correctly.AUTO_UNWRAP_TRANSFORMED_ENV variable without leaking the literal string "None" into child processes.pyvers>=0.2.3, ensuring lazy implement_for wrappers are pickleable before their first dispatch.RewardScaling round-trip assertion failure near zero.value_chunk_dim, including the matching trainer configuration field.No breaking changes are intended in this release.
No new package-level public symbols are exported by this patch. The additive value_chunk_dim constructor/configuration option is documented under #4003.
Thanks to @Agade09, @coder-jayp, @discobot, @fallintoplace, @lin-erica, @theap06, @vmoens, and @younik for the changes included in this release.
TorchRL 0.13.2 is a patch release focused on regression fixes, release reliability, and CI stability. It intentionally avoids new feature backports.
TorchRL 0.13.2 is a patch release focused on regression fixes, release reliability, and CI stability. It intentionally avoids new feature backports.
reset_to edge cases.SliceSampler and EnvBase.step regressions affecting compile compatibility and batch-locked environments.MultiSyncCollector worker-output gathering and improved preemption throughput without busy-waiting._skip_tensordict handling in EnvBase.step.4679d0a Fix SliceSampler compile compatibility.MultiSyncCollector preemption.12093ae Improve release, benchmark, auto-tag, and docs CI reliability.No new public API exports are intended in this patch release.
TorchRL v0.13.1 is a maintenance release for the 0.13 line. It carries post-0.13.0 RNN backend fixes and performance improvements, compile-friendlines
TorchRL v0.13.1 is a maintenance release for the 0.13 line. It carries post-0.13.0 RNN backend fixes and performance improvements, compile-friendliness fixes, SOTA example dependency refreshes, and documentation improvements.
to_module FutureWarning graph break under torch.compile while preserving the previous state-preserving conversion behavior.vllm 0.20.0.transformers 5.0.0rc3.pytorch_sphinx_theme2 and fixes tutorial Colab, Notebook, and GitHub links.torchrl.cuda_memory_profile (code, docs) — context manager/decorator for scoped CUDA memory profiling.torchrl.cuda_memory_stats (code, docs) — helper for reading current and peak CUDA allocation/reservation statistics.torchrl.reset_cuda_peak_stats (code, docs) — helper for resetting CUDA peak memory counters.torchrl.modules.tensordict_module.canonicalize_rnn_subset (also re-exported as torchrl.modules.canonicalize_rnn_subset; specific export, package export, docs) — canonicalizes only the recurrent keys used by selected RNN modules.torch.compile behavior for state-preserving module conversion.pip install torchrl==0.13.1
For CUDA wheel variants, follow the install index documented in the TorchRL README for the desired CUDA runtime.
https://github.com/pytorch/rl/compare/v0.13.0...v0.13.1
Remove v0.12 deprecated APIs (#3670) @vmoens
New algorithms. Five new config-based trainers — DQN, DDPG, IQL, CQL, and TD3 — are built on a new configuration system for reproducible algorithm setups (@vmoens, @bsprenger). PILCO (Probabilistic Inference for Learning Control) is now available as a built-in algorithm (@PSXBRosa, @vmoens). For diffusion-based behavioral cloning, a new DDPMModule diffusion actor and DiffusionBCLoss are included (@theap06). Async PPO infrastructure overlaps data collection and optimization (@vmoens).
Collector and data-flow improvements. A new high-throughput auto-batching inference server automatically batches requests from multiple environments, with pluggable transport backends (threading, multiprocessing, Ray, Monarch) and built-in weight-sync integration. Paired with the new AsyncBatchedCollector, it enables asynchronous data collection with automatic batching for maximum GPU utilization (@vmoens). The new TrajectoryBatcher and AsyncTrajectoryBatcher assemble trajectories efficiently from streaming environment transitions, including variable-length trajectories and padding (@theap06). On the parallel environment side, shared-memory done flags replace mp.Event for lower-latency step synchronization, and a fast-path device-transfer optimization reduces overhead in step_and_maybe_reset (@vmoens).
Inference backends. This release adds full SGLang integration alongside vLLM, with an SGLangWrapper policy module, an AsyncSGLang server-based inference path, NCCL weight synchronization, and GRPO support (@vmoens).
Replay buffer. StoreStorage is a new Redis/Dragonfly-backed storage backend that lets replay buffers share experience across processes and nodes (@vmoens).
Evaluation. A new Evaluator class provides a unified API for synchronous and asynchronous policy evaluation during training, with a process backend, collector-based stepping, weight sync via WeightSyncScheme, multi-model support, and a RayEvalWorker for distributed evaluation (@vmoens).
Environments and platform support. A new GenesisEnv wrapper integrates the Genesis physics simulator (@ParamThakkar123). Dreamer now supports pre-vectorized environments and ships with an IsaacLab environment factory, training script, and integration guide (@vmoens). MPS support improves through float64-to-float32 downcasting in ParallelEnv, SerialEnv, and collectors, fixing previously broken Apple Silicon GPU workflows (@bsprenger).
pip install torchrl==0.12.0
Requires PyTorch >= 2.1 and TensorDict >= 0.12.0.
local_init_rb parameter has been removed from Collector and MultiCollector. Storage-level initialization is now the only behavior.TransformedEnv(env=...) now raises TypeError. Use TransformedEnv(base_env=...) instead.A new inference server that automatically batches requests from multiple environments for efficient GPU inference. This is a key building block for scaling RL training with many parallel environments.
AsyncBatchedCollectorA new collector that combines async environments with the auto-batching inference server for maximum throughput.
TrajectoryBatcher for assembling trajectories from streaming transitions (#3584) @theap06AsyncTrajectoryBatcher for asynchronous trajectory assembly (#3592) @theap06Full SGLang support for LLM inference, mirroring the existing vLLM integration:
AsyncSGLang server-based inference service (#3429)SGLangWrapper policy module (#3430)DDPMModule diffusion actor for denoising diffusion probabilistic models (#3596) @theap06DiffusionBCLoss for diffusion-based behavioral cloning (#3604) @theap06EvaluatorEvaluator class for sync/async evaluation (#3594)WeightSyncScheme + multi-model support (#3627)Evaluator tests + init_fn plumbing for process backend (#3663)RayEvalWorker for distributed async evaluation (#3474)from_name for RayEvalWorker (#3488)New trainers with integrated configuration system:
StoreStorage for Redis/Dragonfly-backed replay buffers (#3516)set_at_, set_, update_ methods on ReplayBuffer (#3590) @jashshah999trajs_per_batch with replay_buffer on multi-process and distributed collectors (#3618)GenesisEnv: wrapper for the Genesis physics simulator (#3536) @ParamThakkar123FinancialRegimeEnv: a vectorized financial environment (#3384) @aneesh223num_workers parameter for HabitatEnv (#3383) @ParamThakkar123VecNorm and VecNormV2 transforms (#3416) @ParamThakkar123prevent_leaking_rng utility (#3401) @ParamThakkar123index_select support for TensorSpec (#3406) @ParamThakkar123strict_shape parameter for QValueModule action shape enforcement (#3593) @Lidang-JiangRandomPolicy action_spec from env in collectors (#3664)__getattr__ in _dispatch_caller_parallel for transparent attribute access (#3389) @ParamThakkar123scalar_output_mode for loss modules with reduction='none' (#3426)ObsDecoder: out_channels parameter for grayscale decoding (#3472)memmap value for the CKPT_BACKEND environment variable (#3619) @theap06SliceSampler: GPU-accelerated trajectory computationParallelEnv: fast-path device transfer in step_and_maybe_resetParallelEnv: replace mp.Event with shared-memory done flags for lower latencylog_metrics usage in sota-implementations (#3454)ParallelEnv/SerialEnv on MPS (#3551) @bsprengermasked_scatter shape preservation on MPS in collectors (#3473)MultiCollector weight sync after device-cast (#3587)__torch_function__ return, and off-by-one in RayCollector (#3530) @jashshah999check_env_specs() when state_spec contains keys not in observation_spec (#3581) @theap06BraxEnv rejecting camera_id and render_kwargs (#3533)StepCounter now tracks nested truncated and done states (#3405)ParallelEnv shutdown hang with shared-memory done flags (#3464)SACLoss when there is more than one qvalue_network (#3500) @ParamThakkar123GPT2RewardModel.compute_reward_loss (#3521)ImpalaNet (#3522)CompilationConfig compatibility and Windows CI pybind11 (#3673)MultiOneHot.to_numpy() returning scalar instead of array (#3589) @jashshah999_set_index_in_td with trailing dims of 1 (#3517)Composite.encode (#3411) @tobiabirLineariseRewards should not squeeze trailing dim (#3614) @mathieuorhanR3MTransform in training mode (#3607) @theap06DataLoadingPrimer batch_size detection for NonTensorStack (#3532)Bounded spec projection (#3453)VideoRecorder support for grayscale (1-channel) observations (#3471)functools.partial warnings (#3465) @ParamThakkar123pendulum.py tutorial (#3595) @theap06StepCounter._reset should not use output_spec (#3626)torchcodec for video export (#3540)NoisyLinear (#3082)meshgrid usage to address deprecation warning (#3412)RayEvalWorker API reference docs (#3487)TransformersWrapper/ChatEnv integration documentation (#3377)Welcome to the following first-time contributors!
FinancialRegimeEnvRayCollector fixes, ReplayBuffer API additions, MultiOneHot fixQValueModule strict_shape parameterLineariseRewards fixR3MTransform fix, CKPT_BACKEND memmapComposite.encode batch size fixhttps://github.com/pytorch/rl/compare/v0.11.0...v0.12.0
No breaking changes in this release.
This patch release includes several important bug fixes and performance improvements:
Composite.encode() to correctly set the batch size of the output TensorDictStepCounter to properly track nested truncated and done states in multi-agent environments_repr_html_ dispatch in parallel environments that was causing doc CI failuresscalar_output_mode to loss modules for proper handling of reduction='none'torch.compile configuration for DreamerNo breaking changes in this release.
Fixed batch size in Composite.encode: The Composite.encode() method now correctly sets the batch_size of the output TensorDict to match the shape of the tensor spec, rather than returning an empty batch size. (#3411) - @tobiabir
Previously, calling Composite.encode(raw_vals) would return a TensorDict with batch_size=torch.Size([]) regardless of the spec's shape. This is now fixed to return the correct batch size matching the spec shape.
Fixed StepCounter nested done/truncated tracking in multi-agent environments: StepCounter now properly updates nested truncated and done keys for multi-agent environments. (#3405) - @vmoens
When using StepCounter with multi-agent environments (e.g., PettingZoo), the transform now correctly propagates truncated/done signals to agent-specific keys (e.g., ("agent", "truncated")) in addition to the root-level keys.
Fixed shared memory weight updater with multiple policies: The shared memory weight updater now correctly handles collectors that use multiple policies. (#3442) - @vmoens
Fixed _repr_html_ dispatch in parallel environments: Parallel environments no longer incorrectly dispatch private/special attribute access (like _repr_html_) to worker processes. (#3441) - @vmoens
Added scalar_output_mode to loss modules: Loss modules (SAC, IQL, CQL, CrossQ, REDQ, DecisionTransformer) now support scalar_output_mode parameter for proper handling of reduction='none'. (#3426) - @vmoens
Fixed torch.compile configuration for Dreamer: Fixed compilation settings for Dreamer world model training. - @vmoens
meshgrid usage to address PyTorch deprecation warning (#3412) - @vmoensThanks to all contributors to this release:
```bash pip install torchrl==0.11.1 ```
Or with conda:
```bash conda install -c pytorch torchrl=0.11.1 ```
Deprecated APIs removed - All deprecation warnings from v0.10 have been promoted to hard errors for v0.11. (#3369) @vmoens
torchrl.weight_update) with SharedMem, MultiProcess, and vLLM-specific (NCCL, double-buffer) schemes. Collectors now integrate seamlessly with weight sync schemes for distributed training. @vmoenscollectors.py is now split into focused modules (_single.py, _multi_base.py, _multi_sync.py, _multi_async.py, _runner.py, base.py), with cleaner separation of concerns. (#3233) @vmoensProcgenEnv wrapper. (#3331) @ParamThakkar123num_envs/num_workers parameter to run multiple environments in a single call via ParallelEnv. (#3343, #3370, #3337) @ParamThakkar123pip install torchrl==0.11.0
[v0.11] Remove deprecated features and replace warnings with errors (#3369) @vmoens
KLRewardTransform from transforms/llm.py (use torchrl.envs.llm.KLRewardTransform)LogReward and Recorder classes from trainers (use LogScalar and LogValidationReward)unbatched_*_spec properties from VmasWrapper/VmasEnv (use full_*_spec_unbatched)rlhf.py modules (data/rlhf.py, envs/transforms/rlhf.py, modules/models/rlhf.py)replay_buffer_chunk parameter from MultiCollectorminimum/maximum deprecation warnings with TypeError in Bounded speccritic_coef/entropy_coef deprecation warnings with TypeError in PPO and A2C losses[Major] Major refactoring of collectors (#3233) @vmoens
collectors.py into focused modules for single/multi sync/async collectors_constants.py, _runner.py, base.py modulesThese changes significantly improve Dreamer training performance, torch.compile compatibility, and usability. @vmoens
[Feature] Refactor Dreamer training with async collectors, profiling, and improved config (cc917bae)
[Refactor] Dreamer implementation updates (3ab4b306)
[Feature] Add noise argument and scan mode to RSSMRollout (8653b6e6)
noise argument to control stochastic sampling during rollout[Feature] Add explicit dimensions and device support to RSSM modules (f350fe06)
[Feature] Logging & RSSM fixes (d0829794)
[Refactor] Use compile-aware helpers in Dreamer objectives (d8c3887d)
[BugFix] Optimize DreamerEnv to avoid CUDA sync in done checks (d57fdecf)
[BugFix] Fix ModelBasedEnvBase for torch.compile compatibility (7ff663d6)
[Feature] Add allow_done_after_reset parameter to ModelBasedEnvBase (24ae0424)
[BugFix] Final polish for Dreamer utils and Collector tests (5aefdd5c)
New modular infrastructure for weight synchronization between training and inference workers. @vmoens
[Feature] Weight Synchronization Schemes - Core Infrastructure (e8f6fa5a)
torchrl.weight_update module with 2400+ lines of weight sync infrastructureSharedMemWeightSyncScheme: Uses shared memory for fast intra-node syncMultiProcessWeightSyncScheme: Uses multiprocessing queues for cross-process syncdocs/source/reference/collectors.rst[Feature] vLLM Weight Synchronization Schemes (d0c8b7e9)
VllmNCCLWeightSyncScheme: NCCL-based weight sync for vLLM distributed inferenceVllmDoubleBufferWeightSyncScheme: Double-buffered async weight updates[Feature] Collectors - Weight Sync Scheme Integration (a7707ca0)
[Refactor] Weight sync schemes refactor (ae0ae064)
New policy optimization algorithms for LLM fine-tuning. @vmoens
[Feature] DAPO (9d5c2767)
torchrl/objectives/llm/grpo.py[Feature] CISPO (ed0d8dc6)
[Refactor] Refactor GRPO as a separate class (2bc3cb70)
New trainer algorithms, configuration system, and utilities. @vmoens
[Trainers] SAC Trainer and algorithms (02d4bfd1)
sota-implementations/sac_trainer/[Feature] Trainer Algorithms - Configuration System (6bc201a5)
torchrl/trainers/algorithms/configs/[Feature] Trainer Infrastructure - Timing and Utilities (dc215236)
[Feature] Async collection within trainers (5f1eb2c5)
[Feature] PPO Trainer Updates (129f3d50)
New infrastructure for tool-augmented LLM agents. @vmoens
[Feature] Tool services (9ca0e407)
torchrl/services/ module for tool executiondocs/source/reference/services.rsttest/test_services.py[Feature] Transform Module - ModuleTransform and Ray Service Refactor (7b85c716)
ModuleTransform for applying nn.Modules as transformsray_service.py to torchrl/envs/transforms/Fixes to enable torch.compile with various TorchRL components. @vmoens
[BugFix] Fix value functions for torch.compile compatibility (3bdc7b18)
test/compile/test_value.py[BugFix] Fix TDLambdaEstimator for torch.compile compatibility (11e22ee9)
[Feature] Add compile-aware timing and profiling helpers (764d9f71)
[Refactor] Add compile-aware timing and profiling helpers (c82447e6)
[Feature] Added EXP3 Scoring function (#3013) @ParamThakkar123
torchrl/data/map/ module with hash-based storage, query utilities, and tree structures for efficient state lookups[Feature] Add num_workers parameter to BraxEnv (#3370) @ParamThakkar123
ParallelEnv by specifying num_workers > 1[Feature] Added num_envs parameter in GymEnv (#3343) @ParamThakkar123
num_envs > 1, GymEnv automatically returns a lazy ParallelEnv wrapping multiple environment instances[Environments] Added Procgen environments (#3331) @ParamThakkar123
ProcgenWrapper and ProcgenEnv classes to wrap OpenAI Procgen environments[Feature] Loss make_value_estimator takes a ValueEstimatorBase class (#3336) @ParamThakkar123
ValueEstimatorBase class directly to make_value_estimator() for more flexible value estimation configuration[Feature] Make custom_range public in ActionDiscretizer (#3333) @vmoens
custom_range parameter in ActionDiscretizer for user-defined discretization ranges[Feature] Added num_envs parameter in DMControlEnv (#3337) @ParamThakkar123
num_workers parameterParallelEnv when num_workers > 1[Feature] Auto-configure exploration module specs from environment (#3317) @bsprenger
AdditiveGaussianModule) from the environment's action specset_exploration_modules_spec_from_env() utility function[Feature] Add NPU Support for Single Agent (#3229) @lowdy1
[Feature] Add support for trackio (#3196) @Xmaster6y
TrackioLogger class for seamless integration[Feature] Add a new Trainer hook point process_loss (#3259) @Xmaster6y
[Feature] Ensure MultiSyncDataCollectors returns data ordered by worker id (#3243) @LCarmi
[Feature] Add TanhModuleConfig (#3255) @bsprenger
[Feature] add TensorDictSequentialConfig (#3248) @bsprenger
[Feature] Weight loss outputs when using prioritized sampler (#3235) @vmoens
importance_key parameter to loss functions (DQN, DDPG, SAC, TD3, TD3+BC)[Feature] Composite specs can create named tensors with 'zero' and 'rand' (#3214) @louisfaury
zero() and rand()[Feature] Enable storing rollouts on a different device (#3199) @Xmaster6y
[Feature] Named dims in Composite (#3174) @vmoens
names property and refine_names() method[BugFix] Fix TransformersWrapper ChatHistory.full not being set (#3375) @vmoens
[BugFix] Fix AsyncEnvPool + LLMCollector with yield_completed_trajectories (#3373) @vmoens
[BugFix] Fix LLM test failures (#3360) @vmoens
[BugFix] Fixed MultiSyncCollector set_seed and split_trajs issue (#3352) @ParamThakkar123
[BugFix] Fix VecNormV2 device gathering (#3368) @vmoens
[BugFix] Fix AsyncEnv - LLMCollector integration (#3365) @vmoens
[BugFix] Fix VecNormV2 GPU device handling for stateful mode (#3364) @BY571
[BugFix] Fix Ray collector iterator bug and process group cleanup (#3363) @vmoens
[BugFix] Fix LLM CI by replacing uvx with uv package (#3356) @vmoens
[Bugfix] Wrong minari download first element (#3106) @marcosgalleterobbva
[BugFix,Test] Fix envpool failing tests (#3345) @vmoens
[BugFix] Fixed ParallelEnv + aSyncDataCollector / MultiSyncDataCollector not working if replay_buffer is given (#3341) @ParamThakkar123
[BugFix] treat 1-D MultiDiscrete as MultiCategorical and accept flattened masks (#3342) @ParamThakkar123
[BugFix] Fixes register_save_hook bug (#3340) @ParamThakkar123
[BugFix,Test] recompiles with string failure (#3338) @vmoens
[BugFix] Fix Safe modules annotation and doc for losses (#3334) @vmoens
[BugFix] Fix ray modules circular import (#3319) @vmoens
[BugFix] Fix SACLoss target_entropy="auto" ignoring action space dimensionality (#3292) @vmoens
[BugFix] Fix agent_dim in multiagent nets & account for neg dims (#3290) @vmoens
[BugFix] Added a missing .to(device) call in _from_transformers_generate_history (#3289) @michalgregor
[BugFix] Fix old pytorch dependencies (#3266) @vmoens
[BugFix] RSSMRollout not advancing state/belief across time steps in Dreamer (#3236) @cmdout
[Doc,BugFix] Fix doc and collectors (#3250) @vmoens
[BugFix] use correct field names in InitTrackerConfig (#3245) @bsprenger
[BugFix] Use torch.zeros for argument in torch.where (#3239) @sebimarkgraf
[BugFix] Fix wrong assertion about collector and buffer (#3176) @vmoens
[BugFix] AttributeError in accept_remote_rref_udf_invocation (#3168) @vmoens
[BugFix] Final polish for Collector and Exploration (558396d7) @vmoens
[BugFix] Collector robustness & Async fixes (cadf23bc) @vmoens
[BugFix] TensorStorage key filtering (0667a580) @vmoens
[BugFix] Replay Buffer prefetch & SliceSampler (9d34dbe7) @vmoens
[BugFix] Fix target_entropy computation for composite action specs (06960c64) @vmoens
[BugFix] Fix device initialization in CrossQLoss.maybe_init_target_entropy (416e4545) @vmoens
[BugFix] Fix schemes and refactor collectors to make them readable (364e0386) @vmoens
[BugFix] Fix collector devices (888095fd) @vmoens
[BugFix] Fix tests (963fdd43) @vmoens
[BugFix] Fix GRPO tests and runs (47ad9d8f) @vmoens
[BugFix] Handle Lock/RLock types in EnvCreator (6985ca25) @vmoens
[BugFix] Defer filter_warnings import to avoid module load issues (8ea954cd) @vmoens
[BugFix] Fix CUDA sync in forked subprocess (4a2a2749) @vmoens
[BugFix] Fix unique ref to lambda func (b6fe45ee) @vmoens
[BugFix] Add pybind11 check and Windows extension pattern fix (5e89e4e7) @vmoens
[Feature] Storage Shared Initialization for Multiprocessing (61c178e5) @vmoens
[Feature] Memmap storage cleanup (7f9ea748) @vmoens
[Feature] Collector Profiling (02ed47ed) @vmoens
[Feature] auto_wrap_envs in PEnv (d781f9e9) @vmoens
[Feature] Auto-wrap lambda functions with EnvCreator for spawn compatibility (d250c182) @vmoens
[Feature] Collectors' getattr_policy and getattr_env (fbdbb617) @vmoens
[Feature] track_policy_version in collectors.py (a089cc48) @vmoens
[Feature] Aggregation strategies (0e6a3565) @vmoens
[Feature] kl_mask_threshold (7ab48a4c) @vmoens
[Feature] Add timing options (13434ebb) @vmoens
[Feature] float32 patch (01d2801b) @vmoens
[Feature] Support callable scale in IndependentNormal and TanhNormal distributions (d3aba7d1) @vmoens
[Doc] Huge doc refactoring (3d5dd1ac) @vmoens
collectors_basics.rst, collectors_single.rst, collectors_distributed.rstcollectors_weightsync.rst, collectors_replay.rstdata_datasets.rst, data_replaybuffers.rst, data_samplers.rst[Docs] Update LLM_TEST_ISSUES.md with fix status (#3374) @vmoens
[Docs] Enable doc builds and tutorial runs (#3335) @vmoens
[CI] Speed up slow tests in tests-gpu/tests-cpu (#3395) @vmoens
[CI] Bump version to 0.11.0 (#3392) @vmoens
[CI] Fix auto-label workflow for fork PRs (#3388) @vmoens
[CI] Better release workflow (#3386) @vmoens
[CI] Fix auto-labelling (#3387) @vmoens
[CI] Auto-tag PRs (#3381) @vmoens
[CI] Add release agent prompt for LLM-assisted releases (#3380) @vmoens
[CI] Add release workflow with PyPI trusted publishing (#3379) @vmoens
[CI] Add granular label support for environment and data workflow jobs (#3371) @vmoens
[CI] Fix PettingZoo CI by updating Python 3.9 to 3.10 (#3362) @vmoens
[CI] Fix GenDGRL CI by adding missing requests dependency (#3361) @vmoens
[CI] Fix IsaacLab tests by using explicit conda Python path (#3358) @vmoens
[CI] Fix M1 build pip command not found (#3359) @vmoens
[CI] Fix Windows build for free-threaded Python (3.13t, 3.14t) (#3357) @vmoens
[CI,BugFix] Fix Habitat CI by upgrading to Python 3.10 (#3346) @vmoens
[CI] Fix Jumanji CI by adding missing requests dependency (#3349) @vmoens
[CI] Fix Chess CI by correcting test file path typo (#3353) @vmoens
[CI] Skip Windows-incompatible tests in optional deps CI (#3348) @vmoens
[CI] Fix GPU benchmark failures (#3347) @vmoens
[CI] Upgrade doc python version (#3222) @vmoens
[CI] Use pip install (#3200) @vmoens
[Setup] Python 3.14 in, python 3.9 out (ff86ab78) @vmoens
[CI] LLM tests integration (cccfaa69) @vmoens
[CI] Fix benchmarks for LLMs (e7ec9c31) @vmoens
[CI] Test setup (2644b899) @vmoens
[CI] Isolate distributed tests (5c5992d9) @vmoens
[CI] Fix uv install with --no-deps (546a1b73) @vmoens
[CI] Fix SOTA runs (57000fcc) @vmoens
[CI] Fix libs (071d079f) @vmoens
[CI] Fix missing librhash0 in doc CI (8d0f0613) @vmoens
[CI] Add --upgrade flag for torch installs and install ffmpeg/xvfb (ce9931943) @vmoens
[CI] Quiet coverage combine (1ac0fd0a) @vmoens
[CI] Fix tensordict install in lib tests (8f487d40) @vmoens
[Tests] Check traj_ids shape with unbatched envs (#3393) @vmoens
[Tests] Fix test isolation in test_set_gym_environments and related tests (#3382) @vmoens
[Test] Use mock Llama tokenizer instead of skipping gated model test (#3376) @vmoens
[Test] Skip Llama test when tokenizer unavailable instead of xfail (#3372) @vmoens
[Tests] Remove _utils_internal.py in tests (#3281) @vmoens
[Test] Check LinearizeReward obs transform (#3241) @vmoens
[Test,Benchmark] Move mocking classes file and bench for non-tensor env (#3257) @vmoens
[Tests] Fix vmas seeding test (#3210) @matteobettini
[Tests] Reintroduce VMAS football (#3178) @matteobettini
[Test] Use spawn in older pytorch (9b0492906) @vmoens
[Test] Fix test_num_threads that instantiate the env in the main process (852dd61b) @vmoens
[Test] Use class references instead of lambdas in transform tests (7dd1e61d) @vmoens
[Test] Add test for lambda wrapping in ParallelEnv (4b2b2279) @vmoens
[Test] Ensure collector shutdown with try/finally (aa2b0311) @vmoens
[Test] Add filterwarnings for unclosed resources (b493ab34) @vmoens
[Test] Add retry decorator to flaky vecnorm test (6d908b69) @vmoens
[Test] Simplify num_threads test to check threads in env factory (295cc20e) @vmoens
[Test] Ignore script_method deprecation warning (eaaf97a4) @vmoens
[Test] Fix failing 3.14 tests (66a48f3c) @vmoens
[Test] Remove the forked decorator of the distributed checks (4db59660) @vmoens
[Test] Test RB+Isaac+Ray (a0d650f9) @vmoens
[Test] Fix flaky parallel test (4f013a81) @vmoens
[Test] Fix vc1 (bd93f136) @vmoens
[Quality] Fix flaky test (03ffc820) @vmoens
[Refactor] Updated num_envs parameter to num_workers for consistency (#3354) @ParamThakkar123
num_envs to num_workers for consistency across environment wrappers[Quality] replace reduce to reduction, better error message for invalid mask (#3179) @Kayzwer
[Refactor] Move WEIGHT_SYNC_TIMEOUT to collectors._constants (3ceb6b9a) @vmoens
*[Refactor] Remove TensorSpec classes (7ea8d861) @vmoens
[Refactor] Rename collectors (8779f2a3) @vmoens
[Refactor] Non-daemonic processes in PEnv (aeb2e9b2) @vmoens
[Refactor] Make env creator optional for Ray (b599d9b8) @vmoens
[Refactor] Move decorate_thread_sub_func to torchrl.testing.mp_helpers (e3e9e6a7) @vmoens
[Refactor] Use non_blocking transfers in distribution modules (5c75777f) @vmoens
[Refactor] Remove MUJOCO_EGL_DEVICE_ID auto-setting from torchrl init (8fac4c54) @vmoens
[Refactor] Add WEIGHT_SYNC_TIMEOUT constant for collector weight synchronization (ab3768aa) @vmoens
[BugFix,Test,Refactor] Refactor tests (eb8a885b) @vmoens
[Refactor,Test] Move compile test to dedicated folder (253b8ddd) @vmoens
[Dependencies] Bump ray from 2.46.0 to 2.52.1 (#3258) @dependabot
[Environment] Fix envpool wrapper (#3339) @vmoens
Revert "replace reduce to reduction, better error message for invalid mask" (#3182) @vmoens
Fix (#3180) @matteobettini
[Versioning] Fix doc versioning (#3175) @vmoens
Thanks to all contributors to this release:
This patch release includes bug fixes, type annotation improvements, and CI enhancements cherry-picked from main.
This patch release includes bug fixes, type annotation improvements, and CI enhancements cherry-picked from main.
#3174 - @vmoens - [Feature] Named dims in Composite
#3214 - @louisfaury - [Feature] Composite specs can create named tensors with 'zero' and 'rand'
zero() and rand() methods@vmoens - [Typing] Edit wrongfully set str type annotations
#3175 - @vmoens - [Versioning] Fix doc versioning
#3200 - @vmoens - [CI] Use pip install
@vmoens - [CI] Fix missing librhash0 in doc CI
@vmoens - [CI] Fix benchmarks for LLMs
#3222 - @vmoens - [CI] Upgrade doc python version
TorchRL 0.10.0 introduces significant advancements in Large Language Model (LLM) support, new algorithms, enhanced environment integrations, and numer
TorchRL 0.10.0 introduces significant advancements in Large Language Model (LLM) support, new algorithms, enhanced environment integrations, and numerous performance improvements and bug fixes.
AddThinkingPrompt transform for reasoning prompts (#3027) @vmoensMCPToolTransform for tool integration (#2993) @vmoensPythonInterpreter transform for code execution (#2988) @vmoensLLMMaskedCategorical for masked categorical distributions (#3041) @vmoensContentBase system for structured content handling (#2985) @vmoensSpecial thanks to all contributors who made this release possible:
pip install torchrl==0.10.0
For the latest features:
pip install git+https://github.com/pytorch/rl.git@release/0.10.0
This release focuses on bug fixes, performance improvements, and code quality enhancements.
This release focuses on bug fixes, performance improvements, and code quality enhancements.
clone() operations (#3077)TransformedEnv (#3076)pip install torchrl==0.9.2
Thanks to all contributors: @felixy12, @Xmaster6y, @louisfaury and @LCarmi
Fixes an critical issue with vLLMWrapper and TransformersWrapper, where a stack of History objects is resent to stack, resulting in a bug.
Fixes an critical issue with vLLMWrapper and TransformersWrapper, where a stack of History objects is resent to stack, resulting in a bug.
Fix deprecated list index (#3005) - Updated deprecated list indexing
We are excited to announce the release of TorchRL 0.9.0! This release introduces a comprehensive LLM API for language model fine-tuning, extensive torch.compile compatibility across all algorithms, and numerous performance improvements.
TorchRL now includes a comprehensive LLM API for post-training and fine-tuning of language models! This new framework provides everything you need for RLHF, supervised fine-tuning, and tool-augmented training:
torchrl.modules.llm: Seamless integration with Hugging Face models and vLLM inference enginestorchrl.data.llm.history: Advanced History class for multi-turn dialogue with automatic chat template detectiontorchrl.envs.llm.transforms: Built-in support for Python code execution, function calling, and custom tool transformstorchrl.objectives.llm: GRPO (Group Relative Policy Optimization) and SFT loss functions optimized for language modelstorchrl.collectors.llm: Async data collection with distributed training supporttorchrl.envs.llm: Transform-based architecture for reward computation, data loading, and conversation augmentationThe LLM API follows TorchRL's modular design principles, allowing you to mix and match components for your specific use case. Check out the complete documentation and GRPO implementation example to get started!
A full GRPO implementation is provided in sota-implementations/grpo/ with:
distutils imports (#2836) - Modern Python compatibilityThe new LLM API provides a complete framework for language model fine-tuning. Key components include:
from torchrl.envs.llm import ChatEnv
from torchrl.modules.llm import TransformersWrapper
from torchrl.objectives.llm import GRPOLoss
from torchrl.collectors.llm import LLMCollector
# Create environment with Python tool execution
env = ChatEnv(
tokenizer=tokenizer,
system_prompt="You are an assistant that can execute Python code.",
batch_size=[1]
).append_transform(PythonInterpreter())
# Wrap your language model
llm = TransformersWrapper(
model=model,
tokenizer=tokenizer,
input_mode="history"
)
# Set up GRPO training
loss_fn = GRPOLoss(llm, critic, gamma=0.99)
collector = LLMCollector(env, llm, frames_per_batch=100)
# Training loop
for data in collector:
loss = loss_fn(data)
loss.backward()
optimizer.step()
auto_unwrap has been modifiedVecNorm transform.We would like to thank all contributors who made this release possible, especially those who contributed to the LLM API framework and the comprehensive testing and documentation improvements.
For detailed usage examples and tutorials, please refer to the TorchRL documentation and the LLM API reference.
This release major upgrades is a patch to collector.start() to allow collectors (single or multi-proc) to run asynchronously. #2935
This release major upgrades is a patch to collector.start() to allow collectors (single or multi-proc) to run asynchronously. #2935
An example is provided in the async SAC example. #2946
Fixes #2958 where partial resets are not handled correctly when a BatchedEnv is transformed - as the "done" checks were inconsistent. We now enforce that root "_reset" entries always precede their respective leaves.
Using an LSTM within GAE is facilitated by ensuring that shifted=True and shifted=False work properly (with appropriate warnings/errors if other hyperparameters need to be set). #2941
Full Changelog: https://github.com/pytorch/rl/compare/v0.8.0...v0.8.1
…1b5e13c44f5dff1a55f9c78a174f7164f37d76a1 [Deprecation] Enact deprecations (#2917) (b247526a1) by @vmoens ghstack-source-id: 690a9f62e274e9f14a89532dd7…
"threading" or "multiprocessing"). Instantiating an async env is roughly the same as a parallel one:from torchrl.envs import AsyncEnvPool
env = AsyncEnvPool([partial(GymEnv, "Pendulum-v1"), partial(GymEnv, "Pendulum-v1")], backend="threading")
These environments support the regular environment methods (reset, step or rollout) but their main advantage lies
in their new async methods:s0 = env.rand_action(env.reset())
env.async_step_send(s0)
# receive
result = env.async_step_recv()
In this example, result will contain the results of the call to step for one or two environments. The environment indices
can be found in the result['env_index'] entry (the name of that key is stored in env._env_idx_key).RayReplayBuffer (#2835)VecNorm by passing the new_api keyword argument.torchrl.collectors.WeightUpdaterBase
abstract class. This should the entry point for any user wanting to implement their own weight update strategy, alleviating
the need to subclass or patch the collector or the policy directly.We relaxed TorchRL dependency to make it compatible with any pytorch version. The current status is:
[Feature] Add EnvBase.all_actions (#2780) (67c3e9a4a) by @kurtamohler ghstack-source-id: 7abf9d469f740be5f14daffa2330811f7572dad9
[Feature] Add MCTSForest/Tree.to_string (#2794) (f8626690c) by @kurtamohler ghstack-source-id: 2127bf24d66e44fb310d12ff5f72e92aa0371cd7
[Feature] Add include_hash_inv arg to ChessEnv (#2766) (3be85c691) by @kurtamohler ghstack-source-id: f6920d781835902a6db02f74c5e5a3041243c5e3
[Feature] Add option for auto-resetting envs in GAE (#2851) (f5f3ae4e2) by @lin-erica Co-authored-by: Erica Lin elin@theaiinstitute.com
[Feature] Async environments (#2864) (4f00025af) by @vmoens ghstack-source-id: 0a70ce0129d2ee6f85bb22adda3c332ff65e7501
[Feature] Capture wrong spec transforms (1/N) (#2805) (d3dca73f3) by @vmoens ghstack-source-id: f2d938b3dfe88af66622099f60cd7e3026289a02
[Feature] Collectors for async envs (#2893) (4ba50667e) by @vmoens ghstack-source-id: 764c21d0f2c3b217440e1a6f12ee797b17820c1d
[Feature] DensifyReward postproc (#2823) (53065cf56) by @vmoens ghstack-source-id: ef6a0f52601642c8944f63f9e3ac9e963425734e
[Feature] Dynamic specs for make_composite_from_td (#2829) (413571b8d) by @vmoens ghstack-source-id: 79e31e737c9f67ff20ce9fe32081e5b0a83de947
[Feature] Enable Hash.inv (#2757) (32c4623b3) by @kurtamohler ghstack-source-id: 956708121067855e519382a37764f06f53b16aa7
[Feature] Env with tensorclass attributes (#2788) (ab76027c1) by @vmoens ghstack-source-id: dc00ea3d23e015756974cd5c2ce638b55e5f6f92
[Feature] Gymnasium 1.1 compatibility (#2898) (78cd7550d) by @vmoens ghstack-source-id: e0891867f4318380f01c15449f9f26070b78536d
[Feature] History API (#2890) (fd10fe213) by @vmoens ghstack-source-id: 5b9723f6e1c327625e1a9be6f6eac68b91ed8492
[Feature] History.default_spec (#2894) (8ce11a859) by @vmoens ghstack-source-id: 40b8a492765a85adaccb591f1bc173754bacc313
[Feature] Local and Remote WeightUpdaters (#2848) (27d3680e7) by @vmoens ghstack-source-id: 2962530f87b596d038e3a13a934ea09064af2964
[Feature] Make PPO ready for text-based data (#2857) (595ddb4fa) by @vmoens ghstack-source-id: eeda5e2355e573e74cf7c080994cd47520ecd45b
[Feature] MultiAction transform (#2779) (621776a21) by @vmoens ghstack-source-id: 0a6f7f916ee6f9c6d450c511385bdfdb1d911da0
[Feature] NonTensor batched arg (#2816) (b97bdb5e6) by @vmoens ghstack-source-id: c6de1bd1f1475b8d02df2ff3eb7438a50f2ae450
[Feature] Pass lists of policy_factory (#2888) (82f8ec26d) by @vmoens ghstack-source-id: e42b100096c6e38365f8a80681473746f51d8a77
[Feature] RayReplayBuffer (#2835) (50af98432) by @vmoens ghstack-source-id: 32eff06494037a1a30e532539794035c035f1e81
[Feature] Set padded token log-prob to 0.0 (#2856) (b9ddfa967) by @vmoens ghstack-source-id: 2b2993e0b15afae17326e6583390d57068712d4f
[Feature] Support lazy tensordict inputs in ppo loss (#2883) (c9caf3d9c) by @vmoens ghstack-source-id: 89098ba3ca61b1524aeddc68f54c377f29c8dc8b
[Feature] TensorDictPrimer with single default_value callable (#2732) (59e85458b) by @vmoens ghstack-source-id: a9a677f24fc1e6a47312d0a96ab60daae543ff78
[Feature] Timer transform (#2806) (104b88092) by @vmoens ghstack-source-id: e42f2aece15f90afc457e1fb3e41a1f7be1a6a85
[Feature] Transform for partial steps (#2777) (7c034e331) by @vmoens ghstack-source-id: 587f91e33dfe1d59b73c4b2f2f1c21760ee79d2e
[Feature] VecNormV2 (#2867) (40fcdb6bb) by @vmoens ghstack-source-id: 639d07ff54be200d54621c2c4619ebd0d3d7d79e
[Feature] VecNormV2: Usage with batched envs (#2901) (b08e7ace3) by @vmoens ghstack-source-id: 5e14ed982b71b0e5192b0687c5259a3b49a81157
[Feature] pass policy-factory in mp data collectors (#2859) (31af2c529) by @vmoens ghstack-source-id: bce8abe9853d5ec187f91ffbcd8b940fa18ec8ab
[Feature] policy factory for collectors (#2841) (49a8a42a0) by @vmoens ghstack-source-id: 96b928e938b8b07fc7de23483358202737571f8e
[Feature] reset_time in Timer (#2807) (5a4637969) by @vmoens ghstack-source-id: 36a74fd20b78e1cdde6bca19b4f95c3d9062d761
[Feature] transformers policy (#2825) (eea932c3c) by @vmoens ghstack-source-id: 870c221b4ebae132a44944f0be0ee78da540d115
[BugFix] Apply inverse transform to input of TransformedEnv._reset (#2787) (1ed5d293a) by @kurtamohler ghstack-source-id: 5f7c1fbd19b716f2b1602c34cf2ae1362f7bc7f6
[BugFix] Avoid calling reset during env init (#2770) (09e93c19c) by @vmoens ghstack-source-id: 5ab8281c34aacfd7dbbfc0e285d88bcae0aededf
[BugFix] Ensure that Composite.set returns self as TensorDict does (#2784) (e084c02dd) by @vmoens ghstack-source-id: 23fe46b61dc2c9548fd9de7e4100431918fd0370
[BugFix] Fix .item() warning on tensors that require grad (#2885) (b66fcd4cf) by @vmoens ghstack-source-id: 502bdda3f5700dc900cf5c748839c965b1d67c1b
[BugFix] Fix KL penalty (#2908) (96c300322) by @vmoens ghstack-source-id: 475dccb0bcddbfe3bd2d826c5389834fb95e1ab8
[BugFix] Fix MultiAction reset (#2789) (76aa9bc0c) by @kurtamohler ghstack-source-id: a2f7bfdd7522a214430182dac65687a977b1a10d
[BugFix] Fix PEnv device copies (#2840) (6e40548ab) by @vmoens ghstack-source-id: df39fd2e4cd72f24c645b0ac32b46ab3e8d847fc
[BugFix] Fix batch_locked check in check_env_specs + error message callable (#2817) (9c98b82c3) by @vmoens ghstack-source-id: c722b164133c27c05dd21add3e7f3158189dd515
[BugFix] Fix calls to _reset_env_preprocess (#2798) (ea76ffb62) by @vmoens ghstack-source-id: 59925635a87b196a5bcb0fb251afe4cc7b8b103e
[BugFix] Fix collector timeouts (#2774) (f6084b657) by @vmoens ghstack-source-id: cb71d95143beb22db1fe1752e72f70c19f43be79
[BugFix] Fix collector with no buffers and devices (#2809) (d4f88460a) by @vmoens ghstack-source-id: 5367df9fcfdf549108be852476b049a0b978e348
[BugFix] Fix compile compatibility of PPO losses (#2889) (9bc85f4f0) by @vmoens ghstack-source-id: b346033641e5d27560fbfa011a006446e56a4e31
[BugFix] Fix composite setitem (#2778) (c2a149d66) by @vmoens ghstack-source-id: f33b49beb4cf8c0c8b156559b1abbee8ac77db20
[BugFix] Fix env.full_done_spec~s~ (#2815) (f5c0666c2) by @vmoens ghstack-source-id: ba0d371d10b3f46ec1172fbec639ccc4d5559659
[BugFix] Fix forced batch-size in _skip_tensordict (#2808) (3acf49101) by @vmoens ghstack-source-id: dac84e8b8835e870bce1772d7893c30b6f9af59c
[BugFix] Fix gc import (#2862) (a183f02a5) by @vmoens ghstack-source-id: b732d4f805d98ceaaa45326d619fce623c10482f
[BugFix] Fix lazy-stack in RBs (#2880) (e80732ede) by @vmoens ghstack-source-id: 38399ee991bc065445f4eb1c84b71e7d844d794c
[BugFix] Fix property getter in RayReplayBuffer (#2869) (04d70c1e4) by @vmoens
[BugFix] Fix slow and flaky non-tensor parallel env test (#2926) by @vmoens ghstack-source-id: fcb5caa56e05176958b3468a7d6f69e363cfe558
[BugFix] Fix update shape mismatch in _skip_tensordict (#2792) (3e42e7a2f) by @vmoens ghstack-source-id: 27e7d444c126e48fdb70d951a0cc7beaee1db3a8
[BugFix] Fixed VideoRecorder crash when passing fps (#2827) (5ec9bc56e) by Alexandre Brown
[BugFix] GAE warning when gamma/lmbda are tensors (#2838) (d56111599) by @louisfaury Co-authored-by: Louis Faury louis.faury@helsing.ai
[BugFix] Keep original class in LazyStackStorage through lazy_stack (#2873) (70f5c0649) by @vmoens ghstack-source-id: 661cd65c86648ffb2ee6ead40110ac3d57477514
[BugFix] NonTensor should not convert anything to numpy (#2771) (3da27506c) by @vmoens ghstack-source-id: 7644f6c695490f34d6455703418c59cfa718a9f0
[BugFix] PPOs with composite distribution (#2791) (edfa25d96) by @louisfaury Co-authored-by: Louis Faury louis.faury@helsing.ai
[BugFix] Refactor _skip_tensordict to avoid update calls (#2802)" (#2802) (e0d3eee3b) by @vmoens ghstack-source-id: 0f31b879f1e4643080530db8f7c7091e281b560f
[BugFix] Remove neg dim checks in expand for all specs (#2906) (c5afe3c90) by @vmoens ghstack-source-id: f718328527275e0be591c5d12c334add8f65f7a4
[BugFix] Right log-prob size in transformer wrapper (#2854) (f81deacd1) by @vmoens ghstack-source-id: 98baa635ca07d5bf7e69a9e3bc43012ae2d91bf0
[BugFix] Test and fix life cycle of env with dynamic non-tensor spec (#2812) (b538c66c5) by @vmoens ghstack-source-id: 77da3a6baf0cb42525dd3a564b36ac03a531d17a
[BugFix] Tree make node fix (#2839) (ba8be9c44) by Rolo
[BugFix] Use brackets to get non-tensor data in gym envs (#2769) (84f6b0483) by @vmoens ghstack-source-id: 3101141eb5b7435c7a4047f5ee84b66c1d74af13
[BugFix] correct dim for resolving dtype in _split_and_pad_sequence (#2801) (21c4d87c7) by KubaMichalczyk Co-authored-by: Jakub Michalczyk jakub.michalczyk@sportec-solutions.com
[Refactor] Avoid padding in transformer wrapper (#2881) (9c4c086a5) by @vmoens ghstack-source-id: de28bab17fc3d59889ea9f2fd152de5001b92320 [Refactor] Fix repeats order (#2887) (93ba865c2) by @vmoens ghstack-source-id: 0bedd5c756f92d23083905ae8a6ddd992ba0b415 [Refactor] MaskedCategorical cross_entropy usage for faster loss (#2882) (3e1f4ff1c) by @vmoens ghstack-source-id: 84330cf08ad8798e2cd4f6a8f3ec146a9de8e1e4 [Refactor] Refactor the weight update logic (#2914) (0da904425) by @vmoens ghstack-source-id: 72b710ab1788090364c068c59b28a21e09221236 [Refactor] Rename weight updaters (#2892) (efe938956) by @vmoens ghstack-source-id: 8889046277b94db0076fa72787295fd9419ab183 [Refactor] TransformersWrapper class (#2871) (5d7256122) by @vmoens ghstack-source-id: 8d5442611e9f1cf499cd59ed3e61a0602459c94d [Refactor] VecNormV2: update before norm, bias_correction at the right time (#2900) (c3310b87f) by @vmoens ghstack-source-id: a90aeb268a83dc2e45735f7b6b19b4e63e572ba7
[BE] Ensure abstractmethods are implemented for specs (#2790) (bd78913fe) by @vmoens ghstack-source-id: 7b943aa84bc497e7e8195f633cb15105de137f04
[BE] Fix some typos (#2811) (0ae140568) by @antoinebrl
[BE] Make better logits in cost tests (#2775) (42ed42c71) by @vmoens ghstack-source-id: be9ea92b3f3d2592e426eaeaff7b81e50472cf16
[BE] Remove deprec specs from tests (#2767) (27a8ecc29) by @vmoens ghstack-source-id: 717bb31b1773c5c8b180c456f1bbad8a022dc55a
[BE] _set_seed returns None + type annotations (#2903) (3a9f244de) by @antoinebrl
[CI] Fix envnames in SOTA tests (#2921) by @vmoens ghstack-source-id: 3b518e2a81e9d988db2fbd12883eabbe486d32db
[CI] Fix libs workflows (#2800) (8dd1be7c3) by @vmoens
[CI] Fix nightly and benchmark CIs (#2930) by @vmoens ghstack-source-id: f39b2573ba58e7808389af3782aed8809759fa2b
[CI] Fix old deps (#2916) (4162db690) by @vmoens ghstack-source-id: 109de71c622760679d449d906b0f33b3f1866975
[CI] Fix wheels (#2876) (e1d3fd488) by @vmoens ghstack-source-id: 0f2602146c4371d2fc6ac33f139b1eebb0829559
[CI] Upgrade to cuda 12.8 (#2820) (8c9dc050f) by @vmoens ghstack-source-id: e0ad7d6c00d53b74b23022836535c453a37df238
[CI] egl for all (#2915) (425952b96) by @vmoens ghstack-source-id: 1b5e13c44f5dff1a55f9c78a174f7164f37d76a1
[Deprecation] Enact deprecations (#2917) (b247526a1) by @vmoens ghstack-source-id: 690a9f62e274e9f14a89532dd7d07176188560e9
[Deprecation] Softly change default behavior of auto_unwrap (#2793) (2046bc536) by @vmoens ghstack-source-id: c28c11ecf68fba0ffde652205ea8e46f8da07cf1
[Doc] Add docstring for MCTSForest.extend (#2795) (a3a1ebefe) by @kurtamohler ghstack-source-id: 7fa8834376a1afd9187d7f1d43a97f70d713a160
[Doc] Better doc for Transform class (#2797) (dd59290d9) by @vmoens ghstack-source-id: 16e563bc810586d31772b58f9923439b632985c7
[Doc] Fix (and deactivate) tutorials (#2785) (f1c42e083) by @vmoens ghstack-source-id: 56c7757c36a2d609688ce0777a49d54763d3e691
[Doc] Fix Doc (#2919) by @vmoens ghstack-source-id: dacd1e7467994c73b22b5f111ac6c486d43d7b58
[Doc] Fix EnvCreator's doc (#2868) (586a5413d) by @louisfaury Co-authored-by: Louis Faury louis.faury@helsing.ai
[Doc] Fix doc CI (#2932) by @vmoens ghstack-source-id: 5a27f9153c77d659ee691117c6af60a2d4b022bf
[Doc] Fix formatting errors (#2786) (03d658630) by @vmoens ghstack-source-id: ac1f3da66c1374d3d19fed88e80f8ed5407b3459
[Doc] Fix tutorials (#2768) (75f113ff5) by @vmoens
[Doc] Fix tutos (#2772) (b27ee6d8d) by @vmoens
[Doc] Solve ref issues in docstrings (#2776) (f5445a4bd) by @vmoens ghstack-source-id: 09823fa85a94115291e7434478776fb0834f9b39
[Doc] Update discrete.py (#2850) (619fec69c) by oswald
[Docs] Fix doc setup (#2922) by @vmoens ghstack-source-id: 91d359ad591a7f8062191825d04ebda112f2cf7d
[Environment] Fix lib CI failures (#2923) by @vmoens ghstack-source-id: c046f5ded86a3a07d66eaddcaef24b69c2d77c01
[Environment] Fix lib CI failures (#2929) by @vmoens ghstack-source-id: febe20b0915025cc253f6aa0404258f3a020e1e6
[Lint] pyupgrade (#2819) (40b147e81) by @vmoens ghstack-source-id: dcdf51db31b8f6bcfad7fd4dc53f5b5ad8098c5d
[Minor] Quick edits to .md files (#2931) by @vmoens ghstack-source-id: d0fbd3da72d41e6cdbfc4761990cb939993eb816
[Performance] Memoize calls to encode and related methods within step (#2907) (0475cbf64) by @vmoens ghstack-source-id: 8acd4839d4ba5f45373d0a0fcb52b15c149d37f1
[Performance] Use TensorDict._new_unsafe in step (#2905) (e5cba04df) by @vmoens ghstack-source-id: 8a117fb7f9c5b24173408d217f59ec23da7db33c
[Quality] Better device checks (#2909) (382430db3) by @vmoens ghstack-source-id: 7174415de2b4221c6c5fca4a31525ed26bc8d6f9
[Quality] Limit warning filter to torchrl (#2762) (85d1e70d3) by @antoinebrl
[Quality] Remove redundant return (#2925) (21ef7253c) by @b10902118
[Setup] Fix no_python_abi_suffix error (#2863) (7df831753) by @vmoens ghstack-source-id: 55c845efd936558116f8fdc356f22aca88943f99
[Setup] Remove distutils imports (#2836) (4c55b65b9) by @antoinebrl
[Test] Capture deprec warnings (#2799) (fb641de13) by @vmoens ghstack-source-id: bcbf41c245c979d0f21524889ad2be8ef4c10c40
[Test] Fix warnings in tests (#2886) (6f634c6fb) by @vmoens ghstack-source-id: d4ed75d4dae2f0d62adff567d5dcc5fd2f98ce3a
[Versioning] Bump 0.8.0 (#2920) by @vmoens ghstack-source-id: 74b3ef75b2b911c097c1d985068958b697a00134
A big thanks to the community supporting this project! There would be no TorchRL if it wasn't for its users.
We are releasing TorchRL 0.7.2, a minor update that addresses several important bug fixes to improve the stability and reliability of our library.
We are releasing TorchRL 0.7.2, a minor update that addresses several important bug fixes to improve the stability and reliability of our library.
This release is particularly crucial as it resolves a critical issue (#2840) where, under certain conditions, the device setting of the parallel environment would prevent the tensors in the buffers from being properly cloned. This resulted in rollouts returning the same tensor instances across steps, potentially leading to incorrect behavior and results.
Due to the severity of this bug, we strongly recommend that all users upgrade to TorchRL 0.7.2 to ensure the accuracy and reliability of their experiments.
The full list of changes can be found below:
Full Changelog: https://github.com/pytorch/rl/compare/v0.7.1...v0.7.2
We are pleased to announce the release of torchrl v0.7.1, which includes several bug fixes, documentation updates, and backend improvements.
We are pleased to announce the release of torchrl v0.7.1, which includes several bug fixes, documentation updates, and backend improvements.
Thank you to @antoinebrl, and @louisfaury for contributing to this release!
Full Changelog: https://github.com/pytorch/rl/compare/v0.7.0...v0.7.1
[Deprecation] Change the default MLP depth (#2746) (12e6bce60) by @vmoens ghstack-source-id: bd34b8e9112c4fc3a30bd095e3ac073a7d0b5469 [Deprecation] Gr…
As always, we want to warmly thank the RL community who's supporting this project. A special thanks to our first time contributors:
as well as all the users who wrote issues, suggestions, started discussions here, on discord, on the pytorch forum or elsewhere! We value your feedback!
As announced, we removed the following classes:
The default MLP depth has passed from 3 to 0 (i.e., now MLP(in_features=3, out_features=4) is equivalent to a regular
nn.Linear layer).
Environments specs are now carefully locked by default (#2729, #2730). This means that
env.observation_spec = spec
is allowed (specs will be unlocked/re-locked automatically) but
env.observation_spec["value"] = spec
will not work. The core idea here is that we want to cache as much info as we can, such as action keys or whether the env has dynamic specs. We can only do that if we can guarantee that the env has not been modified. Locking the specs provides us such guarantee. Note that a version of this already existed but it was not as robust as the new one.
TL;DR: We're changing the way log-probs and entropies are collected and written in ProbabilisticTensorDictModule and
in CompositeDistribution. The "sample_log_prob" default key will soon be "<value>_log_prob (or
("path", "to", "<value>_log_prob") for nested keys). For CompositeDistribution, a different log-prob will be
written for each leaf tensor in the distribution. This new behavior is controlled by the
tensordict.nn.set_composite_lp_aggregate(mode: bool) function or by the COMPOSITE_LP_AGGREGATE environment variable.
We strongly encourage users to adopt the new behavior by setting tensordict.nn.set_composite_lp_aggregate(False).set()
at the beginning of their training script.
The behavior of CompositeDistribution and its interaction with on-policy losses such as PPO has changed.
The PPO documentation now includes a section about multi-head policies and the examples also give such information.
See the tensordict v0.7.0 release notes or #2707 to know more.
[Deprecation] Change the default MLP depth (#2746) (12e6bce60) by @vmoens ghstack-source-id: bd34b8e9112c4fc3a30bd095e3ac073a7d0b5469 [Deprecation] Gracing old *Spec with v0.8 versioning (#2751) (fa697fe59) by @vmoens ghstack-source-id: e7c6e0a4b8520da887fe7e602a351c3c72a08c4c [Deprecation] Remove AdditiveGaussianWrapper (#2748) (6c7f4fbda) by @vmoens ghstack-source-id: 78f248e1239a04fc5213aa4418a158f741679593 [Deprecation] Remove InPlaceSampler (#2750) (0feef11f9) by @vmoens ghstack-source-id: eeae1bf0611a5d293f533767eee7b9700e720cc8 [Deprecation] Remove NormalParamWrapper (#2747) (a38604e47) by @vmoens ghstack-source-id: 4a70178f54f9e25d602c86a0b61248d66f3e39bd [Deprecation] Remove OrnsteinUhlenbeckProcessWrapper (#2749) (0111a8795) by @vmoens ghstack-source-id: 401fdfaca2e27122d5a67fc7177e1015047f0098
We gave a strong focus on a better compatibility with torch.compile across the SOTA training scripts which now
all accept a compile=1 argument. The overall speedups range from 1 to 4x
<img width="566" alt="Screenshot 2025-02-05 at 21 20 54" src="https://github.com/user-attachments/assets/bd36ce4d-426e-4d7d-a4da-ff62eee78240" />
Loss module speedups are displayed in the README.md page.
Replay buffers are also mostly compatible with compile now (with the notable exception of distributed and memmaped ones).
<attr>_spec_unbatchedYou can now use env.auto_spec_ to set the specs automatically based on a dummy rollout.
For batched environments, the unbatched spec can now be accessed via env.<attr>_spec_unbatched. This is useful to
create random policies, for example.
We added TrajCounter (#2532), Hash and Tokenizer (#2648, #2700) and LineariseReward (#2681).
We provide a new ListStorage-based storage (LazyStackStorage) that automatically represents samples as a LazyStackedTensorDict
which makes it easy to store ragged tensors (although not contiguously in memory) #2723.
A new torchrl.envs.ChessEnv allows users to train agents to play chess!
We also opensourced a tutorial to export TorchRL modules on hardware: #2557
[Feature, Test] Adding tests for envs that have no specs (#2621) (c72583f75) by @vmoens ghstack-source-id: 4c75691baa1e70f417e518df15c4208cff189950
[Feature,Refactor] Chess improvements: fen, pgn, pixels, san, action mask (#2702) (d425777b8) by @vmoens ghstack-source-id: f294a2bc99a17911c9b62558d530b148d3c0350f
[Feature] A2C compatibility with compile (#2464) (507766a88) by @vmoens ghstack-source-id: 66a7f0d1dd82d6463d61c1671e8e0a14ac9a55e7
[Feature] ActionDiscretizer custom sampling (#2609) (3da76f006) @oslumbers Co-authored-by: Oliver Slumbers oliver.slumbers@helsing.ai
[Feature] Add Hash transform (#2648) (50011dcf1) @kurtamohler ghstack-source-id: dccf63fe4f9d5f76947ddb7d5dedcff87ff8cdc5
[Feature] Add Choice spec (#2713) (9368ca68e) @kurtamohler ghstack-source-id: afa315a311845ab39ade3e75046f32757f9d94f1
[Feature] Add LossModule.reset_parameters_recursive (#2546) (218d5bf70) by @kurtamohler
[Feature] Add Stack transform (#2567) (594462d6b) by @kurtamohler
[Feature] Add deterministic_sample to masked categorical (#2708) (49d9897af) by @vmoens ghstack-source-id: d34fcf9b44d7a7c60dbde80b0835189f990ef226
[Feature] Adds ordinal distributions (#2520) (c851e1698) by @louisfaury Co-authored-by: @louisfaury
[Feature] Avoid some recompiles of ReplayBuffer.extend/sample (#2504) (0f29c7e93) @kurtamohler
[Feature] CQL compatibility with compile (#2553) (e2be42e82) by @vmoens ghstack-source-id: d362d6c17faa0eb609009bce004bb4766e345d5e
[Feature] CROSSQ compatibility with compile (#2554) (01a421e76) by @vmoens ghstack-source-id: 98a2b30e8f6a1b0bc583a9f3c51adc2634eb8028
[Feature] CatFrames.make_rb_transform_and_sampler (#2643) (9ee1ae7ee) by @vmoens ghstack-source-id: 7ecf952ec9f102a831aefdba533027ff8c4c29cc
[Feature] ChessEnv (#2641) (17983d43e) by @vmoens ghstack-source-id: 087c3b12cd621ea11a252b34c4896133697bce1a
[Feature] Composite.batch_size (#2597) (2e82cab19) by @vmoens ghstack-source-id: 621884a559a71e80a4be36c7ba984fd08be47952
[Feature] Composite.pop (#2598) (8d16c12bd) by @vmoens ghstack-source-id: 64d5bd736657ef56e37d57726dfcfd25b16b699f
[Feature] Composite.separates (#2599) (83e0b0568) by @vmoens ghstack-source-id: fbfc4308a81cd96ecc61723df8c0eb870c442def
[Feature] Custom conversion tool for gym specs (#2726) (dbc8e2ee0) by @vmoens ghstack-source-id: d38bb02f15267a9b1637b3ed25fb44ef013e2456
[Feature] DDPG compatibility with compile (#2555) (7d7cd9538) by @vmoens ghstack-source-id: f18928a419f81794d6870fd4e9fe1205c1b137e1
[Feature] DQN compatibility with compile (#2571) (f149811da) by @vmoens ghstack-source-id: 113dc8c4a5562d217ed867ace1942b2f6b8a39f9
[Feature] DT compatibility with compile (#2556) (fbfe10488) by @vmoens ghstack-source-id: 362b6e88bad4397f35036391729e58f4f7e4a25d
[Feature] Discrete SAC compatibility with compile (#2569) (9e2d214fa) by @vmoens ghstack-source-id: ddc131acedbbe451b28758e757a8c240ebd72b80
[Feature] Ensure out-place policy compatibility in rollout and collectors (#2717) (ec370c6b6) by @vmoens ghstack-source-id: 41a6aa56e0a045a20224b96f9537a7ae3ae14494
[Feature] EnvBase.auto_specs_ (#2601) (d537dcb63) by @vmoens ghstack-source-id: 329679238c5172d7ff13097ceaa189479d4f4145
[Feature] EnvBase.check_env_specs (#2600) (00d3199ec) by @vmoens ghstack-source-id: 332dbf92db496c71c5ce6aba340ad123eac0f5d6
[Feature] GAIL compatibility with compile (#2573) (6482766b8) by @vmoens ghstack-source-id: 98c7602ec0343d7a83cb19bddeb579484c42e77e
[Feature] IQL compatibility with compile (#2649) (2cfc2abd6) by @vmoens ghstack-source-id: 77bca166701d28dd69ef3964f55ab4f3e4b17fed
[Feature] LLMHashingEnv (#2635) (30d21e599) by @vmoens ghstack-source-id: d1a20ecd023008683cf18cf9e694340cfdbdac8a
[Feature] LazyStackStorage (#2723) (fe3f00c6c) by @vmoens ghstack-source-id: e9c031470aa0bdafbb2b26c73c06b25685a128e5
[Feature] Linearise reward transform (#2681) (ff1ff7e9c) by @louisfaury Co-authored-by: @louisfaury
[Feature] Log each entropy for composite distributions in PPO (#2707) (319bb68f0) by @louisfaury Co-authored-by: @louisfaury
[Feature] Log pbar rate in SOTA implementations (#2662) (1ce25f19a) by @vmoens ghstack-source-id: 283cc1bb4ad2d60281296d2cfb78ec41c77f4129
[Feature] MCTSForest (#2307) (e9d167711) by @vmoens ghstack-source-id: 9ac5cd3de39a4dbe1c7c33cb71ff6f45a886ae65
[Feature] Make PPO compatible with composite actions and log-probs (#2665) (256a7002c) by @vmoens ghstack-source-id: c41718e697f9b6edda17d4ddb5bd6d41402b7c30
[Feature] PPO compatibility with compile (#2652) (f5a187d7d) by @vmoens ghstack-source-id: 0ed29f352fcd85f0dc0683d90e95bdbecf6c14f9
[Feature] Re-enable cache for specs (#2730) (4262ab91e) by @vmoens ghstack-source-id: 797132312bfd9749f8926a2dd0b03eff65b8f51c
[Feature] SAC compatibility with compile (#2655) (87a59fb30) by @vmoens ghstack-source-id: b57caeaf6e2d3690fb3311f4c9b8cca8575d3974
[Feature] Send info dict to the storage device in RBs (#2527) (d524d0d6b) by @vmoens ghstack-source-id: 4ed60d649b17f96b49f90d234e679937c60a3c32
[Feature] TD3 compatibility with compile (#2658) (1b7eda199) by @vmoens ghstack-source-id: fb94307557f2b8604403b48211e3da6fb2139e28
[Feature] TD3-bc compatibility with compile (#2657) (91064bc27) by @vmoens ghstack-source-id: 8a33e39829f620c1e1a579a0255162ba93eaca91
[Feature] TensorSpec.enumerate() (#2354) (14b63e4f0) by @vmoens ghstack-source-id: 9db2f5ee47a197eb0403cb4622266fb03b99360f
[Feature] TrajCounter transform (#2532) (05aeb8975) by @vmoens ghstack-source-id: 62a3091e5c9072f26266143319f30de1729c0d4e
[Feature] UnaryTransform for input entries (#2700) (093a1599f) by @vmoens ghstack-source-id: bb0ea97f47bdad6ba5e73692969fece4e2efbfb4
[Feature] example_data for NonTensor spec (#2698) (80690d221) by @vmoens ghstack-source-id: 6fe5d82763dfcc9044d6debe88f0f34bb739c987
[Feature] automatically determine return_contiguous (#2724) (cac93eb0e) by @vmoens ghstack-source-id: 6d1fc31d87cb021e6286cdb07db2d9b0e2302f7d
[Feature] env.step_mdp (#2636) (4bc40a808) by @vmoens ghstack-source-id: 145e37cd772fdd74e35e5ffe6accc5c81ad689f3
[Feature] flexible batch_locked for jumanji (#2382) (35a78139b) by @vmoens ghstack-source-id: e356b6511ff3da8a6c583747214cfa90f42c9083
[Feature] lock_ / unlock_ graphs (#2729) (601483e71) by @vmoens ghstack-source-id: 01e375e636b97b26a89f9bbab2e955db6c85978a
[Feature] multiagent data standardization: PPO advantages (#2677) (b7a0d11e5) by @matteobettini Co-authored-by: Vincent Moens vmoens@meta.com
[Feature] no_cuda_sync arg in collectors (#2727) (280297aee) by @vmoens ghstack-source-id: 9baba31b3ee844882fd4b6a6f69874946caf3b3e
[Feature] single_<attr>_spec (#2549) (58c384713) by @vmoens ghstack-source-id: 27e247ea1775e455999a114dd6d95fac748376c4
[Feature] spec.cardinality (#2638) (dd26ae79f) by @vmoens ghstack-source-id: 1160900f8a81dd51dc72436e1af69c8248bff162
[Feature] spec.is_empty(recurse) (#2596) (097d8ad98) by @vmoens ghstack-source-id: faa3b1df5133c77462d6dd013d3854d684cc7e94
[Feature] timeit.printevery (#2653) (187de7c8b) by @vmoens ghstack-source-id: 19165bbfbea5cdc0a6b159493fb02571bab872f3
[Minor,Feature] Add prefix arg to timeit.todict (#2576) (7bc84d15d) by @vmoens ghstack-source-id: f1ff685caf6e8950d02dfc44ad2c1eb496495ad1
[Minor,Feature] group_optimizers (#2577) (7829bd3f3) by @vmoens ghstack-source-id: 81a94ed641544a420bb1c455921ca6a17ecd6a22
[Doc] Add AOTInductor back (#2564) (9f8f77cdb) by @vmoens ghstack-source-id: 774eb5973045861f284fdc67f74945b1eecdeaf2
[Doc] Add Tokenizer and auto-reset doc link (#2754) (ee4006a6b) by @vmoens ghstack-source-id: 90f55b568e85ae151bea4370025144c19e74602b
[Doc] Add Stack transform link in docs (#2689) (c5f1565de) by @kurtamohler
[Doc] Adding recurrent policies to export tutorial (#2559) (705123870) by @vmoens ghstack-source-id: 1f1af399b120db8bbb1789748641f44fd3b1bd5e
[Doc] Better doc for SliceSampler (#2607) (90572ac11) by @vmoens ghstack-source-id: 7d79ef7d37c4dc2ffbdff5b422cf5da24d93c0da
[Doc] Fix broken links and formatting issues in doc (#2574) (5a2d9e205) by @vmoens ghstack-source-id: 4e3f84fe436de6a6e9696894cd06318a98e4a23b
[Doc] Fix modules doc (#2531) (edbf3dee3) by @vmoens
[Doc] Fix tutorials (#2560) (2f3b4cd4d) by @vmoens ghstack-source-id: 6c9114384015e76e96b3bbd0c8893cc42344537a
[Doc] Fix typo in torchrl/modules/distributions/continuous.py (#2624) (b2e9f291a) by @Mana
[Doc] Fix typos (#2682) (f672c708f) by Nils Kiele Co-authored-by: Vincent Moens vincentmoens@gmail.com
[Doc] MADDPG bug fix of buffer device and improve explaination (#2519) (3e4b2928e) by @matteobettini
[Doc] Minor fixes to the docs and type hints (#2548) (50a35f69b) by @thomasbbrunner
[Doc] Tutorial on exporting TorchRL models (#2557) (c0187a93e) by @vmoens ghstack-source-id: b93146e22d8376563e7ac302b5cff95f09ae50d4
[Doc] Typo in docs for actors.py (#2545) (19dbeebf0) by @carschandler
[Doc] Update docstring for TruncatedNormal with correct parameter names (#2625) (d22266d05) by @valterschutz Co-authored-by: Valter Schutz valterschutz@proton.me
[Doc] actor docstrings (#2626) (825779935) by @valterschutz Co-authored-by: Valter Schutz valterschutz@proton.me
[Doc] fix several typos (#2603) (de153bf45) by @carschandler
[Doc] torchrl_demo.py revamp (#2561) (304e707ef) by @vmoens ghstack-source-id: 2f0087850e4a7d4d4393f0662156af9bfca8e3e1
[Example] Efficient Trajectory Sampling with CompletedTrajRepertoire (#2642) (b840a772c) by @vmoens ghstack-source-id: 4d5c587c69230aa8f3a1b9b6fe19f52fa683d703
[Example] RNN-based policy example (#2675) (d009835b4) by @vmoens ghstack-source-id: ef0087e9b5cba40be428f57ef70ecd2f63483d03
[Example] Using Collector's device args (#2705) (539c2158d) by @vmoens ghstack-source-id: 9aec8daa53000bdfd6091be706c7bc46778d5983
[Performance] Accelerate slice sampler on GPU (#2672) (84c3ec322) by @vmoens ghstack-source-id: a4dc1515d8b51f5ec150b2fae4e1a84254f2af09
[Performance] Avoid cloning trajs in SliceSampler (#2671) (4fd54fef4) by @vmoens ghstack-source-id: 2e133fcea716b202694cfa84df3f6e4ba3507bbc
[Performance] Improve performance of compiled ReplayBuffer (#2529) (2a07f4c0f) by @kurtamohler
[Benchmark] Add benchmark for compiled ReplayBuffer.extend/sample (#2514) (5e03a5518) @kurtamohler ghstack-source-id: d4562697e2c1a8392cf5bdcadb50f8b7b6939e41
[BE] Add trailing spaces when necessary (#2581) (600760f5b) by @vmoens ghstack-source-id: 198b5b5668cce8336d44206c10dacb8a9b1a9785 [BE] Add type annotation for tensor_keys to facilitate auto-complete (#2696) (4b3279a3f) by @vmoens ghstack-source-id: b4a8fe38e7c6b028759eef082f65f26036bc0250 [Refactor,CI] Refactor SOTA tests (#2583) (c0ba3ff54) by @vmoens ghstack-source-id: b14c59bb1ca7bf056bde05fa0abd01fa7e9b3710 [Refactor] Allow safe-tanh for torch >= 2.6.0 (#2580) (1474f8517) by @vmoens ghstack-source-id: 92df1954451453ee051bbde499f6db5ebaafed49 [Refactor] Deprecate recurrent_mode API to use decorators/CMs instead (#2584) (14b277513) by @vmoens ghstack-source-id: 80f705e022abc111df3960fc09576d5e266ed4dd [Refactor] Refactor trees (#2634) (57dc25a44) by @vmoens ghstack-source-id: 368ba4c4402b6db0bc8b0688802ce161db9776b7 [Refactor] Rename Recorder and LogReward (#2616) (607ebc52d) by Goia Rares Dan Tiago [Refactor] Use <spec>_unbatched in VMAS (#2593) (a126a6f94) by @vmoens ghstack-source-id: 2190278de44ba59a3bc8d38398fddae9ecc42a84 [Refactor] Use default device instead of CPU in losses (#2687) (c3b9d1dc7) by @vmoens ghstack-source-id: 8b98062c3ae88d8780ef7428fdfa07e305c790b9 [Refactor] compile compatibility improvements (#2578) (db7f08d76) by @vmoens ghstack-source-id: 95f8241b56e42b80e828485cb5f377288bff6f5e [Quality,BE] Better doc for step_mdp (#2639) (ef5a37d8a) by @vmoens ghstack-source-id: 1f5aed6fb2e97ead9d379f9545ae742f7728c585 [Quality] Better TD construction in codebase (#2565) (a4c1ee3b3) by @vmoens ghstack-source-id: 9e280d9d7d4a735e5055beb0450d933547530e55 [Quality] Better warning when c++ binaries failed to be imported (#2541) (0a13cbd5e) by @vmoens [Quality] IMPALA auto-device (#2654) (526b38d5c) by @vmoens ghstack-source-id: abbb3048f33c9f7f6a623e32e139871093ea74fa [Minor] Fix doc and MARL tests (#2759) (ad7d2a10b) by @vmoens ghstack-source-id: 9308be3ebc7fac30b5bde321792eb97069d55996 [Minor] Fix fbcode imports of mocking classes (#2526) (da0bf1897) by @vmoens ghstack-source-id: 74f9f3bedf8f48988a1956084548f6cd2f720934 [Minor] Make fbcode happy with imports (#2517) (a70b258cd) by @vmoens ghstack-source-id: d4bfce9d51269bc0ab6154ee4c2d1e1ff7af0895
[BugFix, BE] Document and fix fps passing in recorder and loggers (#2694) (61e05b3d9) by @vmoens ghstack-source-id: b3996a9a27643eb5da8a78135f6b9fcef3685f17
[BugFix,Doc] Fix BATCHED_PIPE_TIMEOUT refs and doc (#2695) (dc25a55a7) by @vmoens ghstack-source-id: 6e43c4ff1c319545cf0952abf6f35f3e7ed473e0
[BugFix,Doc] Revert dynamic shape in export tutorial (#2563) (9d292a007) by @vmoens ghstack-source-id: fc856218e840469a5bb0143241d100e9cc612538
[BugFix,Test,Benchmark] Fix graph breaks induced by device context manager (#2602) (152bc81b7) by @vmoens ghstack-source-id: 0df2728928280a43de4abd30afed20826b0af091
[BugFix,Test] test chess rendering (#2721) (ddbb6fdd5) by @vmoens ghstack-source-id: 59b37e6fa2f8c11f600eea334da0bd8257ed382c
[BugFix] Account for composite actions in gym (#2718) (1246db197) by @vmoens ghstack-source-id: c09b59904a89d45fa24a61a5e8a24fe307320794
[BugFix] Account for terminating data in SAC losses (#2606) (c8676f4a8) by @vmoens ghstack-source-id: dc1870292786c262b4ab6a221b3afb551e0efb9b
[BugFix] ActionDiscretizer scalar integration (#2619) (830f2f26c) by @vmoens ghstack-source-id: b22102f3730914b125ef0f813f4d2f22dec0b26e
[BugFix] Allow expanding TensorDictPrimer transforms shape with parent batch size (#2552) (83a7a57da) by Albert Bou Co-authored-by: Vincent Moens vmoens@meta.com
[BugFix] Avoid KeyError in slice sampler (for compile) (#2670) (21eeca42c) by @vmoens ghstack-source-id: 6e2a3036f0e50d365387cced50a761b97a47317d
[BugFix] Better account of composite distributions in PPO (#2622) (90c8e40f6) by @vmoens ghstack-source-id: 3d86f99bc5b20a53e4092d786e96a5f7e83405ac
[BugFix] Compatibility of tensordict primers with batched envs (specifically for LSTM and GRU) (#2668) (f4709c143) by @vmoens ghstack-source-id: e1da58ecfd36ca01b8a11fe90e5f3c5fe77f064c
[BugFix] Fix MARL PPO tutorial action_spec call (#2628) (1ca134cc3) by @vmoens ghstack-source-id: 1d9058c45b28c0f0279e4243a2a0f96c622a51d8
[BugFix] Fix batching envs with non tensor data (#2674) (ab4250ec7) by @vmoens ghstack-source-id: daba8a95459cfa978da09291757b6380fab4f308
[BugFix] Fix call to tree.plot in tests (#2547) (09d6866e0) by @vmoens ghstack-source-id: 4a5babbf46294ab6ed4a791e26cfacaf3a41a2e0
[BugFix] Fix collector length with non-empty batch size (#2575) (b87597922) by @vmoens ghstack-source-id: 0c6a7a49f0570fad083340a64dd89c0f4c220c06
[BugFix] Fix compile weakrefs errors (#2742) (ffa99b2a2) by @vmoens ghstack-source-id: 3cb4c62f465a3c0581064b3ff89290b9d225eb3f
[BugFix] Fix device transfer for collectors with init_random_frames mixed devices (#2704) (1d45117ba) by @vmoens ghstack-source-id: 1684399a7c84dd19b396db6c903fbf68c971c73d
[BugFix] Fix export aoti_compile_and_package API change (#2629) (1cffffee9) by @vmoens ghstack-source-id: 07a0f063f8955815157c2a3eac02c6460a82f672
[BugFix] Fix failing tests (#2582) (863121a27) by @vmoens ghstack-source-id: a43a2e3dbf76cd63c57ae00028df04b41a4e2f2b
[BugFix] Fix get_default_device calls in older PT versions (#2586) (705ecc2bb) by @vmoens ghstack-source-id: fd3a739d38feba075073801dda362be598822a94
[BugFix] Fix imports (#2605) (d90b9e3d1) by @vmoens ghstack-source-id: db85f2611c1c0b22e9179b4fdd6c2dcea78ac8dd
[BugFix] Fix init_random_frames=0 (#2645) (19dfefc84) by @vmoens ghstack-source-id: 38a544ea15631f9affb4c385c09e7c4df94af55d
[BugFix] Fix missing min/max alpha clamps in losses (#2684) (ed656a15f) by @vmoens
[BugFix] Fix output of SipHash(as_tensor=False) (#2664) (1fc9577c4) by @kurtamohler
[BugFix] Fix partial device transfers in collector (#2703) (afb81de51) by @vmoens ghstack-source-id: 2cd74c2d6fceaf079122ae801b67bdbfc29cddaf
[BugFix] Fix pendulum device (#2516) (6799a7f5d) by @vmoens ghstack-source-id: bcaf20de6e317d4bda0e1511e0b1e46653a6f352
[BugFix] Fix safe probabilistic backward by removing in-place modif (#2755) (2f8c118e3) by @vmoens ghstack-source-id: 574eb1f9b662c1eb5be25e97020e11b3fadf625e
[BugFix] Fix tests failing because of https://github.com/pytorch/pytorch/pull/137602 (165163abe) by @vmoens
[BugFix] Fix typing for python 3.9 (#2631) (e7062a1d6) by @vmoens ghstack-source-id: 663da84096214611804a726e2d38d27a6f21c958
[BugFix] Fix typing in chess env (#2646) (cb8e241b2) by @vmoens ghstack-source-id: ad6086bbb7d1ee528ca24ec1d1232da47372e2b5
[BugFix] Fix typing in llm env (#2647) (e3c304733) by @vmoens ghstack-source-id: b5608f91756b5a81141941903158417a111e0710
[BugFix] Fix version parsing in extensions (#2542) (997d90e1b) by @vmoens ghstack-source-id: 903f2b01b508b81b1b4f92c4297d390da79fe8a2
[BugFix] PettingZoo dict action spaces (#2692) (1a6c9e2d0) by @matteobettini
[BugFix] Remove erroneous python 3.8 compatibility classifier (#2540) (528875a9f) by @vmoens
[BugFix] Remove raisers in specs (#2651) (bb6f87adb) by @vmoens ghstack-source-id: a005a62847aa2ff1d286f2c4ad13fd14f9e631d3
[BugFix] Rename RayCollector example file to avoid ImportError (#2525) (8eac84ad2) by Albert Bou
[BugFix] Support for tensor collection in the PPOLoss (#2543) (0eabb7897) by Pau Riba Co-authored-by: Pau Riba pau.riba@helsing.ai
[BugFix] Temporarily remove unsafe caching in envs (#2728) (dc63e820d) by @vmoens ghstack-source-id: a139cf6dc9fcfcfa525a6aa6375163d379593550
[BugFix] Wrong spec returned (#2604) (a1e21f598) by @matteobettini
[BugFix] action_spec_unbatched whenever necessary (#2592) (d30599ec0) by @vmoens ghstack-source-id: ec87794dabaf5023dac85cfc898a7c000e93331d
[BugFix] adapt log-prob TD batch-size to advantage shape in PPO (#2756) (cb37521e1) by @vmoens ghstack-source-id: 8ccd12f65f4a74a42356a630e0e5a1f015337d4a
[BugFix] make buffers zero-dim in exploration modules (#2591) (a47b32c07) by @vmoens ghstack-source-id: fd2705eb9132169da4871b27b354f7895c644061
[BugFix] patch rand_action in TransformedEnv to read the base_env method (#2699) (2c19fcc70) by @vmoens ghstack-source-id: 04e2e85e2675cf34c349ebadb8fa85a5aff2e532
[BugFix] requested_frames_per_batch in distributed collectors (#2579) (408cf7d04) by @vmoens ghstack-source-id: 49289de6956460d9aed13d982eb8003eafc35118
[BugFix] skip_done_states in SAC (#2613) (de61e4d5e) by @vmoens ghstack-source-id: 39d97360e3b0e45dd8c327487eac50ddafe2254d
[Test] Add tests and a few fixes for ChessEnv (#2661) (7bbd7e3b6) @kurtamohler ghstack-source-id: d0fbb520e35c74305041340722a7560ac2f958f2 [Test] Add tests for CatFrames with PermuteTransform (#2715) (d4e401993) @kurtamohler ghstack-source-id: e554d1cda8d7e4458c9397f1f93345c855e68e5c [Test] Add tests for Tree (#2738) (bb9440b40) @kurtamohler ghstack-source-id: 8f7aa07a4d36aa3664eaa19cc35bd66fb9e61c24 [Test] Fix warnings in SOTA tests (#2710) (a90106475) by @vmoens ghstack-source-id: c79223b5d6548a6c5a6ef649f6eb8e1703258815 [Test] More comprehensive tests for auto_spec (#2640) (6c7d233a4) by @vmoens ghstack-source-id: 75352490436fd706af3d36f9b8016e80a8a3f46a [Test] Skip tokenizer tests if transformers is not in workspace (#2744) (20a19fe2a) by @vmoens ghstack-source-id: b92facfd14cba62511e7888567c94d3986419ab5 [Test] Str2StrEnv test (#2725) (5fd509232) by @vmoens ghstack-source-id: 45a0e5f4b33c4624758171b9fe31f1e3932ff5e4 [CI, BugFix] Py3.8 for old deps (#2568) (f3275dab0) by @vmoens ghstack-source-id: 13c7923c0e5c8725c12c3bacc6c21b250d9f7457 [CI] Change doc image (#2632) (2511c04a5) by @vmoens ghstack-source-id: eceab242294ec55135d79f29e848345a5d5d455e [CI] Cuda 12.4 (#2733) (37a514d6c) by @vmoens ghstack-source-id: 2f3842a17d03e530add9608ee4525347a7c6a0e5 [CI] Fix Cairo-2 Chess import error (#2743) (10f015e0c) by @vmoens ghstack-source-id: c2bcbfc4522bd1b4f1fea3dbb006dc9552b09cb4 [CI] Fix docs upload (#2587) (0f592266f) by @vmoens ghstack-source-id: 49d7df06340fc432c29cd9f2d0ed2ae3d5619a38 [CI] Fix dreamer run in SOTA tests (#2627) (aed03fda4) by @vmoens ghstack-source-id: dfe3ab6fe0d29fcdcaf57f31f84d04e07e36bad3 [CI] Fix nightly build (#2666) (133d70936) by @vmoens ghstack-source-id: 5502fa94b6abcc154e020dcb165093fdc30ca025 [CI] Fix olddeps dm_control (#2734) (3ac61270f) by @vmoens ghstack-source-id: 750edcb8cd6b17167f77fb7c9ebd538608cfbde6 [CI] Fix windows build (#2760) (03f56ffb0) by @vmoens [CI] Install stable torch/tv in docs when on release branch (#2761) (57bdc6aec) by @vmoens ghstack-source-id: 7c39c049c7cff0ee112be2d07597f2e291d2fafd [CI] Local import of PIL (#2720) (d628a507f) by @vmoens ghstack-source-id: 6eb4ace11022632e902a7277dd51344bb9fe1f65 [CI] Longer timeout for windows (#2765) (4c06ce2b8) by @vmoens ghstack-source-id: 381e7e39d650e0178178a78076321a2210237b39 [CI] Make MAX_IDLE_COUNT a feature of tests (#2752) (963f3cdf6) by @vmoens ghstack-source-id: 2bf31dfff3d7862a54abeea86c8c5cc47a0f302d [CI] Remove gym import in test_libs.py (#2719) (f2cf5e044) by @vmoens ghstack-source-id: b0474588cfc81ed135d70efb58203c0b503f4ff0 [CI] Revert upgrade of upload image in docs (#2585) (236d38f8a) by @vmoens ghstack-source-id: f323dd2667a073b6c763ed17a793ecd0eec6b7be [CI] Upgrade GHA versions (#2740) (cd4f359ef) by @vmoens ghstack-source-id: 1876f1f0c18cb11c74edc9d96c17fdc985bc7b1a [CI] Upgrade cu121 to cu124 (#2764) (5da1f6522) by @vmoens ghstack-source-id: 4b3c9c0c31a60a5e151ff13b21e54853dc426416 [CI] Upgrade to v0.7 (#2745) (0ecfbe36e) by @vmoens ghstack-source-id: e548bbbb4578d44a8eee000ab0a40c89713afc27 [CI] linux_job_v2.yml (#2570) (527a26a27) by @vmoens ghstack-source-id: ae13b53bd2885263e80019c087171421f5f7d0d5 [CI] minari[hf] (#2722) (dda0df165) by @vmoens ghstack-source-id: 6eb84d906dfbc66839706f328e214014aef7b65f [CI] workflow permissions (#2706) (b000685f3) by @vmoens ghstack-source-id: f520a1b1e7697b1147cb29e66e2ecb1d07cb4cbc
[Deprecation] Deprecate default num_cells in MLP (#2395) by @vmoens
We introduce wrappers for ML-Agents and OpenSpiel. See the doc here for OpenSpiel and here for MLAgents.
We introduce support for [partial steps](#2377, #2381), allowing you to run rollouts that ends only when all envs are done without resetting those who have reached a termination point.
We add the capability of passing replay buffers directly to data collectors, to avoid inter-process synced communications - thereby drastically speeding up data collection. See the doc of the collectors for more info.
The GAIL algorithm has also been integrated in the library (#2273).
We ensure that all loss modules are compatible with torch.compile without graph breaks (for a typical built). Execution of compiled losses is usually in the range of 2x faster than its eager counterpart.
Finally, we have sadly decided not to support Gymnasium v1.0 and future releases as the new autoreset API is fundamentally incompatible with TorchRL. Furthermore, it does not guarantee the same level of reproducibility as previous releases. See this discussion for more information.
We provide wheels for aarch64 machines, but not being able to upload them to PyPI we provide them attached to these release notes.
OpenSpielWrapper and OpenSpielEnv (#2345) by @kurtamohlergroup_map support to MLAgents wrappers (#2491) by @kurtamohlerhold_out_net #2499 by @vmoensTD_GET_DEFAULTS_TO_NONE=1 in all CIs (#2363) by @vmoensMultiCategorical support in PettingZoo action masks (#2485) Co-authored-by: Vincent Moens by @matteobettinireshape(-1) for inputs to DreamerActorLoss (#2496) by @kurtamohlerreshape(-1) for inputs to objectives modules (#2494) Co-authored-by: Vincent Moens by @kurtamohlerCatFrames.unfolding with padding="same" (#2407) by @kurtamohlerPrioritizedSliceSampler._padded_indices (#2433) by @kurtamohlerSliceSampler._tensor_slices_from_startend (#2423) by @kurtamohlerknowledge_base entry (#2383) by @depictigeradvantages.py (#2492) Co-authored-by: Louis Faury by @louisfauryAs always, we want to show how appreciative we are of the vibrant open-source community that keeps TorchRL alive.
Full Changelog: https://github.com/pytorch/rl/compare/v0.5.0...v0.6.0
[BugFix] missing deprecated kwargs by @fedebotu in https://github.com/pytorch/rl/pull/2125
This new release makes it possible to run environments that output non-tensor data. #1944
We also introduce dynamic specs, allowing environments to change the size of the observations / actions during the course of a rollout. This feature is compatible with parallel environment and collectors! #2143
Additionally, it is now possible to update a Replay Buffer in-place by assigning values at a given index. #2224
Finally, TorchRL is now compatible with Python 3.12 (#2282, #2281).
As always, a huge thanks to the vibrant OSS community that helps us developt this library!
modules.AdditiveGaussianModule by @kurtamohler in https://github.com/pytorch/rl/pull/2296modules.OrnsteinUhlenbeckProcessModule by @kurtamohler in https://github.com/pytorch/rl/pull/2297to in MultiDiscreteTensorSpec by @Quinticx in https://github.com/pytorch/rl/pull/2204True by @vmoens in https://github.com/pytorch/rl/pull/2145_reset follow done shape by @matteobettini in https://github.com/pytorch/rl/pull/2189EnvBase._complete_done to complete "terminated" key properly by @kurtamohler in https://github.com/pytorch/rl/pull/2294LazyTensorStorage only allocates data on the given device by @matteobettini in https://github.com/pytorch/rl/pull/2188done = done | truncated in collector by @vmoens in https://github.com/pytorch/rl/pull/2333__iter__ for samplers without replacement + prefetch by @JulianKu in https://github.com/pytorch/rl/pull/2178__iter__ for samplers without replacement + prefetch" by @vmoens in https://github.com/pytorch/rl/pull/2182_run_checks from TensorDict.__init__ by @vmoens in https://github.com/pytorch/rl/pull/2256*Wrapper to *Module by @kurtamohler in https://github.com/pytorch/rl/pull/2298Full Changelog: https://github.com/pytorch/rl/compare/v0.4.0...v0.5.0
[Deprecation] Deprecate in prep for release by @vmoens in https://github.com/pytorch/rl/pull/1820
reset for batched envs by @vmoens in https://github.com/pytorch/rl/pull/2076reset_parameters for multiagent nets by @matteobettini in https://github.com/pytorch/rl/pull/1970split_trajectories's reference to ("next", "done"). by @initmaks in https://github.com/pytorch/rl/pull/2094normalize_advantage to False. by @DobromirM in https://github.com/pytorch/rl/pull/2050A big thanks to our dear contributors as well as the entire user base for helping with this lib!
Full Changelog: https://github.com/pytorch/rl/compare/v0.3.0...v0.4.0
This release provides a bunch of bug fixes and speedups.
This release provides a bunch of bug fixes and speedups.
[BugFix] Fix broken gym tests (#1980) [BugFix,CI] Fix Windows CI (#1983) [Minor] Cleanup [CI] Install stable torch and tensordict for release tests (#1978) [Refactor] Remove remnant legacy functional calls (#1973) [Minor] Use the main branch for the M1 build wheels (#1965) [BugFix] Fixed import for importlib (#1914) [BugFix] Fix offline CatFrames for pixels (#1964) [BugFix] Fix offline CatFrames (#1953) [BugFix] Fix batch-size expansion in functionalization (#1959) [BugFix] Update iql docstring example (#1950) [BugFix] Update cql docstring example (#1951) [BugFix] Fix examples (#1945) [BugFix] Remove reset on last step of a rollout (#1936) [BugFix] Vmap randomness for value estimator (#1942) [BugFix] Fix multiple context syntax in multiagent examples (#1943) [BugFix] Fix habitat (#1941) [BugFix] Fix env.shape regex matches (#1940) [Minor] Add env.shape attribute (#1938) [BugFix] Fix replay buffer extension with lists (#1937) [BugFix] No grad on collector reset (#1927) [BugFix] fix trunc normal device (#1931) [BugFix, Performance] Fewer imports at root (#1930) [BugFix] Fix OOB TruncatedNormal LP (#1924) [BugFix] Fix KLPENPPOLoss KL computation (#1922) [Doc] Fix onw typo (#1917) [BugFix] Make sure ParallelEnv does not overflow mem when policy requires grad (#1909) [BugFix] Non exclusive terminated and truncated (#1911) [BugFix] Use setdefault in _cache_values (#1910) [BugFix] Fix Ray collector example error (#1908) [BugFix] Make KL-controllers independent of the model (#1903) [Minor] Remove warnings in test_cost (#1902) [BugFix] Adaptable non-blocking for mps and non cuda device in batched-envs (#1900) [BugFix] Fix flaky rb tests (#1901) [BugFix] Fix exploration in losses (#1898) [BugFix] Solve recursion issue in losses hook (#1897) [Doc] Update getting-started-5.py (#1894) [Doc] Getting started tutos (#1886) [BugFix] Use traj_terminated in SliceSampler (#1884) [Doc] Improve PrioritizedSampler doc and get rid of np dependency as much as possible (#1881) [BugFix] Fix _reset data passing in parallel env (#1880) [BugFix] state typo in RNG control module (#1878) [BugFix] Fix a bug in SliceSampler, indexes outside sampler lengths were produced (#1874) [BugFix] check_env_specs seeding logic (#1872) [BugFix] Fix update in serial / parallel env (#1866) [Doc] Installation instructions in API ref (#1871) [BugFix] better device consistency in EGreedy (#1867) [BugFix] Fix load_state_dict and is_empty td bugfix impact (#1869) [Doc] Fix tutos (#1863)
Full Changelog: https://github.com/pytorch/rl/compare/v0.3.0...v0.3.1
[BugFix] Fix incorrect deprecation warning by @mikemykhaylov in https://github.com/pytorch/rl/pull/1655
In this release, we focused on building a Data Hub for offline RL, providing a universal <lib>2gym conversion tool (#1795) and improving the doc.
TorchRL now offers many offline datasets in robotics and control or gaming, all under a single data format (TED for TorchRL Episode Data Format). All datasets are one step away of being downloaded: dataset = <Name>ExperienceReplay(dataset_id, root="/path/to/storage", download=True) is all you need to get started.
This means that you can now download OpenX #1751 or Roboset #1743 datasets and combine them in a single replay buffer #1768 or swap one another in no time and with no extra code.
We allow many new sampling techniques, like sampling slices of trajectories with or without repetition etc.
As always you can append your favourite transform to these transforms.
#1795 introduces a new universal converter for simulation libraries to gym. As RL practitioner, it's sometimes difficult to accommodate for the many different environment APIs that exist. TorchRL now provides a way of registering any env in gym(nasium). This allows users to build their dataset in torchrl and integrate them in their code base with no effort if they are already using gym as a backend. It also allows to transform DMControl or Brax envs (among others) to gym without the need for an extra library.
Functional calls can now be turned off for PPO and A2C loss modules, allowing users to run RLHF training loops at scale! #1804
## TensorDict-free replay buffers
You can now use TorchRL's replay buffer with ANY tensor-based structure, whether it involves dict, tuples or lists. In principle, storing data contiguously on disk given any gym environment is as simple as
rb = ReplayBuffer(storage=LazyMemmapStorage(capacity))
obs_, reward, terminal, truncated, info = env.step(action)
rb.add((obs, obs_, reward, terminal, truncated, info, action))
# sampling a tuple obs, reward, terminal, truncated, info
obs, obs_, reward, terminal, truncated, info = rb.sample()
This is independent of TensorDict and it supports many components of our replay buffers as well as transforms. Check the doc here.
## Multiprocessed replay buffers
TorchRL's replay buffers can now be shared across processes. Multiprocessed RBs can not only be read from but also extended on different workers. #1724
We introduce a list of scripts to check that our training scripts work ok before each release: #1822
We removed loads of checks in GymLikeEnv if some basic conditions are met, which improves the throughput significantly for simple envs. #1803
## Algorithms
We introduce discrete CQL #1666 , discrete IQL #1793 and Impala #1506.
action_spec instead of input_spec by @FrankTianTT in https://github.com/pytorch/rl/pull/1657TanhDelta docstring by @matteobettini in https://github.com/pytorch/rl/pull/1683RewardSum key check by @matteobettini in https://github.com/pytorch/rl/pull/1718SamplerWithoutReplacement state dictionary by @matteobettini in https://github.com/pytorch/rl/pull/1788TransformedEnv mirror allow_done_after_reset property of base env by @matteobettini in https://github.com/pytorch/rl/pull/1810serial_for_single arg in batched envs by @vmoens in https://github.com/pytorch/rl/pull/1846Full Changelog: https://github.com/pytorch/rl/compare/v0.2.1...v0.3.0
[Deprecation] Deprecate ambiguous device for memmap replay buffer by @vmoens in https://github.com/pytorch/rl/pull/1624
init_random_frames rounding in collectors by @matteobettini in https://github.com/pytorch/rl/pull/1616Full Changelog: https://github.com/pytorch/rl/compare/v0.2.0...v0.2.1
One column per quarter.