NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
PyPI, #2808 most downloaded on PyPI
Train transformer language models with reinforcement learning.
Latest stable
1.18.0
Last release 7 days ago
13 Nov 2026
Ships on a steady schedule
a new release about every 2 weeks
Nearly every release is documented
notes for 60 of the last 60 stable releases
Nothing withdrawn
no release was ever pulled
7 years old
109 releases · first in 2020
Drag across the ruler to compare, or tap a release to open it.
A new long-context guide plus a runnable example that trains a book-length sequence per step on a single 8×H100 node.
A new long-context guide plus a runnable example that trains a book-length sequence per step on a single 8×H100 node.
Measured: Qwen3-8B at 1,048,576 tokens, 380 s/step, 56.2 GB per GPU (bf16, per_device_train_batch_size=1, loss_type="chunked_nll"). Runs end to end and saves. The guide documents the levers in the order you hit them (chunked_nll, gradient-checkpointing offload, YaRN RoPE) and the constraints (full attention only, no packing). Needs transformers >= 5.16 for checkpointing offload.
lm_head projection now runs on tensor coresThe inner loop of the default loss_type="chunked_nll" was doing h.float() @ w.float().t(). Both operands are already bf16, so the upcast bought no information: it moved the GEMM off the tensor cores onto the fp32 SIMT path and materialized an fp32 copy of the whole lm_head weight (2.03 GB for a 248k vocabulary), rebuilt for every chunk and again on every gradient-checkpoint recompute. In an 8×H100 profile of trl sft on Qwen3.6-35B-A3B those two fp32 SIMT GEMMs were 21.6% of all GPU kernel time.
One chunk, 256 tokens × vocab 248,320 × hidden 2048, 1×H100, bf16, fwd+bwd:
| fwd+bwd | peak mem | |
|---|---|---|
| before | 23.37 ms | 5.99 GB |
| after | 3.86 ms (6.0×) | 3.03 GB |
End to end, 16,384 tokens per step, tokens/s/GPU:
| model | mode | before | after | |
|---|---|---|---|---|
| gemma-3-270m (vocab 262k) | full FT, 1×H100 | 24036 | 31609 | 1.32× |
| Qwen3-0.6B (vocab 152k) | full FT, 1×H100 | 21276 | 26641 | 1.25× |
| Qwen3-8B | full FT, 2×H100 FSDP2 | 3554 | 6009 | 1.69× |
| Qwen3-8B | LoRA r16, 2×H100 FSDP2 | 4531 | 7125 | 1.57× |
| Qwen3-30B-A3B (MoE) | LoRA attn, 2×H100 FSDP2 | 4251 | 5101 | 1.20× |
Same fix applied to the distillation trainers (which paid it twice per chunk, student and teacher). Under accelerate mixed precision the numerics are bit-identical (autocast was already casting the operands back down); without autocast over the projection the GEMM moves to bf16 tensor cores, which is exactly what loss_type="nll" already does.
by @qgallouedec in #6863
trl.lossesTRL's only import from liger-kernel was liger_kernel.chunked_loss (the fused linear DPO / KTO / GRPO / JSD losses). That module is pure PyTorch, and most of the GRPO and DPO variants in it were written for TRL. Upstream reviews had stalled, so the code comes home as trl.losses: FusedLinearDPOLoss, FusedLinearKTOLoss, FusedLinearGRPOLoss, FusedLinearJSDLoss (copied from Liger-Kernel v0.8.2, BSD-2 notice kept, bitwise identical to the installed Liger on random inputs).
No new config flag: use_liger_kernel=True keeps selecting the fused loss exactly as before, and still needs liger-kernel installed because transformers' Trainer patches the model kernels with it.
trl.losses by @qgallouedec in #7059DistillationTrainer by @kashif in #7064.agents/skillsAgent skills for contributing to TRL now ship in the repo at .agents/skills.
by @qgallouedec in #6901
quantization_config argumentThe QLoRA test suites move onto the quantization_config trainer argument added in v1.8, and gain coverage on the VLM paths.
quantization_config by @albertvillanova in #6910quantization_config trainer argument by @albertvillanova in #6927peft>=0.13.0 and drop the 0.12 version guards by @qgallouedec in #7102deepspeed>=0.18.6 and drop the 0.16.4 guard by @albertvillanova in #7118target_parameters use for peft<0.17.0 by @albertvillanova in #7119num_tokens under tensor parallelism by @qgallouedec in #7101output_hidden_states when only last_hidden_state is used by @ciaoyizhen in #4755processing_class by @kashif in #6978AsyncDistillationTrainer up with recent AsyncGRPO changes by @kashif in #6823 and #6987dataset_formatting module by @albertvillanova in #7117FLASH_ATTENTION_VARIANTS constant from the DPO trainer by @albertvillanova in #7013PPOTrainer, PPOConfig, and modeling_value_head.py (PreTrainedModelWrapper, AutoModelForCausalLMWithValueHead, AutoModelForSeq2SeqLMWithValueHead) are gone, along with their tests, doc page, and examples.
Bit of a moment for TRL: PPOTrainer predates every PR in this repo. It landed in dfb6a580 on 2020-03-28, the commit that first added the library, back when the package was called lm_ppo. It was the oldest thing in TRL and the last piece of the original codebase.
Why now: unmaintained (no feature work in over a year, every PPO commit since #4482 a drive-by fix), the only trainer never aligned on the input format (it still took tokenized input_ids), near-zero recorded usage, and a magnet for automated bug hunters filing real reports against code nobody runs. Nobody loses anything: from trl import PPOTrainer already stopped working in v1.10, so anyone actually running PPO is pinned to an older TRL and those installs keep resolving exactly as they do today.
create_reference_model stays (BCO, A2PO and Online DPO use it).
by @qgallouedec in #7020
_ChunkedLogProbFunction backward — backward accepted grad_entropy but never used it, so any loss backpropagating through the entropy output silently got zero gradient from that term, with no error or warning. Only AsyncGRPOTrainer read entropy (under no_grad, for logging), so no user-facing regression today, but this is exported non-experimental code and an entropy-bonus loss term would have hit it silently. By @verma8076 in #6625paper_index paths and add GOLD, IW-OPD, and AsyncDistillation telemetry by @YeonwooSung in #7051A docstring and docs consistency pass across the repo, all by @qgallouedec:
DistillationTrainer signature annotations (#7152), docs formatting pass (#7154), tidy pyproject / Makefile / .gitignore (#7157)Docker images — released and dev builds are now separate workflows, the image version comes from the release tag, and the installed TRL version is pinned:
docker-build.yml by @hf-security-analysis[bot] in #6997Tiny-model config alignment — a sweep bringing every tiny test model's config in line with its reference model, all by @albertvillanova:
layer_types so tiny Gemma3 / Olmo3 cover both attention types (#6962) and tiny Cohere2 (#6963)Other CI, mostly by @albertvillanova:
select instead of extend-select to keep the rule set explicit (#6964), stop dependabot from opening ruff pre-commit bumps (#6965), reformat lambda functions for code clarity (#7092)select instead of extend-select to keep the rule set explicit by @albertvillanova in #6964Note truncated.
Your coding agent can read these notes before it upgrades. Set up the MCP server →
⚠️ v1.12.0 is an accidental duplicate of v1.11.0
During the v1.11.0 release, VERSION was bumped to 1.12.0 instead of 1.12.0.dev0. Our publish workflow fires on any push to main that touches VERSION and only skips the upload when the version contains "dev", so trl 1.12.0 was auto-published to PyPI at 19:46 on 2026-08-26, 90 seconds after 1.11.0, with identical code.
trl==1.12.0 on PyPI is bit-identical to trl==1.11.0. There are no features, fixes, or behavior changes in this version.
The version number is burned (PyPI does not allow yanking + re-uploading the same version), so we skip 1.12 entirely for real development. The next feature release is v1.13.0.
Full Changelog: v1.11.0...v1.12.0 (empty)
Ignore benign NumPy __array_wrap__ deprecation and torch inductor TF32 advisory warnings in CI by @albertvillanova in https://github.com/huggingface/t…
trl vllm-serve used to run a custom FastAPI app around vLLM's LLM, with its own data-parallel fan-out and a worker extension for weight sync. vLLM's own server covers all of that. The custom server is gone: trl/scripts/vllm_serve.py drops from 1218 to ~130 lines and now translates its flags into vllm serve, prints the equivalent command, forwards any extra argument, and sets the three settings TRL needs (--weight-transfer-config, --logprobs-mode processed_logprobs, --max-logprobs -1). fastapi, uvicorn and pydantic leave the vllm extra.
Weight sync now uses vLLM's NCCL weight-transfer engine — tensors are announced once and streamed in a single packed broadcast, instead of one HTTP request + broadcast per tensor. Distillation teacher logprobs come from prompt_logprobs.
Benchmark (GRPO server-mode on H100, Qwen2.5-1.5B / Qwen2.5-VL-3B, same seed, same data):
| Setup | before | after | speedup |
|---|---|---|---|
| Text, TP=1 | 1.08, 0.91 s/step | 0.63, 0.62 s/step | 1.59× |
| Text, TP=2 | 0.96, 0.96 s/step | 0.62, 0.60 s/step | 1.57× |
| VLM, TP=1 | 2.14, 2.13 s/step | 1.48, 1.48 s/step | 1.44× |
Bit-identical completions vs the old server (16/16 over 4 prompts × n=4, logprobs identical to 0.0 over 499 tokens). trl vllm-serve still accepts the same flags — it just forwards to vllm serve now.
by @qgallouedec in https://github.com/huggingface/trl/pull/6765
AsyncDistillationTrainer (+ multi-teacher MOPD)Async on-policy distillation, architected like AsyncGRPOTrainer: a background rollout worker generates the student's own completions and scores them against a teacher served over HTTP, so generation and training overlap instead of alternating. The teacher is never loaded locally — just a vLLM server URL.
Also supports MOPD (multi-teacher on-policy distillation): pass more than one entry in teacher_server_urls and route each sample to a teacher via a teacher_id column (e.g. a math teacher and a code teacher, each served independently).
from trl.experimental.async_distillation import AsyncDistillationConfig, AsyncDistillationTrainer
trainer = AsyncDistillationTrainer(
model="Qwen/Qwen3-4B",
args=AsyncDistillationConfig(
teacher_server_urls=["http://teacher-math:8000", "http://teacher-code:8000"],
teacher_top_k=16,
),
train_dataset=dataset, # rows carry a `teacher_id` column for MOPD
)
trainer.train()
by @kashif in https://github.com/huggingface/trl/pull/6705
examples/ was split by format (scripts/ vs notebooks/), which scattered related files (e.g. sft_nemotron_3 existed as both a script and a notebook, in different folders; OpenEnv notebooks lived apart from OpenEnv scripts). Everything is now one folder per story — the name says the method and the task (grpo_wordle, sft_gpt_oss, ppo_tldr), and the folder holds every file the example needs (scripts, notebooks, prompts, chat templates, eval code). Thin single-trainer demo scripts are removed (they live in trl/scripts/ and each trainer doc page has a runnable snippet). Old examples/scripts/… and examples/notebooks/… GitHub / Colab links will 404 — worth flagging in downstream posts.
by @qgallouedec in https://github.com/huggingface/trl/pull/6820
AsyncGRPOConfig by @AmineDiro in https://github.com/huggingface/trl/pull/6774AsyncGRPOTrainer — drop redundant metric guards, merge train-begin callbacks by @qgallouedec in https://github.com/huggingface/trl/pull/6378trl skills installThe TRL skill is now written around the Python API — trainer selection, dataset format, load-bearing config fields, GRPO reward-function contract — with the CLI as a footnote. trl skills install is dropped and the skill source moves out of the package.
trl skills install and move the skill source out of the package by @qgallouedec in https://github.com/huggingface/trl/pull/6793DistillationTrainer supports tool callingContinuing the graduation cycle from v1.10.
by @qgallouedec in https://github.com/huggingface/trl/pull/6723
DistillationTrainer when the GKD config is fully coveredGKDTrainer now emits a hint when the config is fully covered by the stable DistillationTrainer — no behavior change, just a nudge for users still on the older API.
by @qgallouedec in https://github.com/huggingface/trl/pull/6847
position_ids from the SFT collator when sequence parallelism is enabled by @qgallouedec in https://github.com/huggingface/trl/pull/6812chunked_nll loss by @qgallouedec in https://github.com/huggingface/trl/pull/6842dataset_source to telemetry by @qgallouedec in https://github.com/huggingface/trl/pull/6599liger-kernel 0.8.2 and drop the SAPO warning filter by @albertvillanova in https://github.com/huggingface/trl/pull/6768None token logprobs from vLLM in GRPO importance sampling by @behroozazarkhalili in https://github.com/huggingface/trl/pull/6693[GRPO] Match the Liger dapo / cispo / vespo normalizer to the non-Liger path — completes the fix from v1.10 (#6024) on the Liger path. By @kashif in https://github.com/huggingface/trl/pull/5890[SFT] Treat past_key_values and attentions as optional on the model output by @Hakureirm in https://github.com/huggingface/trl/pull/6686AsyncGRPOTrainer crashing on environment-owned datasets by @albertvillanova in https://github.com/huggingface/trl/pull/6824scale_rewards CLI parsing and warn on ignored feedback by @sergiopaniego in https://github.com/huggingface/trl/pull/6341opencode.py proxy trace / log paths after OpenEnv moved them to functions by @sergiopaniego in https://github.com/huggingface/trl/pull/6672DistillationTrainer when GKD config is covered (see above)Big CI-hygiene cycle by @albertvillanova. Highlights:
head_dim when the model config does not declare it by @albertvillanova in https://github.com/huggingface/trl/pull/6855__array_wrap__ deprecation and torch inductor TF32 advisory warnings in CI by @albertvillanova in https://github.com/huggingface/trl/pull/6850 and https://github.com/huggingface/trl/pull/6851lob / postgres detectors by @albertvillanova in https://github.com/huggingface/trl/pull/6920, https://github.com/huggingface/trl/pull/6921 (+ #6926)GITHUB_ACTIONS by @albertvillanova in https://github.com/huggingface/trl/pull/6726, https://github.com/huggingface/trl/pull/6753, https://github.com/huggingface/trl/pull/6749VERSION env from the Docker build step / fix the Slack title of the TRL Docker image build job by @albertvillanova in https://github.com/huggingface/trl/pull/6931, https://github.com/huggingface/trl/pull/6932RUN_SLOW env variable from the slow tests workflow by @albertvillanova in https://github.com/huggingface/trl/pull/6748docstyle-ignore tag from docstrings by @albertvillanova in https://github.com/huggingface/trl/pull/6785dataset_source to telemetry by @qgallouedec in https://github.com/huggingface/trl/pull/6599__array_wrap__ deprecation warning in CI by @albertvillanova in https://github.com/huggingface/trl/pull/6850trl skills install and move the skill source out of the package by @qgallouedec in https://github.com/huggingface/trl/pull/6793Full Changelog: https://github.com/huggingface/trl/compare/v1.10.0...v1.11.0
Also lands a trl distillation CLI and moves the tests to tests/. The full refactor arc: switched signature columns to prompt (deprecated messages-form…
DistillationTrainer is now a stable trainerAfter a ~30-PR refactor that reshaped its data contract, loss path, generation stack, config surface, and tests, DistillationTrainer and DistillationConfig graduate from trl.experimental.distillation to the top-level trl package. Same import surface as SFT / DPO / GRPO / KTO. The old experimental path still works and emits a FutureWarning (removal in v2.0.0).
# Before
from trl.experimental.distillation import DistillationConfig, DistillationTrainer
# Now
from trl import DistillationConfig, DistillationTrainer
Also lands a trl distillation CLI and moves the tests to tests/. The full refactor arc: switched signature columns to prompt (deprecated messages-format), pinned the generation stack to GRPO's, wired the chunked JSD loss and deleted the full-logit path, cleaned the Liger path to share extraction with the chunked path, adopted GRPO's log() / training_step timing / _save_checkpoint, rebuilt the test suite to GRPO shape, and much more.
by @qgallouedec across ~30 PRs (#6479, #6480, #6481, #6482, #6484, #6487, #6497, #6508, #6509, #6510, #6511, #6512, #6513, #6521, #6522, #6523, #6524, #6525, #6526, #6530, #6537, #6604, #6605, #6606, #6607, #6609, #6610, #6611, #6612, #6613, #6614, #6629, #6630, #6632, #6633, #6634, #6639, #6640, #6641, #6642, #6643, #6644, #6645, #6647, #6653).
DistillationTrainer supports Vision Language ModelsAlongside the graduation, VLMs work end-to-end in DistillationTrainer.
by @qgallouedec in https://github.com/huggingface/trl/pull/6650
New experimental loop-owning (black-box) path for AsyncGRPOTrainer — for training external agents like opencode that run their own tool loop, rather than TRL driving each turn as in the environment_factory white-box path.
The agent runs in an OpenEnv session in transparent_proxy mode; an in-sandbox proxy captures each turn's token ids and logprobs. On completion, TRL reads the trace, rebuilds per-turn training rows, and scores the workspace with the session's verify(). Ships with three caller-supplied policy hooks (rollout_reward_fn, train_turn_fn, agent_turn_fn) so you can drop framework aux calls (title generator, context summarizer) and reinforce only the turns you want.
Includes a self-contained examples/scripts/openenv/opencode.py (local subprocess sandbox + DeepCoder held-out stdin/stdout verifier), validated end-to-end on Qwen3-4B.
by @AmineDiro in https://github.com/huggingface/trl/pull/6420, plus HF-sandbox variant by @sergiopaniego in https://github.com/huggingface/trl/pull/6565
top_p / top_k / min_p / repetition_penalty in AsyncGRPOConfig by @AmineDiro in https://github.com/huggingface/trl/pull/6608A new SFT example for google/diffusiongemma-26B-A4B-it implementing the reference recipe: one response block per step, uniform random token corruption with t ~ U(eps, 1), two-pass self-conditioning, flat cross-entropy over the whole canvas plus an autoregressive co-loss on the encoder. LoRA targets attention + dense MLP linears. Requires transformers >= 5.12.0.
Ships with diffusion_gemma.jinja / diffusion_gemma_training.jinja chat templates (with {% generation %} markers) so assistant_only_loss=True works out of the box.
by @kashif in https://github.com/huggingface/trl/pull/6003
Two default flips this cycle — pinning them explicitly is recommended if you want the old behavior.
SFTConfig.max_completion_length / GRPOConfig.max_completion_length: default bumped from 256 to 512. By @dhruvnigam93 in https://github.com/huggingface/trl/pull/6264GRPOConfig.use_bias_correction_kl now defaults to True. By @gowtham-sai-yadav in https://github.com/huggingface/trl/pull/6503VLM configurations no longer reject packing / padding-free when the batch happens to be text-only.
by @cris96spa in https://github.com/huggingface/trl/pull/6547
target_parameters on peft>=0.20.0 by @albertvillanova in https://github.com/huggingface/trl/pull/6591train_dataset types in core trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6493 and https://github.com/huggingface/trl/pull/6494f_divergence_type / loss type combinations in DPO by @qgallouedec in https://github.com/huggingface/trl/pull/6559TQDM_DISABLE in DPO/KTO/BCO reference log-prob loops by @qgallouedec in https://github.com/huggingface/trl/pull/6507use_cpu in create_model_from_path device_map default by @Strongich in https://github.com/huggingface/trl/pull/6295dtype instead of torch_dtype (deprecated) by @qgallouedec in https://github.com/huggingface/trl/pull/6560test_train_moe_peft_model for KTO by @albertvillanova in https://github.com/huggingface/trl/pull/6589steps_per_generation != gradient_accumulation_steps — gradients were silently mis-scaled by gradient_accumulation_steps / steps_per_generation (e.g. 0.5× or 4× the intended token-mean in common configs). Default-equivalent configs are unaffected. By @0xadvait in https://github.com/huggingface/trl/pull/6024vllm_enable_sleep_mode=True by @muupan in https://github.com/huggingface/trl/pull/5313cuda in FSDP2 vLLM weight sync by @verma8076 in https://github.com/huggingface/trl/pull/6592prepare_deepspeed crash with CPU offload optimizer by @roycho96 in https://github.com/huggingface/trl/pull/5916fix(ppo): exclude padding tokens from entropy calculation by @mukund1985 in https://github.com/huggingface/trl/pull/6121fix: avoid CopySlices when scaling policy logits by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6554[GRPO] Apply the completion mask elementwise in the LuSPO loss aggregation by @YaseenBashaT in https://github.com/huggingface/trl/pull/6654[GRPO] Fix entropy bonus normalization inconsistency across loss types by @YaseenBashaT in https://github.com/huggingface/trl/pull/6648huggingface/OpenEnv by @sergiopaniego in https://github.com/huggingface/trl/pull/6529GRPOConfig loss_type help by @gowtham-sai-yadav in https://github.com/huggingface/trl/pull/6477opencode_hf_sandbox.py: loop-owning training in remote HF sandboxes by @sergiopaniego in https://github.com/huggingface/trl/pull/6565test: don't hard-code bf16=True on devices that lack bf16 support by @behroozazarkhalili in https://github.com/huggingface/trl/pull/6036use_cpu in tests when no CUDA device is available by @albertvillanova in https://github.com/huggingface/trl/pull/6504LOAD REPORT tables in test output by @albertvillanova in https://github.com/huggingface/trl/pull/6502jmespath only on transformers below 5.13.0 by @albertvillanova in https://github.com/huggingface/trl/pull/6597train_dataset=None raises for core trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6492loss_type in KTO precompute ref log-probs test by @albertvillanova in https://github.com/huggingface/trl/pull/6501test_peft_with_quantization tests for bitsandbytes 0.50.0 by @albertvillanova in https://github.com/huggingface/trl/pull/6546torch.get_autocast_gpu_dtype() warning from mamba-ssm kernel by @albertvillanova in https://github.com/huggingface/trl/pull/6556add_response_schema when the model already ships a response template by @qgallouedec in https://github.com/huggingface/trl/pull/6574openenv_harness.py by @albertvillanova in https://github.com/huggingface/trl/pull/6531liger-kernel < 0.8.1 by @albertvillanova in https://github.com/huggingface/trl/pull/6517bitsandbytes < 0.50.0 by @albertvillanova in https://github.com/huggingface/trl/pull/6544ci: exclude the buggy TruffleHog LOB detector by @behroozazarkhalili in https://github.com/huggingface/trl/pull/6674["prompt", "image", "images"] by @qgallouedec in https://github.com/huggingface/trl/pull/6481completion_mask alongside labels by @qgallouedec in https://github.com/huggingface/trl/pull/6482completion_mask by @qgallouedec in https://github.com/huggingface/trl/pull/6484max_completion_length from 256 to 512 by @dhruvnigam93 in https://github.com/huggingface/trl/pull/6264_compute_prompt_length by @qgallouedec in https://github.com/huggingface/trl/pull/6509_tokenize_prompts (unwired) by @qgallouedec in https://github.com/huggingface/trl/pull/6510_generate_single_turn (unwired) by @qgallouedec in https://github.com/huggingface/trl/pull/6513_generate + _generate_and_score_completions (unwired) by @qgallouedec in https://github.com/huggingface/trl/pull/6521TQDM_DISABLE in DPO/KTO/BCO reference log-prob loops by @qgallouedec in https://github.com/huggingface/trl/pull/6507vllm_enable_sleep_mode=True by @muupan in https://github.com/huggingface/trl/pull/5313dtype instead of torch_dtype (deprecated) by @qgallouedec in https://github.com/huggingface/trl/pull/6560add_response_schema when the model already ships a response template by @qgallouedec in https://github.com/huggingface/trl/pull/6574_get_last_hidden_state (unwired) by @qgallouedec in https://github.com/huggingface/trl/pull/6525use_bias_correction_kl=True in GRPOConfig by @gowtham-sai-yadav in https://github.com/huggingface/trl/pull/6503device deprecation warning in CI by @albertvillanova in https://github.com/huggingface/trl/pull/6585test_train_moe_peft_model for KTO by @albertvillanova in https://github.com/huggingface/trl/pull/6589f_divergence_type / loss type combinations in DPO by @qgallouedec in https://github.com/huggingface/trl/pull/6559target_parameters on peft>=0.20.0 by @albertvillanova in https://github.com/huggingface/trl/pull/6591disable_dropout to False by @qgallouedec in https://github.com/huggingface/trl/pull/6604model by @qgallouedec in https://github.com/huggingface/trl/pull/6609advantages optional in print_prompt_completions_sample by @qgallouedec in https://github.com/huggingface/trl/pull/6611print_prompt_completions_sample by @qgallouedec in https://github.com/huggingface/trl/pull/6612log() + bounded _logs deques by @qgallouedec in https://github.com/huggingface/trl/pull/6613training_step timing + _save_checkpoint by @qgallouedec in https://github.com/huggingface/trl/pull/6614use_liger_kernel with logit_scale / final_logit_softcapping by @qgallouedec in https://github.com/huggingface/trl/pull/6629trl distillation CLI by @qgallouedec in https://github.com/huggingface/trl/pull/6642tests/ by @qgallouedec in https://github.com/huggingface/trl/pull/6647Full Changelog: https://github.com/huggingface/trl/compare/v1.9.0...v1.10.0
Revert xfail for NemotronH GRPO/RLOO tests now that transformers#47569 fixed the kernels bug by @albertvillanova in https://github.com/huggingface/trl
Full Changelog: https://github.com/huggingface/trl/compare/v1.9.1...v1.9.2
Fix vLLM server-mode communicator initialization to use the current A… by @sywangyi in https://github.com/huggingface/trl/pull/6417
Full Changelog: https://github.com/huggingface/trl/compare/v1.9.0...v1.9.1
Deprecate & remove `lmbda` on/off-policy mixing and the off-policy training branch (#6458, #6460); GKD paper reproduction now points at GKDTrainer
Long-standing request finally landed. GRPO and RLOO rely on RepeatSampler to repeat each prompt num_generations times and group them across processes — but samplers can't attach to iterable datasets (no length, no indexing), and the old behavior silently bypassed the sampler, quietly corrupting advantage computation.
A new repeat_iterable_dataset generator reproduces RepeatSampler's exact ordering by transforming the stream itself instead of reordering indices. Trainers wrap iterable datasets via IterableDataset.from_generator, force dispatch_batches=False, and shard the stream per-process to preserve prompt groups after cross-process gathering.
from datasets import load_dataset
from trl import GRPOConfig, GRPOTrainer
dataset = load_dataset("trl-lib/DeepMath-103K", split="train", streaming=True)
trainer = GRPOTrainer(
model="Qwen/Qwen3-4B",
args=GRPOConfig(max_steps=1000), # required for iterable datasets
train_dataset=dataset,
reward_funcs=accuracy_reward,
)
max_steps is required for iterable datasets (no length). A clear error also fires if dispatch_batches=True is combined with an iterable dataset.
by @albertvillanova in https://github.com/huggingface/trl/pull/6351
When an environment_factory is provided, the environment can now own the data. train_dataset becomes optional: reset() returns the prompt. No more fabricating a dummy dataset just to drive the loop.
class WordleEnv:
def reset(self, **kwargs) -> str: # returns the prompt
self._target = sample(words)
return f"Guess a {len(self._target)}-letter word."
def get_reward(self) -> float:
return 1.0 if self._solved else 0.0
def guess(self, word: str) -> str: # exposed tool
self._solved = word == self._target
return _feedback(word, self._target)
trainer = GRPOTrainer(
model="Qwen/Qwen3-4B",
args=GRPOConfig(max_steps=1000),
environment_factory=WordleEnv, # no train_dataset needed
)
Applies to both GRPOTrainer and AsyncGRPOTrainer. A provided train_dataset still works exactly as before; multi-environment routing datasets can now be environment-only (no prompt column required).
by @qgallouedec in https://github.com/huggingface/trl/pull/6349
An opt-in way to build training rows from a multi-turn conversation. Message mode keeps the conversation as messages and re-tokenizes the whole thing each turn, then checks whether the fresh tokens still start with the tokens held so far. If yes → append the new part (same as token mode). If no → a rewrite happened → close the row and open a new one that matches what the model actually read.
AsyncGRPOConfig(
rollout_protocol="message", # "token" (default) | "message"
fork_threshold_tokens=1024, # message-mode only
)
Per-turn drift is classified as CLEAN (append), REALIGN (last-answer tail wobble → overwrite same row), or FORK (real rewrite → new row). One advantage per conversation, stamped on every row it produced; under token-mean loss a fork is invisible. Sets up the plumbing for future tree-trajectory support — scoring is already branch-agnostic (groups by rollout_id).
by @AmineDiro in https://github.com/huggingface/trl/pull/6250
VLLMClient and decoupled weight syncA follow-up to v1.8's rollout worker split. A small VLLMClient becomes the single place that speaks HTTP to the vLLM server (named methods for wait_for_server_ready, get_max_model_len, pause/resume, weight-update endpoints — stateless, so picklable). Generation deliberately stays on its own async aiohttp session in the rollout child.
Weight sync is now built from config independent of rollout_worker (previously a custom rollout worker force-set it to None), and is injectable via a new WeightTransferProtocol. token_budget now waits for /health before defaulting to get_max_model_len(), so a slow-starting vLLM waits instead of failing the run.
by @AmineDiro in https://github.com/huggingface/trl/pull/6269
DistillationTrainer refactorA ~13-PR sweep that reshapes the experimental DistillationTrainer for future stability:
ServerDistillationTrainer for the teacher-server path (#6454)lmbda on/off-policy mixing and the off-policy training branch (#6458, #6460); GKD paper reproduction now points at GKDTrainer (#6459)prompt column alongside messages (#6461); deprecate messages-format datasets (#6474)num_items_in_batch to count generated completion tokens — the loss denominator was wrong for on-policy training (counted dataset completions, not generated ones) and NaN'd on prompt-only datasets. This changes loss values for anyone on the old path (#6478)All by @qgallouedec.
Importance-Weighted On-Policy Distillation lands as an optional objective on the experimental DistillationTrainer — detached IW-OPD advantages built from sampled-token teacher and rollout logprobs (with cached vLLM rollout logprobs when use_vllm=True).
DistillationConfig(
distillation_objective="iw_opd",
iw_opd_gamma=1.0,
iw_opd_epsilon=0.2,
lmbda=1.0, reverse_kl_top_1_mode="sampled", # required
)
Now split into its own experimental trainer as part of the refactor above.
by @kashif in https://github.com/huggingface/trl/pull/6191
log_multimodal for GRPO / RLOONew config toggle to control whether images from multimodal completions are logged to trackers. Also applied to experimental DPPO and GRPO-with-replay-buffer.
by @apardyl in https://github.com/huggingface/trl/pull/5408 and by @albertvillanova in https://github.com/huggingface/trl/pull/6352
generate_rollout_completions by @sergiopaniego in https://github.com/huggingface/trl/pull/5870top_p in GKD on-policy generation by @sergiopaniego in https://github.com/huggingface/trl/pull/6337DatasetDict and IterableDatasetDict as eval_dataset in evaluate by @albertvillanova in https://github.com/huggingface/trl/pull/6326AsyncGRPOTrainer args handling with stable trainers by @qgallouedec in https://github.com/huggingface/trl/pull/6377auto_find_batch_size in GRPO and RLOO by @qgallouedec in https://github.com/huggingface/trl/pull/6411GMPOTrainer to the telemetry allowlist by @qgallouedec in https://github.com/huggingface/trl/pull/6410[GRPO] Treat explicit max_tool_calling_iterations=0 as zero, not unlimited by @Strongich in https://github.com/huggingface/trl/pull/6423hf CLI by @davanstrien in https://github.com/huggingface/trl/pull/6422messages-format datasets in DistillationTrainer — use prompt column instead — by @qgallouedec in https://github.com/huggingface/trl/pull/6474lmbda on/off-policy mixing in DistillationTrainer — off-policy path fully removed by @qgallouedec in https://github.com/huggingface/trl/pull/6458 and https://github.com/huggingface/trl/pull/6460clip_ratio/low_min / high_max to be per-completion instead of per-rank-batch-mean (DP-layout-independent). Logging-only, no loss change. By @qgallouedec in https://github.com/huggingface/trl/pull/6380use_liger_kernel under DeepSpeed ZeRO-3 by @qgallouedec in https://github.com/huggingface/trl/pull/6372use_reentrant=True for PEFT + ZeRO-3 + gradient checkpointing across all trainers by @qgallouedec in https://github.com/huggingface/trl/pull/6356precompute_ref_log_probs=True with DeepSpeed in DPO and KTO by @qgallouedec in https://github.com/huggingface/trl/pull/6403cast_lm_head_to_fp32 under FSDP2 / DeepSpeed ZeRO-3 by @qgallouedec in https://github.com/huggingface/trl/pull/6330</think>-only check. By @qgallouedec in https://github.com/huggingface/trl/pull/6363pad_token_id == eos_token_id in GKD and GOLD by @roycho96 in https://github.com/huggingface/trl/pull/6357apply_chat_template dropping leading tokens from chosen when the prompt common prefix shrinks against rejected by @sohumt123 in https://github.com/huggingface/trl/pull/6433functools.partial or callable instance by @qgallouedec in https://github.com/huggingface/trl/pull/6313prepare_peft_model by @sergiopaniego in https://github.com/huggingface/trl/pull/6343FileNotFoundError in DPO/KTO precompute_ref_log_probs by @Functionhx in https://github.com/huggingface/trl/pull/6348GeometricMixtureWrapper.generate() crash after transformers' MoE decode optimization change by @albertvillanova in https://github.com/huggingface/trl/pull/6413vocab_size check for multimodal model configs by @sergiopaniego in https://github.com/huggingface/trl/pull/6340IterableDataset in GRPO and RLOO by @albertvillanova in https://github.com/huggingface/trl/pull/6324eval_dataset and precompute_ref_log_probs by @albertvillanova in https://github.com/huggingface/trl/pull/6370eval_dataset dict passed to evaluate() by @albertvillanova in https://github.com/huggingface/trl/pull/6434hasattr guards on lm_head.bias in the Liger path by @qgallouedec in https://github.com/huggingface/trl/pull/6425logits/chosen and logits/rejected metrics by @qgallouedec in https://github.com/huggingface/trl/pull/6424eos_token_id in non-vLLM on-policy GOLD test stubs by @kashif in https://github.com/huggingface/trl/pull/6394BEMACallback bias_power docstring by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6390max_steps requirement for iterable datasets in GRPO and RLOO by @albertvillanova in https://github.com/huggingface/trl/pull/6432AsyncGRPOConfig.max_completion_length description with GRPOConfig by @qgallouedec in https://github.com/huggingface/trl/pull/6374AsyncGRPOTrainer.processing_class docstring with GRPOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/6375AsyncGRPOTrainer clip-ratio comment with GRPOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/6376loss_type by @qgallouedec in https://github.com/huggingface/trl/pull/6373BEMACallback bias_power=0.0 test by @albertvillanova in https://github.com/huggingface/trl/pull/6464eval_dataset input types in core trainer init / evaluate() tests by @albertvillanova in https://github.com/huggingface/trl/pull/6336 and https://github.com/huggingface/trl/pull/6371eval_dataset init tests for experimental GRPO / Online-DPO / SFT & distillation / preference / BCO+CPO+ORPO+PRM trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6395, https://github.com/huggingface/trl/pull/6396, https://github.com/huggingface/trl/pull/6397, https://github.com/huggingface/trl/pull/6398 and https://github.com/huggingface/trl/pull/6399environment_factory warning in GRPO tests by @albertvillanova in https://github.com/huggingface/trl/pull/6445ensure_weight_tying warning on newer PEFT in GRPO tests / DPO+KTO liger tests by @albertvillanova in https://github.com/huggingface/trl/pull/6446 and https://github.com/huggingface/trl/pull/6463bitsandbytes _check_is_size FutureWarning in CI by @albertvillanova in https://github.com/huggingface/trl/pull/6448IndexPutBackward0 warning from Liger SAPO loss by @albertvillanova in https://github.com/huggingface/trl/pull/6436torch.jit.script_method warning on Python 3.14 by @albertvillanova in https://github.com/huggingface/trl/pull/6465AnnAssign DeprecationWarning on Python 3.14 by @albertvillanova in https://github.com/huggingface/trl/pull/6467create_reference_model deprecation warning by @albertvillanova in https://github.com/huggingface/trl/pull/6418dpo invariant reference after #6321 by @qgallouedec in https://github.com/huggingface/trl/pull/6421torch.tensor re-wrapping of batch labels in KTO by @albertvillanova in https://github.com/huggingface/trl/pull/6431log_multimodal param to GRPOConfig and RLOOConfig to control image logging by @apardyl in https://github.com/huggingface/trl/pull/5408generate_rollout_completions by @sergiopaniego in https://github.com/huggingface/trl/pull/5870functools.partial or callable instance by @qgallouedec in https://github.com/huggingface/trl/pull/6313cast_lm_head_to_fp32 under FSDP2 / DeepSpeed ZeRO-3 by @qgallouedec in https://github.com/huggingface/trl/pull/6330use_liger_kernel under DeepSpeed ZeRO-3 by @qgallouedec in https://github.com/huggingface/trl/pull/6372use_reentrant=True for PEFT + ZeRO-3 + gradient checkpointing in all trainers by @qgallouedec in https://github.com/huggingface/trl/pull/6356GMPOTrainer to the telemetry allowlist by @qgallouedec in https://github.com/huggingface/trl/pull/6410dpo invariant reference after #6321 by @qgallouedec in https://github.com/huggingface/trl/pull/6421hasattr guards on lm_head.bias in the Liger path by @qgallouedec in https://github.com/huggingface/trl/pull/6425logits/chosen and logits/rejected metrics by @qgallouedec in https://github.com/huggingface/trl/pull/6424AsyncGRPOTrainer clip-ratio comment with GRPOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/6376AsyncGRPOConfig.max_completion_length description with GRPOConfig by @qgallouedec in https://github.com/huggingface/trl/pull/6374AsyncGRPOTrainer.processing_class docstring with GRPOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/6375train_dataset optional by @qgallouedec in https://github.com/huggingface/trl/pull/6349AsyncGRPOTrainer args handling with stable trainers by @qgallouedec in https://github.com/huggingface/trl/pull/6377auto_find_batch_size in GRPO and RLOO by @qgallouedec in https://github.com/huggingface/trl/pull/6411precompute_ref_log_probs=True with DeepSpeed in DPO and KTO by @qgallouedec in https://github.com/huggingface/trl/pull/6403apply_chat_template dropping leading tokens from chosen when the prompt common prefix shrinks against rejected by @sohumt123 in https://github.com/huggingface/trl/pull/6433lmbda on/off-policy mixing by @qgallouedec in https://github.com/huggingface/trl/pull/6458GKDTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/6459lmbda and the off-policy training branch by @qgallouedec in https://github.com/huggingface/trl/pull/6460prompt column alongside messages by @qgallouedec in https://github.com/huggingface/trl/pull/6461messages-format datasets by @qgallouedec in https://github.com/huggingface/trl/pull/6474num_items_in_batch to count generated completion tokens ⚠️ changes loss values by @qgallouedec in https://github.com/huggingface/trl/pull/6478Full Changelog: https://github.com/huggingface/trl/compare/v1.8.0...v1.9.0
Fix deprecation in CI Sync TRL skills by @albertvillanova in https://github.com/huggingface/trl/pull/6200
After many cycles of KTOTrainer ↔ DPOTrainer alignment work, KTO graduates from trl.experimental.kto to the top-level trl package. Same API as DPO/GRPO/SFT — imports move from experimental, tests move to the main test tree, docs no longer flag it as experimental. The experimental path still works and emits a FutureWarning (removal in v2.0.0).
# Before
from trl.experimental.kto import KTOConfig, KTOTrainer
# Now
from trl import KTOConfig, KTOTrainer
Per our telemetry, KTO is the 4th most used trainer in TRL — this graduation was overdue.
by @albertvillanova in https://github.com/huggingface/trl/pull/6175, https://github.com/huggingface/trl/pull/6287 and https://github.com/huggingface/trl/pull/6345
Three interrelated changes make agentic RL training with environments substantially more ergonomic.
Environment-owned reward. If your environment_factory env defines a reserved get_reward() method (no args → float), it's called once per completed rollout and treated as a reward source. reward_funcs becomes optional — no more leaking env state back out to trainer-owned reward funcs.
class WordleEnv:
def reset(self, **kwargs):
self._target = sample(words); self._solved = False
def get_reward(self) -> float: # optional, reserved (not a tool)
return 1.0 if self._solved else 0.0
def guess(self, word: str) -> str: # exposed as a tool
self._solved = word == self._target; ...
trainer = GRPOTrainer(
model=model,
train_dataset=dataset,
environment_factory=WordleEnv, # no reward_funcs needed
)
Multi-environment support. environment_factory now accepts dict[str, factory] in addition to a single callable. Each dataset row selects its environment via an environment column, and only that env's tools are exposed in that row's prompt — so a coding task and a game can train together in one run without leaking each other's tool schemas. Single-callable usage is unchanged.
Same wiring lands in GRPO, AsyncGRPO, DPPO, and GRPO-with-replay-buffer.
Env-owned reward by @qgallouedec in https://github.com/huggingface/trl/pull/6238; multi-env in https://github.com/huggingface/trl/pull/6001 and https://github.com/huggingface/trl/pull/6002
GRPOConfig now supports both static and adaptive entropy regularization (Skywork-OR1). The bonus encourages exploration and helps prevent premature policy collapse.
GRPOConfig(
entropy_coef=0.01, # static
# or:
use_adaptive_entropy=True, # adjust to target entropy
entropy_target=1.5,
entropy_coef_delta=0.01,
entropy_coef_min=0.0,
entropy_coef_max=0.1,
)
Adaptive mode adjusts the coefficient once per optimizer step from window-aggregated entropy (gradient accumulation-aware) and persists entropy_coef in the checkpoint for resume. Not compatible with the Liger kernel.
by @albertvillanova in https://github.com/huggingface/trl/pull/6140
quantization_config trainer argument (streamlined QLoRA)QLoRA no longer requires reaching into model_init_kwargs or pre-loading the model manually.
SFTTrainer(
model="meta-llama/Llama-2-7b-hf",
quantization_config=BitsAndBytesConfig(load_in_4bit=True),
peft_config=LoraConfig(),
train_dataset=dataset,
)
Added to SFTTrainer, DPOTrainer, GRPOTrainer, RLOOTrainer, RewardTrainer, and KTOTrainer. Sits next to peft_config (the other non-serializable QLoRA ingredient), flows into from_pretrained, and raises if also set in args.model_init_kwargs. Also drops the redundant get_kbit_device_map() line — QLoRA trains identically without it across all tested configurations.
by @qgallouedec in https://github.com/huggingface/trl/pull/6157 and https://github.com/huggingface/trl/pull/6276
v1.7 added the router load-balancing auxiliary loss to GRPOTrainer / RLOOTrainer / AsyncGRPOTrainer. It's now on DPOTrainer and KTOTrainer too, so post-training MoE models with preference data keeps experts balanced.
by @qgallouedec in https://github.com/huggingface/trl/pull/6208 and https://github.com/huggingface/trl/pull/6275
chunked_nll via static-shape token packingThe chunked NLL path had data-dependent indexing that broke XLA/Neuron compilation. This PR reworks it to pack valid tokens to the front via a stable argsort on the validity mask, iterate over ceil(n_valid / chunk_size) * chunk_size whole chunks (a tensor, no Python-int sync), and use ignore_index=-100 inside each chunk. GPU behavior is unchanged; Neuron now works too — same tiny memory footprint (~1.33 GiB vs the naive 5.94 GiB alternative) with no per-mask recompilation.
by @michaelbenayoun in https://github.com/huggingface/trl/pull/6314
AsyncGRPO micro-batching becomes packing-aware and token-bounded on top of the padding-free path from v1.7:
token_budget tokens with a variable sample count, decoupling peak memory from per_device_train_batch_size. Useful in memory-bound regimes (no gradient checkpointing, long context, very large models).Both ride HF Trainer's existing gradient accumulation — no training-loop surgery, FSDP/EP collectives stay in lockstep.
by @AmineDiro in https://github.com/huggingface/trl/pull/6092
GOLDTrainerGOLDTrainer now supports vision-language models end-to-end.
by @Strongich in https://github.com/huggingface/trl/pull/5969
by @qgallouedec in https://github.com/huggingface/trl/pull/6259
by @albertvillanova in https://github.com/huggingface/trl/pull/6159 and https://github.com/huggingface/trl/pull/6277
fraction in dataset mixturesYou can now weight a DatasetMixtureConfig by fraction instead of only by explicit counts.
by @qgallouedec in https://github.com/huggingface/trl/pull/6199
Continuing the label-refactor from v1.7 (#6037), truncation moves out of the collator and into dataset preparation. SFTTrainer and DPOTrainer now truncate consistently at the same phase.
by @qgallouedec in https://github.com/huggingface/trl/pull/6155
A big refactor pass merging DPO / SFT / Reward / KTO tokenization into one shared implementation:
is_vlm parameter — https://github.com/huggingface/trl/pull/6298_tokenize a module-level function — https://github.com/huggingface/trl/pull/6301_tokenize into a single shared function — https://github.com/huggingface/trl/pull/6302apply_chat_template kwargs — https://github.com/huggingface/trl/pull/6305chat_template as apply_chat_template_kwargs — https://github.com/huggingface/trl/pull/6315All by @albertvillanova. Related: align collators across DPO / SFT / Reward / KTO by @qgallouedec in https://github.com/huggingface/trl/pull/6178.
DatasetDict and IterableDatasetDict as eval_dataset in trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6322processing_class in CPO/ORPO trainers when omitted by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6087trust_remote_code to GRPOConfig by @muupan in https://github.com/huggingface/trl/pull/4186[async GRPO] Per-generation reset() observation by @qgallouedec in https://github.com/huggingface/trl/pull/6072token_budget to vLLM max_model_len by @AmineDiro in https://github.com/huggingface/trl/pull/6218entropy and num_tokens metrics in KTO by @qgallouedec in https://github.com/huggingface/trl/pull/6257 and https://github.com/huggingface/trl/pull/6256TrainerCallback import to the public top-level path by @qgallouedec in https://github.com/huggingface/trl/pull/6249get_kbit_device_map() by @qgallouedec in https://github.com/huggingface/trl/pull/6158get_dataset_column_names from core and experimental trainers by @albertvillanova in https://github.com/huggingface/trl/pull/6272 and https://github.com/huggingface/trl/pull/6273GFPOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/6309 — no known usage, and its behavior can be reproduced by adjusting reward_funcs.PAPOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/6235sft_video_llm.py script by @qgallouedec in https://github.com/huggingface/trl/pull/6193chunked_nll patch hiding VLM kwargs from generate by @Strongich in https://github.com/huggingface/trl/pull/6156OnlineDPOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/6228precompute_ref_log_probs guard for iterable datasets nested in a dict by @albertvillanova in https://github.com/huggingface/trl/pull/6307quantization_config + already-instantiated model in DPOTrainer / KTOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/6312KTOTrainer._prepare_dataset type hint by @qgallouedec in https://github.com/huggingface/trl/pull/6207ensure_weight_tying warning in liger + PEFT GRPO tests by @albertvillanova in https://github.com/huggingface/trl/pull/6188add_response_schema tests for the new parse_response prefix requirement by @qgallouedec in https://github.com/huggingface/trl/pull/6236num_key_value_heads = num_attention_heads by @albertvillanova in https://github.com/huggingface/trl/pull/6268ValueError instead of NotImplementedError for Liger DPO with PEFT by @albertvillanova in https://github.com/huggingface/trl/pull/6225max_steps is required for iterable train datasets by @albertvillanova in https://github.com/huggingface/trl/pull/6333epsilon help/docstring wording by @qgallouedec in https://github.com/huggingface/trl/pull/6014 (v1.7 window)max_length docstring (truncation direction) by @qgallouedec in https://github.com/huggingface/trl/pull/6254compute_metrics type hint (EvalLoopOutput → EvalPrediction) by @qgallouedec in https://github.com/huggingface/trl/pull/6253huggingface/doc-builder by @albertvillanova in https://github.com/huggingface/trl/pull/6304doc-builder pin by @mishig25 in https://github.com/huggingface/trl/pull/6317test_chatml_collator_truncates_keeping_completion_end by @albertvillanova in https://github.com/huggingface/trl/pull/6318compute_metrics and Liger by @albertvillanova in https://github.com/huggingface/trl/pull/6231 and https://github.com/huggingface/trl/pull/6232TestKTOTrainerSlow with test_train_vlm_gemma_3n by @albertvillanova in https://github.com/huggingface/trl/pull/6162DataCollatorForChatML raises when completion fills window by @albertvillanova in https://github.com/huggingface/trl/pull/6319_get_kl_completion_ids into _get_kl_dataset by @qgallouedec in https://github.com/huggingface/trl/pull/6181processing_class in CPO/ORPO trainers when omitted by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6087add_response_schema tests for the new parse_response prefix requirement by @qgallouedec in https://github.com/huggingface/trl/pull/6236policy_ prefix from KTO loss-internal logps/logits by @qgallouedec in https://github.com/huggingface/trl/pull/6204get_kbit_device_map() by @qgallouedec in https://github.com/huggingface/trl/pull/6158OnlineDPOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/6228is_liger_kernel_available in DPOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/6247DPOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/6208quantization_config trainer argument (streamline QLoRA) by @qgallouedec in https://github.com/huggingface/trl/pull/6157entropy metric by @qgallouedec in https://github.com/huggingface/trl/pull/6257_get_train_sampler by @qgallouedec in https://github.com/huggingface/trl/pull/6255max_length docstring (truncation direction) by @qgallouedec in https://github.com/huggingface/trl/pull/6254compute_metrics type hint (EvalLoopOutput → EvalPrediction) by @qgallouedec in https://github.com/huggingface/trl/pull/6253F import with the rest of the repo by @qgallouedec in https://github.com/huggingface/trl/pull/6246model docstring by @qgallouedec in https://github.com/huggingface/trl/pull/6278num_tokens metric by @qgallouedec in https://github.com/huggingface/trl/pull/6256TrainerCallback import to the public top-level path by @qgallouedec in https://github.com/huggingface/trl/pull/6249fraction to dataset mixtures by @qgallouedec in https://github.com/huggingface/trl/pull/6199quantization_config trainer argument by @qgallouedec in https://github.com/huggingface/trl/pull/6276quantization_config + already-instantiated model in DPOTrainer/ KTOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/6312chunked_nll via static-shape token packing by @michaelbenayoun in https://github.com/huggingface/trl/pull/6314Full Changelog: https://github.com/huggingface/trl/compare/v1.7.0...v1.8.0
Fix GRPO + vLLM colocate + PEFT hang on non-NVLink hardware by @albertvillanova in https://github.com/huggingface/trl/pull/6139
add_response_schema tests for the new parse_response prefix requirement by @qgallouedec in https://github.com/huggingface/trl/pull/6236Full Changelog: https://github.com/huggingface/trl/compare/v1.7.0...v1.7.1
use_transformers_paged was deprecated in v1.4; it's now replaced with proper transformers continuous batching. The old branch silently bypassed import…
loss_type is now "chunked_nll"The flip announced in v1.6 has landed. Setting loss_type is optional, and the default now resolves to "chunked_nll" — giving every SFTTrainer run ~30% less peak VRAM on average (up to ~50% on large-vocab models) with wall-clock time neutral or slightly faster. No action needed.
The auto-resolve falls back to "nll" when use_liger_kernel=True (the two paths are incompatible). If you want the old behavior — e.g. for custom heads — pin it explicitly:
SFTConfig(loss_type="nll")
by @qgallouedec in https://github.com/huggingface/trl/pull/5846
Post-training MoE models now correctly include the router load-balancing auxiliary loss, matching the model's own reference forward and SFTTrainer. Enable via model_init_kwargs:
GRPOConfig(
...,
model_init_kwargs={"output_router_logits": True, "router_aux_loss_coef": 0.001},
)
Plumbed through _get_per_token_logps_and_entropies (now returns a 3-tuple including aux_loss), folded into the policy loss with grad-accum scaling matched per trainer, and logged as aux_loss. AsyncGRPO recomputes it via load_balancing_loss_func in the chunked LM-head path (same as SFT's chunked path).
by @AmineDiro in https://github.com/huggingface/trl/pull/6083, plus router_aux_loss_coef config wiring by @qgallouedec in https://github.com/huggingface/trl/pull/6085
Geometric-Mean Policy Optimization lands as an experimental trainer. Replaces GRPO's per-token arithmetic mean of importance ratios with a sequence-level geometric mean (mean of clipped log-ratios, then exp); clipping is one-sided by advantage sign and applied in log space. Default epsilon=0.4 per the paper.
from trl.experimental.gmpo import GMPOConfig, GMPOTrainer
trainer = GMPOTrainer(
model="Qwen/Qwen3-4B",
args=GMPOConfig(epsilon=0.4),
reward_funcs=accuracy_reward,
train_dataset=dataset,
)
by @raghulchandramouli in https://github.com/huggingface/trl/pull/6078
use_transformers_paged was deprecated in v1.4; it's now replaced with proper transformers continuous batching. The old branch silently bypassed importance-sampling correction (logprobs = None); the new path captures logprobs from output.logprobs and exposes a ContinuousBatchingConfig for KV-cache tuning.
GRPOConfig(
...,
use_transformers_continuous_batching=True,
transformers_continuous_batching_config={
"use_cuda_graph": False,
"max_memory_percent": 0.4, # leave headroom for training
},
)
Benchmark (Llama-3.2-1B-Instruct, A100 80GB, GSM8K): 1.25× faster at N=64 generations with -16 GB peak VRAM vs default generate(). Use when N ≥ 32 with variable completion lengths.
use_transformers_paged=True still works and forwards to the new flag with a FutureWarning. Requires transformers>=5.8.0.
by @sergiopaniego in https://github.com/huggingface/trl/pull/5765
WeightTransferClient now drives vLLM's native 4-phase RL weight-transfer API instead of the older 2-call flow: pause(mode="keep") → start_weight_update → threaded update_weights + NCCL broadcast → finish_weight_update → resume. Validated end-to-end on H100 across single-node, FSDP2×4 + TP=4, and 2-node FSDP2×4 + DP=2×TP=4 (weight-sync time ≈ 0.18-0.8 s).
by @AmineDiro in https://github.com/huggingface/trl/pull/5892
AsyncGRPO now supports the same padding-free path SFT already had. Flattens the batch and uses position_ids-based document boundaries instead of right-padding to the longest sequence — meaningful speedup and memory savings on heterogeneous-length workloads.
by @qgallouedec in https://github.com/huggingface/trl/pull/5854
A new trl.experimental.harbor adapter plugs Harbor agentic task suites into GRPOTrainer via environment_factory. Same pattern as the OpenReward integration — one spec wires all three trainer slots:
from trl import GRPOConfig, GRPOTrainer
from trl.experimental.harbor import HarborSpec
spec = HarborSpec("AdithyaSK/data_agent_rl_environment_train", agent="bash", num_tasks=64)
trainer = GRPOTrainer(
model="Qwen/Qwen3-4B",
args=GRPOConfig(num_generations=8, max_steps=50, max_tool_calling_iterations=25),
train_dataset=spec.train_dataset,
environment_factory=spec.environment_factory,
reward_funcs=spec.reward_funcs,
)
Built-in bash harness, plus jupyter and terminal_notes example harnesses. Gated by the new trl[harbor] extra.
by @adithya-s-k in https://github.com/huggingface/trl/pull/6018
trust_remote_code in trainer configsA single trust_remote_code: bool = False field on the trainer configs now covers the whole load surface — model, processor / tokenizer, reference model, reward model, reward tokenizer, teacher — instead of forcing users to thread it through several independent kwarg dicts.
SFTConfig(trust_remote_code=True)
ModelConfig.trust_remote_code is removed to avoid duplicate --trust_remote_code when combining dataclasses; CLI behavior is unchanged.
by @qgallouedec in https://github.com/huggingface/trl/pull/5802
The last alignment cycle before graduation: KTO now has parity with DPO on pad_to_multiple_of, sync_ref_model, evaluate(), method order/signature, metric placement (all moved into _compute_loss), and a real text + VLM (including multi-image) test suite.
PRs all by @albertvillanova: #6029, #6030, #6033, #6034, #6035, #6080, #6093, #6148, #6149, #6150, #6152, #6160, #6163.
GRPO and RLOO now support LFM2-VL multimodal inputs end-to-end.
by @zwischenraum in https://github.com/huggingface/trl/pull/6114
get_repetition_penalty_reward by @qgallouedec in https://github.com/huggingface/trl/pull/6058get_cosine_scaled_reward by @qgallouedec in https://github.com/huggingface/trl/pull/6066Label construction moves out of the collator and into dataset preparation, so "what's trainable" is defined in exactly one place. A single batched map produces a labels column where each token keeps its ID when every applicable mask is 1, else -100. Plain LM stays storage-neutral; pre-tokenized datasets with mask columns now go through the same path. Step 1 toward fixing #3927.
by @0xadvait in https://github.com/huggingface/trl/pull/6037
{% generation %}-marker training template for Idefics3, enabling assistant_only_loss=True.
by @aazizyan in https://github.com/huggingface/trl/pull/5871
num_items_in_batch for gradient accumulation by @behroozazarkhalili in https://github.com/huggingface/trl/pull/6006[AsyncGRPO] Rollout worker: set aiohttp limit to max(100, max_inflight_tasks) by @ggcr in https://github.com/huggingface/trl/pull/5861evaluate() accept the same dataset types as the trainer by @qgallouedec in https://github.com/huggingface/trl/pull/6116unpair_preference_dataset by @albertvillanova in https://github.com/huggingface/trl/pull/6161fix(profiling): log ProfilingContext metrics to Trackio backend by @Anai-Guo in https://github.com/huggingface/trl/pull/5979.contiguous() calls by @qgallouedec in https://github.com/huggingface/trl/pull/6045 and https://github.com/huggingface/trl/pull/6046create_reference_model with num_shared_layers was double-allocating the "shared" frozen layers because the loop never assigned _ref_param back. Now it does, so shared layers are held once. By @behroozazarkhalili in https://github.com/huggingface/trl/pull/6053chunked_nll mixed Tensor/DTensor error under FSDP2 + PEFT by @albertvillanova in https://github.com/huggingface/trl/pull/6065lm_head.weight all-gathers under FSDP2 + chunked_nll by @albertvillanova in https://github.com/huggingface/trl/pull/6077target_parameters by @discobot in https://github.com/huggingface/trl/pull/6043use_liger_kernel is combined with a PEFT adapter on lm_head by @akshansh47 in https://github.com/huggingface/trl/pull/5977[fix] GLM-4-MoE template: turn-terminating token to the turn itself by @qgallouedec in https://github.com/huggingface/trl/pull/6044unpair_preference_dataset dropping extra columns by @albertvillanova in https://github.com/huggingface/trl/pull/6059fix(gold): preserve vllm prompt special tokens by @he-yufeng in https://github.com/huggingface/trl/pull/6063fix(sft): reject transformed datasets during preparation by @he-yufeng in https://github.com/huggingface/trl/pull/6054_unpair_row by @albertvillanova in https://github.com/huggingface/trl/pull/6062__init__ signatures by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6011paper_index (experimental page was split) by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6070docs(rloo): add clarifying comment on KL penalty formula divergence from GRPO by @abderahmane-ai in https://github.com/huggingface/trl/pull/6096vllm>=0.22.0 by @sergiopaniego in https://github.com/huggingface/trl/pull/6101epsilon help/docstring wording by @qgallouedec in https://github.com/huggingface/trl/pull/6014_push_param_to_vllm helper in VLLMGeneration by @albertvillanova in https://github.com/huggingface/trl/pull/6004async_grpo to match the rest of trl/experimental by @qgallouedec in https://github.com/huggingface/trl/pull/6012num_completions_to_print, epsilon_high fallback, logging_steps docstring, loss variable names, clip-ratio metrics by @qgallouedec in https://github.com/huggingface/trl/pull/6020, https://github.com/huggingface/trl/pull/6019, https://github.com/huggingface/trl/pull/6016, https://github.com/huggingface/trl/pull/6013 and https://github.com/huggingface/trl/pull/6021chunked_nll loss by @albertvillanova in https://github.com/huggingface/trl/pull/6074test: bound memory in test_gkd_trainer_with_liger to avoid OOM on shared runners by @behroozazarkhalili in https://github.com/huggingface/trl/pull/6103sft_fa2 invariant test by @qgallouedec in https://github.com/huggingface/trl/pull/6069actions/checkout / astral-sh/setup-uv / actions/setup-python / pre-commit/action by @albertvillanova in https://github.com/huggingface/trl/pull/6097, https://github.com/huggingface/trl/pull/6098, https://github.com/huggingface/trl/pull/6099 and https://github.com/huggingface/trl/pull/6100parse_version with Version by @albertvillanova in https://github.com/huggingface/trl/pull/6164deepspeed < 0.19.2 by @albertvillanova in https://github.com/huggingface/trl/pull/6090chore: update tests_transformers_branch.yml by @hf-security-analysis[bot] in https://github.com/huggingface/trl/pull/6051chore: update clear_cache.yml by @hf-security-analysis[bot] in https://github.com/huggingface/trl/pull/6047chore: update docker-build.yml by @hf-security-analysis[bot] in https://github.com/huggingface/trl/pull/6048pr_style_bot workflow by @albertvillanova in https://github.com/huggingface/trl/pull/6082_get_train_sampler comment to mention num_iterations > 1 by @anidoesdev in https://github.com/huggingface/trl/pull/6125.contiguous() calls by @qgallouedec in https://github.com/huggingface/trl/pull/6045unpair_preference_dataset dropping extra columns by @albertvillanova in https://github.com/huggingface/trl/pull/6059sft_fa2 invariant test by @qgallouedec in https://github.com/huggingface/trl/pull/6069target_parameters by @discobot in https://github.com/huggingface/trl/pull/6043use_transformers_paged with transformers continuous batching by @sergiopaniego in https://github.com/huggingface/trl/pull/5765epsilon help/docstring wording by @qgallouedec in https://github.com/huggingface/trl/pull/6014router_aux_loss_coef by @qgallouedec in https://github.com/huggingface/trl/pull/6085.contiguous() from the shift logits/labels pattern by @qgallouedec in https://github.com/huggingface/trl/pull/6046get_repetition_penalty_reward by @qgallouedec in https://github.com/huggingface/trl/pull/6058trust_remote_code to trainer configs by @qgallouedec in https://github.com/huggingface/trl/pull/5802evaluate() accept the same dataset types as the trainer by @qgallouedec in https://github.com/huggingface/trl/pull/6116get_cosine_scaled_reward by @qgallouedec in https://github.com/huggingface/trl/pull/6066_get_train_sampler comment to mention num_iterations > 1 by @anidoesdev in https://github.com/huggingface/trl/pull/6125Full Changelog: https://github.com/huggingface/trl/compare/v1.6.0...v1.7.0
The old vllm_importance_sampling_cap is deprecated and maps to clip_max.
AsyncRolloutWorker is no longer a thread — it's a spawned child process with its own GIL. The trainer's autograd engine no longer competes with recursive_parse / accuracy_reward for the GIL, which was causing 1-5s stalls in real Qwen3-30B-A3B @ 16k runs and ultimately NCCL watchdog timeouts on other ranks.
Architectural changes:
AsyncRolloutWorker (parent) owns the child process + shared mp.Queue / mp.Value / mp.Event._AsyncRolloutLoop (child-only) handles tokenization, dataset iteration, reward funcs, and asyncio loops.WeightTransferClient owns the NCCL group with vLLM (/pause, /resume, /init_weight_transfer_engine, /update_weights); the rollout child only talks to /v1/completions.Two correctness fixes shipped alongside (they would have conflicted otherwise): broader aiohttp retry (now catches ClientPayloadError) with bounded exponential backoff, and all-NaN reward columns are now preserved — np.nansum was silently returning 0, giving unscorable completions a real advantage signal and pushing the policy away from correct answers (~30% of DeepMath / OpenR1-Math rows).
[!NOTE]
reward_funcs/tools/environment_factorymust now be picklable, and the child runs CPU-only (CUDA_VISIBLE_DEVICES="").
by @AmineDiro in https://github.com/huggingface/trl/pull/5749
A new A2POTrainer implements A*-PO from "Accelerating RL for LLM Reasoning with Optimal Advantage Regression". Two stages: an offline V* estimation pass from reference policy samples (with optional filter_all_incorrect to drop prompts where every reference completion fails), then on-policy training with one generation per prompt and a plain least-squares loss on β₂·log(π/π_ref) vs r − V*. No group, no critic, no clipping, no reward normalization.
from trl.experimental.a2po import A2POConfig, A2POTrainer
trainer = A2POTrainer(
model="Qwen/Qwen3-4B",
args=A2POConfig(num_value_samples=8, filter_all_incorrect=True),
train_dataset=dataset,
reward_funcs=accuracy_reward,
)
trainer.train()
Designed for binary verifiable rewards (math/code), not open-ended problems.
by @raghulchandramouli in https://github.com/huggingface/trl/pull/5940
The biggest KTO ↔ DPO alignment cycle yet — KTOTrainer now supports vision-language models, plus a deep restructuring of compute_loss, KL dataset generation, ref-logp precomputation, activation offloading, sampler strategy, metrics, and more. KTO graduation is very close.
from trl.experimental.kto import KTOConfig, KTOTrainer
trainer = KTOTrainer(
model="Qwen/Qwen2.5-VL-3B-Instruct",
args=KTOConfig(...),
train_dataset=vision_kto_dataset,
)
VLM support: by @albertvillanova in https://github.com/huggingface/trl/pull/5939. Plus ~20 alignment PRs all by @albertvillanova: #5820, #5849, #5852, #5850, #5866, #5864, #5856, #5872, #5875, #5900, #5901, #5899, #5906, #5909, #5914, #5982, #5936, #5996, #5998, #5999.
The GOLD distillation trainer used to align student/teacher tokens by extending two decoded strings and flushing on equality. It silently broke on any byte-level disagreement — including the common case of one tokenizer prepending BOS while the other doesn't (Llama-3 ↔ Qwen-3). The X-Token paper called this out by name.
Each side now carries (start_byte, end_byte) spans derived once from the fast tokenizer's char offsets, and the walker syncs on cumulative byte boundaries. On the on-policy path, spans come from piece_byte_len over the sampled token ids (not from re-encoding the decoded completion — BPE makes that round-trip non-injective).
Two related fixes shipped: long rows no longer lose the completion (now keeping the last max_length tokens), and the vLLM on-policy original_prompt_text is now decoded from the truncated ids the student actually consumed.
by @kashif in https://github.com/huggingface/trl/pull/5885
When teacher_model_kind="live" and vllm_mode="server", the vLLM generation server already holds the current student weights (synced every step for rollouts). The new use_teacher_server=True flag scores the teacher's log-probs on that same server instead of running a separate local teacher forward — removing the teacher from the training step entirely.
Supported modes: sampled_token (reverse KL on the realized token) and topk_logits. When buffered batches reuse steps (num_iterations > 1), weights are re-synced before scoring so the teacher never scores stale.
by @kashif in https://github.com/huggingface/trl/pull/5989
vLLM importance sampling in GRPO now uses a two-sided band [C_min, C_max] instead of a single upper cap, aligning TIS/MIS with IcePop's bidirectional handling of train–inference ratio outliers.
from trl import GRPOConfig
config = GRPOConfig(
vllm_importance_sampling_clip_min=0.5,
vllm_importance_sampling_clip_max=2.0,
vllm_importance_sampling_correction="mask", # or "truncate"
)
The old vllm_importance_sampling_cap is deprecated and maps to clip_max.
by @casinca in https://github.com/huggingface/trl/pull/4732
Day-zero training support for NVIDIA's new model families.
Three more model families with {% generation %} markers (assistant-only loss out of the box):
A new trl/distributed.py introduces a single DistributedBackend class that detects ZeRO stage and FSDP version once, then exposes two context managers (gather_params, summon_full_params) used everywhere. Replaces the scattered getattr(state, "fsdp_plugin", None) / gather_if_zero3 / summon_full_params if ... else nullcontext() boilerplate spread across vllm_generation.py, models/utils.py, and the main trainers. Future deprecations land in one place.
by @albertvillanova in https://github.com/huggingface/trl/pull/6000
A two-PR refactor that disentangles SDPO, SDFT, and other self-distillation trainers from their shared base, making each one self-contained and consistent with the rest of the codebase.
by @LeonEricsson in https://github.com/huggingface/trl/pull/5862 and https://github.com/huggingface/trl/pull/5883
loss_type will change in 1.7Setting SFTConfig.loss_type is now optional, and leaving it unset emits a FutureWarning: in TRL 1.7 the default will switch from "nll" to "chunked_nll". No action needed — you'll just get the new default automatically on upgrade — unless you want to pin the current behavior (e.g. for custom models) with loss_type="nll".
by @qgallouedec in https://github.com/huggingface/trl/pull/5997
'None' as CLI value for Optional[T] fields by @qgallouedec in https://github.com/huggingface/trl/pull/5843lm_head output projections in chunked SFT loss (GPTNeoX) by @qgallouedec in https://github.com/huggingface/trl/pull/5857SFTTrainer: merge entropy and accuracy computation to eliminate redundant logits copy by @flutist in https://github.com/huggingface/trl/pull/5897.contiguous() calls in DPOTrainer to reduce peak memory by @flutist in https://github.com/huggingface/trl/pull/5926.contiguous() before entropy_from_logits by @qgallouedec in https://github.com/huggingface/trl/pull/5930None reward completions from GRPO/RLOO advantage baseline by @AmineDiro in https://github.com/huggingface/trl/pull/5902precompute_ref_log_probs with vision datasets in DPO by @albertvillanova in https://github.com/huggingface/trl/pull/5867max_length by @lxk8998 in https://github.com/huggingface/trl/pull/5927loss_type="chunked_nll" under DeepSpeed ZeRO-3 by @qgallouedec in https://github.com/huggingface/trl/pull/5873use_liger_kernel under DeepSpeed ZeRO-3 by @kashif in https://github.com/huggingface/trl/pull/5891async_grpo: don't return on queue.Empty by @AmineDiro in https://github.com/huggingface/trl/pull/5751generate_batch: inference tensors block inplace ops in background thread by @albertvillanova in https://github.com/huggingface/trl/pull/5818 (cross-listed from v1.5 changelog window)encoding="utf-8" when reading .jinja chat templates on Windows by @ColebyPearson in https://github.com/huggingface/trl/pull/5869ValueError by pinning kernels < 0.15.1 by @albertvillanova in https://github.com/huggingface/trl/pull/5880kernels optional dependency via transformers by @albertvillanova in https://github.com/huggingface/trl/pull/5884kernels extra for transformers < 5.1.0 by @albertvillanova in https://github.com/huggingface/trl/pull/5928use_liger_kernel guard to SDPO teacher-server validation by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/5994huggingface.co/docs/openenv by @sergiopaniego in https://github.com/huggingface/trl/pull/5929bnb_4bit_quant_storage and normalize docstring param headers by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/5993online_dpo_vlm example description by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/5978SFTConfig max_length) by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/5970GRPOConfig epsilon_high help by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/5972trl skills install description by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/6008max_prompt_length argument from GRPO example by @DaoyuanLi2816 in https://github.com/huggingface/trl/pull/5964sft.json / dpo.json snapshots after transformers num_items_in_batch fix by @qgallouedec in https://github.com/huggingface/trl/pull/5845huggingface/skills by @albertvillanova in https://github.com/huggingface/trl/pull/5950.agents by @albertvillanova in https://github.com/huggingface/trl/pull/5987docker-build.yml with version parsing by @hf-security-analysis[bot] in https://github.com/huggingface/trl/pull/5920'None' as CLI value for Optional[T] fields by @qgallouedec in https://github.com/huggingface/trl/pull/5843loss_type="chunked_nll" under DeepSpeed ZeRO-3 by @qgallouedec in https://github.com/huggingface/trl/pull/5873sft.json / dpo.json snapshots after transformers num_items_in_batch fix by @qgallouedec in https://github.com/huggingface/trl/pull/5845.contiguous() before entropy_from_logits by @qgallouedec in https://github.com/huggingface/trl/pull/5930.agents by @albertvillanova in https://github.com/huggingface/trl/pull/5987loss_type default change from 'nll' to 'chunked_nll' by @qgallouedec in https://github.com/huggingface/trl/pull/5997Full Changelog: https://github.com/huggingface/trl/compare/v1.5.0...v1.6.0
🔒 Gate trainer telemetry on an explicit class-name allowlist by @qgallouedec in https://github.com/huggingface/trl/pull/5851
Full Changelog: https://github.com/huggingface/trl/compare/v1.5.0...v1.5.1
Replace deprecated torch_dtype with dtype across examples, docs, notebooks, tests, and experimental distillation / gold trainers by @qgallouedec in ht…
Three more model families gain training-compatible templates with {% generation %} markers (so assistant_only_loss=True just works):
The chunked LM-head path used by AsyncGRPOTrainer now supports models that use final_logit_softcapping (notably Gemma 2). _ChunkedLogProbFunction applies logit_scale, optional tanh-based softcapping, and temperature consistently in both forward and backward — softcapped models are no longer rejected.
by @mlarnouhet in https://github.com/huggingface/trl/pull/5691
Two more cycles closer to KTO graduation:
compute_loss flow by @albertvillanova in https://github.com/huggingface/trl/pull/5810_compute_loss_liger flow by @albertvillanova in https://github.com/huggingface/trl/pull/5816_BaseTrainer.__init__ now emits a single anonymous huggingface_hub.send_telemetry ping per trainer instantiation, so we can finally see which trainers / model families / distributed backends are actually being used in practice and prioritize accordingly.
The payload is intentionally minimal — TRL version, trainer class name, model architecture, PEFT yes/no, distributed backend (deepspeed/fsdp/ddp/none), bucketed world size, device type, GPU model when available. No user data, no dataset names, no model paths, no hyperparameter values, never sent in CI / offline / HF_HUB_DISABLE_TELEMETRY mode.
See usage_stats.md for what's collected and how to opt out.
by @qgallouedec in https://github.com/huggingface/trl/pull/5758
OpenRewardSpec: fix omitting task-scoped tools during rollout binding (fixes #5727) by @rycerzes in https://github.com/huggingface/trl/pull/5729GRPOTrainer was hanging indefinitely on truncated <tool_call> blocks (a degenerate case that happens naturally when generation hits max_completion_length mid-tool-call). Rewrote the regex to be non-backtracking — worst case goes from O(2ⁿ) to O(n). By @xodn348 in https://github.com/huggingface/trl/pull/5798OffloadActivations — follow-up to v1.4's activation-offloading leak fix. By @butterwecksolutions in https://github.com/huggingface/trl/pull/5730add_hooks by @roycho96 in https://github.com/huggingface/trl/pull/4693vocab_size for DistillationTrainer and GOLDTrainer by @Beichen-Ma in https://github.com/huggingface/trl/pull/5592empty_cache() by @jamie-peterson-ml in https://github.com/huggingface/trl/pull/5799metric_for_best_model for trainer-specific eval metrics by @qgallouedec in https://github.com/huggingface/trl/pull/5811generate_batch: inference tensors blocking inplace ops in background thread by @albertvillanova in https://github.com/huggingface/trl/pull/5818torch_dtype with dtype across examples, docs, notebooks, tests, and experimental distillation / gold trainers by @qgallouedec in https://github.com/huggingface/trl/pull/5717Glm4MoeForCausalLM / Cohere / Cohere2 / Qwen2.5-VL configs with their reference models by @qgallouedec in https://github.com/huggingface/trl/pull/5638, https://github.com/huggingface/trl/pull/5706, https://github.com/huggingface/trl/pull/5707 and https://github.com/huggingface/trl/pull/5739deepstack_visual_indexes and drop the test skip by @qgallouedec in https://github.com/huggingface/trl/pull/5779fullatt_block_indexes out of range for depth=2 by @albertvillanova in https://github.com/huggingface/trl/pull/5805num_heads key in Qwen VL tiny model scripts by @matdou in https://github.com/huggingface/trl/pull/5792model.visual. skip in GRPO/RLOO Qwen2.5-VL tests by @qgallouedec in https://github.com/huggingface/trl/pull/5780AttributeError: 'GptOssConfig' object has no attribute 'num_experts' by @albertvillanova in https://github.com/huggingface/trl/pull/5756apply_model_revisions by removing _commit_hash kwarg by @albertvillanova in https://github.com/huggingface/trl/pull/5762model.visual params by @albertvillanova in https://github.com/huggingface/trl/pull/5806torch < 2.12.0 (later reverted) by @albertvillanova in https://github.com/huggingface/trl/pull/5769pytest --only-rerun by @albertvillanova in https://github.com/huggingface/trl/pull/5784tests_latest.yml by @hf-security-analysis[bot] in https://github.com/huggingface/trl/pull/5733torch_dtype -> dtype by @qgallouedec in https://github.com/huggingface/trl/pull/5717model.visual. skip in GRPO / RLOO Qwen2.5-VL tests by @qgallouedec in https://github.com/huggingface/trl/pull/5780deepstack_visual_indexes and drop the test skip by @qgallouedec in https://github.com/huggingface/trl/pull/5779metric_for_best_model for trainer-specific eval metrics by @qgallouedec in https://github.com/huggingface/trl/pull/5811OpenRewardSpec omitting task‑scoped tools during rollout binding (fixes #5727) by @rycerzes in https://github.com/huggingface/trl/pull/5729Full Changelog: https://github.com/huggingface/trl/compare/v1.4.0...v1.5.0
A new loss_type="chunked_nll" option drastically reduces peak activation memory in SFT by avoiding the full [batch × seq × vocab] logits tensor. Ignor
<img width="2704" height="1455" alt="chunked_loss_idea" src="https://github.com/user-attachments/assets/3957f39b-3e71-4465-949a-22b2cf894d03" />
A new loss_type="chunked_nll" option drastically reduces peak activation memory in SFT by avoiding the full [batch × seq × vocab] logits tensor. Ignored-label tokens are dropped before the lm_head matmul, and the cross-entropy is computed over the remaining tokens in checkpointed chunks (default chunk_size=256, the sweet spot consistent across model sizes and sequence lengths).
from trl import SFTConfig, SFTTrainer
trainer = SFTTrainer(
model="Qwen/Qwen3-4B",
args=SFTConfig(loss_type="chunked_nll"),
train_dataset=dataset,
)
trainer.train()
Peak GPU memory, AdamW fp32:
| Model | Hardware | Seq | nll |
chunked_nll |
|---|---|---|---|---|
| Qwen3-1.7B + LoRA | 1×H100 80GB | 2048 | 47.9 GB | 12.3 GB (3.9× less) |
| Qwen3-4B | 1×H100 80GB | 16384 | OOM | 63.8 GB |
| Qwen3-14B | 8×H100 FSDP2 | 16384 | 58.9 GB | 38.9 GB (1.5× less) |
| Qwen3-32B | 8×H100 FSDP2 | 8192 | OOM | 71.2 GB |
End-to-end, chunked NLL is consistently as fast or faster than nll — and it unlocks sequence lengths that don't fit at all under the standard path.
The chunked path also supports VLMs (https://github.com/huggingface/trl/pull/5684).
by @qgallouedec in https://github.com/huggingface/trl/pull/5575, https://github.com/huggingface/trl/pull/5676 and https://github.com/huggingface/trl/pull/5684
A new trl.experimental.openreward adapter plugs any environment speaking the Open Reward Standard (ORS) protocol into any TRL trainer accepting an environment_factory (GRPOTrainer, AsyncGRPOTrainer). One identifier wires all three trainer slots — dataset, factory, reward_func:
from trl import GRPOConfig, GRPOTrainer
from trl.experimental.openreward import OpenRewardEnv
env = OpenRewardEnv("Eigent/SETA") # or "http://localhost:8000"
trainer = GRPOTrainer(
model="Qwen/Qwen3-4B",
args=GRPOConfig(...),
train_dataset=env.dataset,
environment_factory=env.factory,
reward_funcs=env.reward_func,
)
Tools are bound dynamically from JSON Schema at construction (no per-env wrapper code), and env.dataset autoderives task lists from the ORS task endpoints. The same code path works for envs hosted on the OpenReward platform, self-hosted on any container service, or running locally on localhost. A SETA training example is included.
by @adithya-s-k in https://github.com/huggingface/trl/pull/5696
Unit tests don't catch trainer-level numerical drift (gradient-accumulation normalization bugs, attention-impl divergence (eager ↔ FA2 / kernels)) they silently shift the loss trajectory and users only notice when their run no longer reproduces. (Cf. last year's transformers grad-accum bug, or the "We found two bugs in DeepSpeed" paper.)
A new opt-in pytest -m invariant suite asserts the loss / grad_norm trajectory of short end-to-end SFT/DPO runs against committed reference snapshots, with equivalence classes for configs that should produce identical trajectories (e.g. pdb=1, gas=8 ≡ default; eager ≡ FA2 ≡ kernels). Hardware-pinned to H100 80GB, real pretrained model, full_determinism, fixed seed. Initial coverage: 2 trainers × 2 invariance axes (grad-accum, attn-impl) × gradient-checkpointing equivalence.
by @qgallouedec in https://github.com/huggingface/trl/pull/5686, https://github.com/huggingface/trl/pull/5688 and https://github.com/huggingface/trl/pull/5689
Three new pure helpers in trl.trainer.utils for measuring training efficiency:
compute_flops_per_token(config, seq_len) — handles dense and MoE (Mixtral, Qwen3-MoE, DeepSeek-V2)compute_mfu(flops_per_token, tps, world_size, peak_flops) — Model FLOPs Utilization as a percentageadjusted_mfu(mfu, config, seq_len) — non-causal → causal-corrected (Llama / DS Ulysses convention)by @AmineDiro in https://github.com/huggingface/trl/pull/5698
GRPO's Liger-kernel integration is updated for Liger 0.8.0: delta two-sided clipping, use_bias_correction_kl, and SAPO/VESPO parameters are now forwarded into LigerFusedLinearGRPOLoss. The previous delta + use_liger_kernel guard is removed — both can be combined.
by @kashif in https://github.com/huggingface/trl/pull/5690
A new loss_type="sigmoid_norm" option for DPOConfig implements the per-token (length-normalized) DPO loss used by Tülu 3 / OLMo (paper §5.1.2 eq. 6) to mitigate length bias.
from trl import DPOConfig, DPOTrainer
trainer = DPOTrainer(
model="Qwen/Qwen3-4B",
args=DPOConfig(loss_type="sigmoid_norm"),
train_dataset=dataset,
)
by @BrownianNotion in https://github.com/huggingface/trl/pull/5406
Four more model families gain training-compatible chat templates with {% generation %} markers (assistant-only loss masking) and/or response schemas (tool-calling parsing):
{% generation %} markers by @qgallouedec in https://github.com/huggingface/trl/pull/5675get_training_chat_template now also accepts a processor (not just a tokenizer) — useful for VLMs (https://github.com/huggingface/trl/pull/5560).
Another batch of alignment PRs this cycle. KTO and DPO are now structurally aligned across PEFT handling, model initialization, training-arg grouping, ref-logp precomputation, and metric handling — promotion of KTO out of experimental is imminent.
PRs (all by @albertvillanova): #5659, #5660, #5661, #5679, #5701, #5702, #5703, #5704, #5705, #5714.
parallelism_config with cp_size>1 or sp_size>1 in GRPO/RLOO — fail fast at config init with a clear error instead of mid-training crash. By @kashif in https://github.com/huggingface/trl/pull/5699model_accepts_loss_kwargs=False in DPO and Reward by @albertvillanova in https://github.com/huggingface/trl/pull/5710_tokenizer attribute in experimental trainers by @albertvillanova in https://github.com/huggingface/trl/pull/5566peft_config handling in core / experimental trainers by @albertvillanova in https://github.com/huggingface/trl/pull/5673 and https://github.com/huggingface/trl/pull/5674isinstance with is_peft_model / drop redundant is_peft_available by @albertvillanova in https://github.com/huggingface/trl/pull/5682 and https://github.com/huggingface/trl/pull/5683parse_response by @qgallouedec in https://github.com/huggingface/trl/pull/5561OffloadActivations.__exit__ now syncs the compute/offload streams and clears the stash dictionaries, preventing orphaned offload tensors from leaking onto a dead stream (~0.2 GiB/step accumulation observed during QLoRA vision training before the fix). By @butterwecksolutions in https://github.com/huggingface/trl/pull/5694 and https://github.com/huggingface/trl/pull/5700DistillationTrainer by @k1064190 in https://github.com/huggingface/trl/pull/5594GKDTrainer: fix return_outputs in the Liger kernel path by @roycho96 in https://github.com/huggingface/trl/pull/4688GKDTrainer: fix seq-KD wasted teacher forward by @roycho96 in https://github.com/huggingface/trl/pull/5726GKDTrainer: fix Liger fused JSD path computing wrong loss by @roycho96 in https://github.com/huggingface/trl/pull/5731peft_config to core / experimental trainers by @albertvillanova in https://github.com/huggingface/trl/pull/5664 and https://github.com/huggingface/trl/pull/5665peft_config type hint in experimental trainers by @albertvillanova in https://github.com/huggingface/trl/pull/5666DistillationTrainer by @cmpatino in https://github.com/huggingface/trl/pull/5615Qwen3-4B-Instruct-2507 by @qgallouedec in https://github.com/huggingface/trl/pull/5586Qwen/Qwen3-30B-A3B by @qgallouedec in https://github.com/huggingface/trl/pull/5716DistillationTrainer by @cmpatino in https://github.com/huggingface/trl/pull/5615{% generation %} markers for Cohere2 chat template by @qgallouedec in https://github.com/huggingface/trl/pull/5675get_training_chat_template by @qgallouedec in https://github.com/huggingface/trl/pull/5560parse_response by @qgallouedec in https://github.com/huggingface/trl/pull/5561Full Changelog: https://github.com/huggingface/trl/compare/v1.3.0...v1.4.0
TRL v1.3 ships training support for the new Qwen 3.6 family (Qwen/Qwen3.6-27B, Qwen/Qwen3.6-35B-A3B). Qwen 3.6 reuses the Qwen3_5Moe* architecture but
<img width="1536" height="1024" alt="ChatGPT Image Apr 26, 2026 at 11_16_18 AM" src="https://github.com/user-attachments/assets/789aad15-03b2-4ece-9828-d5c1dfed1f1e" />
TRL v1.3 ships training support for the new Qwen 3.6 family (Qwen/Qwen3.6-27B, Qwen/Qwen3.6-35B-A3B). Qwen 3.6 reuses the Qwen3_5Moe* architecture but ships a slightly different chat template (adds a preserve_thinking flag, tweaks tool-arg stringification), so exact-string template matching needed updates across the stack.
What landed:
qwen3_6.jinja (verbatim from upstream) and qwen3_6_training.jinja (prefix-preserving + {% generation %} markers for assistant_only_loss=True)qwen3_5_schema for tool-call parsing — output format unchangedtiny-Qwen3_5MoeForConditionalGeneration-3.6 (with MoE-specific shrinking)test_(train|training)_vlm casesfrom trl import SFTConfig, SFTTrainer
trainer = SFTTrainer(
model="Qwen/Qwen3.6-27B",
args=SFTConfig(assistant_only_loss=True), # works out of the box
train_dataset=dataset,
)
trainer.train()
Tool-calling agent training also works end-to-end via the existing Qwen 3.5 response schema:
from trl import GRPOConfig, GRPOTrainer
def multiply(a: int, b: int) -> int:
"""
Multiplies two integers.
Args:
a: The first integer.
b: The second integer.
Returns:
The product of the two integers.
"""
return a * b
trainer = GRPOTrainer(
model="Qwen/Qwen3.6-27B",
reward_funcs=my_reward_fn,
args=GRPOConfig(...),
train_dataset=dataset,
tools=[multiply],
)
trainer.train()
by @qgallouedec in https://github.com/huggingface/trl/pull/5642
<img width="711" height="177" alt="Screenshot 2026-04-26 at 11 37 28 AM" src="https://github.com/user-attachments/assets/6090212e-5c95-45c1-b137-87333d91daa6" />
A new experimental TPOTrainer implements Triple Preference Optimization, which augments DPO with a reference (gold) completion alongside chosen/rejected. The paper reports +7-19 points over DPO/SimPO on Arena-Hard, MixEval-Hard, MMLU-Pro and GSM8K, with less data.
from trl.experimental.tpo import TPOConfig, TPOTrainer
trainer = TPOTrainer(
model="Qwen/Qwen3-0.6B",
args=TPOConfig(output_dir="Qwen3-0.6B-TPO"),
train_dataset=load_dataset("tpo-alignment/triple-preference-ultrafeedback-40K", split="train"),
)
trainer.train()
by @kashif in https://github.com/huggingface/trl/pull/5506
trl vllm-serveA new --speculative_config JSON flag exposes vLLM's speculative decoding directly through trl vllm-serve — works with native MTP heads (Qwen3 Next), Eagle3 drafts, etc. — without forking the serve script.
# Qwen3 native MTP (no extra draft model)
trl vllm-serve --model Qwen/Qwen3-Next-80B-A3B-Instruct \
--speculative_config '{"method": "qwen3_next_mtp", "num_speculative_tokens": 5}'
# Eagle3 draft model
trl vllm-serve --model Qwen/Qwen3-32B \
--speculative_config '{"model": "RedHatAI/Qwen3-32B-speculator.eagle3", "method": "eagle3", "num_speculative_tokens": 3}'
by @Ofir408 in https://github.com/huggingface/trl/pull/5605
Twelve more alignment PRs this cycle, bringing KTOTrainer and DPOTrainer essentially into structural parity. Notable shifts include moving completion assembly out of _prepare_dataset into a new DataCollatorForKTO, inlining the two-pass tokenization into a single pass, removing BOS/EOS handling, and supporting IterableDataset and dict eval_dataset. The goal — promoting KTO out of experimental and into stable — is now within reach for an upcoming release.
PRs (all by @albertvillanova): #5582, #5578, #5579, #5583, #5587, #5599, #5601, #5600, #5606, #5612, #5632, #5635
{% generation %} training chat templatesThree more model families gain training-compatible chat templates with {% generation %} markers, so assistant_only_loss=True works out of the box:
maybe_apply_chat_template by @albertvillanova in https://github.com/huggingface/trl/pull/5567is_chat_template_prefix_preserving by @qgallouedec in https://github.com/huggingface/trl/pull/5558forward_masked_logits by @qgallouedec in https://github.com/huggingface/trl/pull/5626_tokenizer as trainer attribute by @albertvillanova in https://github.com/huggingface/trl/pull/5489PreTrainedTokenizerBase for tokenizer type hints by @qgallouedec in https://github.com/huggingface/trl/pull/5629async_reward_X to async_X by @qgallouedec in https://github.com/huggingface/trl/pull/5616attention_mask instead of label != -100), and wrong cross-rank aggregation (unweighted mean instead of sum/count). The reported entropy under completion_only_loss=True and sequence parallelism is now correct. Same fix applied to DPO entropy logging. By @qgallouedec in https://github.com/huggingface/trl/pull/5620AsyncGRPOTrainer's processing_class to AsyncRolloutWorker by @xuanduy04 in https://github.com/huggingface/trl/pull/5538generate_tiny_models for gpt-oss by @albertvillanova in https://github.com/huggingface/trl/pull/5622TestSupportsToolCalling for improved coverage by @qgallouedec in https://github.com/huggingface/trl/pull/5537is_chat_template_prefix_preserving by @qgallouedec in https://github.com/huggingface/trl/pull/5558TestSupportsToolCalling for improved coverage by @qgallouedec in https://github.com/huggingface/trl/pull/5537{% generation %} markers for training chat template by @casinca in https://github.com/huggingface/trl/pull/5519forward_masked_logits by @qgallouedec in https://github.com/huggingface/trl/pull/5626PreTrainedTokenizerBase for tokenizer type hints by @qgallouedec in https://github.com/huggingface/trl/pull/5629async_reward_X to async_X by @qgallouedec in https://github.com/huggingface/trl/pull/5616Full Changelog: https://github.com/huggingface/trl/compare/v1.2.0...v1.3.0
Deprecate `use_transformers_paged` in GRPOConfig and RLOOConfig (and remove entirely from experimental OnlineDPOConfig, GOLDConfig, SelfDistillationCo…
SSDTrainer — Simple Self-Distillation<img width="778" height="334" alt="Screenshot 2026-04-16 at 9 08 04 PM" src="https://github.com/user-attachments/assets/8ca223f0-6740-48a8-967c-ec10cb262a93" />
A new experimental SSDTrainer implements the method described in Embarrassingly Simple Self-Distillation Improves Code Generation. SSD samples completions from the model itself at a training-time temperature/truncation setting, then fine-tunes on those raw, unverified samples with standard cross-entropy loss. No reward model, verifier, teacher model, or RL: just prompts and the model.
from datasets import Dataset
from trl.experimental.ssd import SSDConfig, SSDTrainer
dataset = Dataset.from_dict({
"prompt": [
[{"role": "user", "content": "Write a function to add two numbers."}],
[{"role": "user", "content": "Write a function to check if a number is prime."}],
],
})
trainer = SSDTrainer(
model="Qwen/Qwen3-4B-Instruct",
args=SSDConfig(
output_dir="ssd-model",
temperature=0.6, # T_train from the paper
top_k=20,
top_p=0.95,
learning_rate=5e-6,
),
train_dataset=dataset,
)
trainer.train()
by @kashif in https://github.com/huggingface/trl/pull/5505
GRPOTrainerWhen tool calls produce more tokens than max_completion_length allows, GRPOTrainer now rolls back the tool messages/images added in the current iteration instead of trying to truncate them. This removes ~80 lines of fragile, image-boundary-aware bookkeeping in favor of a ~15-line snapshot-and-rollback. Since overlong samples almost always get rewarded as failures anyway, the learning signal is effectively unchanged — but the code is dramatically simpler and no longer needs per-VLM-family vision-token lookup tables.
by @qgallouedec in https://github.com/huggingface/trl/pull/5521
Continuing the effort from v1.1:
{% generation %} markers, enabling assistant-only loss masking for DeepSeek-V3 models. By @RudrenduPaul in https://github.com/huggingface/trl/pull/5527As a result of a tightened detection (see fixes below), the list of templates reported as tool-calling capable is now correct — notably, the basic Llama 3 template is no longer falsely classified as tool-calling capable.
A major cleanup sweep keeps KTOTrainer and DPOTrainer in lockstep, same initialization patterns, same config surface, same precompute behavior:
precompute_ref_batch_size to KTO (https://github.com/huggingface/trl/pull/5530)ref_model initialization (https://github.com/huggingface/trl/pull/5534)None args (https://github.com/huggingface/trl/pull/5531)generate_during_eval (https://github.com/huggingface/trl/pull/5551)ref_model when precompute_ref_log_probs is set in DPO/KTO (https://github.com/huggingface/trl/pull/5542)All by @albertvillanova.
prepare_multimodal_messages by @albertvillanova in https://github.com/huggingface/trl/pull/5474prepare_multimodal_messages by @albertvillanova in https://github.com/huggingface/trl/pull/5508supports_tool_calling falsely accepting templates that drop assistant tool_calls by @qgallouedec in https://github.com/huggingface/trl/pull/5517add_response_schema for VLM processors — the schema was being set on the outer processor instead of the inner tokenizer, so it had no effect. This also collapses a handful of __init__/decode-gate workarounds. By @qgallouedec in https://github.com/huggingface/trl/pull/5520use_transformers_paged in GRPOConfig and RLOOConfig (and remove entirely from experimental OnlineDPOConfig, GOLDConfig, SelfDistillationConfig). Will be removed from the remaining configs in v2.0.0. In a small A/B benchmark (Qwen3-0.6B GRPO), the paged path is ~20% slower and uses ~6x more peak VRAM than the default; it's also superseded by transformers continuous batching. By @qgallouedec in https://github.com/huggingface/trl/pull/5544chat_templates/README by @qgallouedec in https://github.com/huggingface/trl/pull/5545supports_tool_calling falsely accepting templates that drop assistant tool_calls by @qgallouedec in https://github.com/huggingface/trl/pull/5517use_transformers_paged by @qgallouedec in https://github.com/huggingface/trl/pull/5544add_response_schema for VLM processors by @qgallouedec in https://github.com/huggingface/trl/pull/5520chat_templates/README by @qgallouedec in https://github.com/huggingface/trl/pull/5545Full Changelog: https://github.com/huggingface/trl/compare/v1.1.0...v1.2.0
Fix SFT deprecation warning by @albertvillanova in https://github.com/huggingface/trl/pull/5466
DistillationTrainer for efficient on-policy distillationRead the blog post: https://huggingface.co/spaces/HuggingFaceTB/trl-distillation-trainer
The new DistillationTrainer implements on-policy knowledge distillation as described in On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes. It extends the ideas from the GKDTrainer with three key optimizations: a generation buffer that decouples the training microbatch size from the generation batch size (up to 40x speedup), external teacher server support so the teacher doesn't need to fit on training GPUs, and binary-encoded logprob payloads that shrink transfer payloads by ~5x.
from datasets import load_dataset
from trl.experimental.distillation import DistillationConfig, DistillationTrainer
dataset = load_dataset("openai/gsm8k", "main", split="train")
dataset = dataset.map(
lambda x: {"messages": [{"role": "user", "content": x["question"]}]},
remove_columns=dataset.column_names,
)
trainer = DistillationTrainer(
model="Qwen/Qwen2.5-1.5B-Instruct",
teacher_model="Qwen/Qwen2.5-7B-Instruct",
args=DistillationConfig(
output_dir="results/distill-qwen-gsm8k",
lmbda=1.0, # fully on-policy (student generates)
beta=1.0, # reverse KL
teacher_model_init_kwargs={"torch_dtype": "bfloat16"},
),
train_dataset=dataset,
)
trainer.train()
by @cmpatino in https://github.com/huggingface/trl/pull/5407, https://github.com/huggingface/trl/pull/5500 and https://github.com/huggingface/trl/pull/5501
AsyncGRPOTrainerAsyncGRPOTrainer now supports a chunked LM-head path that computes per-token log-probs and entropy via online logsumexp without materializing the full [N, V] logits tensor. Combined with completion_mask filtering to skip prompt tokens, this brings massive memory savings on long sequences — up to 44x lower peak-allocated memory on an 8192-token sequence:
chunk_lm_head_size |
Peak Alloc (GB) | Reduction | Wall Time (ms) |
|---|---|---|---|
None (baseline) |
18.55 | 1.00x | 808.7 |
4096 |
0.42 | 44.32x | 459.0 |
8192 |
0.76 | 24.34x | 393.0 |
Enable it via the new chunk_lm_head_size option in AsyncGRPOConfig:
from trl.experimental.async_grpo import AsyncGRPOConfig, AsyncGRPOTrainer
trainer = AsyncGRPOTrainer(
model="Qwen/Qwen2.5-0.5B-Instruct",
args=AsyncGRPOConfig(chunk_lm_head_size=4096),
...
)
Note: mutually exclusive with use_liger_kernel (both replace the LM head forward pass).
by @AmineDiro in https://github.com/huggingface/trl/pull/5349
{% generation %} support in training chat templatesSFT with assistant_only_loss=True requires chat templates to include {% generation %} / {% endgeneration %} markers so that return_assistant_tokens_mask=True produces correct masks. Very few models ship these markers natively, so users hit a cryptic error when enabling assistant-only loss with models like Qwen3, Llama 3 or GPT-OSS.
SFTTrainer now automatically swaps in a patched training chat template when the original template lacks generation markers — no manual template surgery required. Training templates are shipped for Qwen2.5, Qwen3, Llama 3 and GPT-OSS, stored as standalone .jinja files under trl/chat_templates/ for readability, diffability, and editor syntax highlighting.
from trl import SFTConfig, SFTTrainer
trainer = SFTTrainer(
model="Qwen/Qwen3-4B",
args=SFTConfig(assistant_only_loss=True), # now just works
train_dataset=dataset,
)
trainer.train()
by @qgallouedec in https://github.com/huggingface/trl/pull/5459, https://github.com/huggingface/trl/pull/5470, by @RudrenduPaul in https://github.com/huggingface/trl/pull/5493 and https://github.com/huggingface/trl/pull/5522, and by @casinca in https://github.com/huggingface/trl/pull/5484
Agent training now supports a broader family of models via native tool-call response schemas:
A new supports_tool_calling() utility detects whether a tokenizer/processor can render a full tool-calling turn, and GRPOTrainer now validates tool support at initialization — raising a clear error upfront instead of failing cryptically mid-training.
by @qgallouedec in https://github.com/huggingface/trl/pull/5462, https://github.com/huggingface/trl/pull/5464, https://github.com/huggingface/trl/pull/5463, https://github.com/huggingface/trl/pull/5469 and https://github.com/huggingface/trl/pull/5454
environment_factory tool methods can now return multimodal content blocks (images + text) for VLM training. Previously, tool responses were always converted to str(result), discarding any visual information. Now tools can return content block lists with images, and the trainer handles them end-to-end through tokenization, generation, and the forward pass — including correct pixel_values plumbing.
class ScreenshotEnv:
def take_screenshot(self) -> list[dict]:
return [
{"type": "image", "image": self.browser.screenshot()},
{"type": "text", "text": "Current page state"},
]
The OpenEnv browsergym.py example has been migrated to this pattern, and a new carla_vlm.py example demonstrates VLM training against CARLA with camera-image tool responses.
by @sergiopaniego in https://github.com/huggingface/trl/pull/5323 and https://github.com/huggingface/trl/pull/5437, and by @qgallouedec in https://github.com/huggingface/trl/pull/5448
accuracy_reward and reasoning_accuracy_reward now emit extra diagnostic columns (solution, gold_parsed, answer_parsed) via the log_extra callback introduced in v1.0.0. These show up in the rich completions table, making it much easier to debug why a reward was (or wasn't) assigned.
<img width="1627" alt="accuracy reward with extra columns" src="https://github.com/user-attachments/assets/d7f6e9c2-4d7b-4886-ba7a-f58f0ccfcb9b" />
from datasets import load_dataset
from trl import GRPOConfig, GRPOTrainer
from trl.rewards import accuracy_reward
dataset = load_dataset("trl-lib/DeepMath-103K", split="train")
trainer = GRPOTrainer(
model="Qwen/Qwen2-0.5B-Instruct",
reward_funcs=accuracy_reward,
args=GRPOConfig(log_completions=True),
train_dataset=dataset,
)
trainer.train()
by @qgallouedec in https://github.com/huggingface/trl/pull/5308
KTOConfig by @albertvillanova in https://github.com/huggingface/trl/pull/5477prepare_multimodal_messages by @albertvillanova in https://github.com/huggingface/trl/pull/5424prepare_multimodal_messages by @albertvillanova in https://github.com/huggingface/trl/pull/5475pixel_position_ids with image_position_ids for Gemma 4 support by @qgallouedec in https://github.com/huggingface/trl/pull/5452isinstance(part, dict) checks in image extraction by @qgallouedec in https://github.com/huggingface/trl/pull/5439_get_tool_suffix_ids by @qgallouedec in https://github.com/huggingface/trl/pull/5440prepare_deepspeed by @albertvillanova in https://github.com/huggingface/trl/pull/5414ImportError with vllm-0.10.2 in OnlineDPO and OpenEnv by @albertvillanova in https://github.com/huggingface/trl/pull/5423_get_per_token_logps_and_entropies return type by @kashif in https://github.com/huggingface/trl/pull/5456prepare_multimodal_messages not normalizing empty string content for assistant/tool roles by @albertvillanova in https://github.com/huggingface/trl/pull/5496pad_token_id by @albertvillanova in https://github.com/huggingface/trl/pull/5487huggingface-cli references with hf by @hanouticelina in https://github.com/huggingface/trl/pull/5486truncation_mode from experimental truncate_dataset by @albertvillanova in https://github.com/huggingface/trl/pull/5467keep_end truncation mode in DPOConfig and SFTConfig — will be removed in v2.0.0. Use keep_start instead. By @albertvillanova in https://github.com/huggingface/trl/pull/5465pad_token config parameter in DPOConfig, SFTConfig, and RewardConfig — will be removed in v2.0.0. Set tokenizer.pad_token directly on the processing_class instead. By @albertvillanova in https://github.com/huggingface/trl/pull/5480trl.experimental.judges module and all judge support from trainers. Judges were experimental, unused in practice, and llm-blender (backing PairRMJudge) was unmaintained and incompatible with transformers v5 — actively blocking v5 adoption. Everything judges did can be achieved with reward_funcs. OnlineDPOTrainer, NashMDTrainer, and XPOTrainer are now unified on reward-model scoring only. By @qgallouedec in https://github.com/huggingface/trl/pull/5485carla_vlm OpenEnv example by @sergiopaniego in https://github.com/huggingface/trl/pull/5437completion_only_loss in SFT trainer docs by @RudrenduPaul in https://github.com/huggingface/trl/pull/5494DistillationTrainer by @cmpatino in https://github.com/huggingface/trl/pull/5500prepare_multimodal_messages by @albertvillanova in https://github.com/huggingface/trl/pull/5476make precommit to fix docstring style by @albertvillanova in https://github.com/huggingface/trl/pull/5436test_rloo[fsdp2]: replace non-deterministic xfail with skipif for transformers 5.4.0 by @albertvillanova in https://github.com/huggingface/trl/pull/5403input_ids or inputs_embeds by @albertvillanova in https://github.com/huggingface/trl/pull/5422eval_strategy by @SunMarc in https://github.com/huggingface/trl/pull/5426TypeError: 'NoneType' object is not iterable by @albertvillanova in https://github.com/huggingface/trl/pull/5427TypeError: 'NoneType' object is not iterable by @albertvillanova in https://github.com/huggingface/trl/pull/5438test_rloo[fsdp2] after transformers 5.5.0 release by @albertvillanova in https://github.com/huggingface/trl/pull/5442eval_strategy by @SunMarc in https://github.com/huggingface/trl/pull/5426environment_factory for VLM training by @sergiopaniego in https://github.com/huggingface/trl/pull/5323isinstance(part, dict) checks in image extraction by @qgallouedec in https://github.com/huggingface/trl/pull/5439pixel_position_ids with image_position_ids for Gemma4 support by @qgallouedec in https://github.com/huggingface/trl/pull/5452_get_tool_suffix_ids by @qgallouedec in https://github.com/huggingface/trl/pull/5440.jinja files by @qgallouedec in https://github.com/huggingface/trl/pull/5459supports_tool_calling utility and validate tool support at init by @qgallouedec in https://github.com/huggingface/trl/pull/5462{% generation %} support to training chat templates by @qgallouedec in https://github.com/huggingface/trl/pull/5470DistillationTrainer for efficient on-policy distillation by @cmpatino in https://github.com/huggingface/trl/pull/5407huggingface-cli references with hf by @hanouticelina in https://github.com/huggingface/trl/pull/5486{% generation %} markers for training chat template by @casinca in https://github.com/huggingface/trl/pull/5484trl.experimental.judges module and all judge support from trainers by @qgallouedec in https://github.com/huggingface/trl/pull/5485DistillationTrainer by @cmpatino in https://github.com/huggingface/trl/pull/5501DistillationTrainer by @cmpatino in https://github.com/huggingface/trl/pull/5500Full Changelog: https://github.com/huggingface/trl/compare/v1.0.0...v1.1.0
A `MIGRATION.md` guide has been added covering all breaking changes when upgrading from TRL v0 to v1. If you're already on v0.29, the changes are mini…
<img width="1800" height="1013" alt="thumbnail-2" src="https://github.com/user-attachments/assets/5c55b86a-0600-4f70-bf37-41ab240af851" />
Read our blog post for an overview of TRL v1.
Asynchronous GRPO decouples generation from the gradient update loop by offloading rollouts to an external vLLM server. Generation runs in parallel while training continues, eliminating idle GPU time and improving hardware utilization.
from trl.experimental.async_grpo import AsyncGRPOTrainer
from trl.rewards import accuracy_reward
from datasets import load_dataset
dataset = load_dataset("trl-lib/DeepMath-103K", split="train")
trainer = AsyncGRPOTrainer(
model="Qwen/Qwen2.5-0.5B-Instruct",
reward_funcs=accuracy_reward,
train_dataset=dataset,
)
trainer.train()
by @qgallouedec in https://github.com/huggingface/trl/pull/5293
<img width="465" height="279" alt="Screenshot 2026-03-20 at 5 49 50 PM" src="https://github.com/user-attachments/assets/b60c9697-6eb7-498e-95b3-df78c367f5fa" />
VESPO addresses training instability in off-policy RL caused by policy staleness, asynchronous updates, and train-inference mismatches. Rather than relying on heuristic token-level clipping (GRPO) or sequence-length normalization (GSPO), VESPO derives a principled reshaping kernel from a variational framework. In practice, this yields a smooth, asymmetric Gamma weighting function that gracefully suppresses extreme sequence-level importance weights without introducing length bias. It can be enabled via the loss_type parameter of GRPOConfig:
from trl import GRPOConfig, GRPOTrainer
trainer = GRPOTrainer(
model="Qwen/Qwen3-0.6B",
args=GRPOConfig(loss_type="vespo"),
...
)
by @casinca in https://github.com/huggingface/trl/pull/5199
<img width="3180" height="1187" alt="z_TXYw37xZqsQ21YiDkYL" src="https://github.com/user-attachments/assets/40f1d538-82b3-4097-91c6-119ea9f7797b" /> <img width="1189" height="490" alt="SfgWotuuuRKPkg-0bxWv1" src="https://github.com/user-attachments/assets/2b090df3-0bfb-42e4-9f94-15943736e689" />
DPPO is a new experimental trainer that replaces the standard PPO clipping mechanism with divergence constraints, providing more principled trust-region updates.
by @LeonEricsson in https://github.com/huggingface/trl/pull/5117
SDPO is a new experimental trainer that augments on-policy RL with self-distillation from the model's own high-reward trajectories. Instead of using an external teacher, SDPO treats the current model conditioned on feedback as a self-teacher, distilling its feedback-informed predictions back into the policy.
from trl.experimental import SDPOTrainer, SDPOConfig
config = SDPOConfig(
output_dir="./results",
num_generations=8,
success_reward_threshold=1.0,
use_successful_as_teacher=True,
)
trainer = SDPOTrainer(
model="Qwen/Qwen2.5-Math-1.5B-Instruct",
reward_funcs=[accuracy_reward],
args=config,
train_dataset=dataset,
)
trainer.train()
by @MengAiDev in https://github.com/huggingface/trl/pull/4935
Reward functions can return a dictionary of extra values (scalars or per-sample columns) that will be logged alongside the reward. This makes it easier to track intermediate signals without writing custom callbacks.
def my_reward_fn(completions, answer, log_extra=None, log_metric=None, **kwargs):
extracted = [extract_answer(c) for c in completions]
rewards = [1.0 if e == a else 0.0 for e, a in zip(extracted, answer)]
if log_extra:
log_extra("golden_answer", list(answer))
log_extra("extracted_answer", extracted)
if log_metric:
log_metric("accuracy", sum(rewards) / len(rewards))
return rewards
<img width="1400" height="407" alt="image" src="https://github.com/user-attachments/assets/d345b0ac-0d3c-446f-9321-a26e73ee16b4" /> <img width="1353" height="673" alt="image" src="https://github.com/user-attachments/assets/b4c0302b-f69a-4715-9aad-278b4ad13299" />
by @manueldeprada in https://github.com/huggingface/trl/pull/5233
VLLMClient.chat()VLLMClient.chat() now supports tool calling, enabling agentic workflows directly through the vLLM client interface.
by @kansalaman in https://github.com/huggingface/trl/pull/4889
BFD packing is 35% faster. The "bfd-requeue" packing strategy has also been renamed to "bfd_split". See MIGRATION.md for details.
<img width="1784" height="732" alt="benchmark_results" src="https://github.com/user-attachments/assets/8f0a35ad-cf1a-4fe1-a1f4-9b102637bdca" />
by @mariosasko in https://github.com/huggingface/trl/pull/5189
The GKD/GOLD trainer now supports buffered rollout generation, decoupling generation from gradient updates for more efficient distillation. vLLM inference support has also been added to the base self-distillation trainer.
by @cmpatino in https://github.com/huggingface/trl/pull/5137 and https://github.com/huggingface/trl/pull/5388
A MIGRATION.md guide has been added covering all breaking changes when upgrading from TRL v0 to v1. If you're already on v0.29, the changes are minimal.
by @qgallouedec in https://github.com/huggingface/trl/pull/5255
vllm_mode to "colocate" by @qgallouedec in https://github.com/huggingface/trl/pull/5255truncation_mode in SFT by @albertvillanova in https://github.com/huggingface/trl/pull/5306max_length in DPO VLM training by @albertvillanova in https://github.com/huggingface/trl/pull/5284pad_to_multiple_of to GRPOTrainer and RLOOTrainer by @czkkkkkk in https://github.com/huggingface/trl/pull/5180print_prompt_completions_sample to include reasoning content by @qgallouedec in https://github.com/huggingface/trl/pull/5327pixel_position_ids vision key by @qgallouedec in https://github.com/huggingface/trl/pull/5374None to apply_chat_template when it is an empty list by @rabinadk1 in https://github.com/huggingface/trl/pull/5380accuracy_reward crash when called from non-main thread by @qgallouedec in https://github.com/huggingface/trl/pull/5281RewardFunc type alias to reflect actual calling convention by @s-zx in https://github.com/huggingface/trl/pull/5246prepare_multimodal_messages to support tool_calls and tool role by @alvarobartt in https://github.com/huggingface/trl/pull/5212AGENTS.md) by @qgallouedec in https://github.com/huggingface/trl/pull/5236environment_factory by @sergiopaniego in https://github.com/huggingface/trl/pull/5235.ai by @qgallouedec in https://github.com/huggingface/trl/pull/5268pad_to_multiple_of to GRPOTrainer and RLOOTrainer by @czkkkkkk in https://github.com/huggingface/trl/pull/5180prepare_multimodal_messages to support tool_calls and tool role by @alvarobartt in https://github.com/huggingface/trl/pull/5212AGENTS.md) by @qgallouedec in https://github.com/huggingface/trl/pull/5236prompts in vLLM client and server by @qgallouedec in https://github.com/huggingface/trl/pull/5225truncate_prompt_tokens for vLLM 0.17.0 by @winglian in https://github.com/huggingface/trl/pull/5248rollout_func from _generate_single_turn to _generate by @qgallouedec in https://github.com/huggingface/trl/pull/5232RewardFunc type alias to reflect actual calling convention by @s-zx in https://github.com/huggingface/trl/pull/5246_generate_single_turn by @qgallouedec in https://github.com/huggingface/trl/pull/5239_generate_single_turn by @qgallouedec in https://github.com/huggingface/trl/pull/5240bfd-requeue to bfd_split by @mariosasko in https://github.com/huggingface/trl/pull/5189vllm_mode to "colocate" and add v0→v1 migration guide by @qgallouedec in https://github.com/huggingface/trl/pull/5255grpo_trainer.py): Variational Sequence-Level Soft Policy Optimization (VESPO) by @casinca in https://github.com/huggingface/trl/pull/5199accuracy_reward crash when called from non-main thread by @qgallouedec in https://github.com/huggingface/trl/pull/5281hasattr and getattr with defaults in AGENTS.md by @qgallouedec in https://github.com/huggingface/trl/pull/5294RewardFunc type annotation to allow Nonevalues in reward list by @qgallouedec in https://github.com/huggingface/trl/pull/5297Json() type for tool calling dataset format by @lhoestq in https://github.com/huggingface/trl/pull/5307GRPOTrainer/async: fix prefix EOS slicing for tool suffix (with Qwen3/3.5 type of chat templates) by @casinca in https://github.com/huggingface/trl/pull/5330grpo_trainer.py by @casinca in https://github.com/huggingface/trl/pull/5332environment_factory by @sergiopaniego in https://github.com/huggingface/trl/pull/5235print_prompt_completions_sample to include reasoning content by @qgallouedec in https://github.com/huggingface/trl/pull/5327AGENTS.md by @qgallouedec in https://github.com/huggingface/trl/pull/5280pixel_position_ids vision key by @qgallouedec in https://github.com/huggingface/trl/pull/5374.ai by @qgallouedec in https://github.com/huggingface/trl/pull/5268apply_chat_template when it is an empty list by @rabinadk1 in https://github.com/huggingface/trl/pull/5380TRACKIO_SPACE_ID env var from all scripts by @sergiopaniego in https://github.com/huggingface/trl/pull/5365pr_template_check.yml by @qgallouedec in https://github.com/huggingface/trl/pull/5393disable_config=True from generate to GenerationConfig by @qgallouedec in https://github.com/huggingface/trl/pull/5384Full Changelog: https://github.com/huggingface/trl/compare/v0.29.0...v1.0.0
A `MIGRATION.md` guide has been added covering all breaking changes when upgrading from TRL v0 to v1. If you're already on v0.29, the changes are mini…
<img width="465" height="279" alt="Screenshot 2026-03-20 at 5 49 50 PM" src="https://github.com/user-attachments/assets/b60c9697-6eb7-498e-95b3-df78c367f5fa" />
VESPO addresses training instability in off-policy RL caused by policy staleness, asynchronous updates, and train-inference mismatches. Rather than relying on heuristic token-level clipping (GRPO) or sequence-length normalization (GSPO), VESPO derives a principled reshaping kernel from a variational framework. In practice, this yields a smooth, asymmetric Gamma weighting function that gracefully suppresses extreme sequence-level importance weights without introducing length bias. It can be enabled via the loss_type parameter of GRPOConfig:
from trl import GRPOConfig, GRPOTrainer
trainer = GRPOTrainer(
model="Qwen/Qwen3-0.6B",
args=GRPOConfig(loss_type="vespo"),
...
)
by @casinca in https://github.com/huggingface/trl/pull/5199
<img width="3180" height="1187" alt="z_TXYw37xZqsQ21YiDkYL" src="https://github.com/user-attachments/assets/40f1d538-82b3-4097-91c6-119ea9f7797b" /> <img width="1189" height="490" alt="SfgWotuuuRKPkg-0bxWv1" src="https://github.com/user-attachments/assets/2b090df3-0bfb-42e4-9f94-15943736e689" />
DPPO is a new experimental trainer that replaces the standard PPO clipping mechanism with divergence constraints, providing more principled trust-region updates.
by @LeonEricsson in https://github.com/huggingface/trl/pull/5117
Reward functions can return a dictionary of extra values (scalars or per-sample columns) that will be logged alongside the reward. This makes it easier to track intermediate signals without writing custom callbacks.
def my_reward_fn(completions, answer, log_extra=None, log_metric=None, **kwargs):
extracted = [extract_answer(c) for c in completions]
rewards = [1.0 if e == a else 0.0 for e, a in zip(extracted, answer)]
if log_extra:
log_extra("golden_answer", list(answer))
log_extra("extracted_answer", extracted)
if log_metric:
log_metric("accuracy", sum(rewards) / len(rewards))
return rewards
<img width="1400" height="407" alt="image" src="https://github.com/user-attachments/assets/d345b0ac-0d3c-446f-9321-a26e73ee16b4" /> <img width="1353" height="673" alt="image" src="https://github.com/user-attachments/assets/b4c0302b-f69a-4715-9aad-278b4ad13299" />
by @manueldeprada in https://github.com/huggingface/trl/pull/5233
VLLMClient.chat()VLLMClient.chat() now supports tool calling, enabling agentic workflows directly through the vLLM client interface.
by @kansalaman in https://github.com/huggingface/trl/pull/4889
BFD packing is 35% faster. The "bfd-requeue" packing strategy has also been renamed to "bfd_split". See MIGRATION.md for details.
<img width="1784" height="732" alt="benchmark_results" src="https://github.com/user-attachments/assets/8f0a35ad-cf1a-4fe1-a1f4-9b102637bdca" />
by @mariosasko in https://github.com/huggingface/trl/pull/5189
The GKD/GOLD trainer now supports buffered rollout generation, decoupling generation from gradient updates for more efficient distillation.
by @cmpatino in https://github.com/huggingface/trl/pull/5137
A MIGRATION.md guide has been added covering all breaking changes when upgrading from TRL v0 to v1. If you're already on v0.29, the changes are minimal.
by @qgallouedec in https://github.com/huggingface/trl/pull/5255
vllm_mode to "colocate" by @qgallouedec in https://github.com/huggingface/trl/pull/5255truncation_mode in SFT by @albertvillanova in https://github.com/huggingface/trl/pull/5306max_length in DPO VLM training by @albertvillanova in https://github.com/huggingface/trl/pull/5284pad_to_multiple_of to GRPOTrainer and RLOOTrainer by @czkkkkkk in https://github.com/huggingface/trl/pull/5180accuracy_reward crash when called from non-main thread by @qgallouedec in https://github.com/huggingface/trl/pull/5281RewardFunc type alias to reflect actual calling convention by @s-zx in https://github.com/huggingface/trl/pull/5246prepare_multimodal_messages to support tool_calls and tool role by @alvarobartt in https://github.com/huggingface/trl/pull/5212AGENTS.md) by @qgallouedec in https://github.com/huggingface/trl/pull/5236pad_to_multiple_of to GRPOTrainer and RLOOTrainer by @czkkkkkk in https://github.com/huggingface/trl/pull/5180prepare_multimodal_messages to support tool_calls and tool role by @alvarobartt in https://github.com/huggingface/trl/pull/5212AGENTS.md) by @qgallouedec in https://github.com/huggingface/trl/pull/5236prompts in vLLM client and server by @qgallouedec in https://github.com/huggingface/trl/pull/5225truncate_prompt_tokens for vLLM 0.17.0 by @winglian in https://github.com/huggingface/trl/pull/5248rollout_func from _generate_single_turn to _generate by @qgallouedec in https://github.com/huggingface/trl/pull/5232RewardFunc type alias to reflect actual calling convention by @s-zx in https://github.com/huggingface/trl/pull/5246_generate_single_turn by @qgallouedec in https://github.com/huggingface/trl/pull/5239_generate_single_turn by @qgallouedec in https://github.com/huggingface/trl/pull/5240bfd-requeue to bfd_split by @mariosasko in https://github.com/huggingface/trl/pull/5189vllm_mode to "colocate" and add v0→v1 migration guide by @qgallouedec in https://github.com/huggingface/trl/pull/5255grpo_trainer.py): Variational Sequence-Level Soft Policy Optimization (VESPO) by @casinca in https://github.com/huggingface/trl/pull/5199accuracy_reward crash when called from non-main thread by @qgallouedec in https://github.com/huggingface/trl/pull/5281hasattr and getattr with defaults in AGENTS.md by @qgallouedec in https://github.com/huggingface/trl/pull/5294RewardFunc type annotation to allow Nonevalues in reward list by @qgallouedec in https://github.com/huggingface/trl/pull/5297Json() type for tool calling dataset format by @lhoestq in https://github.com/huggingface/trl/pull/5307Full Changelog: https://github.com/huggingface/trl/compare/v0.29.0...v1.0.0rc1
Handle mm_token_type_ids in SFT/GRPO/RLOO to fix IndexError by @albertvillanova in https://github.com/huggingface/trl/pull/5178
prepare_multimodal_messages to support tool_calls and tool role by @alvarobartt in https://github.com/huggingface/trl/pull/5212prompts in vLLM client and server by @qgallouedec in https://github.com/huggingface/trl/pull/5225rollout_func from _generate_single_turn to _generate by @qgallouedec in https://github.com/huggingface/trl/pull/5232_generate_single_turn by @qgallouedec in https://github.com/huggingface/trl/pull/5239_generate_single_turn by @qgallouedec in https://github.com/huggingface/trl/pull/5240Full Changelog: https://github.com/huggingface/trl/compare/v0.29.0...v0.29.1
Remove deprecated BCO after moved to experimental by @albertvillanova in https://github.com/huggingface/trl/pull/5045
environment_factory to GRPOTrainerGRPOTrainer now accepts an environment_factory argument, allowing users to specify a custom environment class for training. This enables more flexible and diverse training scenarios by letting users define their own environments with specific dynamics and reward structures.
from datasets import Dataset
from trl import GRPOConfig, GRPOTrainer
dataset = Dataset.from_dict({
"prompt": [[{"role": "user", "content": f"Increment the counter by {i}."}] for i in range(1, 7)]
})
def reward_func(environments, **kwargs):
return [env.counter for env in environments]
class IncrementEnv:
def reset(self):
self.counter = 0
def increment(self, step: int) -> int:
"""
Increment the internal counter.
Args:
step: Value to add to the counter.
Returns:
The updated counter value.
"""
self.counter += step
return self.counter
trainer = GRPOTrainer(
model="Qwen/Qwen3-0.6B",
args=GRPOConfig(chat_template_kwargs={"enable_thinking": False}),
train_dataset=dataset,
reward_funcs=reward_func,
environment_factory=IncrementEnv,
)
trainer.train()
by @qgallouedec in https://github.com/huggingface/trl/pull/5093
TRL introduces agent-native CLI Integration: trl-training, a first-class Agent Skill that exposes TRL’s training workflows (SFT, DPO, GRPO, etc.) in a structured, agent-readable format. The skill is packaged directly with the trl library and can be installed via the CLI:
# Install into the project's agent directory (default scope=project), by agent name: claude, codex, opencode
trl skills install trl-training --target <agent>
This enables AI agents to safely and reproducibly execute TRL training workflows using a well-defined interface.
Skills can be installed at the project or global scope, and support explicit targets and overwrite controls.
launch_args for all trainers by @qgallouedec in https://github.com/huggingface/trl/pull/5059num_labels to 1 in causal model initialization for RewardTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/5066None in get_trackio_space_url() to prevent errors by @qgallouedec in https://github.com/huggingface/trl/pull/5115trl <command> --help TypeError caused by unescaped % in TrainingArguments help strings by @albertvillanova in https://github.com/huggingface/trl/pull/5135SFTTrainer support for single-image data by @qgallouedec in https://github.com/huggingface/trl/pull/5132grpo_trainer.md by @casinca in https://github.com/huggingface/trl/pull/5047RewardTrainer collator from chosen/rejected_input_ids to chosen/rejected_ids by @qgallouedec in https://github.com/huggingface/trl/pull/5179RewardTrainer tests by @qgallouedec in https://github.com/huggingface/trl/pull/5060get_training_chat_template by @qgallouedec in https://github.com/huggingface/trl/pull/5108train_dataset is required by @albertvillanova in https://github.com/huggingface/trl/pull/5171grpo_trainer.md by @casinca in https://github.com/huggingface/trl/pull/5047launch_args for all trainers by @qgallouedec in https://github.com/huggingface/trl/pull/5059num_labels to 1 in causal model initialization for RewardTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/5066RewardTrainer tests by @qgallouedec in https://github.com/huggingface/trl/pull/5060get_training_chat_template by @qgallouedec in https://github.com/huggingface/trl/pull/5108None in get_trackio_space_url() to prevent errors by @qgallouedec in https://github.com/huggingface/trl/pull/5115trl <command> --help TypeError caused by unescaped % in TrainingArguments help strings by @albertvillanova in https://github.com/huggingface/trl/pull/5135environment_factory to GRPOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/5093SFTTrainer support for single-image data by @qgallouedec in https://github.com/huggingface/trl/pull/5132train_dataset is required by @albertvillanova in https://github.com/huggingface/trl/pull/5171RewardTrainer collator from chosen/rejected_input_ids to chosen/rejected_ids by @qgallouedec in https://github.com/huggingface/trl/pull/5179Full Changelog: https://github.com/huggingface/trl/compare/v0.28.0...v0.29.0
Remove deprecated parameters by @qgallouedec in https://github.com/huggingface/trl/pull/4847
is_conversational by @qgallouedec in https://github.com/huggingface/trl/pull/4923openenv/utils.py: fallback for no vLLM installed case by @Datta0 in https://github.com/huggingface/trl/pull/4868current_gradient_accumulation_steps by @qgallouedec in https://github.com/huggingface/trl/pull/4852get_open_port based on vLLM version by @qgallouedec in https://github.com/huggingface/trl/pull/4883device_map init consistency in GRPO/RLOO/KTO by @qgallouedec in https://github.com/huggingface/trl/pull/4909warnings_issued by @qgallouedec in https://github.com/huggingface/trl/pull/4960DPOConfig by @qgallouedec in https://github.com/huggingface/trl/pull/4969warmup_ratio with warmup_steps by @qgallouedec in https://github.com/huggingface/trl/pull/4983RewardTrainer, RLOOTrainer and GRPOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/4823TestGRPOTrainer.test_training_vlm_and_liger and update version checks by @qgallouedec in https://github.com/huggingface/trl/pull/4898compute_metrics in SFTTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/4950RewardTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/4959compute_metrics in RewardTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/4958CITATION.cff by @qgallouedec in https://github.com/huggingface/trl/pull/4856DataCollatorForVisionLanguageModeling by @qgallouedec in https://github.com/huggingface/trl/pull/4911max_length in RewardConfig and SFTConfig by @qgallouedec in https://github.com/huggingface/trl/pull/4910sync_ref_model in GRPOTrainer and RLOOTrainer when using PEFT models by @qgallouedec in https://github.com/huggingface/trl/pull/4912⬆️ Bump dev version by @qgallouedec in https://github.com/huggingface/trl/pull/4835
Support triggering CI via push to ci-* branches by @albertvillanova in https://github.com/huggingface/trl/pull/4840
Revert CI hotfix pinning transformers 4.57.4 after tiny model regeneration by @albertvillanova in https://github.com/huggingface/trl/pull/4833
Use pytest-datadir in CI tests by @albertvillanova in https://github.com/huggingface/trl/pull/4836
Refactor KTO coordinated with DPO [c/N]: Remove ref_model_init_kwargs by @albertvillanova in https://github.com/huggingface/trl/pull/4837
Fix _patch_transformers_hybrid_cache for peft by @albertvillanova in https://github.com/huggingface/trl/pull/4844
Refactor KTO [4/N]: Remove unused padding_value by @albertvillanova in https://github.com/huggingface/trl/pull/4839
Remove unused padding_value from BCO by @albertvillanova in https://github.com/huggingface/trl/pull/4846
Fix CI with dev dependencies: Mark Qwen3-VL tests as xfail by @albertvillanova in https://github.com/huggingface/trl/pull/4851
Fix: undefined current_gradient_accumulation_steps by @qgallouedec in https://github.com/huggingface/trl/pull/4852
Remove deprecated parameters by @qgallouedec in https://github.com/huggingface/trl/pull/4847
Add Nash Learning from Human Feedback paper to paper index by @kansalaman in https://github.com/huggingface/trl/pull/4860
Use pytest-datadir for accelerate config files by @albertvillanova in https://github.com/huggingface/trl/pull/4861
Update OpenEnv dependency to new version for hf jobs scripts by @sergiopaniego in https://github.com/huggingface/trl/pull/4843
Update CITATION.cff by @qgallouedec in https://github.com/huggingface/trl/pull/4856
[GRPOTrainer]: Agent Training Supports Async Tool Calls by @pramodith in https://github.com/huggingface/trl/pull/4742
Enhance GRPO documentation with scaling notes by @javadtaghia in https://github.com/huggingface/trl/pull/4849
Add retry strategy to vLLM Client for increased robustness by @apalmas-saifh in https://github.com/huggingface/trl/pull/4845
Update generate_tiny_models.py: CohereForAI -> CohereLabs by @Michellehbn in https://github.com/huggingface/trl/pull/4877
Refactor KTO coordinated with DPO [e/N]: Remove label_pad_token_id by @albertvillanova in https://github.com/huggingface/trl/pull/4875
Refactor KTO coordinated with DPO [d/N]: Remove base_model_attribute_name by @albertvillanova in https://github.com/huggingface/trl/pull/4862
Fix type hint in openenv/utils.py: fallback for no vLLM installed case by @Datta0 in https://github.com/huggingface/trl/pull/4868
Update transformer version checks and documentation for lr_scheduler_kwargs workaround by @qgallouedec in https://github.com/huggingface/trl/pull/4876
fix(DeepSeek OPSM): passing correct (vLLM) logprobs by @casinca in https://github.com/huggingface/trl/pull/4857
Remove label_pad_token_id from experimental trainers by @albertvillanova in https://github.com/huggingface/trl/pull/4878
Fix SFT training for prompt-completion type and transformers v5 by @qgallouedec in https://github.com/huggingface/trl/pull/4880
Bugfix: Logprob drift in vLLM serving mode (compared to colocate mode) by @kdubovikov in https://github.com/huggingface/trl/pull/4873
Enable vLLM sleep mode for generation in Online DPO by @winglian in https://github.com/huggingface/trl/pull/4882
Test distributed training for RewardTrainer, RLOOTrainer and GRPOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/4823
Mark ZeRO 2 as xfail in distributed tests due to current failure by @qgallouedec in https://github.com/huggingface/trl/pull/4885
Fix import path for get_open_port based on vLLM version by @qgallouedec in https://github.com/huggingface/trl/pull/4883
Fix RewardTrainer's results not reproducible by @liyc-ai in https://github.com/huggingface/trl/pull/4887
GOLD training speed up by @141forever in https://github.com/huggingface/trl/pull/4888
Transformers v5 release: extend xfail condition for TestGRPOTrainer.test_training_vlm_and_liger and update version checks by @qgallouedec in https://github.com/huggingface/trl/pull/4898
Fix CI NotImplementedError for bfloat16 by @albertvillanova in https://github.com/huggingface/trl/pull/4902
Fix CI AssertionError: Parameter has not changed by @albertvillanova in https://github.com/huggingface/trl/pull/4904
Refactor vLLM generation [1/N]: Extract vLLM generation by @albertvillanova in https://github.com/huggingface/trl/pull/4700
Created new PTT integration docs as requested by @adityachallapally in https://github.com/huggingface/trl/pull/4907
Fix CI TypeError in llm-blender tests by @albertvillanova in https://github.com/huggingface/trl/pull/4919
Rearrange variable assignments in DataCollatorForVisionLanguageModeling by @qgallouedec in https://github.com/huggingface/trl/pull/4911
Fix help text formatting for max_length in RewardConfig and SFTConfig by @qgallouedec in https://github.com/huggingface/trl/pull/4910
device_map init consistency in GRPO/RLOO/KTO by @qgallouedec in https://github.com/huggingface/trl/pull/4909
Comment about overriding prediction_step in GRPOTrainer and RLOOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/4913
Remove gradient checkpointing option from various training scripts by @qgallouedec in https://github.com/huggingface/trl/pull/4905
docs: add DoRA (2402.09353) to Paper Index by @billycrapediem in https://github.com/huggingface/trl/pull/4892
Fix CI AssertionError: assert not True by @albertvillanova in https://github.com/huggingface/trl/pull/4921
Fix CI ValueError for 0 temperature by @albertvillanova in https://github.com/huggingface/trl/pull/4916
Fix extra EOS appended in DPO preprocessing for conversational data by @qgallouedec in https://github.com/huggingface/trl/pull/4908
Remove chat template setup in dpo_vlm.py by @qgallouedec in https://github.com/huggingface/trl/pull/4906
Update learning rate comments and add assertions for reference model parameters in GRPO and RLOO tests by @qgallouedec in https://github.com/huggingface/trl/pull/4914
Add validation for sync_ref_model in GRPOTrainer and RLOOTrainer when using PEFT models by @qgallouedec in https://github.com/huggingface/trl/pull/4912
Support tool call data in is_conversational by @qgallouedec in https://github.com/huggingface/trl/pull/4923
Set model dtype to float32 in tests of trainers by @albertvillanova in https://github.com/huggingface/trl/pull/4924
Require transformers<5 with PairRMJudge by @albertvillanova in https://github.com/huggingface/trl/pull/4926
Move VLLMClient to generation module by @albertvillanova in https://github.com/huggingface/trl/pull/4928
Set model dtype to float32 in experimental tests of trainers by @albertvillanova in https://github.com/huggingface/trl/pull/4925
Fix profiling of VLLMGeneration.sync_weights by @albertvillanova in https://github.com/huggingface/trl/pull/4931
Fix import statement for import_utils in vllm_client.py by @qgallouedec in https://github.com/huggingface/trl/pull/4932
Set default top_k to 0 in VLLMClient by @albertvillanova in https://github.com/huggingface/trl/pull/4927
[GRPO] Add parquet logging for completions with individual rewards by @qgallouedec in https://github.com/huggingface/trl/pull/4818
Fix SFTTrainer init logic: remove TrainingArguments.push_to_hub_token only for transformers < v5 by @albertvillanova in https://github.com/huggingface/trl/pull/4942
Remove ref_model_init_kwargs from experimental BCO by @albertvillanova in https://github.com/huggingface/trl/pull/4946
Update wordle.py example with masking of env tokens by @sergiopaniego in https://github.com/huggingface/trl/pull/4895
Fix PPO run_name parameter not taking effect by @mel3c in https://github.com/huggingface/trl/pull/4945
Minor fix docs style by @albertvillanova in https://github.com/huggingface/trl/pull/4953
Add test for training with compute_metrics in SFTTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/4950
Remove access to warnings_issued by @qgallouedec in https://github.com/huggingface/trl/pull/4960
NeMo-Gym Integration by @cmunley1 in https://github.com/huggingface/trl/pull/4848
Add test for tool call data in RewardTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/4959
Add test for training with compute_metrics in RewardTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/4958
Remove max_prompt_length from experimental PRM by @albertvillanova in https://github.com/huggingface/trl/pull/4963
Remove max_prompt_length from experimental BCO by @albertvillanova in https://github.com/huggingface/trl/pull/4964
Remove max_prompt_length from experimental CPO by @albertvillanova in https://github.com/huggingface/trl/pull/4965
Remove max_prompt_length from experimental ORPO by @albertvillanova in https://github.com/huggingface/trl/pull/4966
Revert change in GRPO from NeMo-Gym Integration by @qgallouedec in https://github.com/huggingface/trl/pull/4970
Fix test_train_with_chat_template_kwargs by @qgallouedec in https://github.com/huggingface/trl/pull/4971
Remove padding_value from experimental CPO and use pad_token_id by @albertvillanova in https://github.com/huggingface/trl/pull/4962
Remove truncation from tokenizer calls if no max_length by @albertvillanova in https://github.com/huggingface/trl/pull/4972
Set specific OpenEnv version when installed by @sergiopaniego in https://github.com/huggingface/trl/pull/4978
Fix add_column in test_train_with_chat_template_kwargs by @albertvillanova in https://github.com/huggingface/trl/pull/4979
Support truncated completions in GRPO multi-turn training by @albertvillanova in https://github.com/huggingface/trl/pull/4976
Replace torch.allclose with torch.testing.assert_close by @qgallouedec in https://github.com/huggingface/trl/pull/4977
Simplify instructions of installation of OpenEnv by @sergiopaniego in https://github.com/huggingface/trl/pull/4980
Deprecate parameters in DPOConfig by @qgallouedec in https://github.com/huggingface/trl/pull/4969
[CI] Disallow installation of transformers 5.1.0 due to compatibility issues with DeepSpeed by @qgallouedec in https://github.com/huggingface/trl/pull/4982
Replace warmup_ratio with warmup_steps by @qgallouedec in https://github.com/huggingface/trl/pull/4983
Pin transformers!=5.1.0 in deepspeed extra due to incompatibility by @albertvillanova in https://github.com/huggingface/trl/pull/4985
Fix passing tokenizer in test_train_with_chat_template_kwargs by @albertvillanova in https://github.com/huggingface/trl/pull/4987
Update dataset configuration name in toolcall dataset loading by @qgallouedec in https://github.com/huggingface/trl/pull/4984
Use local variable instead of attribute in collator tests by @qgallouedec in https://github.com/huggingface/trl/pull/4957
Fix import of AutoModelForCausalLMWithValueHead from experimental by @albertvillanova in https://github.com/huggingface/trl/pull/4990
Assert chat_template is applied in test_train_with_chat_template_kwargs by @albertvillanova in https://github.com/huggingface/trl/pull/4991
Fix deprecation of DPOConfig.max_completion_length by @albertvillanova in https://github.com/huggingface/trl/pull/4992
Fix post_init warning stacklevel to 3 by @albertvillanova in https://github.com/huggingface/trl/pull/4993
Fix ZeRO-3 + PEFT + gradient checkpointing by @qgallouedec in https://github.com/huggingface/trl/pull/4951
Add GitHub Actions workflow for testing against Transformers branch by @qgallouedec in https://github.com/huggingface/trl/pull/4995
Add distributed smoke tests workflow for Transformers branch by @qgallouedec in https://github.com/huggingface/trl/pull/4996
Update NeMo-Gym to use env_mask by @cmunley1 in https://github.com/huggingface/trl/pull/4986
Update sampling mode to token level for safety by @sergiopaniego in https://github.com/huggingface/trl/pull/4989
perf: Qwen SAPO loss optimization by @casinca in https://github.com/huggingface/trl/pull/4956
Fix GRPO tool calling for corrupted tool calls by @akshayballal95 in https://github.com/huggingface/trl/pull/4890
Add sanitize_logprob function for NaN handling in vLLM log probabilities by @qgallouedec in https://github.com/huggingface/trl/pull/5001
[tests] Remove xfail for transformers version >= 5.0.0 due to upstream bug resolution by @qgallouedec in https://github.com/huggingface/trl/pull/5000
docs: add CGPO/Mixture of Judges (2409.20370) to Paper Index + link ref to AllTrueJudge by @nabin2004 in https://github.com/huggingface/trl/pull/5002
Filter CI SWIG deprecation warnings by @albertvillanova in https://github.com/huggingface/trl/pull/5004
Fix CI TRLExperimentalWarning in regular tests by @albertvillanova in https://github.com/huggingface/trl/pull/5007
Add support for nested_gather in OnlineDPOTrainer for transformers v5.2.0 and above by @qgallouedec in https://github.com/huggingface/trl/pull/4981
Fix CI FutureWarning: ref_model_init_kwargs is deprecated by @albertvillanova in https://github.com/huggingface/trl/pull/5009
Fix typo in DPO max_prompt_length deprecation warning message by @albertvillanova in https://github.com/huggingface/trl/pull/5020
Fix vision model prompt truncation bug in DPOTrainer by @albertvillanova in https://github.com/huggingface/trl/pull/5023
Pin transformers < 5 in judges extra due to incompatibility by @albertvillanova in https://github.com/huggingface/trl/pull/5024
Fix CI FutureWarning: generate_during_eval is deprecated by @albertvillanova in https://github.com/huggingface/trl/pull/5017
Fix typo in xfail test reason by @albertvillanova in https://github.com/huggingface/trl/pull/5028
Fix CI FutureWarning: rpo_alpha is deprecated by @albertvillanova in https://github.com/huggingface/trl/pull/5011
Fix CI FutureWarning: use_logits_to_keep is deprecated by @albertvillanova in https://github.com/huggingface/trl/pull/5013
Mark Qwen3VL tests as xfail for transformers 5.0.x by @albertvillanova in https://github.com/huggingface/trl/pull/5029
[CI] Silence PyTorch JIT and DataLoader deprecation warnings by @qgallouedec in https://github.com/huggingface/trl/pull/4999
Add length-unbiased GRPO loss (LUSPO) by @Haseebasif7 in https://github.com/huggingface/trl/pull/4988
Fix CI FutureWarning: tools is deprecated by @albertvillanova in https://github.com/huggingface/trl/pull/5015
Filter max_prompt_length UserWarning in all test cases by @albertvillanova in https://github.com/huggingface/trl/pull/5035
Fix CI FutureWarning: max_prompt_length is deprecated by @albertvillanova in https://github.com/huggingface/trl/pull/5019
Allow testing with transformers 5.1.0 via xfail marks by @albertvillanova in https://github.com/huggingface/trl/pull/5034
Rename AOT loss type 'aot_pair' to 'aot_unpaired' in DPO by @qgallouedec in https://github.com/huggingface/trl/pull/5038
Deprecate string usage for ref_model in DPOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/5040
Deprecate FDivergenceType in DPOConfig; update f_divergence_type to use string values by @qgallouedec in https://github.com/huggingface/trl/pull/5039
Fix multiprocessing start method to 'spawn' for test compatibility with Python 3.12+ by @qgallouedec in https://github.com/huggingface/trl/pull/5036
Add Online Direct Preference Optimization section to paper index by @qgallouedec in https://github.com/huggingface/trl/pull/5037
Release: 0.28 by @albertvillanova in https://github.com/huggingface/trl/pull/5043
Full Changelog: https://github.com/huggingface/trl/compare/v0.27.0...v0.28.0
Remove access to warnings_issued by @qgallouedec in #4960
warnings_issued by @qgallouedec in #4960Full Changelog: https://github.com/huggingface/trl/compare/v0.27.1...v0.27.2
Fix: undefined current_gradient_accumulation_steps by @qgallouedec in https://github.com/huggingface/trl/pull/4852
current_gradient_accumulation_steps by @qgallouedec in https://github.com/huggingface/trl/pull/4852Full Changelog: https://github.com/huggingface/trl/compare/v0.27.0...v0.27.1
Remove deprecated functions and parameters by @qgallouedec in https://github.com/huggingface/trl/pull/4651
vllm_group_port argument to GRPO, RLOO and OnlineDPO configuration by @pointerhacker in https://github.com/huggingface/trl/pull/4545forward_masked_logits function by @qgallouedec in https://github.com/huggingface/trl/pull/4729AutoModelForCausalLMWithValueHead and AutoModelForSeq2SeqLMWithValueHead to experimental by @qgallouedec in https://github.com/huggingface/trl/pull/4654experimental.utils by @qgallouedec in https://github.com/huggingface/trl/pull/4667DataCollatorForChatML to experimental.utils by @qgallouedec in https://github.com/huggingface/trl/pull/4668add_bos_token_if_needed and add_eos_token_if_needed to experimental.utils by @qgallouedec in https://github.com/huggingface/trl/pull/4674truncate_right and SIMPLE_CHAT_TEMPLATE to experimental.utils by @qgallouedec in https://github.com/huggingface/trl/pull/4677prepare_model_for_kbit_training, enable_gradient_checkpointing, prepare_peft_model to experimental.utils by @qgallouedec in https://github.com/huggingface/trl/pull/4704get_reward function to experimental.utils by @qgallouedec in https://github.com/huggingface/trl/pull/4683num_generations_eval=1 in the calculation of the advantage by @qgallouedec in https://github.com/huggingface/trl/pull/4662num_generations_eval is specified and different than num_generations by @apalmas-saifh in https://github.com/huggingface/trl/pull/4682generation_config for tiny model uploads by @qgallouedec in https://github.com/huggingface/trl/pull/4643qwen3_schema by @mattbui in https://github.com/huggingface/trl/pull/4709HybridCache in Liger-Kernel with transformers v5 by @qgallouedec in https://github.com/huggingface/trl/pull/4798args by @carlyou in https://github.com/huggingface/trl/pull/4801grpo_trainer.md): Added Qwen SAPO details under Loss Types by @casinca in https://github.com/huggingface/trl/pull/4681MergeModelCallback from import structure by @qgallouedec in https://github.com/huggingface/trl/pull/4664ChatMlSpecialTokens by @qgallouedec in https://github.com/huggingface/trl/pull/4666_win_rate_completions_df function from callbacks by @qgallouedec in https://github.com/huggingface/trl/pull/4672DbrxForCausalLM support by @qgallouedec in https://github.com/huggingface/trl/pull/4799compute_accuracy to PRM Trainer file by @qgallouedec in https://github.com/huggingface/trl/pull/4656clone_chat_template to chat_template_utils by @qgallouedec in https://github.com/huggingface/trl/pull/4653GeometricMixtureWrapper to nash_md_trainer.py by @qgallouedec in https://github.com/huggingface/trl/pull/4670exact_div, print_rich_table, truncate_response, forward to ppo_trainer by @qgallouedec in https://github.com/huggingface/trl/pull/4676OnPolicyConfig and PPOConfig and move OnlineTrainerState by @qgallouedec in https://github.com/huggingface/trl/pull/4671AutoModelForCausalLMWithValueHead to test_ppo_trainer by @qgallouedec in https://github.com/huggingface/trl/pull/4678generate and batch_generation to ppo_trainer.py by @qgallouedec in https://github.com/huggingface/trl/pull/4675TrainerCallback from top-level transformers by @qgallouedec in https://github.com/huggingface/trl/pull/4694top_k parameter in OnlineDPOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/4714PeftModel + peft_config in trainers by @qgallouedec in https://github.com/huggingface/trl/pull/4713TestParseResponse by @qgallouedec in https://github.com/huggingface/trl/pull/4736GuidedDecodingParams with StructuredOutputsParams in sampling parameter configuration by @qgallouedec in https://github.com/huggingface/trl/pull/4797--
Full Changelog: https://github.com/huggingface/trl/compare/v0.26.0...v0.27.0
Overwrite model default generation config used by model.generate by @albertvillanova in https://github.com/huggingface/trl/pull/4647
Full Changelog: https://github.com/huggingface/trl/compare/v0.26.1...v0.26.2
Fix vLLM error for tools usage not supported when running GRPO training by @apalmas-saifh in https://github.com/huggingface/trl/pull/4663
num_generations_eval is specified and different than num_generations by @apalmas-saifh in https://github.com/huggingface/trl/pull/4682Full Changelog: https://github.com/huggingface/trl/compare/v0.26.0...v0.26.1
Remove deprecations for 0.26 release by @albertvillanova in https://github.com/huggingface/trl/pull/4607
GRPOTrainer now supports training agents using tools. This allows language models to interact with external functions or APIs during training.
from datasets import Dataset
from trl import GRPOTrainer
def multiply(a: int, b: int) -> int:
"""
Multiplies two integers.
Args:
a: The first integer.
b: The second integer.
Returns:
The product of the two integers.
"""
return a * b
dataset = Dataset.from_list(
[
{"prompt": [{"role": "user", "content": "What is 3 multiplied by 4?"}], "answer": 12},
{"prompt": [{"role": "user", "content": "Calculate 7 times 8."}], "answer": 56},
{"prompt": [{"role": "user", "content": "Find the product of 5 and 6."}], "answer": 30},
{"prompt": [{"role": "user", "content": "What do you get when you multiply 9 by 9?"}], "answer": 81},
{"prompt": [{"role": "user", "content": "Compute 12 multiplied by 11."}], "answer": 132},
{"prompt": [{"role": "user", "content": "What is 15 times 14?"}], "answer": 210},
]
)
def accuracy(completions, answer, **kwargs):
predictions = [completion[-1]["content"] for completion in completions]
rewards = [float(str(ans) in pred) for pred, ans in zip(predictions, answer)]
return rewards
trainer = GRPOTrainer(
model="Qwen/Qwen3-0.6B",
train_dataset=dataset,
tools=[multiply],
reward_funcs=accuracy,
)
trainer.train()
by @qgallouedec in https://github.com/huggingface/trl/pull/4300
CISPO Loss was first introduced in the Minimax-M1 paper, the ScaleRL paper subsequently showed that CISPO loss scales the best in terms of performance and efficiency as models are trained for longer.
GRPOTrainer now supports the CISPO loss using loss_type="cispo" in the GRPOConfig.
by @pramodith in https://github.com/huggingface/trl/pull/4495
When the input model is quantized using bitsandbytes, vLLM will now also use quantization when in colocate mode.
by @sergiopaniego in https://github.com/huggingface/trl/pull/4496
TRL nows includes a reasoning reward function
from trl.rewards import reasoning_accuracy_reward
solutions = [r"\frac{1}{3}", r"\frac{1}{3}", r"\frac{1}{3}"]
completions = [
[
{
"role": "assistant",
"content": r"<think> Reasoning content </think> The final answer is \boxed{\frac{1}{3}}",
}
],
[
{
"role": "assistant",
"content": r"<think> Reasoning content </think> The final answer is \boxed{\frac{1}{2}}",
}
],
[
{
"role": "assistant",
"content": r"<think> Reasoning content with partial answers \boxed{\frac{1}{3}} but no final answer",
}
],
]
reasoning_accuracy_reward(completions, solutions) # [1.0, 0.0, 0.0]
As any other reward function, it can be used in GRPOTrainer or RLOOTrainer.
from trl import GRPOTrainer
from trl.rewards import reasoning_accuracy_reward
trainer = GRPOTrainer(
...,
reward_funcs=reasoning_accuracy_reward,
)
by @lewtun in https://github.com/huggingface/trl/pull/4563
shuffle_dataset option to SFTTrainerYou can now shuffle the dataset in SFTTrainer by setting the shuffle_dataset argument to True in SFTConfig. This is useful when the dataset features high similarity between consecutive samples.
from trl import SFTTrainer, SFTConfig
SFTConfig(shuffle_dataset=True)
by @qgallouedec in https://github.com/huggingface/trl/pull/4564
Soft Adaptive Policy Optimization (SAPO), replaces hard clipping with a smooth, temperature-controlled gate that adaptively attenuates off-policy updates while preserving useful learning signals. Compared with GSPO and GRPO, SAPO is both sequence-coherent and token-adaptive. Like GSPO, SAPO maintains sequence-level coherence, but its soft gating forms a continuous trust region that avoids the brittle hard clipping band used in GSPO.
You can now use SAPO loss in GRPOTrainer by setting loss_type="sapo" in the GRPOConfig.
by @pramodith in https://github.com/huggingface/trl/pull/4600
num_generations_eval parameter for efficient evaluation by @mingxuetian in https://github.com/huggingface/trl/pull/4458WinRateCallback to experimental by @qgallouedec in https://github.com/huggingface/trl/pull/4558device_map and dtype to "auto" by default by @qgallouedec in https://github.com/huggingface/trl/pull/4509flash-attn to flash-attn2 by @qgallouedec in https://github.com/huggingface/trl/pull/4514device_map=None for DeepSpeed and add ZeRO paper (1910.02054) to Paper Index by @JenWei0312 in https://github.com/huggingface/trl/pull/4551num_completions to num_generations by @pramodith in https://github.com/huggingface/trl/pull/4515rnj_1_instruct notebook by @sergiopaniego in https://github.com/huggingface/trl/pull/4646wandb_log_unique_prompts with log_unique_prompts by @taha-yassine in https://github.com/huggingface/trl/pull/4508prepare_model_for_kbit_training by @sergiopaniego in https://github.com/huggingface/trl/pull/4457lr_scheduler_kwargs dtype issue in Transformers 4.57.0 by @qgallouedec in https://github.com/huggingface/trl/pull/4513device_map and dtype to "auto" by default by @qgallouedec in https://github.com/huggingface/trl/pull/4509wandb_log_unique_prompts with log_unique_prompts by @taha-yassine in https://github.com/huggingface/trl/pull/4508num_completions to num_generations by @pramodith in https://github.com/huggingface/trl/pull/4515flash-attn to flash-attn2 by @qgallouedec in https://github.com/huggingface/trl/pull/4514prepare_model_for_kbit_training by @sergiopaniego in https://github.com/huggingface/trl/pull/4457device_map=None for DeepSpeed and add ZeRO paper (1910.02054) to Paper Index by @JenWei0312 in https://github.com/huggingface/trl/pull/4551num_generations_eval parameter for efficient evaluation by @mingxuetian in https://github.com/huggingface/trl/pull/4458shuffle_dataset option to SFTTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/4564WinRateCallback to experimental by @qgallouedec in https://github.com/huggingface/trl/pull/4558rnj_1_instruct notebook by @sergiopaniego in https://github.com/huggingface/trl/pull/4646Full Changelog: https://github.com/huggingface/trl/compare/v0.25.0...v0.26.0
Replace accelerate logging with stdlib in CLI by @lewtun in https://github.com/huggingface/trl/pull/4512
lr_scheduler_kwargs dtype issue in Transformers 4.57.0 by @qgallouedec in https://github.com/huggingface/trl/pull/4513Full Changelog: https://github.com/huggingface/trl/compare/v0.25.0...0.25.1
Replace deprecated AutoModelForVision2Seq with AutoModelForImageTextToText by @albertvillanova in https://github.com/huggingface/trl/pull/4353
prepare_model_for_kbit_training to save VRAM by @sergiopaniego in https://github.com/huggingface/trl/pull/4335add_generation_prompt to processor_kwargs in GRPO and RLOO trainer by @qgallouedec in https://github.com/huggingface/trl/pull/4361trl.experimental by @qgallouedec in https://github.com/huggingface/trl/pull/4312add_generation_prompt=True for conversational only by @qgallouedec in https://github.com/huggingface/trl/pull/4362max_length explanation for VLM in online trainers by @sergiopaniego in https://github.com/huggingface/trl/pull/4220_generate in GRPO/RLOO: Move forward_kwargs outside generation method by @qgallouedec in https://github.com/huggingface/trl/pull/4154trl.experimental by @qgallouedec in https://github.com/huggingface/trl/pull/4312_generate in GRPO/RLOO: Insert images in the prompt by @qgallouedec in https://github.com/huggingface/trl/pull/4155prepare_model_for_kbit_training to save VRAM by @sergiopaniego in https://github.com/huggingface/trl/pull/4335add_generation_prompt to processor_kwargs in GRPO and RLOO trainer by @qgallouedec in https://github.com/huggingface/trl/pull/4361add_generation_prompt=True for conversational only by @qgallouedec in https://github.com/huggingface/trl/pull/4362max_length explanation for VLM in online trainers by @sergiopaniego in https://github.com/huggingface/trl/pull/4220Full Changelog: https://github.com/huggingface/trl/compare/v0.24.0...v0.25.0
⚖️ Align SFT and DPO for model creation and deprecate DPOConfig.padding_value in favour or pad_token_id by @qgallouedec in https://github.com/huggingf…
token_type_ids in DPOTrainer by @aweers in https://github.com/huggingface/trl/pull/4285RichProgressCallback enhancement by @qgallouedec in https://github.com/huggingface/trl/pull/4245chat_template_kwargs in apply_chat_template by @cmpatino in https://github.com/huggingface/trl/pull/4233token_type_ids in DataCollatorForVisionLanguageModeling by @qgallouedec in https://github.com/huggingface/trl/pull/4190RewardTrainer refactor by @qgallouedec in https://github.com/huggingface/trl/pull/4093clone_chat_template by @qgallouedec in https://github.com/huggingface/trl/pull/4097vllm_enable_sleep_mode to RLOO Trainer by @sergiopaniego in https://github.com/huggingface/trl/pull/4107batchmean reduce op in GKDTrainer's loss by @cmpatino in https://github.com/huggingface/trl/pull/4105get_high_entropy_mask by @akakakakakaa in https://github.com/huggingface/trl/pull/4041DPOConfig.padding_value in favour or pad_token_id by @qgallouedec in https://github.com/huggingface/trl/pull/4006BestOfNSampler by @qgallouedec in https://github.com/huggingface/trl/pull/4291AlignPropTrainer, DDPOTrainer and IterativeSFTTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/4068trl.experimental Submodule by @August-murr in https://github.com/huggingface/trl/pull/4073make_parser function in multiple scripts by @qgallouedec in https://github.com/huggingface/trl/pull/4050get_high_entropy_mask by @akakakakakaa in https://github.com/huggingface/trl/pull/4041trl.experimental Submodule by @August-murr in https://github.com/huggingface/trl/pull/4073AlignPropTrainer, DDPOTrainer and IterativeSFTTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/4068set to list of tags by @qgallouedec in https://github.com/huggingface/trl/pull/4092DPOConfig.padding_value in favour or pad_token_id by @qgallouedec in https://github.com/huggingface/trl/pull/4006batchmean reduce op in GKDTrainer's loss by @cmpatino in https://github.com/huggingface/trl/pull/4105image_split_sizes in favour of image_grid_thw by @qgallouedec in https://github.com/huggingface/trl/pull/4111vllm_enable_sleep_mode to RLOO Trainer by @sergiopaniego in https://github.com/huggingface/trl/pull/4107backend parameter from GuidedDecodingParams by @qgallouedec in https://github.com/huggingface/trl/pull/4123max_batch_tokens, num_blocks and block_size from generation kwargs by @qgallouedec in https://github.com/huggingface/trl/pull/4065_generate by @qgallouedec in https://github.com/huggingface/trl/pull/4114image_split_sizes in favour of image_grid_thw by @qgallouedec in https://github.com/huggingface/trl/pull/4156_generate for GRPO with replay buffer by @qgallouedec in https://github.com/huggingface/trl/pull/4158<Tip> with new markdown syntax by @qgallouedec in https://github.com/huggingface/trl/pull/4161require_bitsandbytes by @qgallouedec in https://github.com/huggingface/trl/pull/4137clone_chat_template by @qgallouedec in https://github.com/huggingface/trl/pull/4097RewardTrainer refactor by @qgallouedec in https://github.com/huggingface/trl/pull/4093_generate in GRPO/RLOO: list of ints instead of tensors by @qgallouedec in https://github.com/huggingface/trl/pull/4146trainer.tokenizer by trainer.processing_class by @qgallouedec in https://github.com/huggingface/trl/pull/4185sft example script by @sergiopaniego in https://github.com/huggingface/trl/pull/4197Optional from processing_class in PPOTrainer by @sergiopaniego in https://github.com/huggingface/trl/pull/4212sft_video_llm example by @qgallouedec in https://github.com/huggingface/trl/pull/4214trl-internal-testing/tiny-DbrxForCausalLM by @qgallouedec in https://github.com/huggingface/trl/pull/4213_generate in GRPO/RLOO: Use prompt_ids from generation by @qgallouedec in https://github.com/huggingface/trl/pull/4152token_type_ids in DataCollatorForVisionLanguageModeling by @qgallouedec in https://github.com/huggingface/trl/pull/4190_generate in GRPO/RLOO: Rely on generator for prompt truncation by @qgallouedec in https://github.com/huggingface/trl/pull/4153chat_template_kwargs in apply_chat_template by @cmpatino in https://github.com/huggingface/trl/pull/4233RichProgressCallback enhancement by @qgallouedec in https://github.com/huggingface/trl/pull/4245token_type_ids in DPOTrainer by @aweers in https://github.com/huggingface/trl/pull/4285BestOfNSampler by @qgallouedec in https://github.com/huggingface/trl/pull/4291Full Changelog: https://github.com/huggingface/trl/compare/v0.23.0...v0.24.0
♨️ [GRPO] Fix potential hang in get_high_entropy_mask by @akakakakakaa in https://github.com/huggingface/trl/pull/4041
get_high_entropy_mask by @akakakakakaa in https://github.com/huggingface/trl/pull/4041Full Changelog: https://github.com/huggingface/trl/compare/v0.23.0...v0.23.1
SFT now supports Context Parallelism (CP) for training large language models on very large sequences. You can now train with an arbitrarily long seque
SFT now supports Context Parallelism (CP) for training large language models on very large sequences. You can now train with an arbitrarily long sequence length.
<img width="844" height="336" alt="Screenshot 2025-09-09 at 10 39 30 PM" src="https://github.com/user-attachments/assets/f1dfc349-440a-4e05-aac9-439a3c286f08" />
by @kashif in https://github.com/huggingface/trl/pull/3994
Dynamic Fine-Tuning (DFT) is a nnow supported in TRL.
from trl import SFTConfig
training_args = SFTConfig(
loss_type="dft",
...
)
<img width="692" height="472" alt="Screenshot 2025-09-09 at 10 37 36 PM" src="https://github.com/user-attachments/assets/4ee2b4ab-7cc6-4578-bfac-c38124891510" />
by @qgallouedec in https://github.com/huggingface/trl/pull/4042
Different implementations are used for rollout generation (vLLM) and model training. The implementation gap implicitly turns the on-policy RL to be off-policy. Truncated Importance Sampling (TIS) a simple yet effective importance sampling technique for handling such discrepancy. This is now implemented in GRPO.
from trl import GRPOConfig
training_args = GRPOConfig(
...
use_vllm=True,
vllm_importance_sampling_correction=True, # default True
vllm_importance_sampling_cap=2.0, # hyper-parameter C
)
by @LeonEricsson in https://github.com/huggingface/trl/pull/3867
Mixture of Experts (MoE) models require an auxiliary loss to ensure that the different experts are used evenly. This auxiliary loss is now supported in SFTTrainer.
training_args = SFTConfig(
model_init_kwargs={"output_router_logits": True},
...
)
by @pramodith in https://github.com/huggingface/trl/pull/4012
When running GRPO (or RLOO) with vLLM in colocated mode, the vLLM server consume VRAM during optimization while not being used. We now have an option to put the vLLM server to sleep during optimization to free up VRAM.
from trl import GRPOConfig
training_args = GRPOConfig(..., vllm_sleep_enabled=True)
by @edbeeching in https://github.com/huggingface/trl/pull/3968
You can now use vLLM server mode with OnlineDPOTrainer. Additionally, VLM models are now supported.
by @vaelev in https://github.com/huggingface/trl/pull/3783
The paper index has been significantly enhanced with the addition of 9+ new algorithm implementations, providing a more comprehensive resource for users.
by @behroozazarkhalili in https://github.com/huggingface/trl/pull/3990
get_soft_overlong_punishment by @qgallouedec in https://github.com/huggingface/trl/pull/3972args.gradient_checkpointing = False instead of args = dataclasses.replace(args, gradient_checkpointing=False) by @qgallouedec in https://github.com/huggingface/trl/pull/3981torch_dype to dtype everywhere by @sergiopaniego in https://github.com/huggingface/trl/pull/4000average_tokens_across_devices default replacement by @qgallouedec in https://github.com/huggingface/trl/pull/4039AutoModelForImageTextToText for DPO and Online DPO by @sergiopaniego in https://github.com/huggingface/trl/pull/4049SFTTrainer for compatibility with CP by @qgallouedec in https://github.com/huggingface/trl/pull/4038quantization_config=None by @qgallouedec in https://github.com/huggingface/trl/pull/4019Full Changelog: https://github.com/huggingface/trl/compare/v0.22.0...v0.23.0
⚖️ Fix scale_rewards issue in GRPO by @Peter-Chou in https://github.com/huggingface/trl/pull/3992
Full Changelog: https://github.com/huggingface/trl/compare/v0.22.1...v0.22.2
Refactor version retrieval to use importlib.metadata by @qgallouedec
importlib.metadata by @qgallouedecFull Changelog: https://github.com/huggingface/trl/compare/v0.22.0...v0.22.1
🗑️ Deprecate setup_chat_format by @qgallouedec in https://github.com/huggingface/trl/pull/3929
SFTTrainerSFTTrainer now natively supports Vision-Language Models (VLMs). This includes support for both languauge modeling, prompt-completion data.
It also supports training on completion-only.
<img width="1136" height="586" alt="Group 291-6" src="https://github.com/user-attachments/assets/2629b8e7-d853-4b7c-91d5-f4c128287e04" />
from trl import SFTConfig, SFTTrainer
from datasets import load_dataset
trainer = SFTTrainer(
model="Qwen/Qwen2.5-VL-3B-Instruct",
args=SFTConfig(max_length=None),
train_dataset=load_dataset("trl-lib/llava-instruct-mix", split="train"),
)
trainer.train()
by @qgallouedec in https://github.com/huggingface/trl/pull/3862, https://github.com/huggingface/trl/pull/3907 and https://github.com/huggingface/trl/pull/3908
RLOOTrainer refactorRLOOTrainer has been refactored to align with the design principles of other other trainers in the library. You can now use this trainer exactly like GRPO.
from datasets import load_dataset
from trl import RLOOConfig, RLOOTrainer
dataset = load_dataset("trl-lib/ultrafeedback-prompt", split="train")
# Dummy reward function for demonstration purposes
def reward_num_unique_letters(completions, **kwargs):
"""Reward function that rewards completions with more unique letters."""
completion_contents = [completion[0]["content"] for completion in completions]
return [float(len(set(content))) for content in completion_contents]
trainer = RLOOTrainer(
model="Qwen/Qwen2-0.5B-Instruct",
reward_funcs=reward_num_unique_letters,
train_dataset=dataset,
)
trainer.train()
by @shirinyamani in https://github.com/huggingface/trl/pull/3801
You can now levarage Hugging Face Jobs to easily train and deploy your models with TRL.
hf jobs uv run --flavor a100-large --secrets HF_TOKEN "https://raw.githubusercontent.com/huggingface/trl/main/trl/scripts/sft.py" --model_name_or_path Qwen/Qwen2-0.5B --dataset_name trl-lib/Capybara
A guide is available in the docs.
by @sergiopaniego in https://github.com/huggingface/trl/pull/3890
GRPOTrainer now supports DAPO loss type, which aggregates token-level losses by normalizing with the number of active token in the global accumulated batch. This method was introduced to eliminate length bias. Simply use
from trl import GRPOConfig, GRPOTrainer
training_args = GRPOConfig(
loss_type="dapo",
...
)
by @qgallouedec in https://github.com/huggingface/trl/pull/3938
The authors of Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoning (Lite PPO) find that the combination of:
can unlock the learning capability of critic-free policies using vanilla PPO loss. Their results demonstrate that this simple combination consistently improves performance, surpassing strategies like GRPO and DAPO.
TRL supports using these learnings to train a GRPO model by:
from trl import GRPOConfig
training_args = GRPOConfig(
scale_rewards="batch",
loss_type="dapo",
...
)
by @pramodith in https://github.com/huggingface/trl/pull/3935
Bias-Corrected Exponential Moving Average (BEMA) improves the stability and efficiency of language model fine-tuning by reducing stochasticity and eliminating bias. To use BEMA with SFT as described in the paper, you can now use the [BEMACallback]:
from trl import BEMACallback, SFTTrainer
trainer = SFTTrainer(
...
callbacks=[BEMACallback()],
)
by @kashif in https://github.com/huggingface/trl/pull/3855
gradient_checkpointing=True by @qgallouedec in https://github.com/huggingface/trl/pull/3510position_ids for flash_attention_3 by @jue-jue-zi in https://github.com/huggingface/trl/pull/3942setup_chat_format by @qgallouedec in https://github.com/huggingface/trl/pull/3929IterativeSFTTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/3905AutoModelForVision2Seq to AutoModelForImageTextToText by @qgallouedec in https://github.com/huggingface/trl/pull/3836--bf16 value in scripts by @sergiopaniego in https://github.com/huggingface/trl/pull/3869gradient_checkpointing=True by @qgallouedec in https://github.com/huggingface/trl/pull/3510vllm_mode param in GRPO by @sergiopaniego in https://github.com/huggingface/trl/pull/3866unittest.TestCase with TrlTestCase that handles tmp dir by @qgallouedec in https://github.com/huggingface/trl/pull/3863SFTTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/3862IterativeSFTTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/3905use_cache should be set in the forward pass by @qgallouedec in https://github.com/huggingface/trl/pull/3891logger.warning instead of warnings.warn by @qgallouedec in https://github.com/huggingface/trl/pull/3923SFTTrainer in GRPOTrainer by @MengAiDev in https://github.com/huggingface/trl/pull/3919setup_chat_format by @qgallouedec in https://github.com/huggingface/trl/pull/3929position_ids for flash_attention_3 by @jue-jue-zi in https://github.com/huggingface/trl/pull/3942trackio to docs by @sergiopaniego in https://github.com/huggingface/trl/pull/3965Full Changelog: https://github.com/huggingface/trl/compare/v0.21.0...v0.22.0
One column per quarter.