NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
PyPI · #2346 most downloaded on PyPI
Train transformer language models with reinforcement learning.
Last release 4 days ago
25 Sep 2026
Ships on a steady schedule
a new release about every 2 weeks
Nearly every release is documented
notes for 60 of the last 60 stable releases
Nothing withdrawn
no release was ever pulled
6 years old
102 releases · first in 2020
One column per quarter.
TRL v1.3 ships training support for the new Qwen 3.6 family (Qwen/Qwen3.6-27B, Qwen/Qwen3.6-35B-A3B). Qwen 3.6 reuses the Qwen3_5Moe* architecture but
<img width="1536" height="1024" alt="ChatGPT Image Apr 26, 2026 at 11_16_18 AM" src="https://github.com/user-attachments/assets/789aad15-03b2-4ece-9828-d5c1dfed1f1e" />
TRL v1.3 ships training support for the new Qwen 3.6 family (Qwen/Qwen3.6-27B, Qwen/Qwen3.6-35B-A3B). Qwen 3.6 reuses the Qwen3_5Moe* architecture but ships a slightly different chat template (adds a preserve_thinking flag, tweaks tool-arg stringification), so exact-string template matching needed updates across the stack.
What landed:
qwen3_6.jinja (verbatim from upstream) and qwen3_6_training.jinja (prefix-preserving + {% generation %} markers for assistant_only_loss=True)qwen3_5_schema for tool-call parsing — output format unchangedtiny-Qwen3_5MoeForConditionalGeneration-3.6 (with MoE-specific shrinking)test_(train|training)_vlm casesfrom trl import SFTConfig, SFTTrainer
trainer = SFTTrainer(
model="Qwen/Qwen3.6-27B",
args=SFTConfig(assistant_only_loss=True), # works out of the box
train_dataset=dataset,
)
trainer.train()
Tool-calling agent training also works end-to-end via the existing Qwen 3.5 response schema:
from trl import GRPOConfig, GRPOTrainer
def multiply(a: int, b: int) -> int:
"""
Multiplies two integers.
Args:
a: The first integer.
b: The second integer.
Returns:
The product of the two integers.
"""
return a * b
trainer = GRPOTrainer(
model="Qwen/Qwen3.6-27B",
reward_funcs=my_reward_fn,
args=GRPOConfig(...),
train_dataset=dataset,
tools=[multiply],
)
trainer.train()
by @qgallouedec in https://github.com/huggingface/trl/pull/5642
<img width="711" height="177" alt="Screenshot 2026-04-26 at 11 37 28 AM" src="https://github.com/user-attachments/assets/6090212e-5c95-45c1-b137-87333d91daa6" />
A new experimental TPOTrainer implements Triple Preference Optimization, which augments DPO with a reference (gold) completion alongside chosen/rejected. The paper reports +7-19 points over DPO/SimPO on Arena-Hard, MixEval-Hard, MMLU-Pro and GSM8K, with less data.
from trl.experimental.tpo import TPOConfig, TPOTrainer
trainer = TPOTrainer(
model="Qwen/Qwen3-0.6B",
args=TPOConfig(output_dir="Qwen3-0.6B-TPO"),
train_dataset=load_dataset("tpo-alignment/triple-preference-ultrafeedback-40K", split="train"),
)
trainer.train()
by @kashif in https://github.com/huggingface/trl/pull/5506
trl vllm-serveA new --speculative_config JSON flag exposes vLLM's speculative decoding directly through trl vllm-serve — works with native MTP heads (Qwen3 Next), Eagle3 drafts, etc. — without forking the serve script.
# Qwen3 native MTP (no extra draft model)
trl vllm-serve --model Qwen/Qwen3-Next-80B-A3B-Instruct \
--speculative_config '{"method": "qwen3_next_mtp", "num_speculative_tokens": 5}'
# Eagle3 draft model
trl vllm-serve --model Qwen/Qwen3-32B \
--speculative_config '{"model": "RedHatAI/Qwen3-32B-speculator.eagle3", "method": "eagle3", "num_speculative_tokens": 3}'
by @Ofir408 in https://github.com/huggingface/trl/pull/5605
Twelve more alignment PRs this cycle, bringing KTOTrainer and DPOTrainer essentially into structural parity. Notable shifts include moving completion assembly out of _prepare_dataset into a new DataCollatorForKTO, inlining the two-pass tokenization into a single pass, removing BOS/EOS handling, and supporting IterableDataset and dict eval_dataset. The goal — promoting KTO out of experimental and into stable — is now within reach for an upcoming release.
PRs (all by @albertvillanova): #5582, #5578, #5579, #5583, #5587, #5599, #5601, #5600, #5606, #5612, #5632, #5635
{% generation %} training chat templatesThree more model families gain training-compatible chat templates with {% generation %} markers, so assistant_only_loss=True works out of the box:
maybe_apply_chat_template by @albertvillanova in https://github.com/huggingface/trl/pull/5567is_chat_template_prefix_preserving by @qgallouedec in https://github.com/huggingface/trl/pull/5558forward_masked_logits by @qgallouedec in https://github.com/huggingface/trl/pull/5626_tokenizer as trainer attribute by @albertvillanova in https://github.com/huggingface/trl/pull/5489PreTrainedTokenizerBase for tokenizer type hints by @qgallouedec in https://github.com/huggingface/trl/pull/5629async_reward_X to async_X by @qgallouedec in https://github.com/huggingface/trl/pull/5616attention_mask instead of label != -100), and wrong cross-rank aggregation (unweighted mean instead of sum/count). The reported entropy under completion_only_loss=True and sequence parallelism is now correct. Same fix applied to DPO entropy logging. By @qgallouedec in https://github.com/huggingface/trl/pull/5620AsyncGRPOTrainer's processing_class to AsyncRolloutWorker by @xuanduy04 in https://github.com/huggingface/trl/pull/5538generate_tiny_models for gpt-oss by @albertvillanova in https://github.com/huggingface/trl/pull/5622TestSupportsToolCalling for improved coverage by @qgallouedec in https://github.com/huggingface/trl/pull/5537is_chat_template_prefix_preserving by @qgallouedec in https://github.com/huggingface/trl/pull/5558TestSupportsToolCalling for improved coverage by @qgallouedec in https://github.com/huggingface/trl/pull/5537{% generation %} markers for training chat template by @casinca in https://github.com/huggingface/trl/pull/5519forward_masked_logits by @qgallouedec in https://github.com/huggingface/trl/pull/5626PreTrainedTokenizerBase for tokenizer type hints by @qgallouedec in https://github.com/huggingface/trl/pull/5629async_reward_X to async_X by @qgallouedec in https://github.com/huggingface/trl/pull/5616Full Changelog: https://github.com/huggingface/trl/compare/v1.2.0...v1.3.0
Deprecate `use_transformers_paged` in GRPOConfig and RLOOConfig (and remove entirely from experimental OnlineDPOConfig, GOLDConfig, SelfDistillationCo…
SSDTrainer — Simple Self-Distillation<img width="778" height="334" alt="Screenshot 2026-04-16 at 9 08 04 PM" src="https://github.com/user-attachments/assets/8ca223f0-6740-48a8-967c-ec10cb262a93" />
A new experimental SSDTrainer implements the method described in Embarrassingly Simple Self-Distillation Improves Code Generation. SSD samples completions from the model itself at a training-time temperature/truncation setting, then fine-tunes on those raw, unverified samples with standard cross-entropy loss. No reward model, verifier, teacher model, or RL: just prompts and the model.
from datasets import Dataset
from trl.experimental.ssd import SSDConfig, SSDTrainer
dataset = Dataset.from_dict({
"prompt": [
[{"role": "user", "content": "Write a function to add two numbers."}],
[{"role": "user", "content": "Write a function to check if a number is prime."}],
],
})
trainer = SSDTrainer(
model="Qwen/Qwen3-4B-Instruct",
args=SSDConfig(
output_dir="ssd-model",
temperature=0.6, # T_train from the paper
top_k=20,
top_p=0.95,
learning_rate=5e-6,
),
train_dataset=dataset,
)
trainer.train()
by @kashif in https://github.com/huggingface/trl/pull/5505
GRPOTrainerWhen tool calls produce more tokens than max_completion_length allows, GRPOTrainer now rolls back the tool messages/images added in the current iteration instead of trying to truncate them. This removes ~80 lines of fragile, image-boundary-aware bookkeeping in favor of a ~15-line snapshot-and-rollback. Since overlong samples almost always get rewarded as failures anyway, the learning signal is effectively unchanged — but the code is dramatically simpler and no longer needs per-VLM-family vision-token lookup tables.
by @qgallouedec in https://github.com/huggingface/trl/pull/5521
Continuing the effort from v1.1:
{% generation %} markers, enabling assistant-only loss masking for DeepSeek-V3 models. By @RudrenduPaul in https://github.com/huggingface/trl/pull/5527As a result of a tightened detection (see fixes below), the list of templates reported as tool-calling capable is now correct — notably, the basic Llama 3 template is no longer falsely classified as tool-calling capable.
A major cleanup sweep keeps KTOTrainer and DPOTrainer in lockstep, same initialization patterns, same config surface, same precompute behavior:
precompute_ref_batch_size to KTO (https://github.com/huggingface/trl/pull/5530)ref_model initialization (https://github.com/huggingface/trl/pull/5534)None args (https://github.com/huggingface/trl/pull/5531)generate_during_eval (https://github.com/huggingface/trl/pull/5551)ref_model when precompute_ref_log_probs is set in DPO/KTO (https://github.com/huggingface/trl/pull/5542)All by @albertvillanova.
prepare_multimodal_messages by @albertvillanova in https://github.com/huggingface/trl/pull/5474prepare_multimodal_messages by @albertvillanova in https://github.com/huggingface/trl/pull/5508supports_tool_calling falsely accepting templates that drop assistant tool_calls by @qgallouedec in https://github.com/huggingface/trl/pull/5517add_response_schema for VLM processors — the schema was being set on the outer processor instead of the inner tokenizer, so it had no effect. This also collapses a handful of __init__/decode-gate workarounds. By @qgallouedec in https://github.com/huggingface/trl/pull/5520use_transformers_paged in GRPOConfig and RLOOConfig (and remove entirely from experimental OnlineDPOConfig, GOLDConfig, SelfDistillationConfig). Will be removed from the remaining configs in v2.0.0. In a small A/B benchmark (Qwen3-0.6B GRPO), the paged path is ~20% slower and uses ~6x more peak VRAM than the default; it's also superseded by transformers continuous batching. By @qgallouedec in https://github.com/huggingface/trl/pull/5544chat_templates/README by @qgallouedec in https://github.com/huggingface/trl/pull/5545supports_tool_calling falsely accepting templates that drop assistant tool_calls by @qgallouedec in https://github.com/huggingface/trl/pull/5517use_transformers_paged by @qgallouedec in https://github.com/huggingface/trl/pull/5544add_response_schema for VLM processors by @qgallouedec in https://github.com/huggingface/trl/pull/5520chat_templates/README by @qgallouedec in https://github.com/huggingface/trl/pull/5545Full Changelog: https://github.com/huggingface/trl/compare/v1.1.0...v1.2.0
Fix SFT deprecation warning by @albertvillanova in https://github.com/huggingface/trl/pull/5466
DistillationTrainer for efficient on-policy distillationRead the blog post: https://huggingface.co/spaces/HuggingFaceTB/trl-distillation-trainer
The new DistillationTrainer implements on-policy knowledge distillation as described in On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes. It extends the ideas from the GKDTrainer with three key optimizations: a generation buffer that decouples the training microbatch size from the generation batch size (up to 40x speedup), external teacher server support so the teacher doesn't need to fit on training GPUs, and binary-encoded logprob payloads that shrink transfer payloads by ~5x.
from datasets import load_dataset
from trl.experimental.distillation import DistillationConfig, DistillationTrainer
dataset = load_dataset("openai/gsm8k", "main", split="train")
dataset = dataset.map(
lambda x: {"messages": [{"role": "user", "content": x["question"]}]},
remove_columns=dataset.column_names,
)
trainer = DistillationTrainer(
model="Qwen/Qwen2.5-1.5B-Instruct",
teacher_model="Qwen/Qwen2.5-7B-Instruct",
args=DistillationConfig(
output_dir="results/distill-qwen-gsm8k",
lmbda=1.0, # fully on-policy (student generates)
beta=1.0, # reverse KL
teacher_model_init_kwargs={"torch_dtype": "bfloat16"},
),
train_dataset=dataset,
)
trainer.train()
by @cmpatino in https://github.com/huggingface/trl/pull/5407, https://github.com/huggingface/trl/pull/5500 and https://github.com/huggingface/trl/pull/5501
AsyncGRPOTrainerAsyncGRPOTrainer now supports a chunked LM-head path that computes per-token log-probs and entropy via online logsumexp without materializing the full [N, V] logits tensor. Combined with completion_mask filtering to skip prompt tokens, this brings massive memory savings on long sequences — up to 44x lower peak-allocated memory on an 8192-token sequence:
chunk_lm_head_size |
Peak Alloc (GB) | Reduction | Wall Time (ms) |
|---|---|---|---|
None (baseline) |
18.55 | 1.00x | 808.7 |
4096 |
0.42 | 44.32x | 459.0 |
8192 |
0.76 | 24.34x | 393.0 |
Enable it via the new chunk_lm_head_size option in AsyncGRPOConfig:
from trl.experimental.async_grpo import AsyncGRPOConfig, AsyncGRPOTrainer
trainer = AsyncGRPOTrainer(
model="Qwen/Qwen2.5-0.5B-Instruct",
args=AsyncGRPOConfig(chunk_lm_head_size=4096),
...
)
Note: mutually exclusive with use_liger_kernel (both replace the LM head forward pass).
by @AmineDiro in https://github.com/huggingface/trl/pull/5349
{% generation %} support in training chat templatesSFT with assistant_only_loss=True requires chat templates to include {% generation %} / {% endgeneration %} markers so that return_assistant_tokens_mask=True produces correct masks. Very few models ship these markers natively, so users hit a cryptic error when enabling assistant-only loss with models like Qwen3, Llama 3 or GPT-OSS.
SFTTrainer now automatically swaps in a patched training chat template when the original template lacks generation markers — no manual template surgery required. Training templates are shipped for Qwen2.5, Qwen3, Llama 3 and GPT-OSS, stored as standalone .jinja files under trl/chat_templates/ for readability, diffability, and editor syntax highlighting.
from trl import SFTConfig, SFTTrainer
trainer = SFTTrainer(
model="Qwen/Qwen3-4B",
args=SFTConfig(assistant_only_loss=True), # now just works
train_dataset=dataset,
)
trainer.train()
by @qgallouedec in https://github.com/huggingface/trl/pull/5459, https://github.com/huggingface/trl/pull/5470, by @RudrenduPaul in https://github.com/huggingface/trl/pull/5493 and https://github.com/huggingface/trl/pull/5522, and by @casinca in https://github.com/huggingface/trl/pull/5484
Agent training now supports a broader family of models via native tool-call response schemas:
A new supports_tool_calling() utility detects whether a tokenizer/processor can render a full tool-calling turn, and GRPOTrainer now validates tool support at initialization — raising a clear error upfront instead of failing cryptically mid-training.
by @qgallouedec in https://github.com/huggingface/trl/pull/5462, https://github.com/huggingface/trl/pull/5464, https://github.com/huggingface/trl/pull/5463, https://github.com/huggingface/trl/pull/5469 and https://github.com/huggingface/trl/pull/5454
environment_factory tool methods can now return multimodal content blocks (images + text) for VLM training. Previously, tool responses were always converted to str(result), discarding any visual information. Now tools can return content block lists with images, and the trainer handles them end-to-end through tokenization, generation, and the forward pass — including correct pixel_values plumbing.
class ScreenshotEnv:
def take_screenshot(self) -> list[dict]:
return [
{"type": "image", "image": self.browser.screenshot()},
{"type": "text", "text": "Current page state"},
]
The OpenEnv browsergym.py example has been migrated to this pattern, and a new carla_vlm.py example demonstrates VLM training against CARLA with camera-image tool responses.
by @sergiopaniego in https://github.com/huggingface/trl/pull/5323 and https://github.com/huggingface/trl/pull/5437, and by @qgallouedec in https://github.com/huggingface/trl/pull/5448
accuracy_reward and reasoning_accuracy_reward now emit extra diagnostic columns (solution, gold_parsed, answer_parsed) via the log_extra callback introduced in v1.0.0. These show up in the rich completions table, making it much easier to debug why a reward was (or wasn't) assigned.
<img width="1627" alt="accuracy reward with extra columns" src="https://github.com/user-attachments/assets/d7f6e9c2-4d7b-4886-ba7a-f58f0ccfcb9b" />
from datasets import load_dataset
from trl import GRPOConfig, GRPOTrainer
from trl.rewards import accuracy_reward
dataset = load_dataset("trl-lib/DeepMath-103K", split="train")
trainer = GRPOTrainer(
model="Qwen/Qwen2-0.5B-Instruct",
reward_funcs=accuracy_reward,
args=GRPOConfig(log_completions=True),
train_dataset=dataset,
)
trainer.train()
by @qgallouedec in https://github.com/huggingface/trl/pull/5308
KTOConfig by @albertvillanova in https://github.com/huggingface/trl/pull/5477prepare_multimodal_messages by @albertvillanova in https://github.com/huggingface/trl/pull/5424prepare_multimodal_messages by @albertvillanova in https://github.com/huggingface/trl/pull/5475pixel_position_ids with image_position_ids for Gemma 4 support by @qgallouedec in https://github.com/huggingface/trl/pull/5452isinstance(part, dict) checks in image extraction by @qgallouedec in https://github.com/huggingface/trl/pull/5439_get_tool_suffix_ids by @qgallouedec in https://github.com/huggingface/trl/pull/5440prepare_deepspeed by @albertvillanova in https://github.com/huggingface/trl/pull/5414ImportError with vllm-0.10.2 in OnlineDPO and OpenEnv by @albertvillanova in https://github.com/huggingface/trl/pull/5423_get_per_token_logps_and_entropies return type by @kashif in https://github.com/huggingface/trl/pull/5456prepare_multimodal_messages not normalizing empty string content for assistant/tool roles by @albertvillanova in https://github.com/huggingface/trl/pull/5496pad_token_id by @albertvillanova in https://github.com/huggingface/trl/pull/5487huggingface-cli references with hf by @hanouticelina in https://github.com/huggingface/trl/pull/5486truncation_mode from experimental truncate_dataset by @albertvillanova in https://github.com/huggingface/trl/pull/5467keep_end truncation mode in DPOConfig and SFTConfig — will be removed in v2.0.0. Use keep_start instead. By @albertvillanova in https://github.com/huggingface/trl/pull/5465pad_token config parameter in DPOConfig, SFTConfig, and RewardConfig — will be removed in v2.0.0. Set tokenizer.pad_token directly on the processing_class instead. By @albertvillanova in https://github.com/huggingface/trl/pull/5480trl.experimental.judges module and all judge support from trainers. Judges were experimental, unused in practice, and llm-blender (backing PairRMJudge) was unmaintained and incompatible with transformers v5 — actively blocking v5 adoption. Everything judges did can be achieved with reward_funcs. OnlineDPOTrainer, NashMDTrainer, and XPOTrainer are now unified on reward-model scoring only. By @qgallouedec in https://github.com/huggingface/trl/pull/5485carla_vlm OpenEnv example by @sergiopaniego in https://github.com/huggingface/trl/pull/5437completion_only_loss in SFT trainer docs by @RudrenduPaul in https://github.com/huggingface/trl/pull/5494DistillationTrainer by @cmpatino in https://github.com/huggingface/trl/pull/5500prepare_multimodal_messages by @albertvillanova in https://github.com/huggingface/trl/pull/5476make precommit to fix docstring style by @albertvillanova in https://github.com/huggingface/trl/pull/5436test_rloo[fsdp2]: replace non-deterministic xfail with skipif for transformers 5.4.0 by @albertvillanova in https://github.com/huggingface/trl/pull/5403input_ids or inputs_embeds by @albertvillanova in https://github.com/huggingface/trl/pull/5422eval_strategy by @SunMarc in https://github.com/huggingface/trl/pull/5426TypeError: 'NoneType' object is not iterable by @albertvillanova in https://github.com/huggingface/trl/pull/5427TypeError: 'NoneType' object is not iterable by @albertvillanova in https://github.com/huggingface/trl/pull/5438test_rloo[fsdp2] after transformers 5.5.0 release by @albertvillanova in https://github.com/huggingface/trl/pull/5442eval_strategy by @SunMarc in https://github.com/huggingface/trl/pull/5426environment_factory for VLM training by @sergiopaniego in https://github.com/huggingface/trl/pull/5323isinstance(part, dict) checks in image extraction by @qgallouedec in https://github.com/huggingface/trl/pull/5439pixel_position_ids with image_position_ids for Gemma4 support by @qgallouedec in https://github.com/huggingface/trl/pull/5452_get_tool_suffix_ids by @qgallouedec in https://github.com/huggingface/trl/pull/5440.jinja files by @qgallouedec in https://github.com/huggingface/trl/pull/5459supports_tool_calling utility and validate tool support at init by @qgallouedec in https://github.com/huggingface/trl/pull/5462{% generation %} support to training chat templates by @qgallouedec in https://github.com/huggingface/trl/pull/5470DistillationTrainer for efficient on-policy distillation by @cmpatino in https://github.com/huggingface/trl/pull/5407huggingface-cli references with hf by @hanouticelina in https://github.com/huggingface/trl/pull/5486{% generation %} markers for training chat template by @casinca in https://github.com/huggingface/trl/pull/5484trl.experimental.judges module and all judge support from trainers by @qgallouedec in https://github.com/huggingface/trl/pull/5485DistillationTrainer by @cmpatino in https://github.com/huggingface/trl/pull/5501DistillationTrainer by @cmpatino in https://github.com/huggingface/trl/pull/5500Full Changelog: https://github.com/huggingface/trl/compare/v1.0.0...v1.1.0
A `MIGRATION.md` guide has been added covering all breaking changes when upgrading from TRL v0 to v1. If you're already on v0.29, the changes are mini…
<img width="1800" height="1013" alt="thumbnail-2" src="https://github.com/user-attachments/assets/5c55b86a-0600-4f70-bf37-41ab240af851" />
Read our blog post for an overview of TRL v1.
Asynchronous GRPO decouples generation from the gradient update loop by offloading rollouts to an external vLLM server. Generation runs in parallel while training continues, eliminating idle GPU time and improving hardware utilization.
from trl.experimental.async_grpo import AsyncGRPOTrainer
from trl.rewards import accuracy_reward
from datasets import load_dataset
dataset = load_dataset("trl-lib/DeepMath-103K", split="train")
trainer = AsyncGRPOTrainer(
model="Qwen/Qwen2.5-0.5B-Instruct",
reward_funcs=accuracy_reward,
train_dataset=dataset,
)
trainer.train()
by @qgallouedec in https://github.com/huggingface/trl/pull/5293
<img width="465" height="279" alt="Screenshot 2026-03-20 at 5 49 50 PM" src="https://github.com/user-attachments/assets/b60c9697-6eb7-498e-95b3-df78c367f5fa" />
VESPO addresses training instability in off-policy RL caused by policy staleness, asynchronous updates, and train-inference mismatches. Rather than relying on heuristic token-level clipping (GRPO) or sequence-length normalization (GSPO), VESPO derives a principled reshaping kernel from a variational framework. In practice, this yields a smooth, asymmetric Gamma weighting function that gracefully suppresses extreme sequence-level importance weights without introducing length bias. It can be enabled via the loss_type parameter of GRPOConfig:
from trl import GRPOConfig, GRPOTrainer
trainer = GRPOTrainer(
model="Qwen/Qwen3-0.6B",
args=GRPOConfig(loss_type="vespo"),
...
)
by @casinca in https://github.com/huggingface/trl/pull/5199
<img width="3180" height="1187" alt="z_TXYw37xZqsQ21YiDkYL" src="https://github.com/user-attachments/assets/40f1d538-82b3-4097-91c6-119ea9f7797b" /> <img width="1189" height="490" alt="SfgWotuuuRKPkg-0bxWv1" src="https://github.com/user-attachments/assets/2b090df3-0bfb-42e4-9f94-15943736e689" />
DPPO is a new experimental trainer that replaces the standard PPO clipping mechanism with divergence constraints, providing more principled trust-region updates.
by @LeonEricsson in https://github.com/huggingface/trl/pull/5117
SDPO is a new experimental trainer that augments on-policy RL with self-distillation from the model's own high-reward trajectories. Instead of using an external teacher, SDPO treats the current model conditioned on feedback as a self-teacher, distilling its feedback-informed predictions back into the policy.
from trl.experimental import SDPOTrainer, SDPOConfig
config = SDPOConfig(
output_dir="./results",
num_generations=8,
success_reward_threshold=1.0,
use_successful_as_teacher=True,
)
trainer = SDPOTrainer(
model="Qwen/Qwen2.5-Math-1.5B-Instruct",
reward_funcs=[accuracy_reward],
args=config,
train_dataset=dataset,
)
trainer.train()
by @MengAiDev in https://github.com/huggingface/trl/pull/4935
Reward functions can return a dictionary of extra values (scalars or per-sample columns) that will be logged alongside the reward. This makes it easier to track intermediate signals without writing custom callbacks.
def my_reward_fn(completions, answer, log_extra=None, log_metric=None, **kwargs):
extracted = [extract_answer(c) for c in completions]
rewards = [1.0 if e == a else 0.0 for e, a in zip(extracted, answer)]
if log_extra:
log_extra("golden_answer", list(answer))
log_extra("extracted_answer", extracted)
if log_metric:
log_metric("accuracy", sum(rewards) / len(rewards))
return rewards
<img width="1400" height="407" alt="image" src="https://github.com/user-attachments/assets/d345b0ac-0d3c-446f-9321-a26e73ee16b4" /> <img width="1353" height="673" alt="image" src="https://github.com/user-attachments/assets/b4c0302b-f69a-4715-9aad-278b4ad13299" />
by @manueldeprada in https://github.com/huggingface/trl/pull/5233
VLLMClient.chat()VLLMClient.chat() now supports tool calling, enabling agentic workflows directly through the vLLM client interface.
by @kansalaman in https://github.com/huggingface/trl/pull/4889
BFD packing is 35% faster. The "bfd-requeue" packing strategy has also been renamed to "bfd_split". See MIGRATION.md for details.
<img width="1784" height="732" alt="benchmark_results" src="https://github.com/user-attachments/assets/8f0a35ad-cf1a-4fe1-a1f4-9b102637bdca" />
by @mariosasko in https://github.com/huggingface/trl/pull/5189
The GKD/GOLD trainer now supports buffered rollout generation, decoupling generation from gradient updates for more efficient distillation. vLLM inference support has also been added to the base self-distillation trainer.
by @cmpatino in https://github.com/huggingface/trl/pull/5137 and https://github.com/huggingface/trl/pull/5388
A MIGRATION.md guide has been added covering all breaking changes when upgrading from TRL v0 to v1. If you're already on v0.29, the changes are minimal.
by @qgallouedec in https://github.com/huggingface/trl/pull/5255
vllm_mode to "colocate" by @qgallouedec in https://github.com/huggingface/trl/pull/5255truncation_mode in SFT by @albertvillanova in https://github.com/huggingface/trl/pull/5306max_length in DPO VLM training by @albertvillanova in https://github.com/huggingface/trl/pull/5284pad_to_multiple_of to GRPOTrainer and RLOOTrainer by @czkkkkkk in https://github.com/huggingface/trl/pull/5180print_prompt_completions_sample to include reasoning content by @qgallouedec in https://github.com/huggingface/trl/pull/5327pixel_position_ids vision key by @qgallouedec in https://github.com/huggingface/trl/pull/5374None to apply_chat_template when it is an empty list by @rabinadk1 in https://github.com/huggingface/trl/pull/5380accuracy_reward crash when called from non-main thread by @qgallouedec in https://github.com/huggingface/trl/pull/5281RewardFunc type alias to reflect actual calling convention by @s-zx in https://github.com/huggingface/trl/pull/5246prepare_multimodal_messages to support tool_calls and tool role by @alvarobartt in https://github.com/huggingface/trl/pull/5212AGENTS.md) by @qgallouedec in https://github.com/huggingface/trl/pull/5236environment_factory by @sergiopaniego in https://github.com/huggingface/trl/pull/5235.ai by @qgallouedec in https://github.com/huggingface/trl/pull/5268pad_to_multiple_of to GRPOTrainer and RLOOTrainer by @czkkkkkk in https://github.com/huggingface/trl/pull/5180prepare_multimodal_messages to support tool_calls and tool role by @alvarobartt in https://github.com/huggingface/trl/pull/5212AGENTS.md) by @qgallouedec in https://github.com/huggingface/trl/pull/5236prompts in vLLM client and server by @qgallouedec in https://github.com/huggingface/trl/pull/5225truncate_prompt_tokens for vLLM 0.17.0 by @winglian in https://github.com/huggingface/trl/pull/5248rollout_func from _generate_single_turn to _generate by @qgallouedec in https://github.com/huggingface/trl/pull/5232RewardFunc type alias to reflect actual calling convention by @s-zx in https://github.com/huggingface/trl/pull/5246_generate_single_turn by @qgallouedec in https://github.com/huggingface/trl/pull/5239_generate_single_turn by @qgallouedec in https://github.com/huggingface/trl/pull/5240bfd-requeue to bfd_split by @mariosasko in https://github.com/huggingface/trl/pull/5189vllm_mode to "colocate" and add v0→v1 migration guide by @qgallouedec in https://github.com/huggingface/trl/pull/5255grpo_trainer.py): Variational Sequence-Level Soft Policy Optimization (VESPO) by @casinca in https://github.com/huggingface/trl/pull/5199accuracy_reward crash when called from non-main thread by @qgallouedec in https://github.com/huggingface/trl/pull/5281hasattr and getattr with defaults in AGENTS.md by @qgallouedec in https://github.com/huggingface/trl/pull/5294RewardFunc type annotation to allow Nonevalues in reward list by @qgallouedec in https://github.com/huggingface/trl/pull/5297Json() type for tool calling dataset format by @lhoestq in https://github.com/huggingface/trl/pull/5307GRPOTrainer/async: fix prefix EOS slicing for tool suffix (with Qwen3/3.5 type of chat templates) by @casinca in https://github.com/huggingface/trl/pull/5330grpo_trainer.py by @casinca in https://github.com/huggingface/trl/pull/5332environment_factory by @sergiopaniego in https://github.com/huggingface/trl/pull/5235print_prompt_completions_sample to include reasoning content by @qgallouedec in https://github.com/huggingface/trl/pull/5327AGENTS.md by @qgallouedec in https://github.com/huggingface/trl/pull/5280pixel_position_ids vision key by @qgallouedec in https://github.com/huggingface/trl/pull/5374.ai by @qgallouedec in https://github.com/huggingface/trl/pull/5268apply_chat_template when it is an empty list by @rabinadk1 in https://github.com/huggingface/trl/pull/5380TRACKIO_SPACE_ID env var from all scripts by @sergiopaniego in https://github.com/huggingface/trl/pull/5365pr_template_check.yml by @qgallouedec in https://github.com/huggingface/trl/pull/5393disable_config=True from generate to GenerationConfig by @qgallouedec in https://github.com/huggingface/trl/pull/5384Full Changelog: https://github.com/huggingface/trl/compare/v0.29.0...v1.0.0
A `MIGRATION.md` guide has been added covering all breaking changes when upgrading from TRL v0 to v1. If you're already on v0.29, the changes are mini…
<img width="465" height="279" alt="Screenshot 2026-03-20 at 5 49 50 PM" src="https://github.com/user-attachments/assets/b60c9697-6eb7-498e-95b3-df78c367f5fa" />
VESPO addresses training instability in off-policy RL caused by policy staleness, asynchronous updates, and train-inference mismatches. Rather than relying on heuristic token-level clipping (GRPO) or sequence-length normalization (GSPO), VESPO derives a principled reshaping kernel from a variational framework. In practice, this yields a smooth, asymmetric Gamma weighting function that gracefully suppresses extreme sequence-level importance weights without introducing length bias. It can be enabled via the loss_type parameter of GRPOConfig:
from trl import GRPOConfig, GRPOTrainer
trainer = GRPOTrainer(
model="Qwen/Qwen3-0.6B",
args=GRPOConfig(loss_type="vespo"),
...
)
by @casinca in https://github.com/huggingface/trl/pull/5199
<img width="3180" height="1187" alt="z_TXYw37xZqsQ21YiDkYL" src="https://github.com/user-attachments/assets/40f1d538-82b3-4097-91c6-119ea9f7797b" /> <img width="1189" height="490" alt="SfgWotuuuRKPkg-0bxWv1" src="https://github.com/user-attachments/assets/2b090df3-0bfb-42e4-9f94-15943736e689" />
DPPO is a new experimental trainer that replaces the standard PPO clipping mechanism with divergence constraints, providing more principled trust-region updates.
by @LeonEricsson in https://github.com/huggingface/trl/pull/5117
Reward functions can return a dictionary of extra values (scalars or per-sample columns) that will be logged alongside the reward. This makes it easier to track intermediate signals without writing custom callbacks.
def my_reward_fn(completions, answer, log_extra=None, log_metric=None, **kwargs):
extracted = [extract_answer(c) for c in completions]
rewards = [1.0 if e == a else 0.0 for e, a in zip(extracted, answer)]
if log_extra:
log_extra("golden_answer", list(answer))
log_extra("extracted_answer", extracted)
if log_metric:
log_metric("accuracy", sum(rewards) / len(rewards))
return rewards
<img width="1400" height="407" alt="image" src="https://github.com/user-attachments/assets/d345b0ac-0d3c-446f-9321-a26e73ee16b4" /> <img width="1353" height="673" alt="image" src="https://github.com/user-attachments/assets/b4c0302b-f69a-4715-9aad-278b4ad13299" />
by @manueldeprada in https://github.com/huggingface/trl/pull/5233
VLLMClient.chat()VLLMClient.chat() now supports tool calling, enabling agentic workflows directly through the vLLM client interface.
by @kansalaman in https://github.com/huggingface/trl/pull/4889
BFD packing is 35% faster. The "bfd-requeue" packing strategy has also been renamed to "bfd_split". See MIGRATION.md for details.
<img width="1784" height="732" alt="benchmark_results" src="https://github.com/user-attachments/assets/8f0a35ad-cf1a-4fe1-a1f4-9b102637bdca" />
by @mariosasko in https://github.com/huggingface/trl/pull/5189
The GKD/GOLD trainer now supports buffered rollout generation, decoupling generation from gradient updates for more efficient distillation.
by @cmpatino in https://github.com/huggingface/trl/pull/5137
A MIGRATION.md guide has been added covering all breaking changes when upgrading from TRL v0 to v1. If you're already on v0.29, the changes are minimal.
by @qgallouedec in https://github.com/huggingface/trl/pull/5255
vllm_mode to "colocate" by @qgallouedec in https://github.com/huggingface/trl/pull/5255truncation_mode in SFT by @albertvillanova in https://github.com/huggingface/trl/pull/5306max_length in DPO VLM training by @albertvillanova in https://github.com/huggingface/trl/pull/5284pad_to_multiple_of to GRPOTrainer and RLOOTrainer by @czkkkkkk in https://github.com/huggingface/trl/pull/5180accuracy_reward crash when called from non-main thread by @qgallouedec in https://github.com/huggingface/trl/pull/5281RewardFunc type alias to reflect actual calling convention by @s-zx in https://github.com/huggingface/trl/pull/5246prepare_multimodal_messages to support tool_calls and tool role by @alvarobartt in https://github.com/huggingface/trl/pull/5212AGENTS.md) by @qgallouedec in https://github.com/huggingface/trl/pull/5236pad_to_multiple_of to GRPOTrainer and RLOOTrainer by @czkkkkkk in https://github.com/huggingface/trl/pull/5180prepare_multimodal_messages to support tool_calls and tool role by @alvarobartt in https://github.com/huggingface/trl/pull/5212AGENTS.md) by @qgallouedec in https://github.com/huggingface/trl/pull/5236prompts in vLLM client and server by @qgallouedec in https://github.com/huggingface/trl/pull/5225truncate_prompt_tokens for vLLM 0.17.0 by @winglian in https://github.com/huggingface/trl/pull/5248rollout_func from _generate_single_turn to _generate by @qgallouedec in https://github.com/huggingface/trl/pull/5232RewardFunc type alias to reflect actual calling convention by @s-zx in https://github.com/huggingface/trl/pull/5246_generate_single_turn by @qgallouedec in https://github.com/huggingface/trl/pull/5239_generate_single_turn by @qgallouedec in https://github.com/huggingface/trl/pull/5240bfd-requeue to bfd_split by @mariosasko in https://github.com/huggingface/trl/pull/5189vllm_mode to "colocate" and add v0→v1 migration guide by @qgallouedec in https://github.com/huggingface/trl/pull/5255grpo_trainer.py): Variational Sequence-Level Soft Policy Optimization (VESPO) by @casinca in https://github.com/huggingface/trl/pull/5199accuracy_reward crash when called from non-main thread by @qgallouedec in https://github.com/huggingface/trl/pull/5281hasattr and getattr with defaults in AGENTS.md by @qgallouedec in https://github.com/huggingface/trl/pull/5294RewardFunc type annotation to allow Nonevalues in reward list by @qgallouedec in https://github.com/huggingface/trl/pull/5297Json() type for tool calling dataset format by @lhoestq in https://github.com/huggingface/trl/pull/5307Full Changelog: https://github.com/huggingface/trl/compare/v0.29.0...v1.0.0rc1
Handle mm_token_type_ids in SFT/GRPO/RLOO to fix IndexError by @albertvillanova in https://github.com/huggingface/trl/pull/5178
prepare_multimodal_messages to support tool_calls and tool role by @alvarobartt in https://github.com/huggingface/trl/pull/5212prompts in vLLM client and server by @qgallouedec in https://github.com/huggingface/trl/pull/5225rollout_func from _generate_single_turn to _generate by @qgallouedec in https://github.com/huggingface/trl/pull/5232_generate_single_turn by @qgallouedec in https://github.com/huggingface/trl/pull/5239_generate_single_turn by @qgallouedec in https://github.com/huggingface/trl/pull/5240Full Changelog: https://github.com/huggingface/trl/compare/v0.29.0...v0.29.1
Remove deprecated BCO after moved to experimental by @albertvillanova in https://github.com/huggingface/trl/pull/5045
environment_factory to GRPOTrainerGRPOTrainer now accepts an environment_factory argument, allowing users to specify a custom environment class for training. This enables more flexible and diverse training scenarios by letting users define their own environments with specific dynamics and reward structures.
from datasets import Dataset
from trl import GRPOConfig, GRPOTrainer
dataset = Dataset.from_dict({
"prompt": [[{"role": "user", "content": f"Increment the counter by {i}."}] for i in range(1, 7)]
})
def reward_func(environments, **kwargs):
return [env.counter for env in environments]
class IncrementEnv:
def reset(self):
self.counter = 0
def increment(self, step: int) -> int:
"""
Increment the internal counter.
Args:
step: Value to add to the counter.
Returns:
The updated counter value.
"""
self.counter += step
return self.counter
trainer = GRPOTrainer(
model="Qwen/Qwen3-0.6B",
args=GRPOConfig(chat_template_kwargs={"enable_thinking": False}),
train_dataset=dataset,
reward_funcs=reward_func,
environment_factory=IncrementEnv,
)
trainer.train()
by @qgallouedec in https://github.com/huggingface/trl/pull/5093
TRL introduces agent-native CLI Integration: trl-training, a first-class Agent Skill that exposes TRL’s training workflows (SFT, DPO, GRPO, etc.) in a structured, agent-readable format. The skill is packaged directly with the trl library and can be installed via the CLI:
# Install into the project's agent directory (default scope=project), by agent name: claude, codex, opencode
trl skills install trl-training --target <agent>
This enables AI agents to safely and reproducibly execute TRL training workflows using a well-defined interface.
Skills can be installed at the project or global scope, and support explicit targets and overwrite controls.
launch_args for all trainers by @qgallouedec in https://github.com/huggingface/trl/pull/5059num_labels to 1 in causal model initialization for RewardTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/5066None in get_trackio_space_url() to prevent errors by @qgallouedec in https://github.com/huggingface/trl/pull/5115trl <command> --help TypeError caused by unescaped % in TrainingArguments help strings by @albertvillanova in https://github.com/huggingface/trl/pull/5135SFTTrainer support for single-image data by @qgallouedec in https://github.com/huggingface/trl/pull/5132grpo_trainer.md by @casinca in https://github.com/huggingface/trl/pull/5047RewardTrainer collator from chosen/rejected_input_ids to chosen/rejected_ids by @qgallouedec in https://github.com/huggingface/trl/pull/5179RewardTrainer tests by @qgallouedec in https://github.com/huggingface/trl/pull/5060get_training_chat_template by @qgallouedec in https://github.com/huggingface/trl/pull/5108train_dataset is required by @albertvillanova in https://github.com/huggingface/trl/pull/5171grpo_trainer.md by @casinca in https://github.com/huggingface/trl/pull/5047launch_args for all trainers by @qgallouedec in https://github.com/huggingface/trl/pull/5059num_labels to 1 in causal model initialization for RewardTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/5066RewardTrainer tests by @qgallouedec in https://github.com/huggingface/trl/pull/5060get_training_chat_template by @qgallouedec in https://github.com/huggingface/trl/pull/5108None in get_trackio_space_url() to prevent errors by @qgallouedec in https://github.com/huggingface/trl/pull/5115trl <command> --help TypeError caused by unescaped % in TrainingArguments help strings by @albertvillanova in https://github.com/huggingface/trl/pull/5135environment_factory to GRPOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/5093SFTTrainer support for single-image data by @qgallouedec in https://github.com/huggingface/trl/pull/5132train_dataset is required by @albertvillanova in https://github.com/huggingface/trl/pull/5171RewardTrainer collator from chosen/rejected_input_ids to chosen/rejected_ids by @qgallouedec in https://github.com/huggingface/trl/pull/5179Full Changelog: https://github.com/huggingface/trl/compare/v0.28.0...v0.29.0
Remove deprecated parameters by @qgallouedec in https://github.com/huggingface/trl/pull/4847
is_conversational by @qgallouedec in https://github.com/huggingface/trl/pull/4923openenv/utils.py: fallback for no vLLM installed case by @Datta0 in https://github.com/huggingface/trl/pull/4868current_gradient_accumulation_steps by @qgallouedec in https://github.com/huggingface/trl/pull/4852get_open_port based on vLLM version by @qgallouedec in https://github.com/huggingface/trl/pull/4883device_map init consistency in GRPO/RLOO/KTO by @qgallouedec in https://github.com/huggingface/trl/pull/4909warnings_issued by @qgallouedec in https://github.com/huggingface/trl/pull/4960DPOConfig by @qgallouedec in https://github.com/huggingface/trl/pull/4969warmup_ratio with warmup_steps by @qgallouedec in https://github.com/huggingface/trl/pull/4983RewardTrainer, RLOOTrainer and GRPOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/4823TestGRPOTrainer.test_training_vlm_and_liger and update version checks by @qgallouedec in https://github.com/huggingface/trl/pull/4898compute_metrics in SFTTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/4950RewardTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/4959compute_metrics in RewardTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/4958CITATION.cff by @qgallouedec in https://github.com/huggingface/trl/pull/4856DataCollatorForVisionLanguageModeling by @qgallouedec in https://github.com/huggingface/trl/pull/4911max_length in RewardConfig and SFTConfig by @qgallouedec in https://github.com/huggingface/trl/pull/4910sync_ref_model in GRPOTrainer and RLOOTrainer when using PEFT models by @qgallouedec in https://github.com/huggingface/trl/pull/4912⬆️ Bump dev version by @qgallouedec in https://github.com/huggingface/trl/pull/4835
Support triggering CI via push to ci-* branches by @albertvillanova in https://github.com/huggingface/trl/pull/4840
Revert CI hotfix pinning transformers 4.57.4 after tiny model regeneration by @albertvillanova in https://github.com/huggingface/trl/pull/4833
Use pytest-datadir in CI tests by @albertvillanova in https://github.com/huggingface/trl/pull/4836
Refactor KTO coordinated with DPO [c/N]: Remove ref_model_init_kwargs by @albertvillanova in https://github.com/huggingface/trl/pull/4837
Fix _patch_transformers_hybrid_cache for peft by @albertvillanova in https://github.com/huggingface/trl/pull/4844
Refactor KTO [4/N]: Remove unused padding_value by @albertvillanova in https://github.com/huggingface/trl/pull/4839
Remove unused padding_value from BCO by @albertvillanova in https://github.com/huggingface/trl/pull/4846
Fix CI with dev dependencies: Mark Qwen3-VL tests as xfail by @albertvillanova in https://github.com/huggingface/trl/pull/4851
Fix: undefined current_gradient_accumulation_steps by @qgallouedec in https://github.com/huggingface/trl/pull/4852
Remove deprecated parameters by @qgallouedec in https://github.com/huggingface/trl/pull/4847
Add Nash Learning from Human Feedback paper to paper index by @kansalaman in https://github.com/huggingface/trl/pull/4860
Use pytest-datadir for accelerate config files by @albertvillanova in https://github.com/huggingface/trl/pull/4861
Update OpenEnv dependency to new version for hf jobs scripts by @sergiopaniego in https://github.com/huggingface/trl/pull/4843
Update CITATION.cff by @qgallouedec in https://github.com/huggingface/trl/pull/4856
[GRPOTrainer]: Agent Training Supports Async Tool Calls by @pramodith in https://github.com/huggingface/trl/pull/4742
Enhance GRPO documentation with scaling notes by @javadtaghia in https://github.com/huggingface/trl/pull/4849
Add retry strategy to vLLM Client for increased robustness by @apalmas-saifh in https://github.com/huggingface/trl/pull/4845
Update generate_tiny_models.py: CohereForAI -> CohereLabs by @Michellehbn in https://github.com/huggingface/trl/pull/4877
Refactor KTO coordinated with DPO [e/N]: Remove label_pad_token_id by @albertvillanova in https://github.com/huggingface/trl/pull/4875
Refactor KTO coordinated with DPO [d/N]: Remove base_model_attribute_name by @albertvillanova in https://github.com/huggingface/trl/pull/4862
Fix type hint in openenv/utils.py: fallback for no vLLM installed case by @Datta0 in https://github.com/huggingface/trl/pull/4868
Update transformer version checks and documentation for lr_scheduler_kwargs workaround by @qgallouedec in https://github.com/huggingface/trl/pull/4876
fix(DeepSeek OPSM): passing correct (vLLM) logprobs by @casinca in https://github.com/huggingface/trl/pull/4857
Remove label_pad_token_id from experimental trainers by @albertvillanova in https://github.com/huggingface/trl/pull/4878
Fix SFT training for prompt-completion type and transformers v5 by @qgallouedec in https://github.com/huggingface/trl/pull/4880
Bugfix: Logprob drift in vLLM serving mode (compared to colocate mode) by @kdubovikov in https://github.com/huggingface/trl/pull/4873
Enable vLLM sleep mode for generation in Online DPO by @winglian in https://github.com/huggingface/trl/pull/4882
Test distributed training for RewardTrainer, RLOOTrainer and GRPOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/4823
Mark ZeRO 2 as xfail in distributed tests due to current failure by @qgallouedec in https://github.com/huggingface/trl/pull/4885
Fix import path for get_open_port based on vLLM version by @qgallouedec in https://github.com/huggingface/trl/pull/4883
Fix RewardTrainer's results not reproducible by @liyc-ai in https://github.com/huggingface/trl/pull/4887
GOLD training speed up by @141forever in https://github.com/huggingface/trl/pull/4888
Transformers v5 release: extend xfail condition for TestGRPOTrainer.test_training_vlm_and_liger and update version checks by @qgallouedec in https://github.com/huggingface/trl/pull/4898
Fix CI NotImplementedError for bfloat16 by @albertvillanova in https://github.com/huggingface/trl/pull/4902
Fix CI AssertionError: Parameter has not changed by @albertvillanova in https://github.com/huggingface/trl/pull/4904
Refactor vLLM generation [1/N]: Extract vLLM generation by @albertvillanova in https://github.com/huggingface/trl/pull/4700
Created new PTT integration docs as requested by @adityachallapally in https://github.com/huggingface/trl/pull/4907
Fix CI TypeError in llm-blender tests by @albertvillanova in https://github.com/huggingface/trl/pull/4919
Rearrange variable assignments in DataCollatorForVisionLanguageModeling by @qgallouedec in https://github.com/huggingface/trl/pull/4911
Fix help text formatting for max_length in RewardConfig and SFTConfig by @qgallouedec in https://github.com/huggingface/trl/pull/4910
device_map init consistency in GRPO/RLOO/KTO by @qgallouedec in https://github.com/huggingface/trl/pull/4909
Comment about overriding prediction_step in GRPOTrainer and RLOOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/4913
Remove gradient checkpointing option from various training scripts by @qgallouedec in https://github.com/huggingface/trl/pull/4905
docs: add DoRA (2402.09353) to Paper Index by @billycrapediem in https://github.com/huggingface/trl/pull/4892
Fix CI AssertionError: assert not True by @albertvillanova in https://github.com/huggingface/trl/pull/4921
Fix CI ValueError for 0 temperature by @albertvillanova in https://github.com/huggingface/trl/pull/4916
Fix extra EOS appended in DPO preprocessing for conversational data by @qgallouedec in https://github.com/huggingface/trl/pull/4908
Remove chat template setup in dpo_vlm.py by @qgallouedec in https://github.com/huggingface/trl/pull/4906
Update learning rate comments and add assertions for reference model parameters in GRPO and RLOO tests by @qgallouedec in https://github.com/huggingface/trl/pull/4914
Add validation for sync_ref_model in GRPOTrainer and RLOOTrainer when using PEFT models by @qgallouedec in https://github.com/huggingface/trl/pull/4912
Support tool call data in is_conversational by @qgallouedec in https://github.com/huggingface/trl/pull/4923
Set model dtype to float32 in tests of trainers by @albertvillanova in https://github.com/huggingface/trl/pull/4924
Require transformers<5 with PairRMJudge by @albertvillanova in https://github.com/huggingface/trl/pull/4926
Move VLLMClient to generation module by @albertvillanova in https://github.com/huggingface/trl/pull/4928
Set model dtype to float32 in experimental tests of trainers by @albertvillanova in https://github.com/huggingface/trl/pull/4925
Fix profiling of VLLMGeneration.sync_weights by @albertvillanova in https://github.com/huggingface/trl/pull/4931
Fix import statement for import_utils in vllm_client.py by @qgallouedec in https://github.com/huggingface/trl/pull/4932
Set default top_k to 0 in VLLMClient by @albertvillanova in https://github.com/huggingface/trl/pull/4927
[GRPO] Add parquet logging for completions with individual rewards by @qgallouedec in https://github.com/huggingface/trl/pull/4818
Fix SFTTrainer init logic: remove TrainingArguments.push_to_hub_token only for transformers < v5 by @albertvillanova in https://github.com/huggingface/trl/pull/4942
Remove ref_model_init_kwargs from experimental BCO by @albertvillanova in https://github.com/huggingface/trl/pull/4946
Update wordle.py example with masking of env tokens by @sergiopaniego in https://github.com/huggingface/trl/pull/4895
Fix PPO run_name parameter not taking effect by @mel3c in https://github.com/huggingface/trl/pull/4945
Minor fix docs style by @albertvillanova in https://github.com/huggingface/trl/pull/4953
Add test for training with compute_metrics in SFTTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/4950
Remove access to warnings_issued by @qgallouedec in https://github.com/huggingface/trl/pull/4960
NeMo-Gym Integration by @cmunley1 in https://github.com/huggingface/trl/pull/4848
Add test for tool call data in RewardTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/4959
Add test for training with compute_metrics in RewardTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/4958
Remove max_prompt_length from experimental PRM by @albertvillanova in https://github.com/huggingface/trl/pull/4963
Remove max_prompt_length from experimental BCO by @albertvillanova in https://github.com/huggingface/trl/pull/4964
Remove max_prompt_length from experimental CPO by @albertvillanova in https://github.com/huggingface/trl/pull/4965
Remove max_prompt_length from experimental ORPO by @albertvillanova in https://github.com/huggingface/trl/pull/4966
Revert change in GRPO from NeMo-Gym Integration by @qgallouedec in https://github.com/huggingface/trl/pull/4970
Fix test_train_with_chat_template_kwargs by @qgallouedec in https://github.com/huggingface/trl/pull/4971
Remove padding_value from experimental CPO and use pad_token_id by @albertvillanova in https://github.com/huggingface/trl/pull/4962
Remove truncation from tokenizer calls if no max_length by @albertvillanova in https://github.com/huggingface/trl/pull/4972
Set specific OpenEnv version when installed by @sergiopaniego in https://github.com/huggingface/trl/pull/4978
Fix add_column in test_train_with_chat_template_kwargs by @albertvillanova in https://github.com/huggingface/trl/pull/4979
Support truncated completions in GRPO multi-turn training by @albertvillanova in https://github.com/huggingface/trl/pull/4976
Replace torch.allclose with torch.testing.assert_close by @qgallouedec in https://github.com/huggingface/trl/pull/4977
Simplify instructions of installation of OpenEnv by @sergiopaniego in https://github.com/huggingface/trl/pull/4980
Deprecate parameters in DPOConfig by @qgallouedec in https://github.com/huggingface/trl/pull/4969
[CI] Disallow installation of transformers 5.1.0 due to compatibility issues with DeepSpeed by @qgallouedec in https://github.com/huggingface/trl/pull/4982
Replace warmup_ratio with warmup_steps by @qgallouedec in https://github.com/huggingface/trl/pull/4983
Pin transformers!=5.1.0 in deepspeed extra due to incompatibility by @albertvillanova in https://github.com/huggingface/trl/pull/4985
Fix passing tokenizer in test_train_with_chat_template_kwargs by @albertvillanova in https://github.com/huggingface/trl/pull/4987
Update dataset configuration name in toolcall dataset loading by @qgallouedec in https://github.com/huggingface/trl/pull/4984
Use local variable instead of attribute in collator tests by @qgallouedec in https://github.com/huggingface/trl/pull/4957
Fix import of AutoModelForCausalLMWithValueHead from experimental by @albertvillanova in https://github.com/huggingface/trl/pull/4990
Assert chat_template is applied in test_train_with_chat_template_kwargs by @albertvillanova in https://github.com/huggingface/trl/pull/4991
Fix deprecation of DPOConfig.max_completion_length by @albertvillanova in https://github.com/huggingface/trl/pull/4992
Fix post_init warning stacklevel to 3 by @albertvillanova in https://github.com/huggingface/trl/pull/4993
Fix ZeRO-3 + PEFT + gradient checkpointing by @qgallouedec in https://github.com/huggingface/trl/pull/4951
Add GitHub Actions workflow for testing against Transformers branch by @qgallouedec in https://github.com/huggingface/trl/pull/4995
Add distributed smoke tests workflow for Transformers branch by @qgallouedec in https://github.com/huggingface/trl/pull/4996
Update NeMo-Gym to use env_mask by @cmunley1 in https://github.com/huggingface/trl/pull/4986
Update sampling mode to token level for safety by @sergiopaniego in https://github.com/huggingface/trl/pull/4989
perf: Qwen SAPO loss optimization by @casinca in https://github.com/huggingface/trl/pull/4956
Fix GRPO tool calling for corrupted tool calls by @akshayballal95 in https://github.com/huggingface/trl/pull/4890
Add sanitize_logprob function for NaN handling in vLLM log probabilities by @qgallouedec in https://github.com/huggingface/trl/pull/5001
[tests] Remove xfail for transformers version >= 5.0.0 due to upstream bug resolution by @qgallouedec in https://github.com/huggingface/trl/pull/5000
docs: add CGPO/Mixture of Judges (2409.20370) to Paper Index + link ref to AllTrueJudge by @nabin2004 in https://github.com/huggingface/trl/pull/5002
Filter CI SWIG deprecation warnings by @albertvillanova in https://github.com/huggingface/trl/pull/5004
Fix CI TRLExperimentalWarning in regular tests by @albertvillanova in https://github.com/huggingface/trl/pull/5007
Add support for nested_gather in OnlineDPOTrainer for transformers v5.2.0 and above by @qgallouedec in https://github.com/huggingface/trl/pull/4981
Fix CI FutureWarning: ref_model_init_kwargs is deprecated by @albertvillanova in https://github.com/huggingface/trl/pull/5009
Fix typo in DPO max_prompt_length deprecation warning message by @albertvillanova in https://github.com/huggingface/trl/pull/5020
Fix vision model prompt truncation bug in DPOTrainer by @albertvillanova in https://github.com/huggingface/trl/pull/5023
Pin transformers < 5 in judges extra due to incompatibility by @albertvillanova in https://github.com/huggingface/trl/pull/5024
Fix CI FutureWarning: generate_during_eval is deprecated by @albertvillanova in https://github.com/huggingface/trl/pull/5017
Fix typo in xfail test reason by @albertvillanova in https://github.com/huggingface/trl/pull/5028
Fix CI FutureWarning: rpo_alpha is deprecated by @albertvillanova in https://github.com/huggingface/trl/pull/5011
Fix CI FutureWarning: use_logits_to_keep is deprecated by @albertvillanova in https://github.com/huggingface/trl/pull/5013
Mark Qwen3VL tests as xfail for transformers 5.0.x by @albertvillanova in https://github.com/huggingface/trl/pull/5029
[CI] Silence PyTorch JIT and DataLoader deprecation warnings by @qgallouedec in https://github.com/huggingface/trl/pull/4999
Add length-unbiased GRPO loss (LUSPO) by @Haseebasif7 in https://github.com/huggingface/trl/pull/4988
Fix CI FutureWarning: tools is deprecated by @albertvillanova in https://github.com/huggingface/trl/pull/5015
Filter max_prompt_length UserWarning in all test cases by @albertvillanova in https://github.com/huggingface/trl/pull/5035
Fix CI FutureWarning: max_prompt_length is deprecated by @albertvillanova in https://github.com/huggingface/trl/pull/5019
Allow testing with transformers 5.1.0 via xfail marks by @albertvillanova in https://github.com/huggingface/trl/pull/5034
Rename AOT loss type 'aot_pair' to 'aot_unpaired' in DPO by @qgallouedec in https://github.com/huggingface/trl/pull/5038
Deprecate string usage for ref_model in DPOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/5040
Deprecate FDivergenceType in DPOConfig; update f_divergence_type to use string values by @qgallouedec in https://github.com/huggingface/trl/pull/5039
Fix multiprocessing start method to 'spawn' for test compatibility with Python 3.12+ by @qgallouedec in https://github.com/huggingface/trl/pull/5036
Add Online Direct Preference Optimization section to paper index by @qgallouedec in https://github.com/huggingface/trl/pull/5037
Release: 0.28 by @albertvillanova in https://github.com/huggingface/trl/pull/5043
Full Changelog: https://github.com/huggingface/trl/compare/v0.27.0...v0.28.0
Remove access to warnings_issued by @qgallouedec in #4960
warnings_issued by @qgallouedec in #4960Full Changelog: https://github.com/huggingface/trl/compare/v0.27.1...v0.27.2
Fix: undefined current_gradient_accumulation_steps by @qgallouedec in https://github.com/huggingface/trl/pull/4852
current_gradient_accumulation_steps by @qgallouedec in https://github.com/huggingface/trl/pull/4852Full Changelog: https://github.com/huggingface/trl/compare/v0.27.0...v0.27.1
Remove deprecated functions and parameters by @qgallouedec in https://github.com/huggingface/trl/pull/4651
vllm_group_port argument to GRPO, RLOO and OnlineDPO configuration by @pointerhacker in https://github.com/huggingface/trl/pull/4545forward_masked_logits function by @qgallouedec in https://github.com/huggingface/trl/pull/4729AutoModelForCausalLMWithValueHead and AutoModelForSeq2SeqLMWithValueHead to experimental by @qgallouedec in https://github.com/huggingface/trl/pull/4654experimental.utils by @qgallouedec in https://github.com/huggingface/trl/pull/4667DataCollatorForChatML to experimental.utils by @qgallouedec in https://github.com/huggingface/trl/pull/4668add_bos_token_if_needed and add_eos_token_if_needed to experimental.utils by @qgallouedec in https://github.com/huggingface/trl/pull/4674truncate_right and SIMPLE_CHAT_TEMPLATE to experimental.utils by @qgallouedec in https://github.com/huggingface/trl/pull/4677prepare_model_for_kbit_training, enable_gradient_checkpointing, prepare_peft_model to experimental.utils by @qgallouedec in https://github.com/huggingface/trl/pull/4704get_reward function to experimental.utils by @qgallouedec in https://github.com/huggingface/trl/pull/4683num_generations_eval=1 in the calculation of the advantage by @qgallouedec in https://github.com/huggingface/trl/pull/4662num_generations_eval is specified and different than num_generations by @apalmas-saifh in https://github.com/huggingface/trl/pull/4682generation_config for tiny model uploads by @qgallouedec in https://github.com/huggingface/trl/pull/4643qwen3_schema by @mattbui in https://github.com/huggingface/trl/pull/4709HybridCache in Liger-Kernel with transformers v5 by @qgallouedec in https://github.com/huggingface/trl/pull/4798args by @carlyou in https://github.com/huggingface/trl/pull/4801grpo_trainer.md): Added Qwen SAPO details under Loss Types by @casinca in https://github.com/huggingface/trl/pull/4681MergeModelCallback from import structure by @qgallouedec in https://github.com/huggingface/trl/pull/4664ChatMlSpecialTokens by @qgallouedec in https://github.com/huggingface/trl/pull/4666_win_rate_completions_df function from callbacks by @qgallouedec in https://github.com/huggingface/trl/pull/4672DbrxForCausalLM support by @qgallouedec in https://github.com/huggingface/trl/pull/4799compute_accuracy to PRM Trainer file by @qgallouedec in https://github.com/huggingface/trl/pull/4656clone_chat_template to chat_template_utils by @qgallouedec in https://github.com/huggingface/trl/pull/4653GeometricMixtureWrapper to nash_md_trainer.py by @qgallouedec in https://github.com/huggingface/trl/pull/4670exact_div, print_rich_table, truncate_response, forward to ppo_trainer by @qgallouedec in https://github.com/huggingface/trl/pull/4676OnPolicyConfig and PPOConfig and move OnlineTrainerState by @qgallouedec in https://github.com/huggingface/trl/pull/4671AutoModelForCausalLMWithValueHead to test_ppo_trainer by @qgallouedec in https://github.com/huggingface/trl/pull/4678generate and batch_generation to ppo_trainer.py by @qgallouedec in https://github.com/huggingface/trl/pull/4675TrainerCallback from top-level transformers by @qgallouedec in https://github.com/huggingface/trl/pull/4694top_k parameter in OnlineDPOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/4714PeftModel + peft_config in trainers by @qgallouedec in https://github.com/huggingface/trl/pull/4713TestParseResponse by @qgallouedec in https://github.com/huggingface/trl/pull/4736GuidedDecodingParams with StructuredOutputsParams in sampling parameter configuration by @qgallouedec in https://github.com/huggingface/trl/pull/4797--
Full Changelog: https://github.com/huggingface/trl/compare/v0.26.0...v0.27.0
Overwrite model default generation config used by model.generate by @albertvillanova in https://github.com/huggingface/trl/pull/4647
Full Changelog: https://github.com/huggingface/trl/compare/v0.26.1...v0.26.2
Fix vLLM error for tools usage not supported when running GRPO training by @apalmas-saifh in https://github.com/huggingface/trl/pull/4663
num_generations_eval is specified and different than num_generations by @apalmas-saifh in https://github.com/huggingface/trl/pull/4682Full Changelog: https://github.com/huggingface/trl/compare/v0.26.0...v0.26.1
Remove deprecations for 0.26 release by @albertvillanova in https://github.com/huggingface/trl/pull/4607
GRPOTrainer now supports training agents using tools. This allows language models to interact with external functions or APIs during training.
from datasets import Dataset
from trl import GRPOTrainer
def multiply(a: int, b: int) -> int:
"""
Multiplies two integers.
Args:
a: The first integer.
b: The second integer.
Returns:
The product of the two integers.
"""
return a * b
dataset = Dataset.from_list(
[
{"prompt": [{"role": "user", "content": "What is 3 multiplied by 4?"}], "answer": 12},
{"prompt": [{"role": "user", "content": "Calculate 7 times 8."}], "answer": 56},
{"prompt": [{"role": "user", "content": "Find the product of 5 and 6."}], "answer": 30},
{"prompt": [{"role": "user", "content": "What do you get when you multiply 9 by 9?"}], "answer": 81},
{"prompt": [{"role": "user", "content": "Compute 12 multiplied by 11."}], "answer": 132},
{"prompt": [{"role": "user", "content": "What is 15 times 14?"}], "answer": 210},
]
)
def accuracy(completions, answer, **kwargs):
predictions = [completion[-1]["content"] for completion in completions]
rewards = [float(str(ans) in pred) for pred, ans in zip(predictions, answer)]
return rewards
trainer = GRPOTrainer(
model="Qwen/Qwen3-0.6B",
train_dataset=dataset,
tools=[multiply],
reward_funcs=accuracy,
)
trainer.train()
by @qgallouedec in https://github.com/huggingface/trl/pull/4300
CISPO Loss was first introduced in the Minimax-M1 paper, the ScaleRL paper subsequently showed that CISPO loss scales the best in terms of performance and efficiency as models are trained for longer.
GRPOTrainer now supports the CISPO loss using loss_type="cispo" in the GRPOConfig.
by @pramodith in https://github.com/huggingface/trl/pull/4495
When the input model is quantized using bitsandbytes, vLLM will now also use quantization when in colocate mode.
by @sergiopaniego in https://github.com/huggingface/trl/pull/4496
TRL nows includes a reasoning reward function
from trl.rewards import reasoning_accuracy_reward
solutions = [r"\frac{1}{3}", r"\frac{1}{3}", r"\frac{1}{3}"]
completions = [
[
{
"role": "assistant",
"content": r"<think> Reasoning content </think> The final answer is \boxed{\frac{1}{3}}",
}
],
[
{
"role": "assistant",
"content": r"<think> Reasoning content </think> The final answer is \boxed{\frac{1}{2}}",
}
],
[
{
"role": "assistant",
"content": r"<think> Reasoning content with partial answers \boxed{\frac{1}{3}} but no final answer",
}
],
]
reasoning_accuracy_reward(completions, solutions) # [1.0, 0.0, 0.0]
As any other reward function, it can be used in GRPOTrainer or RLOOTrainer.
from trl import GRPOTrainer
from trl.rewards import reasoning_accuracy_reward
trainer = GRPOTrainer(
...,
reward_funcs=reasoning_accuracy_reward,
)
by @lewtun in https://github.com/huggingface/trl/pull/4563
shuffle_dataset option to SFTTrainerYou can now shuffle the dataset in SFTTrainer by setting the shuffle_dataset argument to True in SFTConfig. This is useful when the dataset features high similarity between consecutive samples.
from trl import SFTTrainer, SFTConfig
SFTConfig(shuffle_dataset=True)
by @qgallouedec in https://github.com/huggingface/trl/pull/4564
Soft Adaptive Policy Optimization (SAPO), replaces hard clipping with a smooth, temperature-controlled gate that adaptively attenuates off-policy updates while preserving useful learning signals. Compared with GSPO and GRPO, SAPO is both sequence-coherent and token-adaptive. Like GSPO, SAPO maintains sequence-level coherence, but its soft gating forms a continuous trust region that avoids the brittle hard clipping band used in GSPO.
You can now use SAPO loss in GRPOTrainer by setting loss_type="sapo" in the GRPOConfig.
by @pramodith in https://github.com/huggingface/trl/pull/4600
num_generations_eval parameter for efficient evaluation by @mingxuetian in https://github.com/huggingface/trl/pull/4458WinRateCallback to experimental by @qgallouedec in https://github.com/huggingface/trl/pull/4558device_map and dtype to "auto" by default by @qgallouedec in https://github.com/huggingface/trl/pull/4509flash-attn to flash-attn2 by @qgallouedec in https://github.com/huggingface/trl/pull/4514device_map=None for DeepSpeed and add ZeRO paper (1910.02054) to Paper Index by @JenWei0312 in https://github.com/huggingface/trl/pull/4551num_completions to num_generations by @pramodith in https://github.com/huggingface/trl/pull/4515rnj_1_instruct notebook by @sergiopaniego in https://github.com/huggingface/trl/pull/4646wandb_log_unique_prompts with log_unique_prompts by @taha-yassine in https://github.com/huggingface/trl/pull/4508prepare_model_for_kbit_training by @sergiopaniego in https://github.com/huggingface/trl/pull/4457lr_scheduler_kwargs dtype issue in Transformers 4.57.0 by @qgallouedec in https://github.com/huggingface/trl/pull/4513device_map and dtype to "auto" by default by @qgallouedec in https://github.com/huggingface/trl/pull/4509wandb_log_unique_prompts with log_unique_prompts by @taha-yassine in https://github.com/huggingface/trl/pull/4508num_completions to num_generations by @pramodith in https://github.com/huggingface/trl/pull/4515flash-attn to flash-attn2 by @qgallouedec in https://github.com/huggingface/trl/pull/4514prepare_model_for_kbit_training by @sergiopaniego in https://github.com/huggingface/trl/pull/4457device_map=None for DeepSpeed and add ZeRO paper (1910.02054) to Paper Index by @JenWei0312 in https://github.com/huggingface/trl/pull/4551num_generations_eval parameter for efficient evaluation by @mingxuetian in https://github.com/huggingface/trl/pull/4458shuffle_dataset option to SFTTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/4564WinRateCallback to experimental by @qgallouedec in https://github.com/huggingface/trl/pull/4558rnj_1_instruct notebook by @sergiopaniego in https://github.com/huggingface/trl/pull/4646Full Changelog: https://github.com/huggingface/trl/compare/v0.25.0...v0.26.0
Replace accelerate logging with stdlib in CLI by @lewtun in https://github.com/huggingface/trl/pull/4512
lr_scheduler_kwargs dtype issue in Transformers 4.57.0 by @qgallouedec in https://github.com/huggingface/trl/pull/4513Full Changelog: https://github.com/huggingface/trl/compare/v0.25.0...0.25.1
Replace deprecated AutoModelForVision2Seq with AutoModelForImageTextToText by @albertvillanova in https://github.com/huggingface/trl/pull/4353
prepare_model_for_kbit_training to save VRAM by @sergiopaniego in https://github.com/huggingface/trl/pull/4335add_generation_prompt to processor_kwargs in GRPO and RLOO trainer by @qgallouedec in https://github.com/huggingface/trl/pull/4361trl.experimental by @qgallouedec in https://github.com/huggingface/trl/pull/4312add_generation_prompt=True for conversational only by @qgallouedec in https://github.com/huggingface/trl/pull/4362max_length explanation for VLM in online trainers by @sergiopaniego in https://github.com/huggingface/trl/pull/4220_generate in GRPO/RLOO: Move forward_kwargs outside generation method by @qgallouedec in https://github.com/huggingface/trl/pull/4154trl.experimental by @qgallouedec in https://github.com/huggingface/trl/pull/4312_generate in GRPO/RLOO: Insert images in the prompt by @qgallouedec in https://github.com/huggingface/trl/pull/4155prepare_model_for_kbit_training to save VRAM by @sergiopaniego in https://github.com/huggingface/trl/pull/4335add_generation_prompt to processor_kwargs in GRPO and RLOO trainer by @qgallouedec in https://github.com/huggingface/trl/pull/4361add_generation_prompt=True for conversational only by @qgallouedec in https://github.com/huggingface/trl/pull/4362max_length explanation for VLM in online trainers by @sergiopaniego in https://github.com/huggingface/trl/pull/4220Full Changelog: https://github.com/huggingface/trl/compare/v0.24.0...v0.25.0
⚖️ Align SFT and DPO for model creation and deprecate DPOConfig.padding_value in favour or pad_token_id by @qgallouedec in https://github.com/huggingf…
token_type_ids in DPOTrainer by @aweers in https://github.com/huggingface/trl/pull/4285RichProgressCallback enhancement by @qgallouedec in https://github.com/huggingface/trl/pull/4245chat_template_kwargs in apply_chat_template by @cmpatino in https://github.com/huggingface/trl/pull/4233token_type_ids in DataCollatorForVisionLanguageModeling by @qgallouedec in https://github.com/huggingface/trl/pull/4190RewardTrainer refactor by @qgallouedec in https://github.com/huggingface/trl/pull/4093clone_chat_template by @qgallouedec in https://github.com/huggingface/trl/pull/4097vllm_enable_sleep_mode to RLOO Trainer by @sergiopaniego in https://github.com/huggingface/trl/pull/4107batchmean reduce op in GKDTrainer's loss by @cmpatino in https://github.com/huggingface/trl/pull/4105get_high_entropy_mask by @akakakakakaa in https://github.com/huggingface/trl/pull/4041DPOConfig.padding_value in favour or pad_token_id by @qgallouedec in https://github.com/huggingface/trl/pull/4006BestOfNSampler by @qgallouedec in https://github.com/huggingface/trl/pull/4291AlignPropTrainer, DDPOTrainer and IterativeSFTTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/4068trl.experimental Submodule by @August-murr in https://github.com/huggingface/trl/pull/4073make_parser function in multiple scripts by @qgallouedec in https://github.com/huggingface/trl/pull/4050get_high_entropy_mask by @akakakakakaa in https://github.com/huggingface/trl/pull/4041trl.experimental Submodule by @August-murr in https://github.com/huggingface/trl/pull/4073AlignPropTrainer, DDPOTrainer and IterativeSFTTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/4068set to list of tags by @qgallouedec in https://github.com/huggingface/trl/pull/4092DPOConfig.padding_value in favour or pad_token_id by @qgallouedec in https://github.com/huggingface/trl/pull/4006batchmean reduce op in GKDTrainer's loss by @cmpatino in https://github.com/huggingface/trl/pull/4105image_split_sizes in favour of image_grid_thw by @qgallouedec in https://github.com/huggingface/trl/pull/4111vllm_enable_sleep_mode to RLOO Trainer by @sergiopaniego in https://github.com/huggingface/trl/pull/4107backend parameter from GuidedDecodingParams by @qgallouedec in https://github.com/huggingface/trl/pull/4123max_batch_tokens, num_blocks and block_size from generation kwargs by @qgallouedec in https://github.com/huggingface/trl/pull/4065_generate by @qgallouedec in https://github.com/huggingface/trl/pull/4114image_split_sizes in favour of image_grid_thw by @qgallouedec in https://github.com/huggingface/trl/pull/4156_generate for GRPO with replay buffer by @qgallouedec in https://github.com/huggingface/trl/pull/4158<Tip> with new markdown syntax by @qgallouedec in https://github.com/huggingface/trl/pull/4161require_bitsandbytes by @qgallouedec in https://github.com/huggingface/trl/pull/4137clone_chat_template by @qgallouedec in https://github.com/huggingface/trl/pull/4097RewardTrainer refactor by @qgallouedec in https://github.com/huggingface/trl/pull/4093_generate in GRPO/RLOO: list of ints instead of tensors by @qgallouedec in https://github.com/huggingface/trl/pull/4146trainer.tokenizer by trainer.processing_class by @qgallouedec in https://github.com/huggingface/trl/pull/4185sft example script by @sergiopaniego in https://github.com/huggingface/trl/pull/4197Optional from processing_class in PPOTrainer by @sergiopaniego in https://github.com/huggingface/trl/pull/4212sft_video_llm example by @qgallouedec in https://github.com/huggingface/trl/pull/4214trl-internal-testing/tiny-DbrxForCausalLM by @qgallouedec in https://github.com/huggingface/trl/pull/4213_generate in GRPO/RLOO: Use prompt_ids from generation by @qgallouedec in https://github.com/huggingface/trl/pull/4152token_type_ids in DataCollatorForVisionLanguageModeling by @qgallouedec in https://github.com/huggingface/trl/pull/4190_generate in GRPO/RLOO: Rely on generator for prompt truncation by @qgallouedec in https://github.com/huggingface/trl/pull/4153chat_template_kwargs in apply_chat_template by @cmpatino in https://github.com/huggingface/trl/pull/4233RichProgressCallback enhancement by @qgallouedec in https://github.com/huggingface/trl/pull/4245token_type_ids in DPOTrainer by @aweers in https://github.com/huggingface/trl/pull/4285BestOfNSampler by @qgallouedec in https://github.com/huggingface/trl/pull/4291Full Changelog: https://github.com/huggingface/trl/compare/v0.23.0...v0.24.0
♨️ [GRPO] Fix potential hang in get_high_entropy_mask by @akakakakakaa in https://github.com/huggingface/trl/pull/4041
get_high_entropy_mask by @akakakakakaa in https://github.com/huggingface/trl/pull/4041Full Changelog: https://github.com/huggingface/trl/compare/v0.23.0...v0.23.1
SFT now supports Context Parallelism (CP) for training large language models on very large sequences. You can now train with an arbitrarily long seque
SFT now supports Context Parallelism (CP) for training large language models on very large sequences. You can now train with an arbitrarily long sequence length.
<img width="844" height="336" alt="Screenshot 2025-09-09 at 10 39 30 PM" src="https://github.com/user-attachments/assets/f1dfc349-440a-4e05-aac9-439a3c286f08" />
by @kashif in https://github.com/huggingface/trl/pull/3994
Dynamic Fine-Tuning (DFT) is a nnow supported in TRL.
from trl import SFTConfig
training_args = SFTConfig(
loss_type="dft",
...
)
<img width="692" height="472" alt="Screenshot 2025-09-09 at 10 37 36 PM" src="https://github.com/user-attachments/assets/4ee2b4ab-7cc6-4578-bfac-c38124891510" />
by @qgallouedec in https://github.com/huggingface/trl/pull/4042
Different implementations are used for rollout generation (vLLM) and model training. The implementation gap implicitly turns the on-policy RL to be off-policy. Truncated Importance Sampling (TIS) a simple yet effective importance sampling technique for handling such discrepancy. This is now implemented in GRPO.
from trl import GRPOConfig
training_args = GRPOConfig(
...
use_vllm=True,
vllm_importance_sampling_correction=True, # default True
vllm_importance_sampling_cap=2.0, # hyper-parameter C
)
by @LeonEricsson in https://github.com/huggingface/trl/pull/3867
Mixture of Experts (MoE) models require an auxiliary loss to ensure that the different experts are used evenly. This auxiliary loss is now supported in SFTTrainer.
training_args = SFTConfig(
model_init_kwargs={"output_router_logits": True},
...
)
by @pramodith in https://github.com/huggingface/trl/pull/4012
When running GRPO (or RLOO) with vLLM in colocated mode, the vLLM server consume VRAM during optimization while not being used. We now have an option to put the vLLM server to sleep during optimization to free up VRAM.
from trl import GRPOConfig
training_args = GRPOConfig(..., vllm_sleep_enabled=True)
by @edbeeching in https://github.com/huggingface/trl/pull/3968
You can now use vLLM server mode with OnlineDPOTrainer. Additionally, VLM models are now supported.
by @vaelev in https://github.com/huggingface/trl/pull/3783
The paper index has been significantly enhanced with the addition of 9+ new algorithm implementations, providing a more comprehensive resource for users.
by @behroozazarkhalili in https://github.com/huggingface/trl/pull/3990
get_soft_overlong_punishment by @qgallouedec in https://github.com/huggingface/trl/pull/3972args.gradient_checkpointing = False instead of args = dataclasses.replace(args, gradient_checkpointing=False) by @qgallouedec in https://github.com/huggingface/trl/pull/3981torch_dype to dtype everywhere by @sergiopaniego in https://github.com/huggingface/trl/pull/4000average_tokens_across_devices default replacement by @qgallouedec in https://github.com/huggingface/trl/pull/4039AutoModelForImageTextToText for DPO and Online DPO by @sergiopaniego in https://github.com/huggingface/trl/pull/4049SFTTrainer for compatibility with CP by @qgallouedec in https://github.com/huggingface/trl/pull/4038quantization_config=None by @qgallouedec in https://github.com/huggingface/trl/pull/4019Full Changelog: https://github.com/huggingface/trl/compare/v0.22.0...v0.23.0
⚖️ Fix scale_rewards issue in GRPO by @Peter-Chou in https://github.com/huggingface/trl/pull/3992
Full Changelog: https://github.com/huggingface/trl/compare/v0.22.1...v0.22.2
Refactor version retrieval to use importlib.metadata by @qgallouedec
importlib.metadata by @qgallouedecFull Changelog: https://github.com/huggingface/trl/compare/v0.22.0...v0.22.1
🗑️ Deprecate setup_chat_format by @qgallouedec in https://github.com/huggingface/trl/pull/3929
SFTTrainerSFTTrainer now natively supports Vision-Language Models (VLMs). This includes support for both languauge modeling, prompt-completion data.
It also supports training on completion-only.
<img width="1136" height="586" alt="Group 291-6" src="https://github.com/user-attachments/assets/2629b8e7-d853-4b7c-91d5-f4c128287e04" />
from trl import SFTConfig, SFTTrainer
from datasets import load_dataset
trainer = SFTTrainer(
model="Qwen/Qwen2.5-VL-3B-Instruct",
args=SFTConfig(max_length=None),
train_dataset=load_dataset("trl-lib/llava-instruct-mix", split="train"),
)
trainer.train()
by @qgallouedec in https://github.com/huggingface/trl/pull/3862, https://github.com/huggingface/trl/pull/3907 and https://github.com/huggingface/trl/pull/3908
RLOOTrainer refactorRLOOTrainer has been refactored to align with the design principles of other other trainers in the library. You can now use this trainer exactly like GRPO.
from datasets import load_dataset
from trl import RLOOConfig, RLOOTrainer
dataset = load_dataset("trl-lib/ultrafeedback-prompt", split="train")
# Dummy reward function for demonstration purposes
def reward_num_unique_letters(completions, **kwargs):
"""Reward function that rewards completions with more unique letters."""
completion_contents = [completion[0]["content"] for completion in completions]
return [float(len(set(content))) for content in completion_contents]
trainer = RLOOTrainer(
model="Qwen/Qwen2-0.5B-Instruct",
reward_funcs=reward_num_unique_letters,
train_dataset=dataset,
)
trainer.train()
by @shirinyamani in https://github.com/huggingface/trl/pull/3801
You can now levarage Hugging Face Jobs to easily train and deploy your models with TRL.
hf jobs uv run --flavor a100-large --secrets HF_TOKEN "https://raw.githubusercontent.com/huggingface/trl/main/trl/scripts/sft.py" --model_name_or_path Qwen/Qwen2-0.5B --dataset_name trl-lib/Capybara
A guide is available in the docs.
by @sergiopaniego in https://github.com/huggingface/trl/pull/3890
GRPOTrainer now supports DAPO loss type, which aggregates token-level losses by normalizing with the number of active token in the global accumulated batch. This method was introduced to eliminate length bias. Simply use
from trl import GRPOConfig, GRPOTrainer
training_args = GRPOConfig(
loss_type="dapo",
...
)
by @qgallouedec in https://github.com/huggingface/trl/pull/3938
The authors of Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoning (Lite PPO) find that the combination of:
can unlock the learning capability of critic-free policies using vanilla PPO loss. Their results demonstrate that this simple combination consistently improves performance, surpassing strategies like GRPO and DAPO.
TRL supports using these learnings to train a GRPO model by:
from trl import GRPOConfig
training_args = GRPOConfig(
scale_rewards="batch",
loss_type="dapo",
...
)
by @pramodith in https://github.com/huggingface/trl/pull/3935
Bias-Corrected Exponential Moving Average (BEMA) improves the stability and efficiency of language model fine-tuning by reducing stochasticity and eliminating bias. To use BEMA with SFT as described in the paper, you can now use the [BEMACallback]:
from trl import BEMACallback, SFTTrainer
trainer = SFTTrainer(
...
callbacks=[BEMACallback()],
)
by @kashif in https://github.com/huggingface/trl/pull/3855
gradient_checkpointing=True by @qgallouedec in https://github.com/huggingface/trl/pull/3510position_ids for flash_attention_3 by @jue-jue-zi in https://github.com/huggingface/trl/pull/3942setup_chat_format by @qgallouedec in https://github.com/huggingface/trl/pull/3929IterativeSFTTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/3905AutoModelForVision2Seq to AutoModelForImageTextToText by @qgallouedec in https://github.com/huggingface/trl/pull/3836--bf16 value in scripts by @sergiopaniego in https://github.com/huggingface/trl/pull/3869gradient_checkpointing=True by @qgallouedec in https://github.com/huggingface/trl/pull/3510vllm_mode param in GRPO by @sergiopaniego in https://github.com/huggingface/trl/pull/3866unittest.TestCase with TrlTestCase that handles tmp dir by @qgallouedec in https://github.com/huggingface/trl/pull/3863SFTTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/3862IterativeSFTTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/3905use_cache should be set in the forward pass by @qgallouedec in https://github.com/huggingface/trl/pull/3891logger.warning instead of warnings.warn by @qgallouedec in https://github.com/huggingface/trl/pull/3923SFTTrainer in GRPOTrainer by @MengAiDev in https://github.com/huggingface/trl/pull/3919setup_chat_format by @qgallouedec in https://github.com/huggingface/trl/pull/3929position_ids for flash_attention_3 by @jue-jue-zi in https://github.com/huggingface/trl/pull/3942trackio to docs by @sergiopaniego in https://github.com/huggingface/trl/pull/3965Full Changelog: https://github.com/huggingface/trl/compare/v0.21.0...v0.22.0
🪦 Remove deprecated by @qgallouedec in https://github.com/huggingface/trl/pull/3817
<img width="4544" height="2344" alt="Group 293-2" src="https://github.com/user-attachments/assets/17241da2-1b1d-41bc-a5f8-0983ea46606f" />
Open AI GPT OSS models are here! Check out the OpenAI Cookbook to see an example of how to SFT these models.
by @qgallouedec in https://github.com/huggingface/trl/pull/3848
You can now pass vllm_model_impl to the TRL vLLM server.
Example, for transformers backend:
trl vllm-serve ... --vllm_model_impl transformers
by @merveenoyan in https://github.com/huggingface/trl/pull/3773
using_llama_models.md by @bwook00 in https://github.com/huggingface/trl/pull/3794Full Changelog: https://github.com/huggingface/trl/compare/v0.20.0...v0.21.0
Remove deprecated processor.tokenizer by @Tavish9 in https://github.com/huggingface/trl/pull/3720
GSPO is a GRPO variant that computes importance sampling weights at the sequence level instead of per-token.
<img width="930" height="538" alt="Screenshot 2025-07-28 at 10 54 15 PM" src="https://github.com/user-attachments/assets/923835af-dc61-4fd4-8a99-44242d02bb7b" />
📜 Paper: https://huggingface.co/papers/2507.18071
To reproduce the paper's setting, use this configuration:
from trl import GRPOConfig
training_args = GRPOConfig(
importance_sampling_level="sequence",
loss_type="grpo",
steps_per_generation=...,
beta=0.04, # not explicitly specified in the paper, but they likely used the same value as in the GRPO paper
epsilon=3e-4, # https://x.com/ChujieZheng/status/1948933507696525392
)
by @qgallouedec in https://github.com/huggingface/trl/pull/3775
<img width="1136" height="594" alt="Group 291-4" src="https://github.com/user-attachments/assets/04850e80-9689-472d-acd7-fda331e66dc3" />
The GRPOTrainer can now be used for VLM training. Give a try with this dummy example:
from trl import GRPOTrainer
from datasets import load_dataset
# Dummy vision-language dataset
dataset = load_dataset("trl-internal-testing/zen-image", "conversational_prompt_only", split="train")
# Dummy reward function: count the number of unique characters in the completions
def reward_num_unique_chars(completions, **kwargs):
return [len(set(c[0]["content"])) for c in completions]
trainer = GRPOTrainer(
model="Qwen/Qwen2.5-VL-3B-Instruct",
reward_funcs=[reward_num_unique_chars],
train_dataset=dataset,
)
trainer.train()
by @CompN3rd and @kashif in https://github.com/huggingface/trl/pull/3072 in https://github.com/huggingface/trl/pull/3760
<img width="440" height="438" alt="Screenshot 2025-07-28 at 10 52 15 PM" src="https://github.com/user-attachments/assets/e07a7936-c4c5-480d-9ffd-db5b77a5445e" />
The DPO trainer supports combining multiple loss functions with different weights, enabling more sophisticated optimization strategies. This is particularly useful for implementing algorithms like MPO (Mixed Preference Optimization). MPO is a training approach that combines multiple optimization objectives, as described in the paper Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization.
To combine multiple losses, specify the loss types and corresponding weights as lists:
from trl import DPOConfig
# MPO: Combines DPO (sigmoid) for preference and BCO (bco_pair) for quality
training_args = DPOConfig(
loss_type=["sigmoid", "bco_pair", "sft"], # Loss types to combine
loss_weights=[0.8, 0.2, 1.0] # Corresponding weights, as used in the MPO paper
)
by @qgallouedec in https://github.com/huggingface/trl/pull/2544
Continuous Batching allows for faster generation using the transformers backend. You can now use it with the GRPOTrainer by setting use_transformers_paged=True in the config.
use_transformers_paged = True
from trl import GRPOConfig
training_args = GRPOConfig(
# ... other args
use_transformers_paged=Ture,
)
by @ArthurZucker in https://github.com/huggingface/trl/pull/3471
<img width="788" height="438" alt="Screenshot 2025-07-28 at 10 27 20 PM" src="https://github.com/user-attachments/assets/8073a5db-a98e-4534-aea9-d3dbd2e75f4a" />
In Beyond the 80/20 Rule: High-Entropy Minority Tokens
Drive Effective Reinforcement Learning for LLM Reasoning, it is shown that utilizing only 20% of the highest entropy tokens leads to similar performance as using all tokens. You can now enable this feature in the GRPOTrainer by setting entropy_filtering=True in the config.
from trl import GRPOConfig
training_args = GRPOConfig(
# ... other args
top_entropy_quantile=0.2, # Use only the top 20% of tokens based on entropy
)
by @pramodith in https://github.com/huggingface/trl/pull/3563
GRPO now supports FSDP2 training. Just run your script with an FSDP2 config:
accelerate launch --config_file examples/accelerate_configs/fsdp2.yaml run_grpo.py
by @SalmanMohammadi in https://github.com/huggingface/trl/pull/3687
position_ids computation for FFD packing by @mariosasko in https://github.com/huggingface/trl/pull/3649dpo_trainer.py by @bvantuan in https://github.com/huggingface/trl/pull/3631seq_lengths to signature columns by @LeonEricsson in https://github.com/huggingface/trl/pull/3699processor.tokenizer by @Tavish9 in https://github.com/huggingface/trl/pull/3720--bf16 flag from training scripts by @qgallouedec in https://github.com/huggingface/trl/pull/3724wandb.run.url instead of wandb.run.get_url() (deprecated) by @qgallouedec in https://github.com/huggingface/trl/pull/3726processing_class docs for trainers by @sergiopaniego in https://github.com/huggingface/trl/pull/3737processing_class docs for rest of trainers by @sergiopaniego in https://github.com/huggingface/trl/pull/3745average_tokens_across_devices by @qgallouedec in https://github.com/huggingface/trl/pull/3746steps_per_generation in vllm max_num_seqs by @akakakakakaa in https://github.com/huggingface/trl/pull/3747AlignPropTrainer and DDPOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/3755GeometricMixtureWrapper by @qgallouedec in https://github.com/huggingface/trl/pull/3779clone_chat_template vocab size and support PEFT instruction tuning by @qgallouedec in https://github.com/huggingface/trl/pull/3763pixel_attention_mask (SmolVLM2) and image_sizes (LLaVa-Next) by @kashif in https://github.com/huggingface/trl/pull/3760max_length value and include visualization tool by @qgallouedec in https://github.com/huggingface/trl/pull/3630Full Changelog: https://github.com/huggingface/trl/compare/v0.19.0...v0.20.0
fix grpo generation_kwargs by @ahatamiz in https://github.com/huggingface/trl/pull/3634
Full Changelog: https://github.com/huggingface/trl/compare/v0.19.0...v0.19.1
SFTTrainer now supports training with tools! You just have to add a column tools to your dataset, which contains a list of tool definitions as json sc
SFTTrainer now supports training with tools! You just have to add a column tools to your dataset, which contains a list of tool definitions as json schemas. The tools will be automatically registered and can be used in the training process.
from datasets import Dataset
from transformers.utils import get_json_schema
from trl import SFTTrainer
# Fictitious functions to simulate tool calls
def start_timer(duration: int) -> int:
"""
Starts a timer for the specified duration in seconds.
Args:
duration: Duration in seconds to set the timer for.
Returns:
The duration set for the timer.
"""
return duration
def create_reminder(time: str, note: str) -> str:
"""
Creates a reminder for the specified time and note.
Args:
time: The time for the reminder.
note: The note for the reminder.
Returns:
A confirmation message indicating that the reminder has been set.
"""
return "I'll remind you to call mom at 7 PM."
# Define the JSON schemas for the tools
start_timer = get_json_schema(start_timer)
create_reminder = get_json_schema(create_reminder)
dataset = Dataset.from_dict({
"messages": [
[
{"role": "user", "content": "Set a timer for 10 minutes."},
{"role": "assistant", "tool_calls": [{"type": "function", "function": {"name": "start_timer", "arguments": {"duration": 600}}}]},
{"role": "tool", "name": "start_timer", "content": "600"},
{"role": "assistant", "content": "Timer set for 10 minutes."},
],
...,
],
"tools": [
[start_timer, create_reminder],
...,
]
})
# Initialize the trainer
trainer = SFTTrainer(model="Qwen3-0.6B", train_dataset=dataset)
# Train the model
trainer.train()
by @qgallouedec in https://github.com/huggingface/trl/pull/3597
We introduce a new packing method: FFD (First Fit Decreasing) packing. This method is designed to optimize the packing of sequences in a way that more efficiently reduces the size of the training dataset by grouping examples more effectively. Previously, we used a wrapped packing method, which often truncated sequences even when they were not longer than the maximum sequence length. The new FFD packing method avoids unnecessary truncation by grouping sequences more intelligently. This new packing strategy is now the default when packing is enabled.
training_args = SFTConfig(..., packing=True)
by @qgallouedec in https://github.com/huggingface/trl/pull/3521 and accelerated by @mariosasko in https://github.com/huggingface/trl/pull/3537
The DPOTrainer now supports the Liger-powered DPO loss, enabling faster training with lower memory usage.
training_args = DPOConfig(..., use_liger_loss=True)
by @kashif in https://github.com/huggingface/trl/pull/2568
setup_chat_format and add clone_chat_templateWe introduce clone_chat_template, a more convenient and flexible function for setting up chat templates from any tokenizer that already includes one. It handles EOS tokens and copies all added tokens from the source tokenizer, preserving their "special" status.
You can either use this function directly:
from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import clone_chat_template
model = AutoModelForCausalLM.from_pretrained("facebook/opt-350m")
tokenizer = AutoTokenizer.from_pretrained("facebook/opt-350m")
model, tokenizer = clone_chat_template(model, tokenizer, "Qwen/Qwen3-4B")
or use the chat_template_path parameter in SFTConfig to specify a chat template, which will be automatically cloned when the SFTTrainer is initialized.
from trl import SFTConfig
training_args = SFTConfig(chat_template_path="Qwen/Qwen3-4B")
by @qgallouedec in https://github.com/huggingface/trl/pull/3404 and https://github.com/huggingface/trl/pull/3599
SFTTrainer now supports passing additional keyword arguments to the chat template. This allows for more flexibility in customizing the chat format during training. To enable it, just add a chat_template_kwargs column to your your dataset.
example = {'messages': [{'content': 'What is better than ugly?', 'role': 'user'},
{'content': 'Beautiful.', 'role': 'assistant'}]
'chat_template_kwargs': {'my_template_arg': 'my_value'}}
by @qgallouedec in https://github.com/huggingface/trl/pull/3609
The SFTTrainer now supports training on assistant messages only
example = {'messages': [
{'role': 'user', 'content': 'What is better than ugly?'}, # masked in the loss
{'role': 'assistant', 'content': 'Beautiful.'}, # used in the loss
{'role': 'user', 'content': 'And what is better than implicit?'}, # masked in the loss
{'role': 'assistant', 'content': 'Explicit.'}, # used in the loss
]}
by @qgallouedec in https://github.com/huggingface/trl/pull/3586
generation_kwargs as a property of GRPOConfig to support additional generation argumentsThe GRPOConfig now includes a generation_kwargs property, allowing users to specify additional generation arguments for the GRPOTrainer. This allows for further customization of the generation behavior, such as setting suppress_tokens, num_beams, etc.
Depending on the generation backend used (transformers or vLLM), this property will be passed either to transformers.GenerationConfig (if using transformers) or vllm.SamplingParams (if using vLLM).
from trl import GRPOConfig
training_args = GRPOConfig(..., generation_kwargs={"length_penalty": -0.1})
by @pramodith in https://github.com/huggingface/trl/pull/3617
beta=0.0 for GRPO by @qgallouedec in https://github.com/huggingface/trl/pull/3516logging_steps=10 by @qgallouedec in https://github.com/huggingface/trl/pull/3514bf16=True by @qgallouedec in https://github.com/huggingface/trl/pull/3515IterableDataset in DPO Trainer by @h-tonywu in https://github.com/huggingface/trl/pull/3559labels are retained in self._signature_columns by @sxndqc in https://github.com/huggingface/trl/pull/3589vllm_gpu_memory_utilization recommendation script by @toslali-ibm in https://github.com/huggingface/trl/pull/3554setup.cfg by @qgallouedec in https://github.com/huggingface/trl/pull/3511beta=0.0 for GRPO by @qgallouedec in https://github.com/huggingface/trl/pull/3516logging_steps=10 by @qgallouedec in https://github.com/huggingface/trl/pull/3514getattr to get gradient_checkpointing by @qgallouedec in https://github.com/huggingface/trl/pull/3535_VALID_DICT_FIELDS by @qgallouedec in https://github.com/huggingface/trl/pull/3553torch.autocast and make it cover XPU by @yao-matrix in https://github.com/huggingface/trl/pull/3541IterableDataset in DPO Trainer by @h-tonywu in https://github.com/huggingface/trl/pull/3559bf16=True by @qgallouedec in https://github.com/huggingface/trl/pull/3515setup_chat_format and add clone_chat_template by @qgallouedec in https://github.com/huggingface/trl/pull/3404logging_steps parameter from for simpler setup by @qgallouedec in https://github.com/huggingface/trl/pull/3612Trainer::create_model_card by @LeonEricsson in https://github.com/huggingface/trl/pull/3613enforce_eager default value in vLLM server. by @LeonEricsson in https://github.com/huggingface/trl/pull/3607labels are retained in self._signature_columns by @sxndqc in https://github.com/huggingface/trl/pull/3589vllm_gpu_memory_utilization recommendation script by @toslali-ibm in https://github.com/huggingface/trl/pull/3554max_prompt_length) with vLLM. by @LeonEricsson in https://github.com/huggingface/trl/pull/3601generation_kwargs as a property of GRPOConfig to support additional generation arguments. by @pramodith in https://github.com/huggingface/trl/pull/3617chat_template_path parameter to SFTConfig by @qgallouedec in https://github.com/huggingface/trl/pull/3599Full Changelog: https://github.com/huggingface/trl/compare/v0.18.0...v0.19.0
🏗️ Add test for training with multiple dataloader workers and update worker initialization for compatibility with transformers 4.52.0 by @qgallouedec
Full Changelog: https://github.com/huggingface/trl/compare/v0.18.1...v0.18.2
📎 Fix clip ratio logging by @qgallouedec in https://github.com/huggingface/trl/pull/3506
setup.cfg by @qgallouedec in https://github.com/huggingface/trl/pull/3511Full Changelog: https://github.com/huggingface/trl/compare/v0.18.0...v0.18.1
⚰️ Remove deprecated by @qgallouedec in https://github.com/huggingface/trl/pull/3364
GRPOTrainer and SFTTrainer by @I-l-l-I in https://github.com/huggingface/trl/pull/3337TextEnvironment and tools by @lewtun in https://github.com/huggingface/trl/pull/3389formatting_func is used with completion_only_loss by @LeonEricsson in https://github.com/huggingface/trl/pull/3385setup.py to setup.cfg and make rich an optional dep by @qgallouedec in https://github.com/huggingface/trl/pull/3403trl env on xpu by @yao-matrix in https://github.com/huggingface/trl/pull/3438base_url parameter for vLLM client initialization by @re-imagined in https://github.com/huggingface/trl/pull/3324keep_end leading to zero'd out samples by @LeonEricsson in https://github.com/huggingface/trl/pull/3398Full Changelog: https://github.com/huggingface/trl/compare/v0.17.0...v0.18.0
⚰️ Remove deprecated by @qgallouedec in https://github.com/huggingface/trl/pull/3153
The TRL v0.17 release introduces three major changes that, together, enable significantly faster generation performance in GRPO—up to 10x faster in some configurations.
These three changes are:
Below, we provide a summary of these changes and how to use them.
The TRL vLLM server now supports data parallelism (DP), enabling significantly faster generation speeds—especially for smaller models. This new feature can be used by adding the --data_parallel_size N argument when launching the vLLM server.
trl vllm-serve --model Qwen/Qwen2.5-14B-Instruct --tensor_parallel_size 2 --data_parallel_size 2
by @qgallouedec in https://github.com/huggingface/trl/pull/3310
Previously, GRPO made one generation request per global batch. The global batch is the total of all local batches, without accounting for gradient accumulation. In other words, if the gradient accumulation step was 8, GRPO would make 8 generation requests per training step.
Now, GRPO groups these global batches into a single "effective batch" and makes only one generation request per effective batch. Since vLLM applies optimizations that are especially effective for large batches, this new approach leads to significantly faster training overall.
No changes are required in the training script, as this is handled internally by the GRPO trainer.
by @qgallouedec in https://github.com/huggingface/trl/pull/3283
vLLM provides two versions of its engine (V0 and V1), and V1 is significantly faster. This version is now supported by TRL and requires vLLM version 0.8.3 or higher.
by @I-l-l-I in https://github.com/huggingface/trl/pull/3276
Disabling dropout has shown to stabilize training. You can now disable dropout in GRPO by setting the disable_dropout argument to False in the GRPO config.
from trl import GRPOConfig
training_args = GRPOConfig(..., disable_dropout=True)
by @edbeeching in https://github.com/huggingface/trl/pull/3234
GRPO now supports the various losses proposed in the recent literature, including the Dr. GRPO loss. The loss type can be set in the GRPO config:
from trl import GRPOConfig
training_args = GRPOConfig(..., loss_type="dr_grpo")
by @qgallouedec in https://github.com/huggingface/trl/pull/3256
The GRPO trainer now has an option to disable shuffling of the training dataset. This is useful for curriculum learning, where the order of the training data is important.
from trl import GRPOConfig
training_args = GRPOConfig(..., shuffle_dataset=False)
by @LeonEricsson in https://github.com/huggingface/trl/pull/3334
Overlong filtering has been shown to significantly stabilize learning and improve performance. You can now use it in TRL!
It simply consists in masking the loss of truncated samples
from trl import GRPOConfig
training_args = GRPOConfig(..., mask_truncated_completions=True)
by @shirinyamani in https://github.com/huggingface/trl/pull/3248
Liger allows to significantly reduce the memory peak of the loss computation. You can now use it in TRL with the use_liger_loss argument in the GRPO config:
from trl import GRPOConfig
training_args = GRPOConfig(..., use_liger_loss=True)
by @shivam15s in https://github.com/huggingface/trl/pull/3184
clip_ratio logging and better document logged values by @qgallouedec in https://github.com/huggingface/trl/pull/3145clip_ratio logging and better document logged values by @qgallouedec in https://github.com/huggingface/trl/pull/3145worker_cls as string by @qgallouedec in https://github.com/huggingface/trl/pull/3159ConstantLengthDataset by @qgallouedec in https://github.com/huggingface/trl/pull/3242formatting_func by @YeFD in https://github.com/huggingface/trl/pull/3147is_liger_kernel_available with min version by @qgallouedec in https://github.com/huggingface/trl/pull/3266test_raise_error_not_causallm by @qgallouedec in https://github.com/huggingface/trl/pull/3265_generate_and_score_completions by @syt-nju in https://github.com/huggingface/trl/pull/3336max_prompt_length < max_length by @LeonEricsson in https://github.com/huggingface/trl/pull/3341Full Changelog: https://github.com/huggingface/trl/compare/v0.16.0...v0.17.0
😷 Fix SFT masking EOS when equal to PAD by @qgallouedec in https://github.com/huggingface/trl/pull/3200
learning_rate argument to _maybe_log_save_evaluate by @qgallouedec in https://github.com/huggingface/trl/pull/3206Full Changelog: https://github.com/huggingface/trl/compare/v0.16.0...v0.16.1
⚰️ Remove deprecated by @qgallouedec in https://github.com/huggingface/trl/pull/2894
Previously, vLLM could only be used by dedicating a single GPU, preventing both the scalability benefits of vLLM and multi-node training. This limitation has now been removed!
GRPO can now scale efficiently with models exceeding 70B parameters, supporting multi-node training with super-fast performance.
To take advantage of this, simply launch a vLLM server using the following command:
trl vllm-serve --model <model_name> --tensor_parallel_size <tp_size>
Then, start GRPO training with use_vllm=True.
Below is a comparison of GRPO throughput with and without vLLM, across different TP values and model sizes.
@binary-husky and @qgallouedec in https://github.com/huggingface/trl/pull/3094
This release introduces the multi-step trick, which allows for the reuse of generated data across multiple steps, speeding up the training process.
To support this, we've implemented importance sampling and clipping logic. This enhancement should lead to significant improvements in training speed.
<img width="1097" alt="Screenshot 2025-03-23 at 14 52 28" src="https://github.com/user-attachments/assets/8f1ee339-63c5-43cf-9b0f-5395432513ae" />
To use it, simply set num_iterations to a value greater than 1.
training_args = GRPOConfig(..., num_iterations=4)
by @qgallouedec in https://github.com/huggingface/trl/pull/2899
As demonstrated in Dr GRPO, sequence-level normalization can introduce a response level length bias.
To address this, we have now switched to normalizing the loss and by the total number of tokens in the batch, ensuring more consistent and unbiased training.
- loss = ((per_token_loss * completion_mask).sum(dim=1) / completion_mask.sum(dim=1)).mean()
+ loss = (per_token_loss * completion_mask).sum() / completion_mask.sum()
by @edbeeching in https://github.com/huggingface/trl/pull/2881
As demonstrated in Dr GRPO, scaling rewards can introduce a question-level difficulty bias. To address this, we have now added an option to disable reward scaling in GRPO.
training_args = GRPOConfig(..., scale_rewards=False)
advantages = rewards - mean_grouped_rewards
- advantages = advantages / std_grouped_rewards
+ if self.args.scale_rewards:
+ advantages = advantages / std_grouped_rewards
it's likely that we'll make this (scale_rewards=False) the default behavior in the future.
by @qgallouedec in https://github.com/huggingface/trl/pull/3135
When optimizing across multiple domains, not all reward functions are relevant for every sample. For example, a math verifier's reward does not apply to grammar samples, and a grammar verifier's reward does not apply to math samples.
It is now possible to return None for rewards that do not make sense for a given sample. For instance, when the domain is specified in a column like domain, you can implement it as follows:
def math_reward(completions, domain, **kwargs):
rewards = []
for completion, dom in zip(completions, domain):
if dom == "math":
rewards.append(verify(completion))
else:
rewards.append(None)
return rewards
This allows for more domain-specific reward handling, ensuring that irrelevant rewards are ignored and don’t interfere with optimization.
by @shirinyamani in https://github.com/huggingface/trl/pull/3079
beta == 0.0It has been observed that not minimizing the KL divergence between the trained model and the reference model can still yield good results, while significantly reducing memory usage and compute. This is because there is no need to store the reference model in memory or perform a forward pass for it.
When beta is set to 0.0, the reference model is not loaded, and the KL divergence is not computed, leading to savings in both time and memory.
training_args = GRPOConfig(..., beta=0.0)
by @ingambe in https://github.com/huggingface/trl/pull/2806
Padding-free batching is an alternative approach to packing for reducing memory usage. In this method, a batch is first sampled and then flattened into a single sequence, avoiding padding. Unlike packing, which can result in incomplete sequences by combining parts of different samples, padding-free batching ensures that all sequences remain complete and intact.
To enable padding-free batching in SFT, simply set padding_free=True in the SFTConfig, and make sure to use flash_attention2 as the attention implementation.
training_args = SFTConfig(..., padding_free=True, model_init_kwargs={"attn_implementation": "flash_attention2"})
by @qgallouedec in https://github.com/huggingface/trl/pull/3076
As outlined in the DAPO paper, increasing the upper bound epsilon leads to higher entropy during generation, promoting better exploration. To enable this, we’ve added support for adjusting the upper bound epsilon directly in the default GRPO trainer.
training_args = GRPOConfig(epsilon_high=0.28)
by @shirinyamani in https://github.com/huggingface/trl/pull/3118
unwrap_model_for_generation for DeepSpeed Stage-3 compatibility by @kiddj in https://github.com/huggingface/trl/pull/2871inference_mode to no_grad when computing old_per_token_logps by @qgallouedec in https://github.com/huggingface/trl/pull/2987maybe_convert_to_chatml map for conversational datasets in SFT by @kashif in https://github.com/huggingface/trl/pull/2862max_seq_length to max_length by @qgallouedec in https://github.com/huggingface/trl/pull/2895 and https://github.com/huggingface/trl/pull/2947num_tokens and some logging fixes by @qgallouedec in https://github.com/huggingface/trl/pull/3006maybe_convert_to_chatml map for conversational datasets in SFT by @kashif in https://github.com/huggingface/trl/pull/2862max_seq_length to max_length by @qgallouedec in https://github.com/huggingface/trl/pull/2895max_seq_length to max_length in SFTConfig by @qgallouedec in https://github.com/huggingface/trl/pull/2947max_tool_response by @shenxiangzhuang in https://github.com/huggingface/trl/pull/2921enable_prefix_caching by @ji-huazhong in https://github.com/huggingface/trl/pull/2900num_iterations by @qgallouedec in https://github.com/huggingface/trl/pull/2966KTOConfig by @sileod in https://github.com/huggingface/trl/pull/2912unwrap_model_for_generation for DeepSpeed Stage-3 compatibility by @kiddj in https://github.com/huggingface/trl/pull/2871inference_mode to no_grad when computing old_per_token_logps by @qgallouedec in https://github.com/huggingface/trl/pull/2987num_tokens and some logging fixes by @qgallouedec in https://github.com/huggingface/trl/pull/3006apply_chat_template by @jamesbraza in https://github.com/huggingface/trl/pull/3005deepspeed>=0.16.4's rename by @jamesbraza in https://github.com/huggingface/trl/pull/2963GPROTrainer.generation_config by @jamesbraza in https://github.com/huggingface/trl/pull/3046SFTTrainer.compute_loss crash with accelerate by @jamesbraza in https://github.com/huggingface/trl/pull/3048SFTTrainer.compute_loss hang from #3048's PR comments by @jamesbraza in https://github.com/huggingface/trl/pull/3056Full Changelog: https://github.com/huggingface/trl/compare/v0.15.0...v0.16.0
♻️ Fix caching in SFT by @qgallouedec in #2945
Full Changelog: https://github.com/huggingface/trl/compare/v0.15.1...v0.15.2
💬 Add maybe_convert_to_chatmlmap for conversational datasets by @kashif in SFT in #2862
maybe_convert_to_chatmlmap for conversational datasets by @kashif in SFT in #2862Full Changelog: https://github.com/huggingface/trl/compare/v0.15.0...v0.15.1
☠️ Remove deprecated by @qgallouedec in https://github.com/huggingface/trl/pull/2692
Coming soon
trl.templates in excluded packages by @qgallouedec in https://github.com/huggingface/trl/pull/2690num_logits_to_keep to logits_to_keep by @qgallouedec in https://github.com/huggingface/trl/pull/2721per_device_batch_size as generations per device by @qgallouedec in https://github.com/huggingface/trl/pull/2776set_seed() call in GRPO to ensure unique seed for each process by @qgallouedec in https://github.com/huggingface/trl/pull/2824tokenizer parameter to processing_class in tests by @qgallouedec in https://github.com/huggingface/trl/pull/2828Full Changelog: https://github.com/huggingface/trl/compare/v0.9.6...v0.15.0
⚰️ Remove deprecated by @qgallouedec in https://github.com/huggingface/trl/pull/2485
by @qgallouedec in https://github.com/huggingface/trl/pull/2565
disable_dropout by @qgallouedec in https://github.com/huggingface/trl/pull/2511PreferenceCollator to DataCollatorForPreference by @qgallouedec in https://github.com/huggingface/trl/pull/2510formatting_func's documentation in ConstantLengthDataset by @SamuelLarkin in https://github.com/huggingface/trl/pull/2549max_prompt_length parameter in tests by @qgallouedec in https://github.com/huggingface/trl/pull/2588max_seq_length instead of max_length by @skandermoalla in https://github.com/huggingface/trl/pull/2590max_prompt_length and loop usage in logp computation by @qgallouedec in https://github.com/huggingface/trl/pull/2598truncation_mode in DPOTrainer by @anakin87 in https://github.com/huggingface/trl/pull/2551num_logits_to_keep to reduce memory usage in GRPO by @qgallouedec in https://github.com/huggingface/trl/pull/2683Full Changelog: https://github.com/huggingface/trl/compare/v0.13.0...v0.14.0
config has been deprecated in favor of args in PPOTrainer.
We introduced a new trainer to train Process-supervised Reward Model (PRM) in TRL. A PRM rewards the quality of intermediate steps, promoting structured reasoning over focusing solely on the final outcome.With this trainer, we introduce a new dataset type: Stepwise supervision, which is a variant of the prompt-completion type, but for which completion is divided into several intermediate steps, and each step is associated with a label. Find out more in the stepwise-supervision section in the TRL documentation.
Here is an example of how to use the PRMTrainer to train a PRM on the Math Shepherd dataset:
# train_prm.py
from datasets import load_dataset
from trl import PRMConfig, PRMTrainer
from transformers import AutoModelForTokenClassification, AutoTokenizer
model = AutoModelForTokenClassification.from_pretrained("Qwen/Qwen2-0.5B", num_labels=2)
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2-0.5B")
train_dataset = load_dataset("trl-lib/math_shepherd", split="train[:10%]")
training_args = PRMConfig(output_dir="Qwen2-0.5B-Reward-Math-Sheperd", logging_steps=10)
trainer = PRMTrainer(model=model, args=training_args, processing_class=tokenizer, train_dataset=train_dataset)
trainer.train()
For more information, check out the PRMTrainer documentation.
by @qgallouedec and @gaetanlop in https://github.com/huggingface/trl/pull/2127 and https://github.com/huggingface/trl/pull/2148
MergeModelCallBackVarious works show that model merging can non-trivially improve performance, especially if the models belong to the same architecture. TRL now features a callback that merges the reference model with the current policy and optionally pushes the merged checkpoint to the Hub. This could be done on step/epoch end and/or the end of training. This callback uses Arcee's mergekit lib: https://github.com/arcee-ai/mergekit
from trl import DPOTrainer, MergeModelCallback
from trl.mergekit_utils import MergeConfig
config = MergeConfig()
merge_callback = MergeModelCallback(config)
trainer = DPOTrainer(..., callbacks=[merge_callback])
by @August-murr in https://github.com/huggingface/trl/pull/2282
TRL preprocessing utils now support tooling. A first step toward agent fine-tuning.
from trl import apply_chat_template
def get_current_temperature(location: str):
"""
Gets the temperature at a given location.
Args:
location: The location to get the temperature for
"""
return 22.0
example = apply_chat_template(example, tokenizer, tools=[get_current_temperature])
by @August-murr in https://github.com/huggingface/trl/pull/2455
DPOTrainerVLMs have their own specificities which require special treatment in the trainer. DPOTrainer now supports LLaVA-Next models natively.
model = model = AutoModelForVision2Seq.from_pretrained("llava-hf/llava-v1.6-mistral-7b-hf")
trainer = DPOTrainer(model=model, ...)
by @chenweize1998 in https://github.com/huggingface/trl/pull/2413
TRL CLI has been refactored to be more user-friendly and easy to extend. We plan to extend the support to all trainers soon.
(simplified output, for readibility)
$ trl dpo --help
usage: trl dpo [-h] --dataset_name DATASET_NAME [--dataset_config DATASET_CONFIG] --output_dir OUTPUT_DIR [--loss_type {sigmoid,hinge,ipo}]
options:
-h, --help show this help message and exit
--dataset_name DATASET_NAME, --dataset-name DATASET_NAME
--dataset_config DATASET_CONFIG, --dataset-config DATASET_CONFIG
--output_dir OUTPUT_DIR, --output-dir OUTPUT_DIR
The output directory where the model predictions and checkpoints will be written. (default: None)
--loss_type {sigmoid,hinge,ipo}, --loss-type {sigmoid,hinge,ipo}
by @qgallouedec in https://github.com/huggingface/trl/pull/2380 and https://github.com/huggingface/trl/pull/2412
TRL features a new judge AllTrueJudge that unifies the decision of multiple binary judges. This judge implements the Mixture of Judges as described in the CGPO paper.
from trl import AllTrueJudge, BaseBinaryJudge
class RandomBinaryJudge(BaseBinaryJudge):
"""
Random binary judge, for testing purposes.
"""
def judge(self, prompts, completions, gold_completions=None, shuffle_order=True):
return [random.choice([0, 1, -1]) for _ in range(len(prompts))]
prompts = ["The capital of France is", "The biggest planet in the solar system is"]
completions = [["Paris", "Marseille"], ["Saturn", "Jupiter"]]
judge = AllTrueJudge(judges=[RandomBinaryJudge(), RandomBinaryJudge()])
judgements = judge.judge(prompts=prompts, completions=completions)
print(judgements) # [0, 1]
by @gaetanlop in https://github.com/huggingface/trl/pull/2159
num_logits_to_keep to save memorySave memory by only keeping the top num_logits_to_keep logits in the DPO trainer.
training_args = DPOConfig(..., use_num_logits_to_keep=True)
by @xyangk in https://github.com/huggingface/trl/pull/2129
The DiscoPOP paper uses LLMs to discover more efficient offline preference optimization losses. In the paper the proposed DiscoPOP loss (which is a log-ratio modulated loss) outperformed other optimization losses on different tasks (IMDb positive text generation, Reddit TLDR summarization, and Alpaca Eval 2.0).
training_args = DPOConfig(..., loss_type="discopop", discopop_tau=0.05)
by @fanconic in https://github.com/huggingface/trl/pull/2323
DPOTrainer for reference modelWe can now control the batch size for precomputing reference model logits.
training_args = DPOConfig(
...
precompute_ref_log_probs=True,
precompute_ref_batch_size=4,
)
by @SwayamInSync in https://github.com/huggingface/trl/pull/2426
SFTTrainer has supported packing datasets for faster training. Now, it support packing tokenized datasets as well.
by @kmehant in https://github.com/huggingface/trl/pull/2011
PPOTrainerPPOTrainer now supports PEFT for efficient training.
PPOTrainer(
...,
peft_config=peft_config,
)
by @ccs96307 in https://github.com/huggingface/trl/pull/2344
config in favor of args in PPOTrainerconfig has been deprecated in favor of args in PPOTrainer.
PPOTrainer(
- config=training_args,
+ args=training_args,
)
by @qgallouedec in https://github.com/huggingface/trl/pull/2384
policy in favor of model in PPOTrainerpolicy has been deprecated in favor of model in PPOTrainer.
PPOTrainer(
- policy=model,
+ model=model,
)
by @qgallouedec in https://github.com/huggingface/trl/pull/2386
0.13.0.dev0 by @qgallouedec in https://github.com/huggingface/trl/pull/2305token_id instead of token in DPOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/2324output_layer to the list of lm_head_namings in AutoModelForCausalLMWithValueHead by @qgallouedec in https://github.com/huggingface/trl/pull/2328tokenizer arg back and add deprecation guidelines by @qgallouedec in https://github.com/huggingface/trl/pull/2348tokenizer argument in BCO, GKD, Iterative SFT, Nash MD and XPO by @qgallouedec in https://github.com/huggingface/trl/pull/2349use_soft_judge option to WinRateCallback by @kashif in https://github.com/huggingface/trl/pull/2347GeometricMixtureWrapper.forward by @kashif in https://github.com/huggingface/trl/pull/2345data_collator in RLOOTrainer and PPOTrainer by @bartoszzuk in https://github.com/huggingface/trl/pull/2360PPOTrainer by @ccs96307 in https://github.com/huggingface/trl/pull/2344require_bitsandbytes by @qgallouedec in https://github.com/huggingface/trl/pull/2370start_time to _maybe_log_save_evaluate by @qgallouedec in https://github.com/huggingface/trl/pull/2373MergeModelCallBack by @August-murr in https://github.com/huggingface/trl/pull/2282start_time parameter by @qgallouedec in https://github.com/huggingface/trl/pull/2381config in favor of args in PPOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/2384policy in favor of model in PPOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/2386KTOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/2394SmolVLM models via standalone script sft_vlm_smol_vlm.py by @sergiopaniego in https://github.com/huggingface/trl/pull/2409AutoModelForCausalLMWithValueHead by @qgallouedec in https://github.com/huggingface/trl/pull/2398DPOTrainer by @chenweize1998 in https://github.com/huggingface/trl/pull/2413DPOTrainer for reference model by @SwayamInSync in https://github.com/huggingface/trl/pull/2426TrlParser by @qgallouedec in https://github.com/huggingface/trl/pull/2412max_steps calculation in RLOOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/2433datast_config to ScriptArguments by @qgallouedec in https://github.com/huggingface/trl/pull/2440ref_model in OnlineDPOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/2417model_args by @qgallouedec in https://github.com/huggingface/trl/pull/2442tests_latest.yml workflow file by @qgallouedec in https://github.com/huggingface/trl/pull/2457BitsAndBytesConfig import in doc by @August-murr in https://github.com/huggingface/trl/pull/2478Full Changelog: https://github.com/huggingface/trl/compare/v0.12.0...v0.13.0
Pin transformers version <4.47 by @kashif in https://github.com/huggingface/trl/pull/2447
Full Changelog: https://github.com/huggingface/trl/compare/v0.12.1...v0.12.2
👈 Add tokenizer arg back and add deprecation guidelines by @qgallouedec in https://github.com/huggingface/trl/pull/2348
tokenizer arg back and add deprecation guidelines by @qgallouedec in https://github.com/huggingface/trl/pull/2348Full Changelog: https://github.com/huggingface/trl/compare/v0.12.0...v0.12.1
The PPOv2 trainer has been renamed to PPO. The old PPO trainer has been removed. PPOv2 is now deprecated and will be removed in the next release.
Online DPO intially only supported a reward model that had the same tokenizer and chat template as the trained model. Now, you can use any reward model.
from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoModelForSequenceClassification, AutoTokenizer
from trl import OnlineDPOConfig, OnlineDPOTrainer
model = AutoModelForCausalLM.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_config.model_name_or_path, padding_side="left")
reward_model = AutoModelForSequenceClassification.from_pretrained(training_args.reward_model_path, num_labels=1)
reward_tokenizer = AutoTokenizer.from_pretrained(reward_model_name, truncation=True, truncation_side="left")
dataset = load_dataset(script_args.dataset_name)
training_args = OnlineDPOConfig(output_dir="...")
trainer = OnlineDPOTrainer(
model=model,
reward_model=reward_model,
args=training_args,
train_dataset=dataset,
processing_class=tokenizer,
reward_processing_class=reward_tokenizer,
)
trainer.train()
by @qgallouedec in https://github.com/huggingface/trl/pull/2276
PPOv2 -> PPOThe PPOv2 trainer has been renamed to PPO. The old PPO trainer has been removed. PPOv2 is now deprecated and will be removed in the next release.
- trainer = PPOv2Trainer(...)
+ trainer = PPOTrainer(...)
by @qgallouedec in https://github.com/huggingface/trl/pull/2174
ScriptArgumentsWe had ScriptArguments, SFTScriptArguments, DPOScriptArguments and RewardScriptArguments. Since they all share mostly the same fields, we've merged them into a single ScriptArguments class.
SFTScriptArguments, DPOScriptArguments and RewardScriptArguments still exist but are deprecated and will be removed in the next release.
- script_args = DPOScriptArguments(...)
+ script_args = ScriptArguments(...)
by @qgallouedec in https://github.com/huggingface/trl/pull/2145
The PairRMJudge now when called via the judge method has a flag return_scores that returns the probability scores of the first completion of the pair (instead of the rank of the preferred completion). The logits for the probability score can be scaled by an optional temperature parameter.
from trl import PairRMJudge
pairrm_judge = PairRMJudge()
prompts = ["Translate 'hello' to French", "What's the capital of Japan?"]
completions = [["Bonjour", "Salut"], ["Kyoto", "Tokyo"]]
results = pairrm_judge.judge(prompts, completions, return_scores=True)
print(results) # [0.7492601275444031, 0.0005497377132996917]
by @kashif in https://github.com/huggingface/trl/pull/2221
The OnlineDPOTrainer and any trainers that inherit from it (NashMDTrainer and XPOTrainer) can now accept an initialized PairwiseJudge instead of a reward model.
from datasets import load_dataset
from trl import OnlineDPOConfig, OnlineDPOTrainer, PairRMJudge
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2-0.5B-Instruct")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2-0.5B-Instruct")
judge = PairRMJudge()
train_dataset = load_dataset("trl-lib/ultrafeedback-prompt", split="train")
training_args = OnlineDPOConfig(output_dir="Qwen2-0.5B-OnlineDPO", logging_steps=10)
trainer = OnlineDPOTrainer(
model=model, judge=judge, args=training_args, processing_class=tokenizer, train_dataset=train_dataset
)
trainer.train()
by @kashif in https://github.com/huggingface/trl/pull/2243
tokenizer to processing_classThe tokenizer argument in the trainers has been renamed to processing_class to better reflect the fact that it can be not only a tokenizer but also a processor.
- trainer = DPOTrainer(model, args=training_args, train_dataset=dataset, tokenizer=tokenizer)
+ trainer = DPOTrainer(model, args=training_args, train_dataset=dataset, processing_class=tokenizer)
tokenizer is still supported for SFTTrainer and DPOTrainer but deprecated and will be removed in the next release.
by @qgallouedec in https://github.com/huggingface/trl/pull/2162
The WPO paper adapts off-policy data to resemble on-policy data more closely by reweighting preference pairs according to their probability under the current policy. To use this method, set the use_weighting flag to True in the [DPOConfig].
DPOConfig(..., use_weighting=True)
<img width="1112" alt="Screenshot 2024-11-04 at 10 59 38" src="https://github.com/user-attachments/assets/544ddc02-bd09-4f21-b8a4-b81c21561a9b"> <img width="539" alt="Screenshot 2024-11-04 at 10 59 22" src="https://github.com/user-attachments/assets/8d5afe9e-89bd-4d00-8483-dd7ba98997e7">
by @gaetanlop in https://github.com/huggingface/trl/pull/2141
Using trainer.push_to_hub() now automatically creates a model card that includes:
All links are properly formatted to allow cross-referencing, enabling traceability back to sources (e.g., the model appears linked on the paper’s page).
https://github.com/user-attachments/assets/b903964e-9087-45cc-8fb0-2418fdd87b72
by @qgallouedec in https://github.com/huggingface/trl/pull/2123
You can now use conversational datasets directly, without needing to apply a chat template beforehand, for the following trainers:
BCOTrainer (by @qgallouedec in PR #2107)CPOTrainer (by @qgallouedec in PR #2144)DPOTrainer (by @qgallouedec in PR #2131)KTOTrainer (by @qgallouedec in PR #2248)ORPOTrainer (by @qgallouedec in PR #2184)from datasets import load_dataset
from transformers import AutoTokenizer, AutoModelForCausalLM
from trl import DPOTrainer
model = AutoModelForCausalLM.from_pretrained(model_id)
tokenizer = AutoTokenizer.from_pretrained(model_id)
dataset = load_dataset(dataset_name, split="train")
# Not needed anymore:
#
# def process(row):
# prompt = tokenizer.apply_chat_template(example["prompt"], tokenize=False, add_generation_prompt=True)
# prompt_chosen = tokenizer.apply_chat_template(example["prompt"] + example["chosen"], tokenize=False)
# chosen = prompt_chosen[len(prompt) :]
# prompt_rejected = tokenizer.apply_chat_template(example["prompt"] + example["rejected"], tokenize=False)
# rejected = prompt_rejected[len(prompt) :]
# return {"prompt": prompt, "chosen": chosen, "rejected": rejected}
#
# dataset = dataset.map(process)
training_args = DPOConfig(output_dir="...")
trainer = DPOTrainer(model, args=training_args, train_dataset=dataset, processing_class=tokenizer)
trainer.train()
For more information, see PR #2209.
trl env for printing system infoYou can now use trl env to print system information, including the platform, Python version, PyTorch version, CUDA device(s), and versions of various libraries.
$ trl env
Copy-paste the following information when reporting an issue:
- Platform: Linux-5.15.0-1048-aws-x86_64-with-glibc2.31
- Python version: 3.11.9
- PyTorch version: 2.4.0
- CUDA device(s): NVIDIA H100 80GB HBM3
- Transformers version: 4.47.0.dev0
- Accelerate version: 0.19.0
- Accelerate config: not found
- Datasets version: 3.0.2
- HF Hub version: 0.26.1
- TRL version: 0.12.0+14ef1ab
- bitsandbytes version: 0.44.1
- DeepSpeed version: 0.15.3
- Diffusers version: 0.30.3
- Liger-Kernel version: 0.3.0
- LLM-Blender version: 0.0.2
- OpenAI version: 1.46.0
- PEFT version: 0.13.2
by @qgallouedec in https://github.com/huggingface/trl/pull/2104
From GKD paper:
Sequence-Level KD (Kim & Rush, 2016). SeqKD maximizes the likelihood of high probability sequences generated by the teacher, and can be viewed as supervised FT on teacher-generated outputs.
SeqKD is taken as a baseline in the paper. It is now possible to use Sequence-Level KD in the GKDTrainer by setting seq_kd=True in the GKDConfig.
training_args = GKDConfig(..., seq_kd=True)
by @mst272 in https://github.com/huggingface/trl/pull/2220
dataset_text_field to "text"Since many users use "text" as the column name for textual data in datasets, we've made it the default (previously a required argument) in SFTConfig. Now, specifying dataset_text_field="text" is no longer necessary.
SFTConfig(
...,
- dataset_text_field="text",
)
by @qgallouedec in https://github.com/huggingface/trl/pull/2078
training_args by @qgallouedec in https://github.com/huggingface/trl/pull/2082trl env for printing system info by @qgallouedec in https://github.com/huggingface/trl/pull/2104BCOTrainer conversational dataset support by @qgallouedec in https://github.com/huggingface/trl/pull/2107max_length from RewardDataCollatorWithPadding by @qgallouedec in https://github.com/huggingface/trl/pull/2119training_step by @qgallouedec in https://github.com/huggingface/trl/pull/2117script_args by @qgallouedec in https://github.com/huggingface/trl/pull/2130WinRateCallback table by @lewtun in https://github.com/huggingface/trl/pull/2134dpo_visual.py example to dpo_vlm.py by @qgallouedec in https://github.com/huggingface/trl/pull/2139eval_strategy="steps" when no eval dataset by @qgallouedec in https://github.com/huggingface/trl/pull/2152DPOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/2131dataset_text_field to "text" by @qgallouedec in https://github.com/huggingface/trl/pull/2078CPOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/2144tokenizer to processing_class by @qgallouedec in https://github.com/huggingface/trl/pull/2162"unsloth" tag by @qgallouedec in https://github.com/huggingface/trl/pull/2173skip_prompt=True in TextIteratorStreamer by @qgallouedec in https://github.com/huggingface/trl/pull/2193decoder_input_ids in DPOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/2208"none" in GKD test by @qgallouedec in https://github.com/huggingface/trl/pull/2214trl env report all cuda devices by @qgallouedec in https://github.com/huggingface/trl/pull/2216ORPOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/2184PPOv2 -> PPO by @qgallouedec in https://github.com/huggingface/trl/pull/2174ScriptArguments by @qgallouedec in https://github.com/huggingface/trl/pull/2145ScriptArguments warning messages by @sergiopaniego in https://github.com/huggingface/trl/pull/2230remove_unused_columns by @qgallouedec in https://github.com/huggingface/trl/pull/2233get_batch_sample and add num_items_in_batch to compute_loss by @qgallouedec in https://github.com/huggingface/trl/pull/2246processing_class instead of tokenizer in LogCompletionsCallback by @qgallouedec in https://github.com/huggingface/trl/pull/2261KTOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/2248max_new_tokens by @qgallouedec in https://github.com/huggingface/trl/pull/2272log_reports.py for Improved Logging, File Processing, and Slack Payload Handling by @Mefisto04 in https://github.com/huggingface/trl/pull/2249eval_dataset in to trainers when no eval strategy by @qgallouedec in https://github.com/huggingface/trl/pull/2270_save_checkpoint for online methods by @qgallouedec in https://github.com/huggingface/trl/pull/2288optimizer_cls_and_kwargs attribute to PPO and RLOO by @qgallouedec in https://github.com/huggingface/trl/pull/2302Full Changelog: https://github.com/huggingface/trl/compare/v0.11.0...v0.12.0
Fix Inconsistency with IsShardedQLoRA Setting by @fabianlim in https://github.com/huggingface/trl/pull/2089
Full Changelog: https://github.com/huggingface/trl/compare/v0.11.3...v0.11.4
[GKD] interpolate in prob. space by @kashif in https://github.com/huggingface/trl/pull/2204
decoder_input_ids in DPOTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/2208Full Changelog: https://github.com/huggingface/trl/compare/v0.11.2...v0.11.3
Fix RLOO checkpointing by @bartoszzuk in https://github.com/huggingface/trl/pull/2114
Full Changelog: https://github.com/huggingface/trl/compare/v0.11.1...v0.11.2
allow parse-args as list of floats for Online DPO, XPO and Nash-MD configs by @kashif in #2108
Full Changelog: https://github.com/huggingface/trl/compare/v0.11.0...v0.11.1
The PPOTrainer is marked for deprecated in favour of PPOv2Trainer to provide a consistent API across TRL's trainers. It will be removed in v0.12.0. By…
We are excited to introduce the new v0.11.0 release, with many new features and post-training algorithms. The highlights are as follows:
<img width="992" alt="Screenshot 2024-09-19 at 10 01 02" src="https://github.com/user-attachments/assets/97afd65d-1a2c-484b-b6dd-b02a2cbe6430">
Generalized Knowledge Distillation (GKD) is a post-training method from Google DeepMind that extends standard knowledge distillation by allowing the student to generate outputs during training and receive online feedback from the teacher. It consistently outperforms SFT and in some cases enables the student model to match the performance of the teacher, but with far fewer parameters.
To train models with this method, check out the GKDTrainer.
<img width="1224" alt="Screenshot 2024-09-19 at 10 13 27" src="https://github.com/user-attachments/assets/36decb24-ef01-41f1-84e8-53b491eb6c86">
Exploratory Preference Optimization is an online post-training method from researchers at Microsoft, MIT, and Wisconsin that extends DPO to incorporate online feedback from reward models or LLM judges. It is similar to online DPO, but has a slightly different theoretical basis concerning sample efficiency.
To train models with this method, check out the XPOTrainer.
<img width="476" alt="Screenshot 2024-09-19 at 10 32 04" src="https://github.com/user-attachments/assets/8e68263f-bf5a-4f68-b451-110c78e27bb6">
Nash Learning with Human Feedback is a novel post-training method from Google DeepMind that uses pairwise preference models which are conditioned on two inputs, instead of the single one used in reward models. These preference models are then used to train a policy that consistently produces responses that are preferred over those from competing policies, thus approximating a Nash equilibrium (i.e. a two player game where actions are responses and payoffs are given by the preference model).
To train models with this method, check out the NashMDTrainer.
OrpoTrainer has better integration with PyTorchXLA for faster step time on TPUs ⚡ . By @wenxindongwork in https://github.com/huggingface/trl/pull/2001PPOTrainer is marked for deprecated in favour of PPOv2Trainer to provide a consistent API across TRL's trainers. It will be removed in v0.12.0. By @qgallouedec in https://github.com/huggingface/trl/pull/2016RichProgressCallback has been removed from the example scripts as it caused a variety of problems with logging in distributed environments. You can still use it by adding it manually to the trainer callbacks. By @lewtun in https://github.com/huggingface/trl/pull/2053prompts arg from WinrateCallback by @qgallouedec in https://github.com/huggingface/trl/pull/2010WinRateCallback to be used without reference model by @qgallouedec in https://github.com/huggingface/trl/pull/2013core.py by @northern-64bit in https://github.com/huggingface/trl/pull/2017packing doc in SFTConfig and fix error when neither dataset_text_field nor formatting_func is provided. by @qgallouedec in https://github.com/huggingface/trl/pull/2035aux_loss_enabled is set to True. by @Jonathanjordan21 in https://github.com/huggingface/trl/pull/2039non_eos_penalty to be consistent across OnPolicy trainers by @RylanSchaeffer in https://github.com/huggingface/trl/pull/2033debug and sanity_check args by @qgallouedec in https://github.com/huggingface/trl/pull/2055SFTTrainer.evaluate() and SFTTrainer.predict() with null train_dataset by @Sohaib9920 in https://github.com/huggingface/trl/pull/2004ConstantLengthDataset (or packing=True) shuffle examples before they are packed by @muupan in https://github.com/huggingface/trl/pull/2037WinRateCallback and LogCompletionsCallback by @lewtun in https://github.com/huggingface/trl/pull/2061transformers utilities when possible by @qgallouedec in https://github.com/huggingface/trl/pull/2064ref_policy and policy have different identities by @RylanSchaeffer in https://github.com/huggingface/trl/pull/2057processor(prompt, images=image) to processor(images=image, text=prompt) by @qgallouedec in https://github.com/huggingface/trl/pull/2076WinRateCallback and set default freq to eval_steps in LogCompletionsCallback` by @lewtun in https://github.com/huggingface/trl/pull/2074logits/chosen and logits/rejected metrics in kto_trainer. by @PhilipMay in https://github.com/huggingface/trl/pull/2077PPOv2Trainer by @qgallouedec in https://github.com/huggingface/trl/pull/2080Full Changelog: https://github.com/huggingface/trl/compare/v0.9.6...v0.11.0
We are excited to introduce the new v0.10.1 release, with many new exciting features and post-training algorithms. The highlights are as follows:
We are excited to introduce the new v0.10.1 release, with many new exciting features and post-training algorithms. The highlights are as follows:
<img width="1210" alt="Screenshot 2024-08-29 at 15 53 29" src="https://github.com/user-attachments/assets/c11863ca-434c-47d7-8436-dc096683075a">
Online DPO is a new alignment method from DeepMind to boost the performance of LLMs. With Online DPO, data is generated on the fly by the trained model (instead of pre-collected). For each prompt, two completions are generated, with a reward model selecting the preferred one. This approach:
To train models with this method, use the OnlineDPOTrainer
SFTTrainer for faster throughput and lower memory usage. To use them, set use_liger_kernel in SFTConfigdpo_visual.py script as followsaccelerate launch examples/scripts/dpo_visual.py \
--dataset_name HuggingFaceH4/rlaif-v_formatted \
--model_name_or_path google/paligemma-3b-pt-224 \
--trust_remote_code \
--per_device_train_batch_size 1 \
--gradient_accumulation_steps 8 \
--output_dir dpo_paligemma_rlaif-v \
--bf16 \
--torch_dtype bfloat16
trainer = DPOTrainer(...)
win_rate_callback = WinRateCallback(..., trainer=trainer)
trainer.add_callback(win_rate_callback)
apo_zero and apo_down. The apo_zero loss increases the likelihood of winning outputs while decreasing the likelihood of losing outputs, making it suitable when the model is less performant than the winning outputs. On the other hand, apo_down decreases the likelihood of both winning and losing outputs, but with a stronger emphasis on reducing the likelihood of losing outputs. This variant is more effective when the model is better than the winning outputs. To use these losses, set loss_type="apo_zero" or loss_type="apo_down" in the DPOConfigmodel_ref naming by @qgallouedec in https://github.com/huggingface/trl/pull/1835CI_HUB_USER_TOKEN by @qgallouedec in https://github.com/huggingface/trl/pull/1852setup_chat_format by @Rishav-hub in https://github.com/huggingface/trl/pull/1862evaluation_strategy -> eval_strategy by @qgallouedec in https://github.com/huggingface/trl/pull/1894setUpClass in reward tester by @qgallouedec in https://github.com/huggingface/trl/pull/1895IterableDataset for SFTTrainer by @qgallouedec in https://github.com/huggingface/trl/pull/1899AlignPropTrainer import by @qgallouedec in https://github.com/huggingface/trl/pull/1908lr_scheduler.step() after optimizer.step() by @qgallouedec in https://github.com/huggingface/trl/pull/1918torch.cuda.amp.autocast() -> torch.amp.autocast("cuda") by @qgallouedec in https://github.com/huggingface/trl/pull/1921dataset_num_proc usage by @qgallouedec in https://github.com/huggingface/trl/pull/1925PartialState().local_main_process_first() when map in examples by @qgallouedec in https://github.com/huggingface/trl/pull/1926push_to_hub by @qgallouedec in https://github.com/huggingface/trl/pull/1945padding_free to DataCollatorForCompletionOnlyLM by @RhuiDih in https://github.com/huggingface/trl/pull/1887PairRMJudge to top-level import by @qgallouedec in https://github.com/huggingface/trl/pull/1985torch.load with weights_only=True by @qgallouedec in https://github.com/huggingface/trl/pull/1988Full Changelog: https://github.com/huggingface/trl/compare/v0.9.6...v0.10
We are excited to introduce the new v0.9.6 release. Many new exciting features and algorithms. The highlights are as follows:
We are excited to introduce the new v0.9.6 release. Many new exciting features and algorithms. The highlights are as follows:
Support for SimPO by @fe1ixxu, a reference-free method that also regularizes output length. To use this loss, the users can input loss_type="simpo" and cpo_alpha=0 in the CPOConfig and use it with the CPOTrainer.
<img width="880" alt="image" src="https://github.com/huggingface/trl/assets/5555347/87551147-3f58-4c6a-9a78-70b513dea76e">
Added AlignProp by @mihirp1998, a method for finetuning Stable Diffusion model using reward gradients.
Added Efficient Exact Optimization (EXO) by @haozheji
We also included many important fixes and improvements such as fixing prints in the CLI with GCP containers by @alvarobartt. Enjoy the release!
numpy to !=2.0.0 for CI and to users by @younesbelkada in https://github.com/huggingface/trl/pull/1747TrlParser: Add ignore extra args option by @younesbelkada in https://github.com/huggingface/trl/pull/1748KTOTrainer: Remove old tests by @younesbelkada in https://github.com/huggingface/trl/pull/1750process function in the example of DPO by @AIR-hl in https://github.com/huggingface/trl/pull/1753evaluation_strategy to eval_strategy by @qgallouedec in https://github.com/huggingface/trl/pull/1771torch_dtype handling in {DPO,SFT}Trainer when provided via CLI by @alvarobartt in https://github.com/huggingface/trl/pull/1807TRL_USE_RICH environment variable handling by @alvarobartt in https://github.com/huggingface/trl/pull/1808Full Changelog: https://github.com/huggingface/trl/compare/v0.9.4...v0.9.6
Mainly backward compatibility fixes with SFTTrainer.
Mainly backward compatibility fixes with SFTTrainer.
TrainingArguments by @younesbelkada in https://github.com/huggingface/trl/pull/1707Full Changelog: https://github.com/huggingface/trl/compare/v0.9.3...v0.9.4
Apply deprecated evaluation_strategy by @muellerzr in https://github.com/huggingface/trl/pull/1559
We are excited to introduce the new v0.9.3 release. Many new exciting features and algorithms. The highlights are as follows:
https://github.com/huggingface/trl/assets/5555347/6575a879-cb2f-4e2e-bb84-a76707f9de84
SFTTrainer] Add warning in SFTTrainer when dataset already processed by @younesbelkada in https://github.com/huggingface/trl/pull/1577SftArgumentParser by @younesbelkada in https://github.com/huggingface/trl/pull/1602KTOTrainer] add BCO (reward shift and underlying distribution matching) by @seanexp in https://github.com/huggingface/trl/pull/1599tests/ from package data by @jamesbraza in https://github.com/huggingface/trl/pull/1607evaluation_strategy by @muellerzr in https://github.com/huggingface/trl/pull/1559enable_input_require_grads issues with PPO models by @younesbelkada in https://github.com/huggingface/trl/pull/1664args=None by @younesbelkada in https://github.com/huggingface/trl/pull/1678Full Changelog: https://github.com/huggingface/trl/compare/v0.8.6...v0.9.2
Nothing published for this version
set dev version by @younesbelkada in https://github.com/huggingface/trl/pull/1556
Full Changelog: https://github.com/huggingface/trl/compare/v0.8.5...v0.8.6
set dev version by @younesbelkada in https://github.com/huggingface/trl/pull/1548
Full Changelog: https://github.com/huggingface/trl/compare/v0.8.4...v0.8.5
This patch release includes important fixes for the CLI and KTO & CPO trainers
This patch release includes important fixes for the CLI and KTO & CPO trainers
dataset_text_field to None to allow ChatML automatic template by @younesbelkada in https://github.com/huggingface/trl/pull/1545Full Changelog: https://github.com/huggingface/trl/compare/v0.8.3...v0.8.4
This is a patch release that includes an import fix for CLIs
This is a patch release that includes an import fix for CLIs
Full Changelog: https://github.com/huggingface/trl/compare/v0.8.2...v0.8.3
This release includes two new trainers: ORPO from KAIST and CPO The release also includes Vision LLM such as Llava support for SFTTrainer, please see:
This release includes two new trainers: ORPO from KAIST and CPO
The release also includes Vision LLM such as Llava support for SFTTrainer, please see: https://github.com/huggingface/trl/blob/main/examples/scripts/vsft_llava.py for more details
use_cache=False in {ORPO,CPO}Trainer.concatenated_forward by @alvarobartt in https://github.com/huggingface/trl/pull/1478input_ids instead by @alvarobartt in https://github.com/huggingface/trl/pull/1516You can now use SFTTrainer to fine-tune VLLMs such as Llava !
See: https://github.com/huggingface/trl/blob/main/examples/scripts/vsft_llava.py for more details
Many fixes were introduced for the KTOTrainer:
RichProgressCallback by @eggry in https://github.com/huggingface/trl/pull/1496Full Changelog: https://github.com/huggingface/trl/compare/v0.8.1...v0.8.2
This patch release includes some important fixes for CLIs
This patch release includes some important fixes for CLIs
Full Changelog: https://github.com/huggingface/trl/compare/v0.8.0...v0.8.1
Fix import error from deprecation in transformers by @lewtun in https://github.com/huggingface/trl/pull/1415
We recently introduced the KTOTrainer in order to run KTO algorithms on LLMs !
Run SFT, DPO and chat with your aligned model directly from the terminal:
SFT:
trl sft --model_name_or_path facebook/opt-125m --dataset_name imdb --output_dir opt-sft-imdb
DPO:
trl dpo --model_name_or_path facebook/opt-125m --dataset_name trl-internal-testing/Anthropic-hh-rlhf-processed --output_dir opt-sft-hh-rlhf
Chat:
trl chat --model_name_or_path Qwen/Qwen1.5-0.5B-Chat
Read more about CLI in the relevant documentation section or use --help for more details.
model --> model_name_or_path by @lvwerra in https://github.com/huggingface/trl/pull/1452SFTTrainer now supports FSDP + QLoRA
SFTTrainer] Add eval_packing by @younesbelkada in https://github.com/huggingface/trl/pull/1369force_use_ref_model for power users by @younesbelkada in https://github.com/huggingface/trl/pull/1367RewardModeling] Fix RM script for PEFT by @younesbelkada in https://github.com/huggingface/trl/pull/1393Full Changelog: https://github.com/huggingface/trl/compare/v0.7.11...v0.8.0
We fixed issues with respect to IPO loss, leading to consistent results according to newest experiements:
We fixed issues with respect to IPO loss, leading to consistent results according to newest experiements:
We also fixed important bugs with respect to DPO / PEFT and Flash Attention
DPOTrainer] Fix DPO trainer + mistral + FA2 by @younesbelkada in https://github.com/huggingface/trl/pull/1290Data processing is now faster for multi-GPU envs
DPOTrainer] Load data only on main process + fix dpo example test by @younesbelkada in https://github.com/huggingface/trl/pull/1291Other DPO bugfixes:
PEFT + DPO] Raise value error if one passes a ref_model and a peft_config by @younesbelkada in https://github.com/huggingface/trl/pull/1289Models now gets tagged correctly even if users do not call trainer.push_to_hub()
core / xxxTrainer] Automatic tagging by @younesbelkada in https://github.com/huggingface/trl/pull/1329DPOTrainer docstrings by @alvarobartt in https://github.com/huggingface/trl/pull/1298core / DDPO] Fix diffusers import issue by @younesbelkada in https://github.com/huggingface/trl/pull/1314CI] Add tests on transformers peft main on push main by @younesbelkada in https://github.com/huggingface/trl/pull/1328Full Changelog: https://github.com/huggingface/trl/compare/v0.7.10...v0.7.11
[core / SFTTrainer] Fix breaking change by @younesbelkada in https://github.com/huggingface/trl/pull/1229
setup_chat_format API, stronger testsThis Patch release adds a new feature in TRL for dealing with chat datasets - you can load a directly formatted dataset without the need of formatting it beforehand.
Read more about it here: https://huggingface.co/docs/trl/sft_trainer#dataset-format-support
The release also introduces a new API setup_chat_format to correctly resize the model embeddings with the target size when adding new tokens to comply with the chat format. Currently we only support chatml format and we can add more formats in the future
Read more about it here: https://huggingface.co/docs/trl/sft_trainer#add-special-tokens-for-chat-format
We also extensively test SFTTrainer and DPOTrainer and the example scripts, dpo.py and sft.py should be well -battletested. If you see any issue with the script, please let us know on GitHub.
core / Docker] Add workflow to build TRL docker images by @younesbelkada in https://github.com/huggingface/trl/pull/1215pad_token_id is not configured by @yumemio in https://github.com/huggingface/trl/pull/1152core / tests ] v1 slow tests by @younesbelkada in https://github.com/huggingface/trl/pull/1218core / SFTTrainer] Fix breaking change by @younesbelkada in https://github.com/huggingface/trl/pull/1229setup_chat_format for adding new special tokens to model for training chat models by @philschmid in https://github.com/huggingface/trl/pull/1242Full Changelog: https://github.com/huggingface/trl/compare/v0.7.9...v0.7.10
This is a patch release that fixes critical issues with SFTTrainer & DPOTrainer, together with minor fixes for PPOTrainer and DataCollatorForCompletio
This is a patch release that fixes critical issues with SFTTrainer & DPOTrainer, together with minor fixes for PPOTrainer and DataCollatorForCompletionOnlyLM
DPOTrainer] Fix peft + DPO + bf16 if one uses generate_during_eval or pre-computed logits by @younesbelkada in https://github.com/huggingface/trl/pull/1203Full Changelog: https://github.com/huggingface/trl/compare/v0.7.8...v0.7.9
Your coding agent can read these notes before it upgrades. Set up the MCP server →