NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
PyPI, #269 most downloaded on PyPI
Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.
Latest stable
5.23.0
Last release 7 days ago
13 Nov 2026
Ships fairly regularly
a new release about every 2 weeks
Nearly every release is documented
notes for 60 of the last 60 stable releases
6 versions withdrawn
withdrawn after publishing
10 years old
253 releases · first in 2016
Drag across the ruler to compare, or tap a release to open it.
OpenAI Privacy Filter is a bidirectional token-classification model for personally identifiable information (PII) detection and masking in text. It is
OpenAI Privacy Filter is a bidirectional token-classification model for personally identifiable information (PII) detection and masking in text. It is intended for high-throughput data sanitization workflows where teams need a model that they can run on-premises that is fast, context-aware, and tunable. The model labels an input sequence in a single forward pass, then decodes coherent spans with a constrained Viterbi procedure, predicting probability distributions over 8 privacy-related output categories for each input token.
Links: Documentation
Privacy Filter] Add model (#45580) by @vasqu in #45580Qianfan-OCR is a 4B-parameter end-to-end document intelligence model developed by Baidu that performs direct image-to-text conversion without traditional multi-stage OCR pipelines. It supports a broad range of prompt-driven tasks including structured document parsing, table extraction, chart understanding, document question answering, and key information extraction all within one unified model. The model features a unique "Layout-as-Thought" capability that generates structured layout representations before producing final outputs, making it particularly effective for complex documents with mixed element types.
Links: Documentation | Paper
SAM3-LiteText is a lightweight variant of SAM3 that replaces the heavy SAM3 text encoder (353M parameters) with a compact MobileCLIP-based text encoder optimized through knowledge distillation, while keeping the SAM3 ViT-H image encoder intact. This reduces text encoder parameters by up to 88% while maintaining segmentation performance comparable to the original model. The model enables efficient vision-language segmentation by addressing the redundancy found in text prompting for segmentation tasks.
Links: Documentation | Paper
SLANet and SLANet_plus are lightweight models designed for table structure recognition, focusing on accurately recognizing table structures in documents and natural scenes. The model improves accuracy and inference speed by adopting a CPU-friendly lightweight backbone network PP-LCNet, a high-low-level feature fusion module CSP-PAN, and a feature decoding module SLA Head that aligns structural and positional information. SLANet was developed by Baidu PaddlePaddle Vision Team as part of their table structure recognition solutions.
Links: Documentation
The internal rotary_fn is no longer registered as a hidden kernel function, so any code referencing self.rotary_fn(...) within an Attention module will break and must be updated to call the function directly instead.
Kernels] Fix kernel function registration (#45420) by @vasquThe transformers serve command received several enhancements, including a new /v1/completions endpoint for legacy text completion, multimodal support for audio and video inputs, improved tool-calling via parse_response, proper forwarding of tool_calls/tool_call_id fields, a 400 error on model mismatch when the server is pinned to a specific model, and fixes for the response API. Documentation was also updated to cover new serving options such as --compile and --model-timeout.
transformers serve (#44558) by @rain-1 in [#44558]transformers serve is pinned (#45443) by @qgallouedec in [#45443]parse_response (#45485) by @SunMarc in [#45485]tool_calls/tool_call_id in processor inputs (#45418) by @qgallouedec in [#45418]Several vision-related bug fixes were applied in this release, including correcting Qwen2.5-VL temporal RoPE scaling for still images, fixing missing/mismatched image processor backends for Emu3 and BLIP, resolving modular image processor class duplication, and preventing accelerate from incorrectly splitting vision encoders in PeVideo/PeAudioVideo models. Image loading performance was also improved by leveraging torchvision's native decode_image in the torchvision backend, yielding up to ~17% speedup over PIL-based loading.
decode_image to load images in the torchvision backend (#45195) by @yonigozlan in [#45195]Fixed several bugs affecting distributed training, including silently wrong results or NaN loss with Expert Parallelism, NaN weights on non-rank-0 FSDP processes, and a resize failure in PP-DocLayoutV3; additionally added support for loading adapters with Tensor Parallelism, added MoE to the Gemma4 TP plan, and published documentation for TP training.
Fixed a docstring typo in streamer classes, resolved a Kimi-K2.5 tokenizer regression and _patch_mistral_regex AttributeError, and patched a streaming generation crash for Qwen3VLProcessor caused by incorrect _tokenizer attribute access. Additional housekeeping included moving the GPT-SW3 instruct tokenizer to an internal testing repo and fixing a global state leak in the tokenizer registry during tests.
Tokenizers] Move gpt sw3 tokenizer out (#45404) by @vasqu in [#45404]test_processors (#45318) by @tarekziade in [#45318]Cache handling was improved for Gemma4 and Gemma3n models by dissociating KV state sharing from the Cache class, ensuring KV states are always shared regardless of whether a Cache is used. Additionally, the image cache for Paddle models was updated to align with the latest API.
Audio models gained vLLM compatibility through targeted fixes across several model implementations, while reliability improvements were also made including exponential back-off retries for audio file downloads, a crash fix in the text-to-speech pipeline when generation configs contain None values, and corrected test failures for Kyutai Speech-To-Text.
text-to-speech pipeline crash when generation config contains None values (#45107) by @jiqing-feng in [#45107]Privacy Filter] Add model (#45580) by @vasqu in [#45580]pass (inherits from DSV3 MoE) (#45572) by @casinca in [#45572]DeepseekV3MoE and remote official implementation (#45441) by @casinca in [#45441]prepare_decoder_input_ids_from_labels (#45516) by @Tokarak in [#45516]TextToAudioPipeline missing <bos> token (#45525) by @jiqing-feng in [#45525]Conversion Mapping] Small fixups (#45483) by @vasqu in [#45483]get_image_size method (#45461) by @JiauZhang in [#45461]fix] Always early return for non-Mistral models in _patch_mistral_regex (#45444) by @tomaarsen in [#45444]fix] Make Qwen2_5OmniProcessor warning a lot less noisy via warning_once (#45455) by @tomaarsen in [#45455]step3_vl to MODELS_WITH_INCORRECT_HUB_TOKENIZER_CLASS (#45449) by @hmellor in [#45449]fix] PEFT integration fixes preventing save/load & integration (#45428) by @tomaarsen in [#45428]apply_chat_template crash on tool_call messages without content (#45348) by @qgallouedec in [#45348]trackio integration to use Buckets and "freeze" Space after training (#45329) by @abidlabs in [#45329]cohere_asr: fix device issue for test_model_parallel_beam_search (#45214) by @kaixuanliu in [#45214][transformers] prefix in non-verbose mode (#45316) by @zucchini-nlp in [#45316]pi0 model (#45011) by @kaixuanliu in [#45011]grouped_mm (#45001) by @Sai-Suraj-27 in [#45001]Wav2Vec2Config.vocab_size type to allow None (#45108) by @jiqing-feng in [#45108]hasattr(torch.backends.cudnn, "conv") to conftest.py (#45263) by @ydshieh in [#45263]SmolVLM video processor resize using wrong interpolation after backend refactor (#45258) by @ydshieh in [#45258]Qwen2IntegrationTest (#45268) by @ydshieh in [#45268]torch.backends.cudnn.conv.fp32_precision explicitly. (#45248) by @ydshieh in [#45248]torch 2.11 (#45243) by @ydshieh in [#45243]get_test_info.py (related to tiny model creation) (#45238) by @ydshieh in [#45238]test_register_result_handler (#45188) by @SunMarc in [#45188]The following contributors have made significant changes to the library over the last release:
Privacy Filter] Add model (#45580)Conversion Mapping] Small fixups (#45483)Kernels] Fix kernel function registration (#45420)Tokenizers] Move gpt sw3 tokenizer out (#45404)transformers serve (#44558)test_processors (#45318)hasattr(torch.backends.cudnn, "conv") to conftest.py (#45263)SmolVLM video processor resize using wrong interpolation after backend refactor (#45258)Qwen2IntegrationTest (#45268)torch.backends.cudnn.conv.fp32_precision explicitly. (#45248)torch 2.11 (#45243)get_test_info.py (related to tiny model creation) (#45238)Your coding agent can read these notes before it upgrades. Set up the MCP server →
This is mostly some fixes that are good to have asap, mostly for tokenizers; Fix Kimi-K2.5 tokenizer regression and _patch_mistral_regex Attribute… (#
This is mostly some fixes that are good to have asap, mostly for tokenizers; ** Fix Kimi-K2.5 tokenizer regression and _patch_mistral_regex Attribute… (#45305) by ArthurZucker
For training: ** Fix #45305 + add regression test GAS (#45349) by florian6973, SunMarc ** Fix IndexError with DeepSpeed ZeRO-3 when kernels rotary is active (#…) by ArthurZucker
And for Qwen2.5-VL : ** Fix Qwen2.5-VL temporal RoPE scaling applied to still images (#45330) by Kash6, zucchini-nlp
Small patch release to fix device_map support for Gemma4! It contains the following commit:
Small patch release to fix device_map support for Gemma4! It contains the following commit:
Small patch dedicated to optimizing gemma4, fixing inference with use_cache=False due to k/v states sharing between layers, as well as conversion mapp
Small patch dedicated to optimizing gemma4, fixing inference with use_cache=False due to k/v states sharing between layers, as well as conversion mappings for some models that would inconsistently serialize their weight names. It contains the following PRs:
This patch is very small and focuses on vLLM and Gemma4!
This patch is very small and focuses on vLLM and Gemma4!
** Fix export for gemma4 and add Integration tests (#45285) by @Cyrilvallez ** Fix vllm cis (#45139) by @ArthurZucker
[FA] Fix BC support for a few versions + add deprecation cycle (#45061) by @vasqu in [#45061]
<img width="2786" height="1504" alt="image" src="https://github.com/user-attachments/assets/6c8c878f-042b-4858-9f64-73fd9ccd7e4b" />
Gemma 4 is a multimodal model with pretrained and instruction-tuned variants, available in 1B, 13B, and 27B parameters. The architecture is mostly the same as the previous Gemma versions. The key differences are a vision processor that can output images of fixed token budget and a spatial 2D RoPE to encode vision-specific information across height and width axis.
<img width="1478" height="1374" alt="image" src="https://github.com/user-attachments/assets/9d88bd1b-02ea-4829-b7d0-fac0e347d436" />
You can find all the original Gemma 4 checkpoints under the Gemma 4 release.
The key difference from previous Gemma releases is the new design to process images of different sizes using a fixed-budget number of tokens. Unlike many models that squash every image into a fixed square (like 224×224), Gemma 4 keeps the image's natural aspect ratio while making it the right size. There a a couple constraints to follow:
[!IMPORTANT] Gemma 4 does not apply the standard ImageNet mean/std normalization that many other vision models use. The model's own patch embedding layer handles the final scaling internally (shifting values to the [-1, 1] range).
The number of "soft tokens" (aka vision tokens) an image processor can produce is configurable. The supported options are outlined below and the default is 280 soft tokens per image.
| Soft Tokens | Patches (before pooling) | Approx. Image Area |
|---|---|---|
| 70 | 630 | ~161K pixels |
| 140 | 1,260 | ~323K pixels |
| 280 | 2,520 | ~645K pixels |
| 560 | 5,040 | ~1.3M pixels |
| 1,120 | 10,080 | ~2.6M pixels |
To encode positional information for each patch in the image, Gemma 4 uses a learned 2D position embedding table. The position table stores up to 10,240 positions per axis, which allows the model to handle very large images. Each position is a learned vector of the same dimensions as the patch embedding. The 2D RoPE which Gemma 4 uses independently rotate half the attention head dimensions for the x-axis and the other half for the y-axis. This allows the model to understand spatial relationships like "above," "below," "left of," and "right of."
NomicBERT is a BERT-inspired encoder model that applies Rotary Position Embeddings (RoPE) to create reproducible long context text embeddings. It is the first fully reproducible, open-source text embedding model with 8192 context length that outperforms both OpenAI Ada-002 and OpenAI text-embedding-3-small on short-context MTEB and long context LoCo benchmarks. The model generates dense vector embeddings for various tasks including search, clustering, and classification using specific instruction prefixes.
Links: Documentation | Paper
Music Flamingo is a fully open large audio–language model designed for robust understanding and reasoning over music. It builds upon the Audio Flamingo 3 architecture by including Rotary Time Embeddings (RoTE), which injects temporal position information to enable the model to handle audio sequences up to 20 minutes. The model features a unified audio encoder across speech, sound, and music with special sound boundary tokens for improved audio sequence modeling.
Links: Documentation | Paper
Mamba and hybrid model caches are now first-class native citizens in the library, so users working with Mamba-based or hybrid (Mamba + attention) models should update their code to use the new native cache classes instead of any previous workarounds.
Remote code execution support has been removed from the native LightGlue integration, so users who were loading LightGlue with trust_remote_code=True must remove that argument and use the model directly through the standard native API.
LightGlue] Remove remote code execution (#45122) by @vasquSeveral vision-related bugs were fixed in this release, including correcting the Gemma vision mask to support video inputs, resolving a dependency issue that incorrectly required torchvision for PIL-based image processors, and patching bugs in the Janus image generation model and image loading. Local code resolution for tokenizers and image processors was also corrected.
Image.open failure (#44645) by @sywangyi in [#44645]Improved the performance of repository checks (check-repo) by introducing file-level and AST-level disk caching, achieving up to a 27x speedup (from ~46s to ~1.6s with a warm cache), and fixed the mlinter cache location in .gitignore.
janus model (#44739) by @kaixuanliu in [#44739]FA] Fix BC support for a few versions + add deprecation cycle (#45061) by @vasqu in [#45061]model_type in AutoConfig.from_pretrained (#45058) by @hmellor in [#45058]SmolLM3IntegrationTest (#45048) by @Sai-Suraj-27 in [#45048]The following contributors have made significant changes to the library over the last release:
Additionally, integration tests for LASR with chunked decoding were added, and outdated references to deprecated pipeline tasks were cleaned up.
<img width="1480" height="460" alt="image" src="https://github.com/user-attachments/assets/bec6fc25-b0ab-4227-8c2b-a838554f37f3" />
Video Encoder-only Mask Transformer (VidEoMT) is a lightweight encoder-only model for online video segmentation built on a plain Vision Transformer (ViT). It eliminates the need for dedicated tracking modules by introducing a lightweight query propagation mechanism that carries information across frames and employs a query fusion strategy that combines propagated queries with temporally-agnostic learned queries. VidEoMT achieves competitive accuracy while being 5x-10x faster than existing approaches, running at up to 160 FPS with a ViT-L backbone.
Links: Documentation | Paper
<img width="1765" height="875" alt="image" src="https://github.com/user-attachments/assets/365e510e-8fb8-46cb-8f4b-e8b7082f0ae2" />
UVDoc is a machine learning model designed for document image rectification and correction. The main purpose of this model is to carry out geometric transformation on images to correct document distortion, inclination, perspective deformation and other problems in document images. It provides both single input and batched inference capabilities for processing distorted document images.
Links: Documentation
<img width="595" height="513" alt="image" src="https://github.com/user-attachments/assets/2aee0692-8286-4c6b-98db-847b95ab2d40" />
The Jina-Embeddings-v3 is a multilingual, multi-task text embedding model designed for a variety of NLP applications. Based on the XLM-RoBERTa architecture, this model supports Rotary Position Embeddings (RoPE) replacing absolute position embeddings to support long input sequences up to 8192 tokens. Additionally, it features 5 built-in Task-Specific LoRA Adapters that allow the model to generate task-specific embeddings (e.g., for retrieval vs. classification) without increasing inference latency significantly.
Links: Documentation | Paper
Jina-Embeddings-V3 Model (#44251) by @Sai-Suraj-27 in #44251<img width="2429" height="1787" alt="image" src="https://github.com/user-attachments/assets/a6feb0da-8504-4eab-be65-22d6c676336f" />
Mistral 4 is a powerful hybrid model with the capability of acting as both a general instruction model and a reasoning model. It unifies the capabilities of three different model families - Instruct, Reasoning (previously called Magistral), and Devstral - into a single, unified model. The model features a MoE architecture with 128 experts and 4 active, 119B parameters with 6.5B activated per token, 256k context length, and supports multimodal input with both text and image processing capabilities.
Links: Documentation
PI0 is a vision-language-action model for robotics manipulation that jointly processes visual observations and language instructions to generate robot actions. It uses a novel flow matching architecture built on top of a pre-trained vision-language model to inherit Internet-scale semantic knowledge. The model can perform complex dexterous tasks like laundry folding, table cleaning, and assembling boxes across multiple robot platforms including single-arm robots, dual-arm robots, and mobile manipulators.
Links: Documentation | Paper
SLANeXt is a series of dedicated lightweight models for table structure recognition, focusing on accurately recognizing table structures in documents and natural scenes. The SLANeXt series is a new generation of table structure recognition models independently developed by the Baidu PaddlePaddle Vision Team, with dedicated weights trained separately for wired and wireless tables. The recognition ability for all types of tables has been significantly improved, especially for wired tables.
Links: Documentation
PP-OCRv5_mobile_rec is a dedicated lightweight model for text recognition, focusing specifically on efficient recognition and understanding of text elements in multi-language documents and natural scenes. It is designed to efficiently and accurately support the recognition of Simplified Chinese, Traditional Chinese, English, Japanese, as well as complex text scenarios such as handwriting, vertical text, pinyin, and rare characters with a single model. While maintaining recognition performance, it also balances inference speed and model robustness, providing efficient and accurate technical support for document understanding in various scenarios.
Links: Documentation
PP-OCRv5_server_rec is a dedicated lightweight model for text recognition, focusing specifically on efficient recognition and understanding of text elements in multi-language documents and natural scenes. It is designed to efficiently and accurately support the recognition of Simplified Chinese, Traditional Chinese, English, Japanese, as well as complex text scenarios such as handwriting, vertical text, pinyin, and rare characters with a single model. While maintaining recognition performance, it also balances inference speed and model robustness, providing efficient and accurate technical support for document understanding in various scenarios.
Links: Documentation
PP-OCRv5_mobile_det is a dedicated lightweight model for text detection, focusing specifically on efficient detection and understanding of text elements in multi-language documents and natural scenes. It is part of the latest generation of text detection models developed by the PaddleOCR team that efficiently and accurately supports the detection of text in diverse scenarios—including handwriting, vertical, rotated, and curved text—across multiple languages such as Simplified Chinese, Traditional Chinese, English, and Japanese. The model features robust handling of complex layouts, varying text sizes, and challenging backgrounds, making it suitable for practical applications like document analysis, license plate recognition, and scene text detection.
Links: Documentation
PP-LCNet is a family of efficient, lightweight convolutional neural networks designed for real-world document understanding and OCR tasks. It balances accuracy, speed, and model size, making it ideal for both server-side and edge deployment. The model has three main variants optimized for specific tasks: document image orientation classification, table classification, and text line orientation classification.
Links: Documentation
PPLCNetV3 is a lightweight CPU-optimized convolutional backbone designed for efficient image classification and downstream vision tasks. It builds on the PP-LCNet architecture with improved training strategies and structural refinements for better accuracy-latency tradeoffs on CPU hardware.
Links: Documentation | Paper
PP-OCRv5_server_det is a high-performance text detection model optimized for server-side applications, focusing on accurate detection of multi-language text in documents and natural scenes. It supports the detection of text in diverse scenarios—including handwriting, vertical, rotated, and curved text—across multiple languages such as Simplified Chinese, Traditional Chinese, English, and Japanese. The model features robust handling of complex layouts, varying text sizes, and challenging backgrounds, making it suitable for practical applications like document analysis, license plate recognition, and scene text detection.
Links: Documentation
CHMv2 is a global, meter-resolution canopy height mapping model that uses DINOv3 to estimate forest canopy heights from high-resolution optical satellite imagery. Building on the original canopy height maps released in 2024, CHMv2 delivers substantial improvements in accuracy, detail, and global consistency by leveraging Meta's self-supervised vision model. The model is trained against airborne laser scanning data and provides essential information for quantifying forest carbon, monitoring restoration and degradation, and assessing habitat structure.
Links: Documentation | Paper | Blog Post
The dual BaseImageProcessor/BaseImageProcessorFast design has been replaced with a unified backend architecture, and the image_processing_utils_fast module has been removed — users should migrate to the new unified image_processing_utils module.
PreTrainedConfig and model config classes have been refactored to use @dataclass and no longer accept positional arguments — users must update any config instantiation calls to use keyword arguments only.
Flash Attention 2 (FA2) support now requires version 2.3.3 or newer, and initial Flash Attention 4 (FA4) support has been added — users on older FA2 versions must upgrade to at least 2.3.3.
FA4] Initial support (#42435) by @vasquWeight tying behavior has changed so that weights are now tied even when both keys are already present in a checkpoint — users relying on the previous behavior (e.g., with .bin checkpoints containing duplicate keys) should verify their models load as expected.
The cache_position argument has been removed from the forward signatures of most major models — users passing cache_position directly to these models should remove it, as it is now handled internally by generate.
Several bug fixes and improvements were made to pipeline parallel (PP) and tensor parallel (TP) support, including fixing supports_tp/pp_plan detection, resolving attribute errors in PP for Qwen2VL-based models, correcting FSDP loading with meta devices, and ensuring TP weight sharding properly updates parent module attributes (e.g., in_features/out_features) to improve compatibility with libraries like PEFT.
supports_{tp/pp}_plan (#44696) by @hmellor in [#44696]torch.distributed.fsdp in trainer_seq2seq.py (#44507) by @0xDELUXA in [#44507]Quantization support was improved with up to 30x faster FP8 grouped and batched matmuls, static FP8 expert support for multi-GPU setups, and a torchao minimum version bump to 0.15.0. Additionally, MXFP4 dependency error messages were made more actionable, and AWQ tests were updated to align with the GPTQModel migration.
Several performance improvements were made to tokenizer loading and saving, including eliminating redundant file parsing and unnecessary deep copies of large vocabularies that caused significant overhead. Additionally, bug fixes were applied for incorrect tokenizer class names on the Hub (DeepSeek V2/V3, ModernBERT), a clean_up_tokenization_spaces misconfiguration in Llama 3 tokenizer conversion, and a string replacement issue in AutoTokenizer class name resolution.
processing_utils.py: avoid deepcopying tokenizer in ProcessorMixin to improve performance (#44894) by @ydshieh in [#44894]clean_up_tokenization_spaces=False in Llama 3 tokenizer conversion (#44914) by @maxsloef-goodfire in [#44914]Kernel support has been expanded with Flash Attention 4 fallback integration, a paged_attention kernel for continuous batching, and Neuron device support for custom kernels. Several stability fixes were also made, including bumping the kernels version dependency to prevent crashes and correcting the LFM2 kernel path.
FA4] Add kernels fallback (#44797) by @vasqu in [#44797]Several cache-related fixes and improvements were made, including aligning LFM2's cache implementation with other Mamba caches, fixing a tensor indexing crash in KV cache continuation for the transformers serve streaming endpoint, and resolving a generation bug in Idefics3 when using use_cache=False. A caching layer was also added to the model linter to skip unchanged valid files and improve build performance.
Fixed backward compatibility for full-path imports of Fast Image Processors and resolved a Llama4 vision rotary embedding initialization error where freqs_ci was not registered as a buffer, causing failures when loading models with device_map="auto".
The cache_position argument has been fully removed from the generation pipeline, as all models have been updated to no longer use it (with a backward-compatibility path retained for remote code models). Additionally, integration tests for LASR with chunked decoding were added, and outdated references to deprecated pipeline tasks were cleaned up.
cache_position anymore in generation (#44816) by @Cyrilvallez in [#44816]text2text-generation, summarization and translation pipeline tasks (#44510) by @math-hiyoko in [#44510]tests_hub if no tests found (#45014) by @ydshieh in [#45014]attention_chunk_size in Llama4TextConfig (#45002) by @hmellor in [#45002]maybe_autocast crashing on meta device tensors (#44984) by @Butanium in [#44984]mm_token_type be non-padded lists (#44563) by @zucchini-nlp in [#44563]Qwen2VL (#44976) by @hmellor in [#44976]check_auto_docstrings (#44803) by @yonigozlan in [#44803]vllm x v5] nit (#44971) by @ArthurZucker in [#44971]T5ModelIntegrationTest (#44934) by @Sai-Suraj-27 in [#44934]Update Transformers metadata after #43514 (#44941) by @ydshieh in [#44941]from_pretrained (url input deprecated) (#44946) by @BSchilperoort in [#44946]image_processing_utils_fast (#44897) by @yonigozlan in [#44897]NemotronH is torch compiled (#44854) by @ydshieh in [#44854]SizeDict (#44884) by @hmellor in [#44884]layer_types type hint for AFMoE and Llama4 (#44874) by @hmellor in [#44874]PreTrainedModel (#44672) by @neo in [#44672]KeyError when patching mistral regex (#43376) by @LeonardoEmili in [#43376]position_ids keys when loading OwlViT models (#44508) by @KartikPawade in [#44508].ai (#44489) by @tarekziade in [#44489]@strict (#44770) by @zucchini-nlp in [#44770]is_causal from EuroBertConfig (#44774) by @ydshieh in [#44774]mlcd auto config/model/mapping issues (#44730) by @ydshieh in [#44730]config class in some model class definitions (#44715) by @ydshieh in [#44715]FA] Fix fa detection (#44703) by @vasqu in [#44703]set_encoder (#44698) by @hmellor in [#44698]parent issue (#44685) by @ydshieh in [#44685]ParallelInterface (#44640) by @michaelbenayoun in [#44640]Chmv2] Fix conversion after capture refactor (#44665) by @vasqu in [#44665]dtype for subconfig when _from_config (#44629) by @zucchini-nlp in [#44629]cache_position in more models (2) (#44602) by @Cyrilvallez in [#44602]VibeVoiceAcousticTokenizer (#44628) by @ydshieh in [#44628]cache_position in more models (#44330) by @Cyrilvallez in [#44330]src/transformers/quantizers (#44412) by @tarekziade in [#44412]fix] Prevent crash with Apertus without xielu installed (#44567) by @tomaarsen in [#44567]MusicgenStereo integration tests (#44527) by @Sai-Suraj-27 in [#44527]higgs_audio_v2 tests (#44482) by @kaixuanliu in [#44482]_prepare_input_fn and _prepare_output_fn instance methods (#44499) by @michaelbenayoun in [#44499]mps device (#44506) by @michaelbenayoun in [#44506]GPTNeoModelLanguageGenerationTest (#44515) by @Sai-Suraj-27 in [#44515]MarianIntegrationTests (#44519) by @Sai-Suraj-27 in [#44519]build_pr_documentation.yml (will be the new required job) (#44538) by @ydshieh in [#44538]build_pr_documentation workflow for merge_group event (#44532) by @ydshieh in [#44532]ty to 0.0.20 (#44494) by @tarekziade in [#44494]diffusers to CI docker file (#44480) by @ydshieh in [#44480]DepthProModelIntegrationTest (#44456) by @Sai-Suraj-27 in [#44456]ProphetNetModelIntegrationTest (#44439) by @Sai-Suraj-27 in [#44439]The following contributors have made significant changes to the library over the last release:
tests_hub if no tests found (#45014)Update Transformers metadata after #43514 (#44941)processing_utils.py: avoid deepcopying tokenizer in ProcessorMixin to improve performance (#44894)NemotronH is torch compiled (#44854)is_causal from EuroBertConfig (#44774)mlcd auto config/model/mapping issues (#44730)config class in some model class definitions (#44715)parent issue (#44685)VibeVoiceAcousticTokenizer (#44628)build_pr_documentation.yml (will be the new required job) (#44538)build_pr_documentation workflow for merge_group event (#44532)diffusers to CI docker file (#44480).ai (#44489)src/transformers/quantizers (#44412)ty to 0.0.20 (#44494)T5ModelIntegrationTest (#44934)Jina-Embeddings-V3 Model (#44251)MusicgenStereo integration tests (#44527)GPTNeoModelLanguageGenerationTest (#44515)MarianIntegrationTests (#44519)DepthProModelIntegrationTest (#44456)ProphetNetModelIntegrationTest (#44439)FA4] Add kernels fallback (#44797)FA] Fix fa detection (#44703)FA4] Initial support (#42435)Chmv2] Fix conversion after capture refactor (#44665)higgs_audio_v2 tests (#44482)text2text-generation, summarization and translation pipeline tasks (#44510)Fixed several kernel-related issues including a security vulnerability, corrected Mamba kernel loading to handle incompatible import structures, ensur…
<img width="1080" height="1080" alt="image" src="https://github.com/user-attachments/assets/33603f42-5435-421a-9641-baf72faacb22" />
EuroBERT is a multilingual encoder model based on a refreshed transformer architecture, akin to Llama but with bidirectional attention. It supports a mixture of European and widely spoken languages, with sequences of up to 8192 tokens.
Links: Documentation | Paper | Blog Post
<img width="673" height="464" alt="image" src="https://github.com/user-attachments/assets/e4093a6b-fc6e-4136-a15d-2fcd7b27a69e" />
VibeVoice ASR is an automatic speech recognition model from Microsoft that combines acoustic and semantic audio tokenizers with a causal language model for robust speech-to-text transcription. The model uses VibeVoice's acoustic and semantic tokenizers that process audio at 24kHz, paired with a Qwen2-based language decoder for generating transcriptions. It can process up to 60 minutes of continuous audio input, supports customized hotwords, performs joint ASR/diarization/timestamping, and handles over 50 languages with code-switching support.
Links: Documentation | Paper
<img width="799" height="497" alt="image" src="https://github.com/user-attachments/assets/1e486803-1b68-496b-aa67-4c3f2055fbeb" />
TimesFM 2.5 is a pretrained time-series foundation model that uses a decoder-only attention architecture with input patching for forecasting. The model is designed to provide accurate zero-shot forecasts across different domains, forecasting horizons and temporal granularities without requiring dataset-specific training. It builds on the original TimesFM architecture with enhancements including rotary attention, QK normalization, per-dimension attention scaling, and continuous quantile prediction.
Links: Documentation | Paper
<img width="1440" height="436" alt="image" src="https://github.com/user-attachments/assets/31d6609b-ef42-4f15-8c34-eeb2c0d679a9" />
PP-DocLayoutV2 is a dedicated lightweight model for layout analysis, focusing specifically on element detection, classification, and reading order prediction. The model is composed of two sequentially connected networks: an RT-DETR-based detection model that performs layout element detection and classification, followed by a pointer network that orders these layout elements. It is designed to analyze document layouts by identifying and organizing various layout components in their proper reading sequence.
Links: Documentation
OLMo Hybrid is a hybrid architecture model from Ai2 that combines standard transformer attention layers with linear attention layers using the Gated Deltanet. This hybrid approach aims to improve efficiency while maintaining model quality by interleaving full attention layers with linear attention layers. The model uses a custom cache system that handles both KV cache for attention layers and recurrent state for linear attention layers.
Links: Documentation
<img width="332" height="343" alt="image" src="https://github.com/user-attachments/assets/23e1e140-9ad2-4144-b5d6-8b8c1e3414c9" />
ModernVBert is a Vision-Language encoder that combines ModernBert with a SigLIP vision encoder. It is optimized for visual document understanding and retrieval tasks, making it suitable for processing documents that contain both text and visual elements.
Links: Documentation | Paper
ColModernVBert is a model for efficient visual document retrieval that leverages ModernVBert to construct multi-vector embeddings directly from document images, following the ColPali approach. The model enables retrieval and scoring of visual documents by processing both text queries and document images to generate embeddings that can be compared for relevance scoring.
Links: Documentation | Paper
<img width="3065" height="1464" alt="image" src="https://github.com/user-attachments/assets/94ad4db1-3c10-43d1-b1f7-2ce01329c8a4" />
Higgs Audio V2 is a powerful audio foundation model developed by Boson AI that was pretrained on over 10 million hours of audio data and diverse text data. Despite having no post-training or fine-tuning, the model excels in expressive audio generation thanks to its deep language and acoustic understanding. The model supports various audio generation tasks including single-speaker and multi-speaker smart voice, zero-shot voice cloning, and multi-speaker voice cloning.
Links: Documentation
The Higgs Audio V2 Tokenizer is an audio tokenization model that operates at a low frame rate of 25 fps while maintaining high audio quality, effectively halving the frame rate of many baseline models. It uses unified 24 kHz training that mixes speech, music, and sound-event clips in one model to capture both semantic and acoustic details, facilitating the training of audio language models. The model enables fast inference by avoiding diffusion steps, with an encoder/decoder architecture that processes batches quickly for real-time or large-scale tasks.
Links: Documentation
Tensor parallelism (TP) support for dense and MoE decoder-only models has been fixed and stabilized, requiring users to update their TP configurations and conversion mappings accordingly.
The Ernie4.5 VL MoE model class and configuration names have been renamed to align with vLLM/SGLang conventions, requiring users to update any references to the old model names in their code.
Ernie 4.5 VL Moe] Fix up namings to vllm/sglang convention (#44299) by @vasquSeveral pipeline tasks have been removed or updated in the V5 cleanup (including question-answering, visual-question-answering, and image-to-image), requiring users to migrate to the replacement pipelines or updated task names.
3D position IDs for vision-language models have been unified under a common interface (sourced from qwen2-vl), requiring users of affected VLMs (e.g., Ernie, GLM4V) to update their processors and any code that manually constructs position IDs.
Unigram tokenizers were missing the spm precompiled charsmap support. We ran an overall v4 vs v5 regression test and fixed what we had missed.
This was done in:
Generation input preparation was significantly refactored to stop relying on cache_position and instead pass pre-sliced input_ids/inputs_embeds directly to prepare_inputs_for_generation, simplifying the generation loop and laying groundwork for broader cache_position removal. Several bug fixes were also applied, including correct sampling for HiggsAudioV2, flaky cache-equality test stabilization for Idefics, and restored generation integration tests.
prepare_inputs_for_generation (#44226) by @Cyrilvallez in [#44226]cache_position to prepare inputs (#44130) by @Cyrilvallez in [#44130]Several tokenization bugs were fixed in this release, including resolving an AttributeError in MLukeTokenizer caused by the v5 rename of additional_special_tokens, correcting the Fuyu tokenizer class mapping, fixing LayoutXLM tokenization test failures from the slow tokenizer removal refactor, and adding olmo_hybrid to the auto-tokenizer mapping. The tokenizer documentation was also updated to reflect the new unified v5 backend architecture and reorganized for clarity.
Fixed several kernel-related issues including a security vulnerability, corrected Mamba kernel loading to handle incompatible import structures, ensured Liger Kernel is properly enabled during hyperparameter search, and expanded Flash Attention to support multiple compatible implementations.
Mamba] Fix kernel loading (#44176) by @vasqu in [#44176]Flash Attn] Enable compatible implementations (#44177) by @vasqu in [#44177]This release adds several new quantization backends and fixes, including MLX quantization support for MPS devices, Four Over Six (4/6) NVFP4 quantization integration for NVIDIA Blackwell GPUs, and CPU support for MXFP4 models, alongside a bug fix for MXFP4 model saving using reverse_op.
Fixed backward compatibility for image processors loaded from older remote code that lack valid_kwargs definitions, and resolved test failures in AMD ROCm CI by adding the missing timm dependency to the Docker image.
from_dict backward compatibility with old remote code (#44245) by @yonigozlan in [#44245]speaking_rate as an optionl forward argument (#43283) by @gau-nernst in [#43283]ProcessingKwargs ImagesKwargs etc. to docs (#44269) by @yonigozlan in [#44269]has_similar_generate_outputs assertions (#44166) by @tarekziade in [#44166]TokenizersBackend for Olmo3 to preserve custom pre_tokenizer (#44294) by @mario-sanz in [#44294]Modular] Fix file type regression (#44283) by @vasqu in [#44283]Trainer class docs (compute_loss & hyperparameter_search) (#44268) by @ethanknights in [#44268]fix] Set input_modalities on various architectures that aren't just text (#44078) by @tomaarsen in [#44078]VersionComparison.from_string return type mismatch (#43709) by @tarekziade in [#43709]AnyToAnyPipeline.__call__ docstring (#44229) by @alvarobartt in [#44229]test_generate_with_and_without_position_ids in GLM ORC (#44173) by @tarekziade in [#44173]Seq2SeqTrainingArguments documentation (#35258) by @qgallouedec in [#35258]__setitem__ on ModelOutput even if the parameter was previously None (#44080) by @tomaarsen in [#44080]simple] Fix up __repr__ whitespace/brackets (#44048) by @tomaarsen in [#44048]chore] Fix incorrect forward type hint for Gemma3n (#44051) by @tomaarsen in [#44051]get_audio_features (#44040) by @zucchini-nlp in [#44040]Kosmos2ModelTest test (#44061) by @tarekziade in [#44061]grouped_mm fallback (#44043) by @IlyasMoutawwakil in [#44043]The following contributors have made significant changes to the library over the last release:
has_similar_generate_outputs assertions (#44166)VersionComparison.from_string return type mismatch (#43709)test_generate_with_and_without_position_ids in GLM ORC (#44173)Kosmos2ModelTest test (#44061)Ernie 4.5 VL Moe] Fix up namings to vllm/sglang convention (#44299)Modular] Fix file type regression (#44283)Mamba] Fix kernel loading (#44176)Flash Attn] Enable compatible implementations (#44177)adding BC for custom toks accessing slow tok attrs deprecated in v5 (#43898) by @itazap
<img width="1920" height="1080" alt="image" src="https://github.com/user-attachments/assets/80e37670-6d70-402b-8c8e-ccfb8c32df2d" />
VoxtralRealtime is a streaming speech-to-text model from Mistral AI, designed for real-time automatic speech recognition (ASR). Unlike the offline Voxtral model which processes complete audio files, VoxtralRealtime is architected for low-latency, incremental transcription by processing audio in chunks as they arrive.
The model combines an audio encoder with a Mistral-based language model decoder, using time conditioning embeddings and causal convolutions with padding caches to enable efficient streaming inference.
<img width="947" height="638" alt="image" src="https://github.com/user-attachments/assets/4c4fff37-7f40-4e86-b4a0-db718f45c93b" />
The zAI team launches GLM-5, and introduces it as such:
GLM-5, targeting complex systems engineering and long-horizon agentic tasks. Scaling is still one of the most important ways to improve the intelligence efficiency of Artificial General Intelligence (AGI). Compared to GLM-4.5, GLM-5 scales from 355B parameters (32B active) to 744B parameters (40B active), and increases pre-training data from 23T to 28.5T tokens. GLM-5 also integrates DeepSeek Sparse Attention (DSA), largely reducing deployment cost while preserving long-context capacity.
Reinforcement learning aims to bridge the gap between competence and excellence in pre-trained models. However, deploying it at scale for LLMs is a challenge due to the RL training inefficiency. To this end, we developed slime, a novel asynchronous RL infrastructure that substantially improves training throughput and efficiency, enabling more fine-grained post-training iterations. With advances in both pre-training and post-training, GLM-5 delivers significant improvement compared to GLM-4.7 across a wide range of academic benchmarks and achieves best-in-class performance among all open-source models in the world on reasoning, coding, and agentic tasks, closing the gap with frontier models.
<img width="1920" height="1080" alt="image" src="https://github.com/user-attachments/assets/b56dcaca-80e7-4b22-80a5-2f767bb65095" />
The Qwen team launches Qwen 3.5, and introduces it as such:
We are delighted to announce the official release of Qwen3.5, introducing the open-weight of the first model in the Qwen3.5 series, namely Qwen3.5-397B-A17B. As a native vision-language model, Qwen3.5-397B-A17B demonstrates outstanding results across a full range of benchmark evaluations, including reasoning, coding, agent capabilities, and multimodal understanding, empowering developers and enterprises to achieve significantly greater productivity. Built on an innovative hybrid architecture that fuses linear attention (via Gated Delta Networks) with a sparse mixture-of-experts, the model attains remarkable inference efficiency: although it comprises 397 billion total parameters, just 17 billion are activated per forward pass, optimizing both speed and cost without sacrificing capability. We have also expanded our language and dialect support from 119 to 201, providing broader accessibility and enhanced support to users around the world.
<img width="821" height="349" alt="image" src="https://github.com/user-attachments/assets/b1433597-b43b-4d2d-a2c7-216d7792b8c9" />
VibeVoice is a novel framework for synthesizing high-fidelity, long-form speech with multiple speakers by employing a next-token diffusion approach within a Large Language Model (LLM) structure. It's designed to capture the authentic conversational "vibe" and is particularly suited for generating audio content like podcasts and multi-participant audiobooks.
One key feature of VibeVoice is the use of two continuous audio tokenizers, one for extracting acoustic features and another for semantic features.
Attn] New attn mask interface everywhere (#42848):rotating_light: This one is quite breaking for super super super old modles: :rotating_light: :rotating_light:
convert_rope_params_to_dict so it uses rope_theta from the config (#43766) by @hmellorAGENTS.md (#43763) by @tarekziadeModular Dependencies] Fixup qwen rms norms (#43772) by @vasquRepo Consistency] Fix rms norm (#43803) by @vasqucheck_model_inputs implementation (#43765) by @Cyrilvallezdo_sample=False to qwen2_5_vl model tests to stablize the output (#43728) by @kaixuanliuJamba] Fallback to slow path and warn instead of error out (#43889) by @vasqufix] Use last_hidden_state key from get_image_features for llama4 (#43882) by @tomaarsencheck_model_inputs into capture_outputs and merge_with_config_defaults + ensure correctness (#43862) by @Cyrilvallez_keys_to_ignore_on_load_missing for now (#43893) by @ArthurZuckerinput_embeds to inputs_embeds everywhere (#43916) by @Cyrilvallezimage_url content support in apply_chat_template (#43786) by @kaixuanliugenerate (#43734) by @zucchini-nlprun_*_no‑trainer.py examples (#42769) by @casincarun_*_no‑trainer.py examples (#43947) by @casincaout_features (#43886) by @zucchini-nlpget_number_of_image_tokens (#43948) by @zucchini-nlpother_workflow_run_ids for issue_comment in utils/notification_service.py (#44036) by @ydshiehThe following contributors have made significant changes to the library over the last release:
Jamba] Fallback to slow path and warn instead of error out (#43889)Attn] New attn mask interface everywhere (#42848)Repo Consistency] Fix rms norm (#43803)Modular Dependencies] Fixup qwen rms norms (#43772)🚨 Remove deprecated AnnotionFormat (#42983) - removes a missnamed class in favour of AnnotationFormat.
<img width="2278" height="1142" alt="image" src="https://github.com/user-attachments/assets/0c3d5341-0483-49c3-8467-f9784ec94b37" />
K-EXAONE is a large-scale multilingual language model developed by LG AI Research. Built using a Mixture-of-Experts architecture, K-EXAONE features 236 billion total parameters, with 23 billion active during inference. Performance evaluations across various benchmarks demonstrate that K-EXAONE excels in reasoning, agentic capabilities, general knowledge, multilingual understanding, and long-context processing.
<img width="6252" height="1892" alt="image" src="https://github.com/user-attachments/assets/b2e58244-8ed3-42c6-80d7-e32842977ddb" />
PP-DocLayoutV3 is a unified and high-efficiency model designed for comprehensive layout analysis. It addresses the challenges of complex physical distortions—such as skewing, curving, and adverse lighting—by integrating instance segmentation and reading order prediction into a single, end-to-end framework.
<img width="564" height="352" alt="image" src="https://github.com/user-attachments/assets/864372be-4ecb-41fd-8c92-f3515be040d3" />
Youtu-LLM is a new, small, yet powerful LLM, contains only 1.96B parameters, supports 128k long context, and has native agentic talents. On general evaluations, Youtu-LLM significantly outperforms SOTA LLMs of similar size in terms of Commonsense, STEM, Coding and Long Context capabilities; in agent-related testing, Youtu-LLM surpasses larger-sized leaders and is truly capable of completing multiple end2end agent tasks.
<img width="3972" height="2352" alt="image" src="https://github.com/user-attachments/assets/a7ddfb4f-42ea-4dc6-bc73-aefb0f750c4e" />
GLM-OCR is a multimodal OCR model for complex document understanding, built on the GLM-V encoder–decoder architecture. It introduces Multi-Token Prediction (MTP) loss and stable full-task reinforcement learning to improve training efficiency, recognition accuracy, and generalization. The model integrates the CogViT visual encoder pre-trained on large-scale image–text data, a lightweight cross-modal connector with efficient token downsampling, and a GLM-0.5B language decoder. Combined with a two-stage pipeline of layout analysis and parallel recognition based on PP-DocLayout-V3, GLM-OCR delivers robust and high-quality OCR performance across diverse document layouts.
🚨 T5Gemma2 model structure (#43633) - Makes sure that the attn implementation is set to all sub-configs. The config.encoder.text_config was not getting its attn set because we aren't passing it to PreTrainedModel.init. We can't change the model structure without breaking so I manually re-added a call to self.adjust_attn_implemetation in modeling code
🚨 Generation cache preparation (#43679) - Refactors cache initialization in generation to ensure sliding window configurations are now properly respected. Previously, some models (like Afmoe) created caches without passing the model config, causing sliding window limits to be ignored. This is breaking because models with sliding window attention will now enforce their window size limits during generation, which may change generation behavior or require adjusting sequence lengths in existing code.
🚨 Delete duplicate code in backbone utils (#43323) - This PR cleans up backbone utilities. Specifically, we have currently 5 different config attr to decide which backbone to load, most of which can be merged into one and seem redundant After this PR, we'll have only one config.backbone_config as a single source of truth. The models will load the backbone from_config and load pretrained weights only if the checkpoint has any weights saved. The overall idea is same as in other composite models. A few config arguments are removed as a result.
🚨 Refactor DETR to updated standards (#41549) - standardizes the DETR model to be closer to other vision models in the library.
🚨Fix floating-point precision in JanusImageProcessor resize (#43187) - replaces an int() with round(), expect light numerical differences
🚨 Remove deprecated AnnotionFormat (#42983) - removes a missnamed class in favour of AnnotationFormat.
feat] Allow loading T5Gemma2Encoder with AutoModel (#43559) by @tomaarsenimage_sizes input param (#43678) by @kaixuanliuAttn] Fixup interface usage after refactor (#43706) by @vasqunum_frames in ASR pipeline (#43546) by @jiqing-fengPreTrainedTokenizerBase (#43675) by @tarekziadeFP8Expert for DeepSeek R1 (#43616) by @yiliu30HunYuan] Fix RoPE init (#43411) by @vasquSam] Fixup training flags (#43567) by @vasquprocess_bad_commit_report.py: avoid items to appear in null author in the report (#43662) by @ydshiehKeyError in check_bad_commit.py (#43655) by @ydshiehtied_weight_keys in-place (#43619) by @zucchini-nlpRope] Revert #43410 and make inheritance implicit again (#43620) by @vasqumake_batched_video with 5D arrays (#43486) by @zucchini-nlputils/fetch_hub_objects_for_ci.py: avoid too many requests and/or timeout (#43584) by @ydshiehMistralConverter.extract_vocab_merges_from_model (#43557) by @tarekziadetemplates folder (#43536) by @CyrilvallezModular] Allow to add new bases that are not present in the inherited class (#43556) by @vasqupad_token_id (#43453) by @Sai-Suraj-27RoPE] Make explicit inheritance (#43410) by @vasquShieldGemma2IntegrationTest::test_model (#43343) by @sywangyiSamHQModelIntegrationTest::test_inference_mask_generation_batched_points_batched_images for XPU (#43511) by @sywangyisuper() (#43280) by @zucchini-nlppytest-random-order for reproducible test randomization (#43483) by @tarekziademarkuplm & perception_lm integration tests (#43464) by @Sai-Suraj-27The following contributors have made significant changes to the library over the last release:
PreTrainedTokenizerBase (#43675)MistralConverter.extract_vocab_merges_from_model (#43557)pytest-random-order for reproducible test randomization (#43483)Attn] Fixup interface usage after refactor (#43706)HunYuan] Fix RoPE init (#43411)Sam] Fixup training flags (#43567)Rope] Revert #43410 and make inheritance implicit again (#43620)Modular] Allow to add new bases that are not present in the inherited class (#43556)RoPE] Make explicit inheritance (#43410)process_bad_commit_report.py: avoid items to appear in null author in the report (#43662)KeyError in check_bad_commit.py (#43655)utils/fetch_hub_objects_for_ci.py: avoid too many requests and/or timeout (#43584)Biggest breaking change is in transformers chat. This command starts a terminal UI to interact with a chat model. It used to also be able to start a C…
<img width="1800" height="1013" alt="image" src="https://github.com/user-attachments/assets/7b5187d7-6945-4108-a546-6d1d7bfb55e3" />
We have a migration guide that will be continuously updated available on the main branch, please check it out in case you're facing issues: migration guide.
We are excited to announce the initial release of Transformers v5. This is the first major release in five years, and the release is significant: 1200 commits have been pushed to main since the latest minor release. This release removes a lot of long-due deprecations, introduces several refactors that significantly simplify our APIs and internals, and comes with a large number of bug fixes.
We give an overview of our focus for this release in the following blogpost. In these release notes, we'll focus directly on the refactors and new APIs coming with v5.
This release is the full V5 release. It sets in motion something bigger: going forward, starting with v5, we'll now release minor releases every week, rather than every 5 weeks. Expect v5.1 to follow next week, then v5.2 the week that follows, etc.
We're moving forward with this change to ensure you have access to models as soon as they're supported in the library, rather than a few weeks after.
In order to install this release, please do so with the following:
pip install transformers
For us to deliver the best package possible, it is imperative that we have feedback on how the toolkit is currently working for you. Please try it out, and open an issue in case you're facing something inconsistent/a bug.
Transformers version 5 is a community endeavor, and we couldn't have shipped such a massive release without the help of the entire community.
We introduce a new weight loading API in transformers, which significantly improves on the previous API. This
weight loading API is designed to apply operations to the checkpoints loaded by transformers.
Instead of loading the checkpoint exactly as it is serialized within the model, these operations can reshape, merge, and split the layers according to how they're defined in this new API. These operations are often a necessity when working with quantization or parallelism algorithms.
This new API is centered around the new WeightConverter class:
class WeightConverter(WeightTransform):
operations: list[ConversionOps]
source_keys: Union[str, list[str]]
target_keys: Union[str, list[str]]
The weight converter is designed to apply a list of operations on the source keys, resulting in target keys. A common operation done on the attention layers is to fuse the query, key, values layers. Doing so with this API would amount to defining the following conversion:
conversion = WeightConverter(
["self_attn.q_proj", "self_attn.k_proj", "self_attn.v_proj"], # The input layers
"self_attn.qkv_proj", # The single layer as output
operations=[Concatenate(dim=0)],
)
In this situation, we apply the Concatenate operation, which accepts a list of layers as input and returns a single
layer.
This allows us to define a mapping from architecture to a list of weight conversions. Applying those weight conversions
can apply arbitrary transformations to the layers themselves. This significantly simplified the from_pretrained method
and helped us remove a lot of technical debt that we accumulated over the past few years.
This results in several improvements:
Linked PR: https://github.com/huggingface/transformers/pull/41580
Just as we moved towards a single backend library for model definition, we want our tokenizers, and the Tokenizer object to be a lot more intuitive. With v5, tokenizer definition is much simpler; one can now initialize an empty LlamaTokenizer and train it directly on your corpus.
Defining a new tokenizer object should be as simple as this:
from transformers import TokenizersBackend, generate_merges
from tokenizers import pre_tokenizers, Tokenizer
from tokenizers.model import BPE
class Llama5Tokenizer(TokenizersBackend):
def __init__(self, unk_token="<unk>",bos_token="<s>", eos_token="</s>", vocab=None, merges=None ):
if vocab is None:
self._vocab = {
str(unk_token): 0,
str(bos_token): 1,
str(eos_token): 2,
}
else:
self._vocab = vocab
self._merges = merges
self._tokenizer = Tokenizer(
BPE(vocab=self._vocab, merges=self._merges, fuse_unk=True)
)
self._tokenizer.pre_tokenizer = pre_tokenizers.Metaspace(
replacement="▁", prepend_scheme=_get_prepend_scheme(self.add_prefix_space, self), split=False
)
super().__init__(
tokenizer_object=self._tokenizer,
unk_token=unk_token,
bos_token=bos_token,
eos_token=eos_token,
)
Once the tokenizer is defined as above, you can load it with the following: Llama5Tokenizer(). Doing this returns you an empty, trainable tokenizer that follows the definition of the authors of Llama5 (it does not exist yet :wink:).
The above is the main motivation towards refactoring tokenization: we want tokenizers to behave similarly to models: trained or empty, and with exactly what is defined in their class definition.
Up to now, transformers maintained two parallel implementations for many tokenizers:
tokenization_<model>.py) - Python-based implementations, often using SentencePiece as the backend.tokenization_<model>_fast.py) - Rust-based implementations using the 🤗 tokenizers library.In v5, we consolidate to a single tokenizer file per model: tokenization_<model>.py. This file will use the most appropriate backend available:
sentencepiece library. It inherits from PythonBackend.tokenizers. Basically allows adding tokens.MistralCommon's tokenization library. (Previously known as the MistralCommonTokenizer)The AutoTokenizer automatically selects the appropriate backend based on available files and dependencies. This is transparent, you continue to use AutoTokenizer.from_pretrained() as before. This allows transformers to be future-proof and modular to easily support future backends.
We enable users and tokenizer builders to define their own tokenizers from top to bottom. Tokenizers are usually defined using a backend such as tokenizers, sentencepiece or mistral-common, but we offer the possibility to design the tokenizer at a higher-level, without relying on those backends.
To do so, you can import the PythonBackend (which was previously known as PreTrainedTokenizer). This class encapsulates all the logic related to added tokens, encoding, and decoding.
If you want something even higher up the stack, then PreTrainedTokenizerBase is what PythonBackend inherits from. It contains the very basic tokenizer API features:
encodedecodevocab_sizeget_vocabconvert_tokens_to_idsconvert_ids_to_tokensfrom_pretrainedsave_pretrainedStarting with v5, we now enable initializing blank, untrained tokenizers-backed tokenizers:
from transformers import LlamaTokenizer
tokenizer = LlamaTokenizer()
This tokenizer will therefore follow the definition of the LlamaTokenizer as defined in its class definition. It can then be trained on a corpus as can be seen in the tokenizers documentation.
These tokenizers can also be initialized from vocab and merges (if necessary), like the previous "slow" tokenizers:
from transformers import LlamaTokenizer
vocab = {"<unk>": 0, "<s>": 1, "</s>": 2, "hello": 3, "world": 4}
merges = [("h", "e"), ("l", "l"), ("o", " ")]
tokenizer = LlamaTokenizer(vocab=vocab, merges=merges)
This tokenizer will behave as a Llama-like tokenizer, with an updated vocabulary. This allows comparing different tokenizer classes with the same vocab; therefore enabling the comparison of different pre-tokenizers, normalizers, etc.
⚠️ The vocab_file (as in, a path towards a file containing the vocabulary) cannot be used to initialize the LlamaTokenizer as loading from files is reserved to the from_pretrained method.
The batch_decode and decode methods have been unified to reflect behavior of the encode method. Both single and batch decoding now use the same decode method. See an example of the new behavior below:
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("t5-small")
inputs = ["hey how are you?", "fine"]
tokenizer.decode(tokenizer.encode(inputs))
Gives:
- 'hey how are you?</s> fine</s>'
+ ['hey how are you?</s>', 'fine</s>']
We expect encode and decode to behave, as two sides of the same coin: encode, process, decode, should work.
[!NOTE] A common use-case would be:
encode,model.generate,decode. However, usinggeneratewould returnlist[list[int]], which would then be incompatible withdecode.
The encode_plus method is deprecated in favor of the single __call__ method.
apply_chat_template returns BatchEncodingPreviously, apply_chat_template returned input_ids for backward compatibility. Starting with v5, it now consistently returns a BatchEncoding dict like other tokenizer methods.
# v5
messages = [
{"role": "user", "content": "Hello!"},
{"role": "assistant", "content": "Hi there!"}
]
# Now returns BatchEncoding with input_ids, attention_mask, etc.
outputs = tokenizer.apply_chat_template(messages, return_tensors="pt")
print(outputs.keys()) # dict_keys(['input_ids', 'attention_mask'])
We simplify the serialization of tokenization attributes:
special_tokens_map.json - special tokens are now stored in tokenizer_config.json.added_tokens.json - added tokens are now stored in tokenizer.json.added_tokens_decoder is only stored when there is no tokenizer.json.When loading older tokenizers, these files are still read for backward compatibility, but new saves use the consolidated format. We're gradually moving towards consolidating attributes to fewer files so that other libraries and implementations may depend on them more reliably.
Several models that had identical tokenizers now import from their base implementation:
These modules will eventually be removed altogether.
Removed T5-specific workarounds
The internal _eventually_correct_t5_max_length method has been removed. T5 tokenizers now handle max length consistently with other models.
A few testing changes specific to tokenizers have been applied:
add_tokens, encode, decode) are now centralized and automatically applied across all tokenizers. This reduces test duplication and ensures consistent behaviorFor legacy implementations, the original BERT Python tokenizer code (including WhitespaceTokenizer, BasicTokenizer, etc.) is preserved in bert_legacy.py for reference purposes.
Special Tokens Structure:
SpecialTokensMixin: Merged into PreTrainedTokenizerBase to simplify the tokenizer architecture.special_tokens_map: Now only stores named special token attributes (e.g., bos_token, eos_token). Use extra_special_tokens for additional special tokens (formerly additional_special_tokens). all_special_tokens includes both named and extra tokens.# v4
tokenizer.special_tokens_map # Included 'additional_special_tokens'
# v5
tokenizer.special_tokens_map # Only named tokens
tokenizer.extra_special_tokens # Additional tokens
special_tokens_map_extended and all_special_tokens_extended: Removed. Access AddedToken objects directly from _special_tokens_map or _extra_special_tokens if needed.additional_special_tokens: Still accepted for backward compatibility but is automatically converted to extra_special_tokens.Deprecated Methods:
sanitize_special_tokens(): Already deprecated in v4, removed in v5.prepare_seq2seq_batch(): Deprecated; use __call__() with text_target parameter instead.# v4
model_inputs = tokenizer.prepare_seq2seq_batch(src_texts, tgt_texts, max_length=128)
# v5
model_inputs = tokenizer(src_texts, text_target=tgt_texts, max_length=128, return_tensors="pt")
model_inputs["labels"] = model_inputs.pop("input_ids_target")
BatchEncoding.words(): Deprecated; use word_ids() instead.Removed Methods:
create_token_type_ids_from_sequences(): Removed from base class. Subclasses that need custom token type ID creation should implement this method directly.prepare_for_model(), build_inputs_with_special_tokens(), truncate_sequences(): Moved from tokenization_utils_base.py to tokenization_python.py for PythonBackend tokenizers. TokenizersBackend provides model-ready input via tokenize() and encode(), so these methods are no longer needed in the base class._switch_to_input_mode(), _switch_to_target_mode(), as_target_tokenizer(): Removed from base class. Use __call__() with text_target parameter instead.# v4
with tokenizer.as_target_tokenizer():
labels = tokenizer(tgt_texts, ...)
# v5
labels = tokenizer(text_target=tgt_texts, ...)
parse_response(): Removed from base class.The v5 release significantly improves the performance of the MoE models, as can be seen in the graphs below. We improve and optimize MoE performance through batched and grouped experts implementations, and we optimize them for decoding using batched_mm.
<img width="2048" height="1451" alt="image" src="https://github.com/user-attachments/assets/c3f2e59f-3026-4f56-9a56-36e4eb0fcf73" />
We focus on improving the performance of loading weights on device (which gives speedups up to 6x in tensor parallel situations); this is preliminary work that we'll continue to work on in the coming weeks. Some notable improvements:
dtype updateWe have updated the default dtype for all models loaded with from_pretrained to be auto. This will lead to model instantiations respecting the dtype in which the model was saved, rather than forcing it to load in float 32.
You can, of course, still specify the dtype in which you want to load your model by specifying it as an argument to the from_pretrained method.
The Hugging Face Hub infrastructure has gradually moved to a XET backend. This will significantly simplify uploads and downloads, with higher download and upload speeds, partial uploads, and, most notably, a higher threshold for accepted file sizes on the Hugging Face Hub.
To reflect this, we're increasing the default shard size of models serialized on the Hub to 50GB (up from 5GB).
use_auth_tokenThe use_auth_token argument/parameter is deprecated in favor of token everywhere.
You should be able to search and replace use_auth_token with token and get the same logic.
Linked PR: https://github.com/huggingface/transformers/pull/41666
We decided to remove some features for the upcoming v5 as they are currently only supported in a few old models and no longer integrated in current model additions. It's recommended to stick to v4.x in case you need them. Following features are affected:
We dropped support for two torch APIs:
torchscript in https://github.com/huggingface/transformers/pull/41688torch.fx in https://github.com/huggingface/transformers/pull/41683Those APIs were deprecated by the PyTorch team, and we're instead focusing on the supported APIs dynamo and export.
We clean up the quantization API in transformers, and significantly refactor the weight loading as highlighted above.
We drop support for two quantization arguments that have been deprecated for some time:
load_in_4bitload_in_8bitWe remove them in favor of the quantization_config argument which is much more complete. As an example, here is how
you would load a 4-bit bitsandbytes model using this argument:
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
quantization_config = BitsAndBytesConfig(load_in_4bit=True)
model_4bit = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-3.2-3B",
device_map="auto",
quantization_config=quantization_config
)
from_xxx_config are deleted. Configs can be init from the __init__ method in the same way. See #41314.mode.rope_parameters, including the rope_theta and rope_type. Model's config.rope_parameters is a simple dictionaty in most cases, and can also be a nested dict in special cases (i.e. Gemma3 and ModernBert) with different rope parameterization for each layer type. Trying to get config.rope_theta will throw an attribute error from now on. See #39847 and #42255config.vocab_size). Users are expected to access keys from their respective sub-configs (config.text_config.vocab_size).model.generate()) will no longer have a generation_config and model.config.generation_config will throw an attribute error.tokenization_<model>.py ) will be removed in favor of using fast tokenizer files tokenization_<model>_fast.py --> will be renamed to tokenization_<model>.py. As fast tokenizers are :hugs:tokenizers - backend, they include a wider range of features that are maintainable and reliable.encode_plus --> __call__batch_decode --> decodeapply_chat_template by default returns naked input_ids rather than a BatchEncoding dict.
This was inconvenient - it should return a BatchEncoding dict like tokenizer.__call__(), but we were stuck with
it for backward compatibility. The method now returns a BatchEncoding.
Linked PRs:
processor_config.json as a nested dict, instead of serializing attributes in their own config files. Loading will be supported for all old format processors (https://github.com/huggingface/transformers/pull/41474)XXXFeatureExtractors classes are completely removed in favor of XXXImageProcessor class for all vision models (https://github.com/huggingface/transformers/pull/41174)XXXFastImageProcessorKwargs is removed in favor of XXXImageProcessorKwargs which will be shared between fast and slow processors (https://github.com/huggingface/transformers/pull/40931)RotaryEmbeddings layers will start returning a dict of tuples, in case the model uses several RoPE configurations (Gemma2, ModernBert). Each value will be a tuple of "cos, sin" per RoPE type.RotaryEmbeddings layer will be unified and accessed via config.rope_parameters. Config attr for rope_theta might not be accessible anymore for some models, and instead will be in config.rope_parameters['rope_theta']. BC will be supported for a while as much as possible, and in the near future we'll gradually move to the new RoPE format (https://github.com/huggingface/transformers/pull/39847)model.language_model. It is recommended to either access the module with model.model.language_model or model.get_decoder(). See #42156kwargs in their forward methodsGreedySearchEncoderDecoderOutput). We now only have 4 output classes built from the following matrix: decoder-only vs encoder-decoder, uses beams vs doesn't use beams (https://github.com/huggingface/transformers/pull/40998)generate doesn't receive any KV Cache argument, the default cache class used is now defined by the model (as opposed to always being DynamicCache) (https://github.com/huggingface/transformers/pull/41505)config.json for any old model, it will be loaded back into model's generation config. Users are expected to access or modify generation parameters only with model.generation_config.do_sample = True.compute_loss_func Handling
compute_loss_func now always takes priority over the model's built-in loss computation, giving users consistent control over custom loss functions.num_items_in_batch in Prediction Step
num_items_in_batch argument is now passed to compute_loss during prediction_step, enabling proper loss scaling during evaluation.report_to now defaults to "none"
TrainingArguments due to low usagemp_parameters -> legacy param that was later on added to the Sagemaker trainer_n_gpu -> not intended for users to set, we will initialize it correctly instead of putting it in the TrainingArgumentsoverwrite_output_dir - > replaced by resume_from_checkpoint, and it was only used in the examples script, no impact on Trainer.logging_dir -> only used for tensorboard, set TENSORBOARD_LOGGING_DIR env var insteadjit_mode_eval -> use use_torch_compile instead, as torchscript is not recommended anymoretpu_num_cores-> It is actually better to remove it, as it is not recommended to set the number of cores. By default, all TPU cores are used . Set TPU_NUM_CORES env var insteadpast_index -> it was only used for a very small number of models that have special architecture like transformersxl + it was not documented at all how to train those modelsray_scope -> only for a minor arg for ray integration. Set RAY_SCOPE var env insteadwarmup_ratio -> use warmup_step instead. We combined both args together by allowing passing float values in warmup_step.TrainingArgumentsfsdp_min_num_params and fsdp_transformer_layer_cls_to_wrap -> use fsdp_configtpu_metrics_debug -> debugpush_to_hub_token -> hub_tokenpush_to_hub_model_id and push_to_hub_organization -> hub_model_idinclude_inputs_for_metrics -> include_for_metricsper_gpu_train_batch_size -> per_device_train_batch_sizeper_gpu_eval_batch_size -> per_device_eval_batch_sizeuse_mps_device -> mps will be used by default if detectedfp16_backend and half_precision_backend -> we will only rely on torch.amp as everything has been upstreamed to torchno_cuda -> use_cpu include_tokens_per_second -> include_num_input_tokens_seenuse_legacy_prediction_loop -> we only use evaluation_loop function from now onTrainertokenizer in initialization -> processing_classmodel_path in train() -> resume_from_checkpointTrainerTraineruse_cache in the model config will be set to False. You can still change the cache value through TrainingArguments usel_cache argument if needed.organization and repo_url from PushToHubMixin. You must pass a repo_id instead.ignore_metadata_errors from PushToMixin. In practice if we ignore errors while loading the model card, we won't be able to push the card back to the Hub so it's better to fail early and not provide the option to fail later.push_to_hub do not accept **kwargs anymore. All accepted parameters are explicitly documented.push_to_hub are now keyword-only to avoid confusion. Only repo_id can be positional since it's the main arg.use_temp_dir argument from push_to_hub. We now use a tmp dir in all cases.Linked PR: https://github.com/huggingface/transformers/pull/42391.
The deprecated transformers-cli ... command was deprecated, transformers ... is now the only CLI entry point.
transformers CLI has been migrated to Typer, making it easier to maintain + adding some nice features out of
the box (improved --help section, autocompletion).
Biggest breaking change is in transformers chat. This command starts a terminal UI to interact with a chat model.
It used to also be able to start a Chat Completion server powered by transformers and chat with it. In this revamped
version, this feature has been removed in favor of transformers serve. The goal of splitting transformers chat
and transformers serve is to define clear boundaries between client and server code. It helps with maintenance
but also makes the commands less bloated. The new signature of transformers chat is:
Usage: transformers chat [OPTIONS] BASE_URL MODEL_ID [GENERATE_FLAGS]...
Chat with a model from the command line.
It works hand in hand with transformers serve, which means that if transformers serve is running on its default endpoint, transformers chat can be launched as follows:
transformers chat HuggingFaceTB/SmolLM3-3B
It can however use any OpenAI API compatible HTTP endpoint:
transformers chat HuggingFaceTB/SmolLM3-3B https://router.huggingface.co/v1
Linked PRs:
run methodThe transformers run (previously transformers-cli run) is an artefact of the past, was not documented nor tested,
and isn't part of any public documentation. We're removing it for now and ask you to please let us know in case
this is a method you are using; in which case we should bring it back with better support.
Linked PR: https://github.com/huggingface/transformers/pull/42447
TRANSFORMERS_CACHE, PYTORCH_TRANSFORMERS_CACHE, and PYTORCH_PRETRAINED_BERT_CACHE have been removed. Please use HF_HOME instead.HUGGINGFACE_CO_EXAMPLES_TELEMETRY, HUGGINGFACE_CO_EXAMPLES_TELEMETRY, HUGGINGFACE_CO_PREFIX, and HUGGINGFACE_CO_RESOLVE_ENDPOINT have been removed. Please use huggingface_hub.constants.ENDPOINT instead.Linked PR: https://github.com/huggingface/transformers/pull/42391.
transformers v5 pins the huggingface_hub version to >=1.0.0. See this migration guide to learn more about this major release. Here are to main aspects to know about:
requests to httpx. This change was made to improve performance and to support both synchronous and asynchronous requests the same way. If you are currently catching requests.HTTPError errors in your codebase, you'll need to switch to httpx.HTTPError.HTTP_PROXY / HTTPS_PROXY environment variableshf_transfer and therefore HF_HUB_ENABLE_HF_TRANSFER have been completed dropped in favor of hf_xet. This should be transparent for most users. Please let us know if you notice any downside!typer-slim has been added as required dependency, used to implement both hf and transformers CLIs.
<img width="809" height="471" alt="image" src="https://github.com/user-attachments/assets/58bb9c70-d481-48ed-ab8f-6553be7c240f" />
The Code World Model (CWM) model was proposed in CWM: An Open-Weights LLM for Research on Code Generation with World Models by Meta FAIR CodeGen Team. CWM is an LLM for code generation and reasoning about code that has, in particular, been trained to better represent and reason about how code and commands affect the state of a program or system. Specifically, we mid-trained CWM on a large number of observation-action trajectories from Python execution traces and agentic interactions in containerized environments. We post-trained with extensive multi-task RL in verifiable coding, math, and multi-turn software engineering environments.
<img width="1505" height="915" alt="image" src="https://github.com/user-attachments/assets/eec48633-f02b-464a-ae5c-c65473387e53" />
SAM3 (Segment Anything Model 3) was introduced in SAM 3: Segment Anything with Concepts.
The SAM3 addition adds four new architectures:
SAM3 performs Promptable Concept Segmentation (PCS) on images. PCS takes text and/or image exemplars as input (e.g., "yellow school bus"), and predicts instance and semantic masks for every single object matching the concept.
Sam3Tracker and Sam3TrackerVideo perform Promptable Visual Segmentation (PVS) on images. PVS takes interactive visual prompts (points, boxes, masks) or text inputs to segment a specific object instance per prompt. This is the task that SAM 1 and SAM 2 focused on, and SAM 3 improves upon it. Sam3Tracker and Sam3TrackerVideo are updated versions of SAM2 Video that maintain the same API while providing improved performance and capabilities.
SAM3 Video performs Promptable Concept Segmentation (PCS) on videos. PCS takes text as input (e.g., "yellow school bus"), and predicts instance and semantic masks for every single object matching the concept, while preserving object identities across video frames. The model combines a detection module (SAM3) with a tracking module (SAM2-style tracker) to enable robust object tracking across video frames using text prompts.
<img width="1080" height="849" alt="image" src="https://github.com/user-attachments/assets/a9fa1b81-114d-4054-9699-5083ac69d830" />
LFM2-MoE is a Mixture-of-Experts (MoE) variant of LFM2. The LFM2 family is optimized for on-device inference by combining short‑range, input‑aware gated convolutions with grouped‑query attention (GQA) in a layout tuned to maximize quality under strict speed and memory constraints.
LFM2‑MoE keeps this fast backbone and introduces sparse MoE feed‑forward networks to add representational capacity without significantly increasing the active compute path. The first LFM2-MoE release is LFM2-8B-A1B, with 8.3B total parameters and 1.5B active parameters. The model excels in quality (comparable to 3-4B dense models) and speed (faster than other 1.5B class models).
<img width="812" height="366" alt="image" src="https://github.com/user-attachments/assets/21c82c6e-cf0a-4d6c-a707-b9e57663ca85" />
The VideoLLaMA3 model is a major update to VideoLLaMA2 from Alibaba DAMO Academy.
<img width="621" height="475" alt="image" src="https://github.com/user-attachments/assets/c9616758-b3aa-41d0-bd58-695966ba146d" />
Audio Flamingo 3 (AF3) is a fully open large audio–language model designed for robust understanding and reasoning over speech, environmental sounds, and music. AF3 pairs a Whisper-style audio encoder with a causal language model and performs replace-in-place audio–text fusion: the processor aligns post-pool audio frames to a dedicated placeholder token and the model replaces those token slots with projected audio embeddings during the forward pass.
The model checkpoint is available at: nvidia/audio-flamingo-3-hf
Highlights:
NanoChat is a compact decoder-only transformer model designed for educational purposes and efficient training. The model features several fundamental architectural innovations which are common in modern transformer models. Therefore, it is a good model to use as a starting point to understand the principles of modern transformer models. NanoChat is a variant of the Llama architecture, with simplified attention mechanism and normalization layers.
<img width="868" height="331" alt="image" src="https://github.com/user-attachments/assets/cd8b82cf-10de-49b0-af2a-28ffbeac6fa7" />
FastVLM is an open-source vision-language model featuring a novel hybrid vision encoder, FastViTHD. Leveraging reparameterizable convolutional layers, scaled input resolution, and a reduced number of visual tokens, FastVLM delivers high accuracy with exceptional efficiency. Its optimized architecture enables deployment even on edge devices, achieving ultra-low TTFT (time to first token) without sacrificing performance.
<img width="3840" height="2160" alt="image" src="https://github.com/user-attachments/assets/712fbe57-a4b6-4bf1-acf6-3ef803f75f0e" />
PaddleOCR-VL is a SOTA and resource-efficient model tailored for document parsing. Its core component is PaddleOCR-VL-0.9B, a compact yet powerful vision-language model (VLM) that integrates a NaViT-style dynamic resolution visual encoder with the ERNIE-4.5-0.3B language model to enable accurate element recognition. This innovative model efficiently supports 109 languages and excels in recognizing complex elements (e.g., text, tables, formulas, and charts), while maintaining minimal resource consumption. Through comprehensive evaluations on widely used public benchmarks and in-house benchmarks, PaddleOCR-VL achieves SOTA performance in both page-level document parsing and element-level recognition. It significantly outperforms existing solutions, exhibits strong competitiveness against top-tier VLMs, and delivers fast inference speeds. These strengths make it highly suitable for practical deployment in real-world scenarios.
<img width="719" height="541" alt="image" src="https://github.com/user-attachments/assets/2fc5ee26-3c15-451f-bc7f-c779c2a78919" />
PE Audio (Perception Encoder Audio) is a state-of-the-art multimodal model that embeds audio and text into a shared (joint) embedding space. The model enables cross-modal retrieval and understanding between audio and text.
Text input
Audio input
The resulting embeddings can be used for:
<img width="2100" height="1154" alt="image" src="https://github.com/user-attachments/assets/a9343e81-903b-4445-ba0c-61c87830776a" />
Jais2 a next-generation Arabic open-weight LLM trained on the richest Arabic-first dataset to date. Built from the ground up with 8B and 70B parameters, Jais 2 understands Arabic the way it's truly spoken across dialects, cuulutre, and modern expression. It is developed by MBZUAI, Inception and Cerebras Systems and based on the transformer architecture with modifications including:
<img width="5478" height="2102" alt="image" src="https://github.com/user-attachments/assets/abbd93d4-8c4c-4fd9-8fab-58d969d8b296" />
Pixio is a vision foundation model that uses ViT as a feature extractor for multiple downstream tasks like depth estimation, semantic segmentation, feed-forward 3D reconstruction, robotics, and image classification. It is built on the Masked Autoencoder (MAE) pre-training framework, with four minimal yet critical updates: 1) deeper decoder, 2) larger masking granularity, 3) more class tokens, and 4) web-scale curated training data.
<img width="848" height="853" alt="image" src="https://github.com/user-attachments/assets/02817ead-7560-4eea-8f2b-0b959553a3cd" />
The Ernie 4.5 VL MoE model was released in the Ernie 4.5 Model Family release by baidu. This family of models contains multiple different architectures and model sizes. The Vision-Language series in specific is composed of a novel multimodal heterogeneous structure, sharing paremeters across modalities and dedicating parameters to specific modalities. This becomes especially apparent in the Mixture of Expert (MoE) which is composed of
This architecture has the advantage to enhance multimodal understanding without compromising, and even improving, performance on text-related tasks. An more detailed breakdown is given in the Technical Report.
Ernie 4.5] Ernie VL models by @vasqu in https://github.com/huggingface/transformers/pull/39585<img width="1600" height="1029" alt="image" src="https://github.com/user-attachments/assets/d630a900-9ef5-467c-93ab-34f064501b8d" />
GLM-ASR-Nano-2512 is a robust, open-source speech recognition model with 1.5B parameters. Designed for real-world complexity, it outperforms OpenAI Whisper V3 on multiple benchmarks while maintaining a compact size.
Key capabilities include:
Exceptional Dialect Support Beyond standard Mandarin and English, the model is highly optimized for Cantonese (粤语) and other dialects, effectively bridging the gap in dialectal speech recognition.
Low-Volume Speech Robustness Specifically trained for "Whisper/Quiet Speech" scenarios. It captures and accurately transcribes extremely low-volume audio that traditional models often miss.
SOTA Performance Achieves the lowest average error rate (4.10) among comparable open-source models, showing significant advantages in Chinese benchmarks (Wenet Meeting, Aishell-1, etc..).
This model was contributed by Eustache Le Bihan and Yuxuan Zhang. you can check the model card for more details and our github repo.
<img width="5038" height="5860" alt="image" src="https://github.com/user-attachments/assets/af0f2821-3831-4afd-a7dc-26a91f17afd0" />
GLM-4.7-Flash offers a new option for lightweight deployment that balances performance and efficiency.
<img width="10101" height="7371" alt="image" src="https://github.com/user-attachments/assets/e2de76ab-35a9-4e42-9252-6b3dbf75e991" />
We present GLM-4.1V-Thinking, GLM-4.5V, and GLM-4.6V, a family of vision-language models (VLMs) designed to advance general-purpose multimodal understanding and reasoning. In this report, we share our key findings in the development of the reasoning-centric training framework. We first develop a capable vision foundation model with significant potential through large-scale pre-training, which arguably sets the upper bound for the final performance. We then propose Reinforcement Learning with Curriculum Sampling (RLCS) to unlock the full potential of the model, leading to comprehensive capability enhancement across a diverse range of tasks, including STEM problem solving, video understanding, content recognition, coding, grounding, GUI-based agents, and long document interpretation. In a comprehensive evaluation across 42 public benchmarks, GLM-4.5V achieves state-of-the-art performance on nearly all tasks among open-source models of similar size, and demonstrates competitive or even superior results compared to closed-source models such as Gemini-2.5-Flash on challenging tasks including Coding and GUI Agents. Meanwhile, the smaller GLM-4.1V-9B-Thinking remains highly competitive-achieving superior results to the much larger Qwen2.5-VL-72B on 29 benchmarks. We open-source both GLM-4.1V-9B-Thinking and GLM-4.5V. We further introduce the GLM-4.6V series, open-source multimodal models with native tool use and a 128K context window. A brief overview is available at this https URL. Code, models and more information are released at https://github.com/zai-org/GLM-V
<img width="706" height="242" alt="image" src="https://github.com/user-attachments/assets/2c4770f4-5d48-4d87-a454-b574b648e5ae" />
LW-DETR proposes a light-weight Detection Transformer (DETR) architecture designed to compete with and surpass the dominant YOLO series for real-time object detection. It achieves a new state-of-the-art balance between speed (latency) and accuracy (mAP) by combining recent transformer advances with efficient design choices.
The LW-DETR architecture is characterized by its simple and efficient structure: a plain ViT Encoder, a Projector, and a shallow DETR Decoder. It enhances the DETR architecture for efficiency and speed using the following core modifications:
Efficient ViT Encoder: Uses a plain ViT with interleaved window/global attention and a window-major organization to drastically reduce attention complexity and latency.
Richer Input: Aggregates multi-level features from the encoder and uses a C2f Projector (YOLOv8) to pass two-scale features ( 1 / 8 and 1 / 32 ).
Faster Decoder: Employs a shallow 3-layer DETR decoder with deformable cross-attention for lower latency and faster convergence.
Optimized Queries: Uses a mixed-query scheme combining learnable content queries and generated spatial queries.
<img width="1172" height="661" alt="image" src="https://github.com/user-attachments/assets/cee98e82-b3d0-42a6-b820-3061752ad4a8" />
LightOnOcr combines a Vision Transformer encoder (Pixtral-based) with a lightweight text decoder (Qwen3-based) distilled from high-quality open VLMs. It is optimized for document parsing tasks, producing accurate, layout-aware text extraction from high-resolution pages.
JetMoe Fix jetmoe after #40132 by @ArthurZucker in #41324gemma3 by @Sai-Suraj-27 in #41354PretrainedConfig to PreTrainedConfig by @Cyrilvallez in #41300ModularChecker] QOL for the modular checker by @ArthurZucker in #41361v5] Remove relative position embeddings (for bert like models) by @vasqu in #41170apply_chat_template by @Samoed in #41355test_longcat_generation_cpu by @ydshieh in #41368CB] Refactors the way we access paged by @ArthurZucker in #41370v5] Sync Bert and Bart eager attention by @vasqu in #41248TypeError exception for invalid type by @Sai-Suraj-27 in #41346update_device_map for GPTQ quantizer by @Sai-Suraj-27 in #41328prune_heads by @gante in #41417JetMoe] Fix KV head repetition and padding free by @vasqu in #41423JetMoeIntegrationTest by @ydshieh in #41377past_key_value in BERT-like models by @zucchini-nlp in #41448utils/tf_ops/ by @gante in #41402Attention Masks] Bidirectional masks for encoder and encoder-decoder models by @vasqu in #41265past_index by @SunMarc in #41384report_to default changed to "none" + cleaning deprecated env var by @SunMarc in #41375overwrite_output_dir by @SunMarc in #41323CI] Fix copies on main by @vasqu in #41486jit_mode_eval by @SunMarc in #41376local_rank arg from TrainingArguments by @SunMarc in #41382pickle - BloomTokenizerFast by @ydshieh in #41466glm4v by @Sai-Suraj-27 in #41483truncation to False in Qwen3Omni to avoid default truncation by @BakerBunker in #41473local_rank deletion and some cleaning by @SunMarc in #41504tpu_num_cores by @SunMarc in #41383HunYuanMoEV1IntegrationTest:test_model_generation by @ydshieh in #41373generate delegates default cache initialization to the model by @gante in #41505from_pretrained] Small refactor from_pretrained: move around unrelated stuff by @ArthurZucker in #41445transformers serve by @LysandreJik in #41446logits_to_keep to many older CausalLM models by @philiproeleveld in #41335torch.compile recompiled part of th… by @sywangyi in #41558Docs] Fix changed references by @vasqu in #41614expand_device_map instead of redefining it by @Cyrilvallez in #41608tp_plan in from_pretrained directly by @Cyrilvallez in #41435Executorch] Simplify for encoder models by @vasqu in #41627Ernie 4.5 Moe] Fix Moe and offloading by @vasqu in #41385Masks] Fix mask handling in eager for vision models by @vasqu in #41625utils/check_bad_commit.py by @ydshieh in #41658use_cache default to False by @SunMarc in #41585chat_extras.md to Korean by @Judy-Choi in #39863big_bird.md to Korean by @ssum21 in #40445code_llama.md to Korean by @Judy-Choi in #40558ko-LFM2.md to Korean by @ssum21 in #41502use_auth_token parameter by @Wauplin in #41666Attn] Allow dynamic causality in SDPA via Kwargs by @vasqu in #41692run_name docs in TrainingArguments by @tobiasofsn in #41705utils/check_bad_commit.py by @ydshieh in #41658)videos from image processing classes by @zucchini-nlp in #41607@staticmethod from module-level get_device_and_memory_breakdown by @albertvillanova in #41747Onnx docs] Remove some traces by @vasqu in #41791utils/check_bad_commit.py by @ydshieh in #41658)Clip] Fix masking and enable flash attention on all model types by @vasqu in #41750test_tensor_parallel.py by @3outeille in #41918detectron2 installation in docker files by @ydshieh in #41975autoawq[kernels] installation in quantization docker file by @ydshieh in #41978torchcodec version in quantization docker file by @ydshieh in #41988run slow v2: empty report when there is only one model by @ydshieh in #42002torch+deepspeed docker file by @ydshieh in #41985logging_dir by @SunMarc in #42013deeepspeed in AMD docker file by @ydshieh in #42025huggingface_hub dependency version by @hanouticelina in #42033pr_slow_ci_suggestion.yml after #42023 by @ydshieh in #42049Argument list too long in pr_slow_ci_suggestion.yml by @ydshieh in #42061setattr as well by @zucchini-nlp in #41808Attn Masks] Non-vmap default for attention masks by @vasqu in #41852image_transforms.py by @yaswanth19 in #42044prepare_inputs_for_generation cache slicing condition by @aNote truncated.
We are getting closer and closer to the official release! This RC is focused on removing more of the deprecated stuff, fixing some minors issues, doc…
We are getting closer and closer to the official release! This RC is focused on removing more of the deprecated stuff, fixing some minors issues, doc updates.
_get_num_multimodal_tokens by @Abhinavexists in https://github.com/huggingface/transformers/pull/43137BartModelIntegrationTest by @Sai-Suraj-27 in https://github.com/huggingface/transformers/pull/43160auto_doctring in Processors by @yonigozlan in https://github.com/huggingface/transformers/pull/42101BitModelIntegrationTest by @Sai-Suraj-27 in https://github.com/huggingface/transformers/pull/43164Fp8] Fix experts by @vasqu in https://github.com/huggingface/transformers/pull/43154salesforce-ctrl, xlm & gpt-neo model generation tests by @Sai-Suraj-27 in https://github.com/huggingface/transformers/pull/43180Generate] Allow custom config values in generate config by @vasqu in https://github.com/huggingface/transformers/pull/43181Pix2StructIntegrationTest by @Sai-Suraj-27 in https://github.com/huggingface/transformers/pull/43229PhiIntegrationTests by @Sai-Suraj-27 in https://github.com/huggingface/transformers/pull/43214HF_TOKEN directly and remove require_read_token by @ydshieh in https://github.com/huggingface/transformers/pull/43233Owlv2ModelIntegrationTest & OwlViTModelIntegrationTest by @Sai-Suraj-27 in https://github.com/huggingface/transformers/pull/43182add_dates by @yonigozlan in https://github.com/huggingface/transformers/pull/43199Vip-llava model integration test by @Sai-Suraj-27 in https://github.com/huggingface/transformers/pull/43252position_ids in all apply_rotary_pos_emb by @Cyrilvallez in https://github.com/huggingface/transformers/pull/43255_get_test_info in testing_utils.py by @ydshieh in https://github.com/huggingface/transformers/pull/43259Hiera, SwiftFormer & LED Model integration tests by @Sai-Suraj-27 in https://github.com/huggingface/transformers/pull/43225_toctree.yml by @Cyrilvallez in https://github.com/huggingface/transformers/pull/43264PegasusX, Mvp & LED model integration tests by @Sai-Suraj-27 in https://github.com/huggingface/transformers/pull/43245Full Changelog: https://github.com/huggingface/transformers/compare/v5.0.0rc2...v5.0.0rc3
This release candidate is focused on fixing AutoTokenizer, expanding the dynamic weight loading support, and improving performances with MoEs!
This release candidate is focused on fixing AutoTokenizer, expanding the dynamic weight loading support, and improving performances with MoEs!
<img width="2048" height="1451" alt="image" src="https://github.com/user-attachments/assets/3ed2508e-3eb1-4f13-8717-cd9027d12a39" />
The main issue with the tokenization refactor is that tokenizer_class are now "enforced" when in most cases they are wrong. This took a while to properly isolate and now we try to use TokenizersBackend whenever we can. #42894 has a much more detailed description of the big changes!
TokenizersBackend by @ArthurZucker in https://github.com/huggingface/transformers/pull/42894Tokenizers] Change treatment of special tokens by @vasqu in https://github.com/huggingface/transformers/pull/42903Here we focused on boosting the performances of loading weights on device!
post_init and fix all of them by @Cyrilvallez in https://github.com/huggingface/transformers/pull/42873_init_weights for ALL models by @Cyrilvallez in https://github.com/huggingface/transformers/pull/42309Ernie 4.5] Ernie VL models by @vasqu in https://github.com/huggingface/transformers/pull/39585Mostly around processors!
convert_segmentation_map_to_binary_masks to EoMT by @simonreise in https://github.com/huggingface/transformers/pull/43073Thanks again to everyone !
Full Changelog: https://github.com/huggingface/transformers/compare/v5.0.0rc1...v5.0.0rc2
This release candidate was focused mostly on quantization support with the new dynamic weight loader, and a few notable 🚨 breaking changes🚨:
This release candidate was focused mostly on quantization support with the new dynamic weight loader, and a few notable 🚨 breaking changes🚨:
from_pretrained is now auto!Mostly QOL and fixed + support back CPU offloading.
Mostly added support for fbgemme , quanto,
The dynamic weight loader broke small things, this adds glue for all models but MoEs.
Tokenization needed more refactoring, this time its a lot cleaner!
rope_parameters to empty dict if there is something to put in it by @hmellor in https://github.com/huggingface/transformers/pull/42651We omitted a lot of other commits for clarity, but thanks to everyone and the new contributors!
Full Changelog: https://github.com/huggingface/transformers/compare/v5.0.0rc0...v5.0.0rc1
Biggest breaking change is in transformers chat. This command starts a terminal UI to interact with a chat model. It used to also be able to start a C…
<img width="1800" height="1013" alt="image" src="https://github.com/user-attachments/assets/7b5187d7-6945-4108-a546-6d1d7bfb55e3" />
We are excited to announce the initial release of Transformers v5. This is the first major release in five years, and the release is significant: 800 commits have been pushed to main since the latest minor release. This release removes a lot of long-due deprecations, introduces several refactors that significantly simplify our APIs and internals, and comes with a large number of bug fixes.
We give an overview of our focus for this release in the following blogpost. In these release notes, we'll focus directly on the refactors and new APIs coming with v5.
This release is a release candidate (RC). It is not the final v5 release, and we will push on pypi as a pre-release. This means that the current release is purely opt-in, as installing transformers without specifying this exact release will install the latest version instead (v4.57.3 as of writing).
In order to install this release, please do so with the following:
pip install transformers --pre
For us to deliver the best package possible, it is imperative that we have feedback on how the toolkit is currently working for you. Please try it out, and open an issue in case you're facing something inconsistent/a bug.
Transformers version 5 is a community endeavor, and this is the last mile. Let's ship this together!
[!NOTE] 👀 Nothing is final and things are still actively in movement. We have a section dedicated to what is planned for future release candidates, yet is known not to work in the RC0. Look for "Disclaimers for the RC0".
We'll be eagerly awaiting your feedback in our GitHub issues!
We introduce a new weight loading API in transformers, which significantly improves on the previous API. This
weight loading API is designed to apply operations to the checkpoints loaded by transformers.
Instead of loading the checkpoint exactly as it is serialized within the model, these operations can reshape, merge, and split the layers according to how they're defined in this new API. These operations are often a necessity when working with quantization or parallelism algorithms.
This new API is centered around the new WeightConverter class:
class WeightConverter(WeightTransform):
operations: list[ConversionOps]
source_keys: Union[str, list[str]]
target_keys: Union[str, list[str]]
The weight converter is designed to apply a list of operations on the source keys, resulting in target keys. A common operation done on the attention layers is to fuse the query, key, values layers. Doing so with this API would amount to defining the following conversion:
conversion = WeightConverter(
["self_attn.q_proj", "self_attn.k_proj", "self_attn.v_proj"], # The input layers
"self_attn.qkv_proj", # The single layer as output
operations=[Concatenate(dim=0)],
)
In this situation, we apply the Concatenate operation, which accepts a list of layers as input and returns a single
layer.
This allows us to define a mapping from architecture to a list of weight conversions. Applying those weight conversions
can apply arbitrary transformations to the layers themselves. This significantly simplified the from_pretrained method
and helped us remove a lot of technical debt that we accumulated over the past few years.
This results in several improvements:
While this is being implemented, expect varying levels of support across different release candidates.
Linked PR: https://github.com/huggingface/transformers/pull/41580
Just as we moved towards a single backend library for model definition, we want our tokenizers, and the Tokenizer object to be a lot more intuitive. With v5, tokenizer definition is much simpler; one can now initialize an empty LlamaTokenizer and train it directly on your corpus.
Defining a new tokenizer object should be as simple as this:
from transformers import TokenizersBackend, generate_merges
from tokenizers import pre_tokenizers, Tokenizer
from tokenizers.model import BPE
class Llama5Tokenizer(TokenizersBackend):
def __init__(self, unk_token="<unk>",bos_token="<s>", eos_token="</s>", vocab=None, merges=None ):
if vocab is None:
self._vocab = {
str(unk_token): 0,
str(bos_token): 1,
str(eos_token): 2,
}
else:
self._vocab = vocab
if merges is not None:
self._merges = merges
else:
self._merges = generate_merges(filtered_vocab)
self._tokenizer = Tokenizer(
BPE(vocab=self._vocab, merges=self._merges, fuse_unk=True)
)
self._tokenizer.pre_tokenizer = pre_tokenizers.Metaspace(
replacement="▁", prepend_scheme=_get_prepend_scheme(self.add_prefix_space, self), split=False
)
super().__init__(
tokenizer_object=self._tokenizer,
unk_token=unk_token,
bos_token=bos_token,
eos_token=eos_token,
)
Once the tokenizer is defined as above, you can load it with the following: Llama5Tokenizer(). Doing this returns you an empty, trainable tokenizer that follows the definition of the authors of Llama5 (it does not exist yet :wink:).
The above is the main motivation towards refactoring tokenization: we want tokenizers to behave similarly to models: trained or empty, and with exactly what is defined in their class definition.
Up to now, transformers maintained two parallel implementations for many tokenizers:
tokenization_<model>.py) - Python-based implementations, often using SentencePiece as the backend.tokenization_<model>_fast.py) - Rust-based implementations using the 🤗 tokenizers library.In v5, we consolidate to a single tokenizer file per model: tokenization_<model>.py. This file will use the most appropriate backend available:
sentencepiece library. It inherits from PythonBackend.tokenizers. Basically allows adding tokens.MistralCommon's tokenization library. (Previously known as the MistralCommonTokenizer)The AutoTokenizer automatically selects the appropriate backend based on available files and dependencies. This is transparent, you continue to use AutoTokenizer.from_pretrained() as before. This allows transformers to be future-proof and modular to easily support future backends.
We enable users and tokenizer builders to define their own tokenizers from top to bottom. Tokenizers are usually defined using a backend such as tokenizers, sentencepiece or mistral-common, but we offer the possibility to design the tokenizer at a higher-level, without relying on those backends.
To do so, you can import the PythonBackend (which was previously known as PreTrainedTokenizer). This class encapsulates all the logic related to added tokens, encoding, and decoding.
If you want something even higher up the stack, then PreTrainedTokenizerBase is what PythonBackend inherits from. It contains the very basic tokenizer API features:
encodedecodevocab_sizeget_vocabconvert_tokens_to_idsconvert_ids_to_tokensfrom_pretrainedsave_pretrainedStarting with v5, we now enable initializing blank, untrained tokenizers-backed tokenizers:
from transformers import LlamaTokenizer
tokenizer = LlamaTokenizer()
This tokenizer will therefore follow the definition of the LlamaTokenizer as defined in its class definition. It can then be trained on a corpus as can be seen in the tokenizers documentation.
These tokenizers can also be initialized from vocab and merges (if necessary), like the previous "slow" tokenizers:
from transformers import LlamaTokenizer
vocab = {"<unk>": 0, "<s>": 1, "</s>": 2, "hello": 3, "world": 4}
merges = [("h", "e"), ("l", "l"), ("o", " ")]
tokenizer = LlamaTokenizer(vocab=vocab, merges=merges)
This tokenizer will behave as a Llama-like tokenizer, with an updated vocabulary. This allows comparing different tokenizer classes with the same vocab; therefore enabling the comparison of different pre-tokenizers, normalizers, etc.
⚠️ The vocab_file (as in, a path towards a file containing the vocabulary) cannot be used to initialize the LlamaTokenizer as loading from files is reserved to the from_pretrained method.
The batch_decode and decode methods have been unified to reflect behavior of the encode method. Both single and batch decoding now use the same decode method. See an example of the new behavior below:
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("t5-small")
inputs = ["hey how are you?", "fine"]
tokenizer.decode(tokenizer.encode(inputs))
Gives:
- 'hey how are you?</s> fine</s>'
+ ['hey how are you?</s>', 'fine</s>']
We expect encode and decode to behave, as two sides of the same coin: encode, process, decode, should work.
[!NOTE] A common use-case would be:
encode,model.generate,decode. However, usinggeneratewould returnlist[list[int]], which would then be incompatible withdecode.
The encode_plus method is deprecated in favor of the single __call__ method.
apply_chat_template returns BatchEncodingPreviously, apply_chat_template returned input_ids for backward compatibility. Starting with v5, it now consistently returns a BatchEncoding dict like other tokenizer methods.
# v5
messages = [
{"role": "user", "content": "Hello!"},
{"role": "assistant", "content": "Hi there!"}
]
# Now returns BatchEncoding with input_ids, attention_mask, etc.
outputs = tokenizer.apply_chat_template(messages, return_tensors="pt")
print(outputs.keys()) # dict_keys(['input_ids', 'attention_mask'])
We simplify the serialization of tokenization attributes:
special_tokens_map.json - special tokens are now stored in tokenizer_config.json.added_tokens.json - added tokens are now stored in tokenizer.json.added_tokens_decoder is only stored when there is no tokenizer.json.When loading older tokenizers, these files are still read for backward compatibility, but new saves use the consolidated format. We're gradually moving towards consolidating attributes to fewer files so that other libraries and implementations may depend on them more reliably.
Several models that had identical tokenizers now import from their base implementation:
These modules will eventually be removed altogether.
Removed T5-specific workarounds
The internal _eventually_correct_t5_max_length method has been removed. T5 tokenizers now handle max length consistently with other models.
A few testing changes specific to tokenizers have been applied:
add_tokens, encode, decode) are now centralized and automatically applied across all tokenizers. This reduces test duplication and ensures consistent behaviorFor legacy implementations, the original BERT Python tokenizer code (including WhitespaceTokenizer, BasicTokenizer, etc.) is preserved in bert_legacy.py for reference purposes.
Special Tokens Structure:
SpecialTokensMixin: Merged into PreTrainedTokenizerBase to simplify the tokenizer architecture.special_tokens_map: Now only stores named special token attributes (e.g., bos_token, eos_token). Use extra_special_tokens for additional special tokens (formerly additional_special_tokens). all_special_tokens includes both named and extra tokens.# v4
tokenizer.special_tokens_map # Included 'additional_special_tokens'
# v5
tokenizer.special_tokens_map # Only named tokens
tokenizer.extra_special_tokens # Additional tokens
special_tokens_map_extended and all_special_tokens_extended: Removed. Access AddedToken objects directly from _special_tokens_map or _extra_special_tokens if needed.additional_special_tokens: Still accepted for backward compatibility but is automatically converted to extra_special_tokens.Deprecated Methods:
sanitize_special_tokens(): Already deprecated in v4, removed in v5.prepare_seq2seq_batch(): Deprecated; use __call__() with text_target parameter instead.# v4
model_inputs = tokenizer.prepare_seq2seq_batch(src_texts, tgt_texts, max_length=128)
# v5
model_inputs = tokenizer(src_texts, text_target=tgt_texts, max_length=128, return_tensors="pt")
model_inputs["labels"] = model_inputs.pop("input_ids_target")
BatchEncoding.words(): Deprecated; use word_ids() instead.Removed Methods:
create_token_type_ids_from_sequences(): Removed from base class. Subclasses that need custom token type ID creation should implement this method directly.clean_up_tokenization(): Removed from base class. Now defined at model class level for models that need it (e.g., PLBart, CLVP, Wav2Vec2).prepare_for_model(), build_inputs_with_special_tokens(), truncate_sequences(): Moved from tokenization_utils_base.py to tokenization_python.py for PythonBackend tokenizers. TokenizersBackend provides model-ready input via tokenize() and encode(), so these methods are no longer needed in the base class._switch_to_input_mode(), _switch_to_target_mode(), as_target_tokenizer(): Removed from base class. Use __call__() with text_target parameter instead.# v4
with tokenizer.as_target_tokenizer():
labels = tokenizer(tgt_texts, ...)
# v5
labels = tokenizer(text_target=tgt_texts, ...)
parse_response(): Removed from base class.Because we are switching from the naive MOE (nn.ModuleList for experts) we currently have an issue with MoEs that have adapters. For more details see https://github.com/huggingface/transformers/issues/42491#issuecomment-3591485649.
We aim for this to be fixed and released in a following release candidate in the week that follows RC0.
We are streamlining the MoE support with vLLM; while this is being implemented, tensor parallelism and expert parallelism aren't working as expected. This is known and actively being worked on.
We aim for this to be fixed and released in a following release candidate in the week that follows RC0.
For anyone inheriting from a transformers PreTrainedModel, the weights are automatically initialized with the common scheme:
@torch.no_grad()
def _init_weights(self, module):
"""
Initialize the weights. This is quite general on purpose, in the spirit of what we usually do. For more complex
initialization scheme, it should be overridden by the derived `PreTrainedModel` class. In case a model adds an explicit
`nn.Parameter`, this method should also be overridden in order to initialize it correctly.
"""
if hasattr(self.config, "initializer_range"):
std = self.config.initializer_range or 0.02
elif hasattr(self.config, "init_std"):
std = self.config.init_std
elif hasattr(self.config, "initializer_factor"):
std = self.config.initializer_factor
else:
# 0.02 is the standard default value across the library
std = getattr(self.config.get_text_config(), "initializer_range", 0.02)
if isinstance(module, (nn.Linear, nn.Conv1d, nn.Conv2d, nn.Conv3d, nn.ConvTranspose1d, nn.ConvTranspose2d)):
if getattr(module, "weight", None) is not None:
init.normal_(module.weight, mean=0.0, std=std)
if getattr(module, "bias", None) is not None:
init.zeros_(module.bias)
elif isinstance(module, nn.Embedding):
if getattr(module, "weight", None) is not None:
init.normal_(module.weight, mean=0.0, std=std)
# Here we need the check explicitly, as we slice the weight in the `zeros_` call, so it looses the flag
if module.padding_idx is not None and not getattr(module.weight, "_is_hf_initialized", False):
init.zeros_(module.weight[module.padding_idx])
elif isinstance(module, nn.MultiheadAttention):
# This uses torch's original init
module._reset_parameters()
# We cannot use `isinstance` on the RMSNorms or LayerNorms, as they usually are custom modules which change names
# between modelings (because they are prefixed with the model name)
elif (
isinstance(module, (nn.GroupNorm, nn.BatchNorm1d, nn.BatchNorm2d, nn.BatchNorm3d))
or "LayerNorm" in module.__class__.__name__
or "RMSNorm" in module.__class__.__name__
):
# Norms can exist without weights (in which case they are None from torch primitives)
if hasattr(module, "weight") and module.weight is not None:
init.ones_(module.weight)
if hasattr(module, "bias") and module.bias is not None:
init.zeros_(module.bias)
If you want to avoid that, for now you should just do:
class CustomModel(Qwen3VLForConditionalGeneration):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.action_head = nn.Linear(1024, 7)
self.positional_embedding = nn.Parameter(torch.randn(16, 1152))
self.post_init()
def _init_weights(self, module):
pass
There is a tracker for that here: https://github.com/huggingface/transformers/issues/42418.
use_auth_tokenThe use_auth_token argument/parameter is deprecated in favor of token everywhere.
You should be able to search and replace use_auth_token with token and get the same logic.
Linked PR: https://github.com/huggingface/transformers/pull/41666
We decided to remove some features for the upcoming v5 as they are currently only supported in a few old models and no longer integrated in current model additions. It's recommended to stick to v4.x in case you need them. Following features are affected:
We dropped support for two torch APIs:
torchscript in https://github.com/huggingface/transformers/pull/41688torch.fx in https://github.com/huggingface/transformers/pull/41683Those APIs were deprecated by the PyTorch team, and we're instead focusing on the supported APIs dynamo and export.
We clean up the quantization API in transformers, and significantly refactor the weight loading as highlighted above.
We drop support for two quantization arguments that have been deprecated for some time:
load_in_4bitload_in_8bitWe remove them in favor of the quantization_config argument which is much more complete. As an example, here is how
you would load a 4-bit bitsandbytes model using this argument:
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
quantization_config = BitsAndBytesConfig(load_in_4bit=True)
model_4bit = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-3.2-3B",
device_map="auto",
quantization_config=quantization_config
)
from_xxx_config are deleted. Configs can be init from the __init__ method in the same way. See #41314.mode.rope_parameters, including the rope_theta and rope_type. Model's config.rope_parameters is a simple dictionaty in most cases, and can also be a nested dict in special cases (i.e. Gemma3 and ModernBert) with different rope parameterization for each layer type. Trying to get config.rope_theta will throw an attribute error from now on. See #39847 and #42255config.vocab_size). Users are expected to access keys from their respective sub-configs (config.text_config.vocab_size).model.generate()) will no longer have a generation_config and model.config.generation_config will throw an attribute error.tokenization_<model>.py ) will be removed in favor of using fast tokenizer files tokenization_<model>_fast.py --> will be renamed to tokenization_<model>.py. As fast tokenizers are :hugs:tokenizers - backend, they include a wider range of features that are maintainable and reliable.encode_plus --> __call__batch_decode --> decodeapply_chat_template by default returns naked input_ids rather than a BatchEncoding dict.
This was inconvenient - it should return a BatchEncoding dict like tokenizer.__call__(), but we were stuck with
it for backward compatibility. The method now returns a BatchEncoding.
Linked PRs:
processor_config.json as a nested dict, instead of serializing attributes in their own config files. Loading will be supported for all old format processors (https://github.com/huggingface/transformers/pull/41474)XXXFeatureExtractors classes are completely removed in favor of XXXImageProcessor class for all vision models (https://github.com/huggingface/transformers/pull/41174)XXXFastImageProcessorKwargs is removed in favor of XXXImageProcessorKwargs which will be shared between fast and slow processors (https://github.com/huggingface/transformers/pull/40931)RotaryEmbeddings layers will start returning a dict of tuples, in case the model uses several RoPE configurations (Gemma2, ModernBert). Each value will be a tuple of "cos, sin" per RoPE type.RotaryEmbeddings layer will be unified and accessed via config.rope_parameters. Config attr for rope_theta might not be accessible anymore for some models, and instead will be in config.rope_parameters['rope_theta']. BC will be supported for a while as much as possible, and in the near future we'll gradually move to the new RoPE format (https://github.com/huggingface/transformers/pull/39847)model.language_model. It is recommended to either access the module with model.model.language_model or model.get_decoder(). See #42156GreedySearchEncoderDecoderOutput). We now only have 4 output classes built from the following matrix: decoder-only vs encoder-decoder, uses beams vs doesn't use beams (https://github.com/huggingface/transformers/pull/40998)generate doesn't receive any KV Cache argument, the default cache class used is now defined by the model (as opposed to always being DynamicCache) (https://github.com/huggingface/transformers/pull/41505)config.json for any old model, it will be loaded back into model's generation config. Users are expected to access or modify generation parameters only with model.generation_config.do_sample = True.compute_loss_func Handling
compute_loss_func now always takes priority over the model's built-in loss computation, giving users consistent control over custom loss functions.num_items_in_batch in Prediction Step
num_items_in_batch argument is now passed to compute_loss during prediction_step, enabling proper loss scaling during evaluation.report_to now defaults to "none"
TrainingArguments due to low usagemp_parameters -> legacy param that was later on added to the Sagemaker trainer_n_gpu -> not intended for users to set, we will initialize it correctly instead of putting it in the TrainingArgumentsoverwrite_output_dir - > replaced by resume_from_checkpoint, and it was only used in the examples script, no impact on Trainer.logging_dir -> only used for tensorboard, set TENSORBOARD_LOGGING_DIR env var insteadjit_mode_eval -> use use_torch_compile instead, as torchscript is not recommended anymoretpu_num_cores-> It is actually better to remove it, as it is not recommended to set the number of cores. By default, all TPU cores are used . Set TPU_NUM_CORES env var insteadpast_index -> it was only used for a very small number of models that have special architecture like transformersxl + it was not documented at all how to train those modelsray_scope -> only for a minor arg for ray integration. Set RAY_SCOPE var env insteadwarmup_ratio -> use warmup_step instead. We combined both args together by allowing passing float values in warmup_step.TrainingArgumentsfsdp_min_num_params and fsdp_transformer_layer_cls_to_wrap -> use fsdp_configtpu_metrics_debug -> debugpush_to_hub_token -> hub_tokenpush_to_hub_model_id and push_to_hub_organization -> hub_model_idinclude_inputs_for_metrics -> include_for_metricsper_gpu_train_batch_size -> per_device_train_batch_sizeper_gpu_eval_batch_size -> per_device_eval_batch_sizeuse_mps_device -> mps will be used by default if detectedfp16_backend and half_precision_backend -> we will only rely on torch.amp as everything has been upstreamed to torchno_cuda -> use_cpu include_tokens_per_second -> include_num_input_tokens_seenuse_legacy_prediction_loop -> we only use evaluation_loop function from now onTrainertokenizer in initialization -> processing_classmodel_path in train() -> resume_from_checkpointTrainerTraineruse_cache in the model config will be set to False. You can still change the cache value through TrainingArguments usel_cache argument if needed.organization and repo_url from PushToHubMixin. You must pass a repo_id instead.ignore_metadata_errors from PushToMixin. In practice if we ignore errors while loading the model card, we won't be able to push the card back to the Hub so it's better to fail early and not provide the option to fail later.push_to_hub do not accept **kwargs anymore. All accepted parameters are explicitly documented.push_to_hub are now keyword-only to avoid confusion. Only repo_id can be positional since it's the main arg.use_temp_dir argument from push_to_hub. We now use a tmp dir in all cases.Linked PR: https://github.com/huggingface/transformers/pull/42391.
The deprecated transformers-cli ... command was deprecated, transformers ... is now the only CLI entry point.
transformers CLI has been migrated to Typer, making it easier to maintain + adding some nice features out of
the box (improved --help section, autocompletion).
Biggest breaking change is in transformers chat. This command starts a terminal UI to interact with a chat model.
It used to also be able to start a Chat Completion server powered by transformers and chat with it. In this revamped
version, this feature has been removed in favor of transformers serve. The goal of splitting transformers chat
and transformers serve is to define clear boundaries between client and server code. It helps with maintenance
but also makes the commands less bloated. The new signature of transformers chat is:
Usage: transformers chat [OPTIONS] BASE_URL MODEL_ID [GENERATE_FLAGS]...
Chat with a model from the command line.
It works hand in hand with transformers serve, which means that if transformers serve is running on its default endpoint, transformers chat can be launched as follows:
transformers chat HuggingFaceTB/SmolLM3-3B
It can however use any OpenAI API compatible HTTP endpoint:
transformers chat HuggingFaceTB/SmolLM3-3B https://router.huggingface.co/v1
Linked PRs:
run methodThe transformers run (previously transformers-cli run) is an artefact of the past, was not documented nor tested,
and isn't part of any public documentation. We're removing it for now and ask you to please let us know in case
this is a method you are using; in which case we should bring it back with better support.
Linked PR: https://github.com/huggingface/transformers/pull/42447
TRANSFORMERS_CACHE, PYTORCH_TRANSFORMERS_CACHE, and PYTORCH_PRETRAINED_BERT_CACHE have been removed. Please use HF_HOME instead.HUGGINGFACE_CO_EXAMPLES_TELEMETRY, HUGGINGFACE_CO_EXAMPLES_TELEMETRY, HUGGINGFACE_CO_PREFIX, and HUGGINGFACE_CO_RESOLVE_ENDPOINT have been removed. Please use huggingface_hub.constants.ENDPOINT instead.Linked PR: https://github.com/huggingface/transformers/pull/42391.
transformers v5 pins the huggingface_hub version to >=1.0.0. See this migration guide to learn more about this major release. Here are to main aspects to know about:
requests to httpx. This change was made to improve performance and to support both synchronous and asynchronous requests the same way. If you are currently catching requests.HTTPError errors in your codebase, you'll need to switch to httpx.HTTPError.HTTP_PROXY / HTTPS_PROXY environment variableshf_transfer and therefore HF_HUB_ENABLE_HF_TRANSFER have been completed dropped in favor of hf_xet. This should be transparent for most users. Please let us know if you notice any downside!typer-slim has been added as required dependency, used to implement both hf and transformers CLIs.
<img width="809" height="471" alt="image" src="https://github.com/user-attachments/assets/58bb9c70-d481-48ed-ab8f-6553be7c240f" />
The Code World Model (CWM) model was proposed in CWM: An Open-Weights LLM for Research on Code Generation with World Models by Meta FAIR CodeGen Team. CWM is an LLM for code generation and reasoning about code that has, in particular, been trained to better represent and reason about how code and commands affect the state of a program or system. Specifically, we mid-trained CWM on a large number of observation-action trajectories from Python execution traces and agentic interactions in containerized environments. We post-trained with extensive multi-task RL in verifiable coding, math, and multi-turn software engineering environments.
<img width="1505" height="915" alt="image" src="https://github.com/user-attachments/assets/eec48633-f02b-464a-ae5c-c65473387e53" />
SAM3 (Segment Anything Model 3) was introduced in SAM 3: Segment Anything with Concepts.
The SAM3 addition adds four new architectures:
SAM3 performs Promptable Concept Segmentation (PCS) on images. PCS takes text and/or image exemplars as input (e.g., "yellow school bus"), and predicts instance and semantic masks for every single object matching the concept.
Sam3Tracker and Sam3TrackerVideo perform Promptable Visual Segmentation (PVS) on images. PVS takes interactive visual prompts (points, boxes, masks) or text inputs to segment a specific object instance per prompt. This is the task that SAM 1 and SAM 2 focused on, and SAM 3 improves upon it. Sam3Tracker and Sam3TrackerVideo are updated versions of SAM2 Video that maintain the same API while providing improved performance and capabilities.
SAM3 Video performs Promptable Concept Segmentation (PCS) on videos. PCS takes text as input (e.g., "yellow school bus"), and predicts instance and semantic masks for every single object matching the concept, while preserving object identities across video frames. The model combines a detection module (SAM3) with a tracking module (SAM2-style tracker) to enable robust object tracking across video frames using text prompts.
<img width="1080" height="849" alt="image" src="https://github.com/user-attachments/assets/a9fa1b81-114d-4054-9699-5083ac69d830" />
LFM2-MoE is a Mixture-of-Experts (MoE) variant of LFM2. The LFM2 family is optimized for on-device inference by combining short‑range, input‑aware gated convolutions with grouped‑query attention (GQA) in a layout tuned to maximize quality under strict speed and memory constraints.
LFM2‑MoE keeps this fast backbone and introduces sparse MoE feed‑forward networks to add representational capacity without significantly increasing the active compute path. The first LFM2-MoE release is LFM2-8B-A1B, with 8.3B total parameters and 1.5B active parameters. The model excels in quality (comparable to 3-4B dense models) and speed (faster than other 1.5B class models).
<img width="812" height="366" alt="image" src="https://github.com/user-attachments/assets/21c82c6e-cf0a-4d6c-a707-b9e57663ca85" />
The VideoLLaMA3 model is a major update to VideoLLaMA2 from Alibaba DAMO Academy.
<img width="621" height="475" alt="image" src="https://github.com/user-attachments/assets/c9616758-b3aa-41d0-bd58-695966ba146d" />
Audio Flamingo 3 (AF3) is a fully open large audio–language model designed for robust understanding and reasoning over speech, environmental sounds, and music. AF3 pairs a Whisper-style audio encoder with a causal language model and performs replace-in-place audio–text fusion: the processor aligns post-pool audio frames to a dedicated placeholder token and the model replaces those token slots with projected audio embeddings during the forward pass.
The model checkpoint is available at: nvidia/audio-flamingo-3-hf
Highlights:
NanoChat is a compact decoder-only transformer model designed for educational purposes and efficient training. The model features several fundamental architectural innovations which are common in modern transformer models. Therefore, it is a good model to use as a starting point to understand the principles of modern transformer models. NanoChat is a variant of the Llama architecture, with simplified attention mechanism and normalization layers.
JetMoe Fix jetmoe after #40132 by @ArthurZucker in #41324gemma3 by @Sai-Suraj-27 in #41354PretrainedConfig to PreTrainedConfig by @Cyrilvallez in #41300ModularChecker] QOL for the modular checker by @ArthurZucker in #41361v5] Remove relative position embeddings (for bert like models) by @vasqu in #41170apply_chat_template by @Samoed in #41355test_longcat_generation_cpu by @ydshieh in #41368CB] Refactors the way we access paged by @ArthurZucker in #41370v5] Sync Bert and Bart eager attention by @vasqu in #41248TypeError exception for invalid type by @Sai-Suraj-27 in #41346update_device_map for GPTQ quantizer by @Sai-Suraj-27 in #41328prune_heads by @gante in #41417JetMoe] Fix KV head repetition and padding free by @vasqu in #41423JetMoeIntegrationTest by @ydshieh in #41377past_key_value in BERT-like models by @zucchini-nlp in #41448utils/tf_ops/ by @gante in #41402Attention Masks] Bidirectional masks for encoder and encoder-decoder models by @vasqu in #41265past_index by @SunMarc in #41384report_to default changed to "none" + cleaning deprecated env var by @SunMarc in #41375overwrite_output_dir by @SunMarc in #41323CI] Fix copies on main by @vasqu in #41486jit_mode_eval by @SunMarc in #41376local_rank arg from TrainingArguments by @SunMarc in #41382pickle - BloomTokenizerFast by @ydshieh in #41466glm4v by @Sai-Suraj-27 in #41483truncation to False in Qwen3Omni to avoid default truncation by @BakerBunker in #41473local_rank deletion and some cleaning by @SunMarc in #41504tpu_num_cores by @SunMarc in #41383HunYuanMoEV1IntegrationTest:test_model_generation by @ydshieh in #41373generate delegates default cache initialization to the model by @gante in #41505from_pretrained] Small refactor from_pretrained: move around unrelated stuff by @ArthurZucker in #41445transformers serve by @LysandreJik in #41446logits_to_keep to many older CausalLM models by @philiproeleveld in #41335torch.compile recompiled part of th… by @sywangyi in #41558Docs] Fix changed references by @vasqu in #41614expand_device_map instead of redefining it by @Cyrilvallez in #41608tp_plan in from_pretrained directly by @Cyrilvallez in #41435Executorch] Simplify for encoder models by @vasqu in #41627Ernie 4.5 Moe] Fix Moe and offloading by @vasqu in #41385Masks] Fix mask handling in eager for vision models by @vasqu in #41625utils/check_bad_commit.py by @ydshieh in #41658use_cache default to False by @SunMarc in #41585chat_extras.md to Korean by @Judy-Choi in #39863big_bird.md to Korean by @ssum21 in #40445code_llama.md to Korean by @Judy-Choi in #40558ko-LFM2.md to Korean by @ssum21 in #41502use_auth_token parameter by @Wauplin in #41666Attn] Allow dynamic causality in SDPA via Kwargs by @vasqu in #41692run_name docs in TrainingArguments by @tobiasofsn in #41705utils/check_bad_commit.py by @ydshieh in #41658)videos from image processing classes by @zucchini-nlp in #41607@staticmethod from module-level get_device_and_memory_breakdown by @albertvillanova in #41747Onnx docs] Remove some traces by @vasqu in #41791utils/check_bad_commit.py by @ydshieh in #41658)Clip] Fix masking and enable flash attention on all model types by @vasqu in #41750test_tensor_parallel.py by @3outeille in #41918detectron2 installation in docker files by @ydshieh in #41975autoawq[kernels] installation in quantization docker file by @ydshieh in #41978torchcodec version in quantization docker file by @ydshieh in #41988run slow v2: empty report when there is only one model by @ydshieh in #42002torch+deepspeed docker file by @ydshieh in #41985logging_dir by @SunMarc in #42013deeepspeed in AMD docker file by @ydshieh in #42025huggingface_hub dependency version by @hanouticelina in #42033pr_slow_ci_suggestion.yml after #42023 by @ydshieh in #42049Argument list too long in pr_slow_ci_suggestion.yml by @ydshieh in #42061setattr as well by @zucchini-nlp in #41808Attn Masks] Non-vmap default for attention masks by @vasqu in #41852image_transforms.py by @yaswanth19 in #42044prepare_inputs_for_generation cache slicing condition by @albertvillanova in #41764T5Gemma] Fix cross attention cache by @vasqu in #41890streaming by @McPatate in #42102pytest<9 for now by @ydshieh in #42162Pop2Piano] Fix cache usage by @vasqu in #42170PEFT] Fix prefix tuning by @vasqu in #41696FqnToConfig by @jcaip in #41894PEFT] Fix the general test for prefix tuning by @vasqu in #42185Pop2Piano] Fix tied weights by @vasqu in #42193BLT] Fix cache usage by @vasqu in #42188test_dynamic_cache_exportability_multiple_run (failing on torch 2.10 nightly) by @ydshieh in #42212AttentionMaskConverter._unmask_unattended for xpu device before by @kaixuanliu in #42230base_model by @zucchini-nlp in #41589batch_size by @ydshieh in #42213batch_size" by @ydshieh in #42258get_decoder() for multimodal and delete redundant code 🔪 by @zucchini-nlp in #42156cwm by @ydshieh in #42261torch.get_autocast_dtype instead of torch.get_autocast_gpu_dtype by @qgallouedec in #42055WhisperFeatureExtractor by @TopCoder2K in #42286CI] Skip EfficientLoFTR test by @vasqu in #42327Attn Masks] Lift bidirectional mask restriction on eager by @vasqu in #42325torch.distributed imports by @Cyrilvallez in #42361Attn Masks] Add skip option for non-packed sequences by @vasqu in #42367Mistral Tokenizers] Fix tokenizer detection by @vasqu in #42389Note truncated.
Another fix for qwen vl models that prevented correctly loading the associated model type - this works together with https://github.com/huggingface/tr
Another fix for qwen vl models that prevented correctly loading the associated model type - this works together with https://github.com/huggingface/transformers/pull/41808 of the previous patch release.
Full Changelog: https://github.com/huggingface/transformers/compare/v4.57.5...v4.57.6
Should not have said last patch :wink: These should be the last remaining fixes that got lost in between patches and the transition to v5.
Should not have said last patch :wink: These should be the last remaining fixes that got lost in between patches and the transition to v5.
Full Changelog: https://github.com/huggingface/transformers/compare/v4.57.4...v4.57.5
Last patch release for v4: We have a few small fixes for remote generation methods (e.g. group beam search), vLLM, and an offline tokenizer fix (if it
Last patch release for v4: We have a few small fixes for remote generation methods (e.g. group beam search), vLLM, and an offline tokenizer fix (if it's already been cached).
Full Changelog: https://github.com/huggingface/transformers/compare/v4.57.3...v4.57.4
There was a hidden bug when loading models with local_files_only=True and a typo related to the recent patch.
There was a hidden bug when loading models with local_files_only=True and a typo related to the recent patch.
The main fix is: https://github.com/huggingface/transformers/commit/b6055550a15a8fab367cf983b743ff68cc58d81a.
We are really sorry that this slipped through, our CIs just did not catch it.
As it affects a lot of users we are gonna yank the previous release
This patch most notably fixes an issue on some Mistral tokenizers. It contains the following commits:
This patch most notably fixes an issue on some Mistral tokenizers. It contains the following commits:
@staticmethod from module-level get_device_and_memory_breakdown (#41747)This patch most notably fixes an issue with an optional dependency (optax), which resulted in parsing errors with poetry. It contains the following fi
This patch most notably fixes an issue with an optional dependency (optax), which resulted in parsing errors with poetry. It contains the following fixes:
[deprecations] Remove generate-related deprecations up to v4.56 by @gante in #40729
<img width="1200" height="511" alt="image" src="https://github.com/user-attachments/assets/3abad6c4-5650-412d-a831-f8a30a5d962e" />
The Qwen3-Next series represents the Qwen team's next-generation foundation models, optimized for extreme context length and large-scale parameter efficiency. The series introduces a suite of architectural innovations designed to maximize performance while minimizing computational cost:
Built on this architecture, they trained and open-sourced Qwen3-Next-80B-A3B — 80B total parameters, only 3B active — achieving extreme sparsity and efficiency.
Despite its ultra-efficiency, it outperforms Qwen3-32B on downstream tasks — while requiring less than 1/10 of the training cost. Moreover, it delivers over 10x higher inference throughput than Qwen3-32B when handling contexts longer than 32K tokens.
For more details, please visit their blog Qwen3-Next (blog post).
<img width="1282" height="392" alt="image" src="https://github.com/user-attachments/assets/9412905b-4083-4994-9000-aa0dbf97eb6f" />
VaultGemma is a text-only decoder model derived from Gemma 2, notably it drops the norms after the Attention and MLP blocks, and uses full attention for all layers instead of alternating between full attention and local sliding attention. VaultGemma is available as a pretrained model with 1B parameters that uses a 1024 token sequence length.
VaultGemma was trained from scratch with sequence-level differential privacy (DP). Its training data includes the same mixture as the Gemma 2 models, consisting of a number of documents of varying lengths. Additionally, it is trained using DP stochastic gradient descent (DP-SGD) and provides a (ε ≤ 2.0, δ ≤ 1.1e-10)-sequence-level DP guarantee, where a sequence consists of 1024 consecutive tokens extracted from heterogeneous data sources. Specifically, the privacy unit of the guarantee is for the sequences after sampling and packing of the mixture.
<img width="3544" height="1886" alt="image" src="https://github.com/user-attachments/assets/5afa70cb-506e-4d56-baa3-30e7522ac653" />
Qwen3-VL is a multimodal vision-language model series, encompassing both dense and MoE variants, as well as Instruct and Thinking versions.
Building upon its predecessors, Qwen3-VL delivers significant improvements in visual understanding while maintaining strong pure text capabilities. Key architectural advancements include: enhanced MRope with interleaved layout for better spatial-temporal modeling, DeepStack integration to effectively leverage multi-level features from the Vision Transformer (ViT), and improved video understanding through text-based time alignment—evolving from T-RoPE to text timestamp alignment for more precise temporal grounding.
These innovations collectively enable Qwen3-VL to achieve superior performance in complex multimodal tasks.
<img width="763" height="468" alt="image" src="https://github.com/user-attachments/assets/289d33e0-6c71-458d-ae07-b7d454ac2adf" />
The LongCatFlash model was proposed in LongCat-Flash Technical Report by the Meituan LongCat Team. LongCat-Flash is a 560B parameter Mixture-of-Experts (MoE) model that activates 18.6B-31.3B parameters dynamically (average ~27B). The model features a shortcut-connected architecture enabling high inference speed (>100 tokens/second) and advanced reasoning capabilities.
The abstract from the paper is the following:
We present LongCat-Flash, a 560 billion parameter Mixture-of-Experts (MoE) language model featuring a dynamic computation mechanism that activates 18.6B-31.3B parameters based on context (average ~27B). The model incorporates a shortcut-connected architecture enabling high inference speed (>100 tokens/second) and demonstrates strong performance across multiple benchmarks including 89.71% accuracy on MMLU and exceptional agentic tool use capabilities.
Tips:
<img width="700" height="414" alt="image" src="https://github.com/user-attachments/assets/7b92ee0f-5f5a-459c-ad4d-e01b5c10202e" />
FlexOlmo is a new class of language models (LMs) that supports (1) distributed training without data sharing, where different model parameters are independently trained on closed datasets, and (2) data-flexible inference, where these parameters along with their associated data can be flexibly included or excluded from model inferences with no further training. FlexOlmo employs a mixture-of-experts (MoE) architecture where each expert is trained independently on closed datasets and later integrated through a new domain-informed routing without any joint training. FlexOlmo is trained on FlexMix, a corpus we curate comprising publicly available datasets alongside seven domain-specific sets, representing realistic approximations of closed sets.
You can find all the original FlexOlmo checkpoints under the FlexOlmo collection.
<img width="2300" height="1400" alt="image" src="https://github.com/user-attachments/assets/ef0605cd-9512-458c-915a-62316e14d90c" />
LFM2-VL first series of vision-language foundation models developed by Liquid AI. These multimodal models are designed for low-latency and device-aware deployment. LFM2-VL extends the LFM2 family of open-weight Liquid Foundation Models (LFMs) into the vision-language space, supporting both text and image inputs with variable resolutions.
LFM2-VL consists of three main components: a language model backbone, a vision encoder, and a multimodal projector. LFM2-VL builds upon the LFM2 backbone, inheriting from either LFM2-1.2B (for LFM2-VL-1.6B) or LFM2-350M (for LFM2-VL-450M). For the vision tower, LFM2-VL uses SigLIP2 NaFlex encoders to convert input images into token sequences. Two variants are implemented:
The encoder processes images at their native resolution up to 512×512 pixels, efficiently handling smaller images without upscaling and supporting non-standard aspect ratios without distortion. Larger images are split into non-overlapping square patches of 512×512 each, preserving detail. In LFM2-VL-1.6B, the model also receives a thumbnail (a small, downscaled version of the original image capturing the overall scene) to enhance global context understanding and alignment. Special tokens mark each patch’s position and indicate the thumbnail’s start. The multimodal connector is a 2-layer MLP connector with pixel unshuffle to reduce image token count.
<img width="1448" height="1062" alt="image" src="https://github.com/user-attachments/assets/af1fbb09-082c-4331-9217-357adb506cbf" />
The BLT model was proposed in Byte Latent Transformer: Patches Scale Better Than Tokens by Artidoro Pagnoni, Ram Pasunuru, Pedro Rodriguez, John Nguyen, Benjamin Muller, Margaret Li1, Chunting Zhou, Lili Yu, Jason Weston, Luke Zettlemoyer, Gargi Ghosh, Mike Lewis, Ari Holtzman†, Srinivasan Iyer. BLT is a byte-level LLM that achieves tokenization-level performance through entropy-based dynamic patching.
The abstract from the paper is the following:
We introduce the Byte Latent Transformer (BLT), a new byte-level LLM architecture that, for the first time, matches tokenization-based LLM performance at scale with significant improvements in inference efficiency and robustness. BLT encodes bytes into dynamically sized patches, which serve as the primary units of computation. Patches are segmented based on the entropy of the next byte, allocating more compute and model capacity where increased data complexity demands it. We present the first flop controlled scaling study of byte-level models up to 8B parameters and 4T training bytes. Our results demonstrate the feasibility of scaling models trained on raw bytes without a fixed vocabulary. Both training and inference efficiency improve due to dynamically selecting long patches when data is predictable, along with qualitative improvements on reasoning and long tail generalization. Overall, for fixed inference costs, BLT shows significantly better scaling than tokenization-based models, by simultaneously growing both patch and model size.
Dual Model Architecture: BLT consists of two separate trained models:
Dynamic Patching: The model uses entropy-based dynamic patching where:
Local Encoder: Processes byte sequences with cross-attention to patch embeddings
Global Transformer: Processes patch-level representations with full attention across patches
Local Decoder: Generates output with cross-attention back to the original byte sequence
Byte-Level Tokenizer: Unlike traditional tokenizers that use learned vocabularies, BLT's tokenizer simply converts text to UTF-8 bytes and maps each byte to a token ID. There is no need for a vocabulary.
<img width="14084" height="7429" alt="image" src="https://github.com/user-attachments/assets/20d46a43-15f2-42bf-9703-9575f5ca4430" />
The Qwen2.5-Omni model is a unified multiple modalities model proposed in Qwen2.5-Omni Technical Report from Qwen team, Alibaba Group.
Qwen2_5OmniForConditionalGeneration] to generate audio and text output. To generate only one output type, use [Qwen2_5OmniThinkerForConditionalGeneration] for text-only and [Qwen2_5OmniTalkersForConditionalGeneration] for audio-only outputs.Qwen2_5OmniForConditionalGeneration] supports only single batch size at the moment.processor.max_pixels. By default the maximum is set to a very arge value and high resolution visuals will not be resized, unless resolution exceeds processor.max_pixels.~ProcessorMixin.apply_chat_template] method to convert chat messages to model inputs.<img width="1431" height="527" alt="image" src="https://github.com/user-attachments/assets/e831f451-9be3-4b5c-a222-b833a50ceb2a" />
Parakeet models, introduced by NVIDIA NeMo, are models that combine a Fast Conformer encoder with connectionist temporal classification (CTC), recurrent neural network transducer (RNNT) or token and duration transducer (TDT) decoder for automatic speech recognition.
Model Architecture
ParakeetEncoder] for the encoder implementation and details).<img width="949" height="537" alt="image" src="https://github.com/user-attachments/assets/5ca4e73d-5aa9-487d-96e1-92d4f2f4739f" />
The EdgeTAM model was proposed in EdgeTAM: On-Device Track Anything Model Chong Zhou, Chenchen Zhu, Yunyang Xiong, Saksham Suri, Fanyi Xiao, Lemeng Wu, Raghuraman Krishnamoorthi, Bo Dai, Chen Change Loy, Vikas Chandra, Bilge Soran.
EdgeTAM is an efficient adaptation of SAM 2 that introduces a 2D Spatial Perceiver architecture to optimize memory attention mechanisms for real-time video segmentation on mobile devices.
More details to come soon :eyes:
We are introducing Continuous Batching (CB) in this release, we consider it a stable feature. The main use case for CB is batched generation, which makes it very efficient in the context of GRPO training or evaluation. Thanks to CB, researchers or model developers are now free to use transformers in these contexts without having to spin up an additional inference engine.
CB currently supports both full attention and sliding window attention: this means that the vast majority of models are supported, like llama, gemma3, gpt-oss.
CB is also integrated with transformers serve, which means that you can deploy transformers as an OpenAI-compatible HTTP server. Here is a small snippet on how to use it:
import datasets
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers.generation import GenerationConfig
model = AutoModelForCausalLM.from_pretrained(
"Qwen/Qwen3-4B-Instruct-2507", dtype=torch.bfloat16, _attn_implementation="sdpa_paged", device_map="auto"
)
model.generation_config.max_new_tokens = 32
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-4B-Instruct-2507", padding_side="left")
dataset = datasets.load_dataset("openai/gsm8k", "socratic", split="test")
tokenized_datasets = dataset.map(lambda x: tokenizer(x["question"]), batched=True)
simple_batch_inputs = [item["input_ids"] for item in tokenized_datasets]
batch_outputs = model.generate_batch(inputs=simple_batch_inputs)
for request in batch_outputs:
print(tokenizer.decode(batch_outputs[request].generated_tokens))
"""
Let's break down the problem step by step:
1. **Total eggs laid per day**:
Janet’s ducks lay **16 eggs per day**
Let's break down the problem step by step:
1. **Blue fiber**: The robe takes **2 bolts** of blue fiber.
2. **White fiber
To determine Josh's profit from flipping the house, let's go step by step.
---
### Step 1: Initial cost of the house
Josh buys the
To find the total distance James runs in a week, we can break down the problem step by step:
1. **Sprints per session**: James runs
To determine how many cups of feed Wendi needs to give her chickens in the final meal of the day, let's go step by step.
"""
check_model_inputs in core VLMs by @zucchini-nlp in #40342input_feature length and attention_mask length in WhisperFeatureExtractor by @BakerBunker in #39221_prepare_generation_config by @manueldeprada in #40715center_crop fast equivalent to slow by @yonigozlan in #40856pytest-rerunfailures<16.0 by @ydshieh in #40561test_all_params_have_gradient=False for DeepseekV2ModelTest by @ydshieh in #40566test_eager_matches_sdpa_inference not run for CLIP by @ydshieh in #40581remi-or to run-slow by @ydshieh in #40590get_*_features methods + update doc snippets by @qubvel in #40555TvpImageProcessingTest::test_slow_fast_equivalence by @ydshieh in #40593siglip flaky test_eager_matches_sdpa_inference by @ydshieh in #40584Tests] Fixup duplicated mrope logic by @vasqu in #40592TokenizerTesterMixin temporarily by @ydshieh in #40611transformers serve by @McPatate in #40479too many request caused by AutoModelTest::test_dynamic_saving_from_local_repo by @ydshieh in #40614JambaModelTest.test_load_balancing_loss by @ydshieh in #40617deepseek_v3.md to Korean by @ssum21 in #39649too many requests in TestMistralCommonTokenizer by @ydshieh in #40623test_prompt_lookup_decoding_matches_greedy_search for voxtral by @ydshieh in #40643LongformerModelTest::test_attention_outputs as flaky by @ydshieh in #40655custom_generate Callables and unify generation args structure by @manueldeprada in #40586check_determinism inside test_determinism by @ydshieh in #40661test_fast_is_faster_than_slow for Owlv2ImageProcessingTest by @ydshieh in #40663test_prompt_lookup_decoding_matches_greedy_search for qwen2_audio by @ydshieh in #40664GitModelTest::test_beam_search_generate by @ydshieh in #40666tolist instead of list comprehension calling .item() by @McPatate in #40646Aimv2ModelTest::test_eager_matches_sdpa_inference_04_fp16_pad_right_sdpa_kernels as flaky by @ydshieh in #40683T5GemmaModelTest::test_eager_matches_sdpa_inference being flaky by @ydshieh in #40702hf_hub_download by @ydshieh in #40710self in post-process methods by @framonmar7 in #40711or for grounding dino mask by @lmarshall12 in #40625Gemma Embedding] Fix SWA by @vasqu in #40700VitMatteImageProcessingTest::test_fast_is_faster_than_slow by @ydshieh in #40713request_id to headers by @McPatate in #40722and/or_mask_function by @Cyrilvallez in #40753--continuous_batching by @McPatate in #40618continue_final_message in apply_chat_template to prevent substring matching issues by @abdokaseb in #40732public.cloud.experiment_url api error by @Zeyi-Lin in #40763PromptLookupCandidateGenerator won't generate forbidden tokens by @gante in #40726test_past_key_values_format and delete overwrites by @gante in #40701generate by @gante in #40375Jetmoe] Fix RoPE by @vasqu in #40819self.loss_function by @qubvel in #40764test_modeling_common.py by @gante in #40854past_key_values by @gante in #40803rsqrt by @thalahors in #40848VaultGemma] Update expectations in integration tests by @vasqu in #40855imageprocessor.md to Korean by @HyunZ118 in #39557Gemma3nAudioFeatureExtractionTest::test_dither by @ydshieh in #40902get_mask_sizes by @Cyrilvallez in #40907Glm4vIntegrationTest by @ydshieh in #40905runner_map by @ydshieh in #40880test_fast_is_faster_than_slow by @ydshieh in #40909Gemma3ForConditionalGeneration compatible with assisted generation by @gante in #40791image_sizes arg and deprecate vision_feature_layer by @yaswanth19 in #40832import torch.utils.checkpoint by @gante in #40934Glm4vMoeIntegrationTest by @ydshieh in #40930Glm4vModelTest::test_eager_matches_fa2_generate by @ydshieh in #40947test_speculative_generation by @ydshieh in #40949The following contributors have made significant changes to the library over the last release:
imageprocessor.md to Korean (#39557)Processor load with multi-processing
This patch most notably fixes an issue with the new dtype argument (replacing torch_dtype) in pipelines!
This patch most notably fixes an issue with the new dtype argument (replacing torch_dtype) in pipelines!
The following commits are breaking changes in workflows that were either buggy or not working as expected.
DINOv3 is a family of versatile vision foundation models that outperforms the specialized state of the art across a broad range of settings, without fine-tuning. DINOv3 produces high-quality dense features that achieve outstanding performance on various vision tasks, significantly surpassing previous self- and weakly-supervised foundation models.
You can find all the original DINOv3 checkpoints under the DINOv3 collection.
<img width="814" height="658" alt="image" src="https://github.com/user-attachments/assets/740a5c3d-a5a1-45d9-9e4c-d9117837205d" />
he X-Codec model was proposed in Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Model by Zhen Ye, Peiwen Sun, Jiahe Lei, Hongzhan Lin, Xu Tan, Zheqi Dai, Qiuqiang Kong, Jianyi Chen, Jiahao Pan, Qifeng Liu, Yike Guo, Wei Xue
The X-Codec model is a neural audio codec that integrates semantic information from self-supervised models (e.g., HuBERT) alongside traditional acoustic information. This enables :
<img width="1958" height="949" alt="image" src="https://github.com/user-attachments/assets/e36552d0-6465-4921-8208-3f7d3c9087f1" />
The Ovis2 is an updated version of the Ovis model developed by the AIDC-AI team at Alibaba International Digital Commerce Group.
Ovis2 is the latest advancement in multi-modal large language models (MLLMs), succeeding Ovis1.6. It retains the architectural design of the Ovis series, which focuses on aligning visual and textual embeddings, and introduces major improvements in data curation and training methods.
<img src="https://cdn-uploads.huggingface.co/production/uploads/637aebed7ce76c3b834cea37/XB-vgzDL6FshrSNGyZvzc.png" width="600">
MetaCLIP 2 is a replication of the original CLIP model trained on 300+ languages. It achieves state-of-the-art (SOTA) results on multilingual benchmarks (e.g., XM3600, CVQA, Babel‑ImageNet), surpassing previous SOTA such as mSigLIP and SigLIP‑2. The authors show that English and non-English worlds can mutually benefit and elevate each other.
<img width="805" height="408" alt="image" src="https://github.com/user-attachments/assets/72eaa441-9362-4a6a-a834-f505d6727a2a" />
Florence-2 is an advanced vision foundation model that uses a prompt-based approach to handle a wide range of vision and vision-language tasks. Florence-2 can interpret simple text prompts to perform tasks like captioning, object detection, and segmentation. It leverages the FLD-5B dataset, containing 5.4 billion annotations across 126 million images, to master multi-task learning. The model's sequence-to-sequence architecture enables it to excel in both zero-shot and fine-tuned settings, proving to be a competitive vision foundation model.
<img width="864" height="565" alt="image" src="https://github.com/user-attachments/assets/d09dfe3a-6dda-45a3-8dd3-0254d8503b4e" />
SAM2 (Segment Anything Model 2) was proposed in Segment Anything in Images and Videos by Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollár, Christoph Feichtenhofer.
The model can be used to predict segmentation masks of any object of interest given an input image or video, and input points or bounding boxes.
<img width="960" height="540" alt="image" src="https://github.com/user-attachments/assets/0ab42e5c-6951-4cbc-9d5d-ff8bf0c2dbf1" />
The Kosmos-2.5 model was proposed in KOSMOS-2.5: A Multimodal Literate Model by Microsoft.
The abstract from the paper is the following:
We present Kosmos-2.5, a multimodal literate model for machine reading of text-intensive images. Pre-trained on large-scale text-intensive images, Kosmos-2.5 excels in two distinct yet cooperative transcription tasks: (1) generating spatially-aware text blocks, where each block of text is assigned its spatial coordinates within the image, and (2) producing structured text output that captures styles and structures into the markdown format. This unified multimodal literate capability is achieved through a shared Transformer architecture, task-specific prompts, and flexible text representations. We evaluate Kosmos-2.5 on end-to-end document-level text recognition and image-to-markdown text generation. Furthermore, the model can be readily adapted for any text-intensive image understanding task with different prompts through supervised fine-tuning, making it a general-purpose tool for real-world applications involving text-rich images. This work also paves the way for the future scaling of multimodal large language models.
<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/kosmos2_5_ocr.png" alt="drawing" width="600"/>
<img width="674" height="402" alt="image" src="https://github.com/user-attachments/assets/230f83f0-870c-4b31-b8b7-738116761457" />
More information at release 🤗
<img width="858" height="537" alt="image" src="https://github.com/user-attachments/assets/29ccc3c2-9b85-4d89-935a-1e1c28d173fd" />
More information at release 🤗
More information at release 🤗
Beyond a large refactor of the caching system in Transformers, making it much more practical and general, models using sliding window attention/chunk attention do not waste memory anymore when caching past states. It was allowed most notable by:
See the following improvements on memory usage for Mistral (using only sliding layers) and GPT-OSS (1 out of 2 layers is sliding) respectively: <img width="569" height="431" alt="image" src="https://github.com/user-attachments/assets/7f1688f4-b077-4840-a62c-bfa6131fe806" /> <img width="574" height="431" alt="image" src="https://github.com/user-attachments/assets/bb4a284f-961e-413d-b7e1-783bb5d8fb39" />
Beyond memory usage, it will also improve generation/forward speed by a large margin for large contexts, as only necessary states are passed to the attention computation, which is very sensitive to the sequence length.
Since the GPT-OSS release which introduced the MXPF4 quantization type, several improvements have been made to the support, which should now stabilize.
swiglu_limit not passed in for MXFP4 by @danielhanchen in #40197Mxfp4] Add a way to save with a quantization method by @ArthurZucker in #40176Now that we deprecated tensorflow and jax, we felt that torch_dtype was not only misaligned with torch, but was redundant and hard to remember. For this reason, we switched to a much more standard dtype argument!
torch_dtype will still be a valid usage for as long as needed to ensure a smooth transition, but new code should use dtype, and we encourage you to update older code as well!
The following commits are breaking changes in workflows that were either buggy or not working as expected.
On models where the hub checkpoint specifies cache_implementation="hybrid" (static sliding window hybrid cache), UNSETS this value. This will make the model use the dynamic sliding window layers by default.
This default meant that there were widespread super slow 1st generate calls on models with hybrid caches, which should nol onger be the case.
cache_implementation="hybrid" hub defaults by @gante in #40135Cache the computation of sine positional embeddings for MaskFormer; results in a 6% performance improvement.
Adds explicit cache initialization to prepare for the deprecation of the from_legacy_cache utility.
fullgraph=FalseHaving fullgraph set to True during compilation ended up being very restrictive, especially with the arrival of widely-used MoEs.
The DoLa decoding strategy has been moved to the following remote-code repository a few versions ago: https://huggingface.co/transformers-community/dola
The Contrastive Search decoding strategy has been moved to the following remote-code repository a few versions ago: https://huggingface.co/transformers-community/contrastive-search
Both have now been removed from the library as a result.
Flash attention has used sliding window sizes which were off by one. This affected generations that had initially bigger contexts than the sliding window size.
Flash Attention] Fix sliding window size by @vasqu in #40163Torch 2.1 support has been unreliable for some time, so we've now made it official and bumped our minimum version to 2.2.
GptOss fixes for green CI by @gante in #39929utils/check_bad_commit.py failing due to rate limit (requesting api.github.com) by @ydshieh in #39918torch.device('cpu').index being None by @manueldeprada in #39933torchcodec is updated by @ydshieh in #39951triton_kernels dep with kernels instead by @SunMarc in #39926fix_and_overwrite mode of utils/check_docstring.py by @manueldeprada in #39369find_file_type by @yonigozlan in #39897past_key_value to past_key_valueS everywhere by @Cyrilvallez in #39956notification_service.py about time_spent by @ydshieh in #40037notification_service.py about time_spent" by @ydshieh in #40044torchcodec==0.5.0 and use torch 2.8 on daily CI by @ydshieh in #40072time_spent in notification_service.py. by @ydshieh in #40081GPT Big Code] Fix attention scaling by @vasqu in #40041ForConditionalGeneration by @qgallouedec in #39973is_fast to ImageProcessor by @MilkClouds in #39603logger.warning with logger.warning_once in GradientCheckpointingLayer by @qgallouedec in #40091Flash Attention] Fix flash attention integration by @vasqu in #40002custom_generate collections by @gante in #39894tiny_agents.md to Korean by @AhnJoonSung in #39913content inputs for LLMs by @gante in #39829decoding_method argument in generate by @manueldeprada in #40085generation_config by @gante in #40127main_classes/processors.md to Korean by @TaskerJang in #39519jamba.md to Korean by @skwh54 in #39890main_classes/optimizer_schedules.md to Korean by @luckyvickyricky in #39713gpt2.md to Korean by @taemincode in #39808optimizers.md to Korean by @chelsseeey in #40011pipelines.md to Korean by @xhaktm00 in #39577gemma3.md to Korean by @seopp in #39865torch_compile_test and torch_export_test by @ydshieh in #39950self.tokenizer by self.processing_class by @qgallouedec in #40119too long with no output by @ydshieh in #40201model_input_names for PixtralImageProcessor by @rohitrango in #40226chat_template (jinja2) as an extra dependency by @tboerstad in #40128CI] Fix repo consistency by @vasqu in #40249k_proj weight and bias slicing in D-FINE by @notkisk in #40257id=usage to <hfoptions> tag in LayoutLM model card by @Jin-HoMLee in #40273torch.compile tests with fullgraph=True by @ydshieh in #40164FA] Fix dtype in varlen with position ids by @vasqu in #40295fix] Pass adamw optimizer parameters to StableAdamW by @emapco in #40184find_executable_batch_size to match new 0.9 ratio by @MilkClouds in #40206Flash Attention] Fix sliding window size by @vasqu in #40163_tp_plan attribute by @rishub-tamirisa in #39944natten by @ydshieh in #40287GPT OSS] Refactor the tests as it was not properly checking the outputs by @ArthurZucker in #40288get_placeholder_mask in Ovis2 by @thisisiron in #40280/en/model_doc by @gante in #40311/en/model_doc by @gante in #40344test_spm_converter_bytefallback_warning by @ydshieh in #40284FA] Fix some model tests by @vasqu in #40350label_names as an argument to TrainingArguments by @huzaifa-jawad367 in #40353skip_special_tokens in the main text generation pipelines by @gante in #40356dtype instead of torch_dtype everywhere! by @Cyrilvallez in #39782tokenizer_kwargs argument to the text generation pipeline by @Joshua-Chin in #40364transformers TF classes/methods by @gante in #40429models.md to Korean by @Judy-Choi in #39518main by @ydshieh in #40451qwen2_moe tests by @ydshieh in #40494merge to main by @ydshieh in #40503The following contributors have made significant changes to the library over the last release:
get_placeholder_mask in Ovis2 (#40280)There was a mick mack on our side when cherry-picking the commit #40197 which led to a wrong commit in the patch! Sorry everyone 😭
There was a mick mack on our side when cherry-picking the commit #40197 which led to a wrong commit in the patch! Sorry everyone 😭
This patch is just the official fix for #40197!
Focused on stabilizing FlashAttention-2 on Ascend NPU, improving FSDP behavior for generic-task models, fixing MXFP4 integration for GPT-OSS
Focused on stabilizing FlashAttention-2 on Ascend NPU, improving FSDP behavior for generic-task models, fixing MXFP4 integration for GPT-OSS
😢 Well sorry everyone, sometimes shit can happen... 4.55.1 was broken because of 🥁 git merge conflict. I cherry-picked https://github.com/huggingface/
FA2 generations!😢 Well sorry everyone, sometimes shit can happen...
4.55.1 was broken because of 🥁 git merge conflict.
I cherry-picked https://github.com/huggingface/transformers/pull/40002 without having https://github.com/huggingface/transformers/pull/40029 , thus from ..modeling_flash_attention_utils import prepare_fa_kwargs_from_position_ids is missing, and since this is a slow test, nothing caught it.
Will work to remediate and write the post-mortem when yanking the release.
Mostly focused around stabalizing the Mxfp4 for GPTOSS model!
Mostly focused around stabalizing the Mxfp4 for GPTOSS model!
Remove all expired deprecation cycles by @Cyrilvallez in #39725
<img width="2320" height="1160" alt="image" src="https://github.com/user-attachments/assets/4a1cd2f6-dde9-445e-83d9-73f6551e2da2" />
For more detailed information about this model, we recommend reading the following blogpost: https://huggingface.co/blog/welcome-openai-gpt-oss
GPT OSS is a hugely anticipated open-weights release by OpenAI, designed for powerful reasoning, agentic tasks, and versatile developer use cases. It comprises two models: a big one with 117B parameters (gpt-oss-120b), and a smaller one with 21B parameters (gpt-oss-20b). Both are mixture-of-experts (MoEs) and use a 4-bit quantization scheme (MXFP4), enabling fast inference (thanks to fewer active parameters, see details below) while keeping resource usage low. The large model fits on a single H100 GPU, while the small one runs within 16GB of memory and is perfect for consumer hardware and on-device applications.
The following snippet shows simple inference with the 20B model. It runs on 16 GB GPUs when using mxfp4, or ~48 GB in bfloat16.
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "openai/gpt-oss-20b"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
device_map="auto",
torch_dtype="auto",
)
messages = [
{"role": "user", "content": "How many rs are in the word 'strawberry'?"},
]
inputs = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
return_tensors="pt",
return_dict=True,
).to(model.device)
generated = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(generated[0][inputs["input_ids"].shape[-1]:]))
The models use attention sinks, a technique the vLLM team made compatible with Flash Attention 3. We have packaged and integrated their optimized kernel in kernels-community/vllm-flash-attn3. At the time of writing, this super-fast kernel has been tested on Hopper cards with PyTorch 2.7 and 2.8. We expect increased coverage in the coming days. If you run the models on Hopper cards (for example, H100 or H200), you need to pip install –upgrade kernels and add the following line to your snippet:
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "openai/gpt-oss-20b"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
device_map="auto",
torch_dtype="auto",
+ # Flash Attention with Sinks
+ attn_implementation="kernels-community/vllm-flash-attn3",
)
messages = [
{"role": "user", "content": "How many rs are in the word 'strawberry'?"},
]
inputs = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
return_tensors="pt",
return_dict=True,
).to(model.device)
generated = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(generated[0][inputs["input_ids"].shape[-1]:]))
Even though the 120B model fits on a single H100 GPU (using mxfp4), you can also run it easily on multiple GPUs using accelerate or torchrun. Transformers provides a default parallelization plan, and you can leverage optimized attention kernels as well. The following snippet can be run with torchrun --nproc_per_node=4 generate.py on a system with 4 GPUs:
from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers.distributed import DistributedConfig
import torch
model_path = "openai/gpt-oss-120b"
tokenizer = AutoTokenizer.from_pretrained(model_path, padding_side="left")
device_map = {
"tp_plan": "auto", # Enable Tensor Parallelism
}
model = AutoModelForCausalLM.from_pretrained(
model_path,
torch_dtype="auto",
attn_implementation="kernels-community/vllm-flash-attn3",
**device_map,
)
messages = [
{"role": "user", "content": "Explain how expert parallelism works in large language models."}
]
inputs = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
return_tensors="pt",
return_dict=True,
).to(model.device)
outputs = model.generate(**inputs, max_new_tokens=1000)
# Decode and print
response = tokenizer.decode(outputs[0])
print("Model response:", response.split("<|channel|>final<|message|>")[-1].strip())
If you have a Hopper GPU or better, we recommend you use mxfp4 for the reasons explained above. If you can additionally use Flash Attention 3, then by all means do enable it!
[!TIP] If your GPU is not compatible with mxfp4, then we recommend you use MegaBlocks MoE kernels for a nice speed bump. To do so, you just need to adjust your inference code like this:
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "openai/gpt-oss-20b"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
device_map="auto",
torch_dtype="auto",
+ # Optimize MoE layers with downloadable MegaBlocksMoeMLP
+ use_kernels=True,
)
messages = [
{"role": "user", "content": "How many rs are in the word 'strawberry'?"},
]
inputs = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
tokenize=True,
return_tensors="pt",
return_dict=True,
).to(model.device)
generated = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(generated[0][inputs["input_ids"].shape[-1]:]))
[!TIP] MegaBlocks optimized MoE kernels require the model to run on bfloat16, so memory consumption will be higher than running on mxfp4. We recommend you use mxfp4 if you can, otherwise opt in to MegaBlocks via use_kernels=True.
You can use transformers serve to experiment locally with the models, without any other dependencies. You can launch the server with just: transformers serve
To which you can send requests using the Responses API.
# responses API
curl -X POST http://localhost:8000/v1/responses \
-H "Content-Type: application/json" \
-d '{"input": [{"role": "system", "content": "hello"}], "temperature": 1.0, "stream": true, "model": "openai/gpt-oss-120b"}'
You can also send requests using the standard Completions API:
# completions API
curl -X POST http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"messages": [{"role": "system", "content": "hello"}], "temperature": 1.0, "max_tokens": 1000, "stream": true, "model": "openai/gpt-oss-120b"}'
<img width="1920" height="960" alt="image" src="https://github.com/user-attachments/assets/5502cc65-2fc9-49ac-8e15-262aa573b68d" />
Command A Vision is a state-of-the-art multimodal model designed to seamlessly integrate visual and textual information for a wide range of applications. By combining advanced computer vision techniques with natural language processing capabilities, Command A Vision enables users to analyze, understand, and generate insights from both visual and textual data.
The model excels at tasks including image captioning, visual question answering, document understanding, and chart understanding. This makes it a versatile tool for AI practitioners. Its ability to process complex visual and textual inputs makes it useful in settings where text-only representations are imprecise or unavailable, like real-world image understanding and graphics-heavy document processing.
Command A Vision is built upon a robust architecture that leverages the latest advancements in VLMs. It's highly performant and efficient, even when dealing with large-scale datasets. The model's flexibility makes it suitable for a wide range of use cases, from content moderation and image search to medical imaging analysis and robotics.
<img width="838" height="266" alt="image" src="https://github.com/user-attachments/assets/4d1e153c-0586-4650-8e18-c9d08145ce49" />
MM Grounding DINO model was proposed in An Open and Comprehensive Pipeline for Unified Object Grounding and Detection by Xiangyu Zhao, Yicheng Chen, Shilin Xu, Xiangtai Li, Xinjiang Wang, Yining Li, Haian Huang>.
MM Grounding DINO improves upon the Grounding DINO by improving the contrastive class head and removing the parameter sharing in the decoder, improving zero-shot detection performance on both COCO (50.6(+2.2) AP) and LVIS (31.9(+11.8) val AP and 41.4(+12.6) minival AP).
You can find all the original MM Grounding DINO checkpoints under the MM Grounding DINO collection. This model also supports LLMDet inference. You can find LLMDet checkpoints under the LLMDet collection.
FastSpeech2Conformer by @bvantuan in #39689classmethod by @zucchini-nlp in #38812CI] Add Eric to comment slow ci by @vasqu in #39601QAPipelineTests::test_large_model_course after #39193 by @ydshieh in #39666Glm4MoeModelTest::test_torch_compile_for_training by @ydshieh in #39670Qwen2AudioForConditionalGeneration.forward() and test_flash_attn_kernels_inference_equivalence by @ebezzam in #39503models/__init__.py for typo checking by @hebangwen in #39745GemmaIntegrationTest::test_model_2b_bf16_dola again by @ydshieh in #39731--gpus all in workflow files by @ydshieh in #39752libcst to extras["testing"] in setup.py by @ydshieh in #39761main_classes/peft.md by @luckyvickyricky in #39515tvp.md to Korean by @Kim-Ju-won in #39578tokenizer.md to Korean by @seopp in #39532pipeline_gradio.md to Korean by @AhnJoonSung in #39520perf_train_gpu_one.md to Korean by @D15M4S in #39552how_to_hack_models.md to Korean by @skwh54 in #39536run_name when none by @qgallouedec in #39695model_results.json by @ydshieh in #39783attn_implementation] remove recursive, allows custom kernels with wrappers by @ArthurZucker in #39823plot_keypoint_matching, make visualize_keypoint_matching as a standard by @sbucaille in #39830TrackioCallback to work when pynvml is not installed by @qgallouedec in #39851is_wandb_available function to verify WandB installation by @qgallouedec in #39875sub_configs by @qubvel in #39855Tokenizer with PreTrainedTokenizerFast in ContinuousBatchProcessor by @qgallouedec in #39858torch.backends.cudnn.allow_tf32 = False for CI by @ydshieh in #39885AutoModelForCausalLM and AutoModelForImageTextToText by @qubvel in #39881ModernBertForMultipleChoice by @netique in #39232Exaone4] Fixes the attn implementation! by @ArthurZucker in #39906The following contributors have made significant changes to the library over the last release:
We had quite a lot of bugs that got through! Release was a bit rushed, sorry everyone! 🤗 Mostly cache fixes, as we now have layered cache, and fixed t
We had quite a lot of bugs that got through! Release was a bit rushed, sorry everyone! 🤗 Mostly cache fixes, as we now have layered cache, and fixed to distributed.
Fix tests due to breaking change in accelerate by @SunMarc in #39451
In order to become the source of truth, we recognize that we need to address two common and long-heard critiques about transformers:
transformers is bloatedtransformers is slowOur team has focused on improving both aspects, and we are now ready to announce this.
The modeling files for the standard Llama models are down to 500 LOC and should be much more readable, keeping just the core of the modeling and hiding the "powerful transformers features."
<img width="1583" height="974" alt="image" src="https://github.com/user-attachments/assets/f1075598-d63e-4184-b3af-c0d4b31cdde5" />
The MoEs are getting some kernel magic, enabling the use of the efficient megablocks kernels, setting a good precedent to allow the community to leverage any of the most powerful kernels developed for quantization as well!
It should also be much more convenient to use with any attention implementation you want. This opens the door to some optimizations such as leveraging flash-attention on Metal (MPS Torch backend).
<img width="2050" height="752" alt="image" src="https://github.com/user-attachments/assets/23ebfb20-7626-46a5-b264-76ffb8b8c811" />
This is but the tip of the iceberg: with the work on kernels that we're heavily pushing forward, expect speed-ups on several backends in the coming months!!
This release also includes the first steps to enabling efficient distributed training natively in transformers. Loading a 100B model takes ~3 seconds on our cluster — we hope this will be the norm for everyone! We are working on distributed checkpointing as well, and want to make sure our API can be easily used for any type of parallelism.
We want the community to benefit from all of the advances, and as always, include all hardware and platforms! We believe the kernels library will give the tools to optimize everything, making a big difference for the industry!
The Ernie 4.5 model was released in the Ernie 4.5 Model Family release by baidu. This family of models contains multiple different architectures and model sizes. This model in specific targets the base text model without mixture of experts (moe) with 0.3B parameters in total. It uses the standard Llama at its core.
Other models from the family can be found at Ernie 4.5 MoE.
<div class="flex justify-center"> <img src="https://ernie.baidu.com/blog/posts/ernie4.5/overview.png"/> </div>
Ernie 4.5] Add ernie text models by @vasqu in #39228Voxtral is an upgrade of Ministral 3B and Mistral Small 3B, extending its language capabilities with audio input support. It is designed to handle tasks such as speech transcription, translation, and audio understanding.
You can read more in Mistral's realease blog post.
The model is available in two checkpoints:
Voxtral builds on Ministral-3B by adding audio processing capabilities:
LFM2 represents a new generation of Liquid Foundation Models developed by Liquid AI, specifically designed for edge AI and on-device deployment.
The models are available in three sizes (350M, 700M, and 1.2B parameters) and are engineered to run efficiently on CPU, GPU, and NPU hardware, making them particularly well-suited for applications requiring low latency, offline operation, and privacy.
The DeepSeek-V2 model was proposed in DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model by DeepSeek-AI Team.
The model uses Multi-head Latent Attention (MLA) and DeepSeekMoE architectures for efficient inference and cost-effective training. It employs an auxiliary-loss-free strategy for load balancing and multi-token prediction training objective. The model can be used for various language tasks after being pre-trained on 14.8 trillion tokens and going through Supervised Fine-Tuning and Reinforcement Learning stages.
ModernBERT Decoder is the same architecture as ModernBERT but trained from scratch with a causal language modeling (CLM) objective. This allows for using the same architecture for comparing encoders and decoders. This is the decoder architecture implementation of ModernBERT, designed for autoregressive text generation tasks.
Like the encoder version, ModernBERT Decoder incorporates modern architectural improvements such as rotary positional embeddings to support sequences of up to 8192 tokens, unpadding to avoid wasting compute on padding tokens, GeGLU layers, and alternating attention patterns. However, it uses causal (unidirectional) attention to enable autoregressive generation.
The Encoder-only Mask Transformer (EoMT) model was introduced in the CVPR 2025 Highlight Paper Your ViT is Secretly an Image Segmentation Model by Tommie Kerssies, Niccolò Cavagnero, Alexander Hermans, Narges Norouzi, Giuseppe Averta, Bastian Leibe, Gijs Dubbelman, and Daan de Geus. EoMT reveals Vision Transformers can perform image segmentation efficiently without task-specific components.
<div style="text-align: center;"> <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/eomt_architecture.png" alt="drawing" width="500"/> </div>
Doge is a series of small language models based on the Doge architecture, aiming to combine the advantages of state-space and self-attention algorithms, calculate dynamic masks from cached value states using the zero-order hold method, and solve the problem of existing mainstream language models getting lost in context. It uses the wsd_scheduler scheduler to pre-train on the smollm-corpus, and can continue training on new datasets or add sparse activation feedforward networks from stable stage checkpoints.
<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/refs%2Fpr%2F426/transformers/model_doc/doge_architecture.png" alt="drawing" width="600"
The AIMv2 model was proposed in Multimodal Autoregressive Pre-training of Large Vision Encoders by Enrico Fini, Mustafa Shukor, Xiujun Li, Philipp Dufter, Michal Klein, David Haldimann, Sai Aitharaju, Victor Guilherme Turrisi da Costa, Louis Béthune, Zhe Gan, Alexander T Toshev, Marcin Eichner, Moin Nabi, Yinfei Yang, Joshua M. Susskind, Alaaeldin El-Nouby.
The abstract from the paper is the following:
We introduce a novel method for pre-training of large-scale vision encoders. Building on recent advancements in autoregressive pre-training of vision models, we extend this framework to a multimodal setting, i.e., images and text. In this paper, we present AIMV2, a family of generalist vision encoders characterized by a straightforward pre-training process, scalability, and remarkable performance across a range of downstream tasks. This is achieved by pairing the vision encoder with a multimodal decoder that autoregressively generates raw image patches and text tokens. Our encoders excel not only in multimodal evaluations but also in vision benchmarks such as localization, grounding, and classification. Notably, our AIMV2-3B encoder achieves 89.5% accuracy on ImageNet-1k with a frozen trunk. Furthermore, AIMV2 consistently outperforms state-of-the-art contrastive models (e.g., CLIP, SigLIP) in multimodal image understanding across diverse settings.
The PerceptionLM model was proposed in PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding by Jang Hyun Cho et al. It's a fully open, reproducible model for transparent research in image and video understanding. PLM consists of a vision encoder with a small scale (<8B parameters) LLM decoder.
The EfficientLoFTR model was proposed in Efficient LoFTR: Semi-Dense Local Feature Matching with Sparse-Like Speed by Yifan Wang, Xingyi He, Sida Peng, Dongli Tan and Xiaowei Zhou.
This model consists of matching two images together by finding pixel correspondences. It can be used to estimate the pose between them. This model is useful for tasks such as image matching, homography estimation, etc.
<img width="2772" height="3276" alt="image" src="https://github.com/user-attachments/assets/8fb76e17-9bff-4edc-9ac8-205f4b58a898" />
Evolla is an advanced 80-billion-parameter protein-language generative model designed to decode the molecular language of proteins. It integrates information from protein sequences, structures, and user queries to generate precise and contextually nuanced insights into protein function. Trained on an unprecedented AI-generated dataset of 546 million protein question-answer pairs and 150 billion word tokens, Evolla significantly advances research in proteomics and functional genomics, providing expert-level insights and shedding light on the molecular logic encoded in proteins.
<img width="824" height="1017" alt="image" src="https://github.com/user-attachments/assets/298074a0-c509-4e0b-9adb-c72bac206b18" />
Deepseek-VL was introduced by the DeepSeek AI team. It is a vision-language model (VLM) designed to process both text and images for generating contextually relevant responses. The model leverages LLaMA as its text encoder, while SigLip is used for encoding images.
<img width="692" height="434" alt="image" src="https://github.com/user-attachments/assets/164243ba-1fd8-48b5-b565-3396d640ce0e" />
The xLSTM model was proposed in xLSTM: Extended Long Short-Term Memory by Maximilian Beck*, Korbinian Pöppel*, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter and Sepp Hochreiter. xLSTM updates the original LSTM architecture to be competitive with Transformer models by introducing exponential gating, matrix memory expansion, and parallelizable training and ingestion.
The 7B model variant was trained by the xLSTM team Maximilian Beck, Korbinian Pöppel, Phillip Lippe, Richard Kurle, Patrick Blies, Sebastian Böck and Sepp Hochreiter at NXAI.
<img width="3750" height="954" alt="image" src="https://github.com/user-attachments/assets/591040d2-d90a-4770-a4a7-0d91e21263fb" />
EXAONE 4.0 model is the language model, which integrates a Non-reasoning mode and Reasoning mode to achieve both the excellent usability of EXAONE 3.5 and the advanced reasoning abilities of EXAONE Deep. To pave the way for the agentic AI era, EXAONE 4.0 incorporates essential features such as agentic tool use, and its multilingual capabilities are extended to support Spanish in addition to English and Korean.
The EXAONE 4.0 model series consists of two sizes: a mid-size 32B model optimized for high performance, and a small-size 1.2B model designed for on-device applications.
We've added Expert Parallel support for Llama4, next release will include it for all model! You can just set a distributed_config with enable_expert_parallel=True. This is enabling efficient training of sparse Mixture-of-Experts (MoE) models across multiple devices. This allows each expert in the MoE layer to run in parallel (instead of previous TP which requires more communication), significantly improving scalability and memory efficiency.
FP-Quant is a quantization method optimized for Blackwell-generation Nvidia GPUs, supporting efficient post-training quantization (PTQ) and quantization-aware training (QAT) of LLMs using MXFP4 and NVFP4 formats.
Currently, only PTQ with MXFP4 is available. You can quantize models on-the-fly using transformers:
from transformers import AutoModelForCausalLM, FPQuantConfig
model = AutoModelForCausalLM.from_pretrained(
"qwen/Qwen3-8B",
quantization_config=FPQuantConfig(),
device_map="cuda",
torch_dtype=torch.bfloat16,
)
FP-Quant requires a Blackwell GPU and runs via the QuTLASS library. No Blackwell GPU? Use FPQuantConfig(pseudoquant=True) to emulate quantization (no QuTLASS needed).
The following results show the inference speedup of QuTLASS MXFP4 over PyTorch BF16 in Transformers. MXFP4 gives consistent speedups across all batch sizes, reaching up to 4× faster at larger scales.
<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/qwen3-8b-end-to-end-prefill-speedup-mxfp4-vs-bf16-on-rtx5090.svg" alt="drawing" width="600">
The kernels project aims to become the single trusted source for high-performance kernels in the Transformers ecosystem. We're working toward centralizing all kernels on the Hub, so updates, bug fixes, and improvements can happen in one place—no more scattered repos and no compilation headaches!
You can already try it out today by setting use_kernels=True in from_pretrained. Any contributor can build their kernel, register it and use it right away—no extra setup, more on this here
Even better: want to use Flash Attention 3? No need to deal with tricky compilation and missing symbols issues! Just drop in:
model.set_attn_implementation("kernels-community/flash-attn3")
This automatically fetches the right build for your setup (e.g. CUDA and PyTorch versions).
We’re also teaming up with amazing kernel devs from Unsloth, Liger, vLLM, and more to bring their work directly to the Hub—making it easier than ever to access amazing performance with a single line of code.
https://github.com/user-attachments/assets/9928f62b-543c-4b8a-b81b-4a6e262c229e
Over the past few months, we have been putting more and more functionality in the transformers chat utility, which offers a CLI-based app to chat with chat models. We've chosen to push this further by splitting the backend of transformers chat in a new, separate utility called transformers serve.
This is ideal for experimentation purposes, or to run models locally for personal and private use. It does not aim to compete with dedicated inference engines such as vLLM or SGLang.
Models of diverse modalities supported by transformers may be served with the transformers serve CLI. It spawns a local server that offers compatibility with the OpenAI SDK, which is the de-facto standard for LLM conversations and other related tasks. This way, you can use the server from many third party applications, or test it using the transformers chat CLI (docs).
The server supports the following REST APIs:
/v1/chat/completions/v1/responses/v1/audio/transcriptions/v1/modelsRelevant commits:
transformers chat and transformers serve by @LysandreJik in #38443transformers serve by @LysandreJik in #39149generation_config by @gante in #39230transformers serve by @LysandreJik in #39155/v1/audio/transcriptions) by @gante in #39434Significant refactors have been underway in transformers, aiming to reduce the complexity of the code. A metric we follow to see how the refactors impact our code is to follow the number of lines in a given model; we try to reduce it as much as possible, while keeping everything related to the forward pass and model definition in that file.
See the evolution here:
<img width="1200" height="600" alt="image" src="https://github.com/user-attachments/assets/c232bc8d-7d7c-4192-baa8-a60efe5eb2ff" />
Some notable refactors:
KV caches are now defined per layer, enabling new hybrid caches that mix different attention types. CacheProcessors also encapsulate cache quantization and offloading, making them easy to customize.
output_attentions or output_hidden_statesSuch attributes require very specific handling within the forward call, while they're not important to understand how the model works. We remove that code but keep the functionality by providing a better utility to handle it.
We refactor the way to explicitly set the attention implementation so that it has a method dedicated to it.
average_tokens_across_devices by default in TrainingArguments by @Krish0909 in #39395Flex Attn] Fix torch 2.5.1 incompatibilities by @vasqu in #37406test_compare_unprocessed_logit_scores by @ydshieh in #39053t5gemma tests by @ydshieh in #39052layoutlmv3 tests by @ydshieh in #39050Gemma3nProcessorTest by @ydshieh in #39068mistral3 tests by @ydshieh in #38989dots1 tests by @ydshieh in #39088test_is_split_into_words in test_pipelines_token_classification.py by @st81 in #39079test_sdpa_can_dispatch_on_flash by @ydshieh in #39092@lru_cache() to @lru_cache to match styles from #38883. by @rasmi in #39093run-slow by @ydshieh in #39100llama tests by @ydshieh in #39161Dia] Change ckpt path in docs by @vasqu in #39181from_pretrained by @qubvel in #39184fastspeech2_conformer tests by @ydshieh in #39229is not None -> isinstance(..., dict) by @qubvel in #39145segmentation_maps support to MobileNetV2ImageProcessor by @simonreise in #37312tests/generation/test_utils.py by @ydshieh in #39254test_eager_matches sdpa generate and update an integration test for blip-like models by @ydshieh in #39248smollm3 by @gante in #39271PretrainedConfig.__init__ method to make it more explicit by @qubvel in #39158test_generate_compile_model_forward by @ydshieh in #39276datasets 4.0 by @lhoestq in #39156aria tests by @ydshieh in #39277test_torchscript_* for now until the majority of the community ask for it by @ydshieh in #39307stevhliu to the list in self-comment-ci.yml by @ydshieh in #39315src/ for doctest (for now) by @ydshieh in #39316max_length_q and max_length_k types to flash_attn_varlen_func by @HollowMan6 in #37206phi3 tests by @ydshieh in #39312position_ids in masking_utils by @Cyrilvallez in #39310test_sdpa_can_dispatch_on_flash by @ydshieh in #39259timm (for perception_lm) by @ydshieh in #39380/v1/models output payload by @alvarobartt in #39414set_tracer_provider and set_meter_provider calls by @McPatate in #39422JetMoeForCausalLM by @Phoenix-Shen in #37830ContinuousBatchProcessor by @qgallouedec in #39372CI] Fix partially red CI by @vasqu in #39448GemmaIntegrationTest::test_model_2b_bf16_dola by @ydshieh in #39362datasets pin by @gante in #39500args_doc.py to auto_docstring.py by @yonigozlan in #39439_supports_flash_attn_2 in examples and tests by @zucchini-nlp in #39471TypeError instead of ValueError for invalid types by @Sai-Suraj-27 in #38660MambaCache to modeling_mamba.py by @manueldeprada in #38086perf_infer_gpu_multi.md to Korean by @luckyvickyricky in #39441CI] Fix post merge ernie 4.5 by @vasqu in #39561docs/source/ko/_toctree.yml by @jungnerd in #39516supports_static_cache to can_compile_fullgraph by @zucchini-nlp in #39505device_mesh have multiple dim by @S1ro1 in #38949test_export_static_cache by @gante in #39662Ernie 4.5] Post merge adaptations by @vasqu in #39664kyutai tests by @ydshieh in #39416typing.Literal as type of tool parameters or return value by @grf53 in #39633The following contributors have made significant changes to the library over the last release:
segmentation_maps support to MobileNetV2ImageProcessor (#37312)docs/source/ko/_toctree.yml (#39516)A small patch for open telemetry fixes! Sorry for the delay!
A small patch for open telemetry fixes! Sorry for the delay!
** refactor: remove set_tracer_provider and set_meter_provider calls (https://github.com/huggingface/transformers/pull/39422) from @McPatate
[sliding window] revert and deprecate
This patch contains the following bug fixes:
smollm3 (#39271)position_ids in masking_utils (#39310)This patch contains several bug fixes. The following commits are included:
This patch contains several bug fixes. The following commits are included:
Several minimal breaking changes aiming to bring clearer defaults while greatly simplifying the library have been merged.
Gemma 3n models are designed for efficient execution on low-resource devices. They are capable of multimodal input, handling text, image, video, and audio input, and generating text outputs, with open weights for pre-trained and instruction-tuned variants. These models were trained with data in over 140 spoken languages.
Gemma 3n models use selective parameter activation technology to reduce resource requirements. This technique allows the models to operate at an effective size of 2B and 4B parameters, which is lower than the total number of parameters they contain. For more information on Gemma 3n's efficient parameter management technology, see the Gemma 3n page.
from transformers import pipeline
import torch
pipe = pipeline(
"image-text-to-text",
torch_dtype=torch.bfloat16,
model="google/gemma-3n-e4b",
device="cuda",
)
output = pipe(
"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg",
text="<image_soft_token> in this image, there is"
)
print(output)
Dia is an opensource text-to-speech (TTS) model (1.6B parameters) developed by Nari Labs. It can generate highly realistic dialogue from transcript including nonverbal communications such as laughter and coughing. Furthermore, emotion and tone control is also possible via audio conditioning (voice cloning).
Model Architecture: Dia is an encoder-decoder transformer based on the original transformer architecture. However, some more modern features such as rotational positional embeddings (RoPE) are also included. For its text portion (encoder), a byte tokenizer is utilized while for the audio portion (decoder), a pretrained codec model DAC is used - DAC encodes speech into discrete codebook tokens and decodes them back into audio.
<img src="https://huggingface.co/datasets/eustlb/documentation-images/resolve/main/kyutai_stt.png"/>
Kyutai STT is a speech-to-text model architecture based on the Mimi codec, which encodes audio into discrete tokens in a streaming fashion, and a Moshi-like autoregressive decoder. Kyutai’s lab has released two model checkpoints:
Read more about the model in the documentation
<div class="flex justify-center"> <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/vjepa.gif" alt="drawing" width="600"/> </div>
V-JEPA 2 is a self-supervised approach to training video encoders developed by FAIR, Meta. Using internet-scale video data, V-JEPA 2 attains state-of-the-art performance on motion understanding and human action anticipation tasks. V-JEPA 2-AC is a latent action-conditioned world model post-trained from V-JEPA 2 (using a small amount of robot trajectory interaction data) that solves robot manipulation tasks without environment-specific data collection or task-specific training or calibration.
Read more about the model in the documentation.
Arcee is a decoder-only transformer model based on the Llama architecture with a key modification: it uses ReLU² (ReLU-squared) activation in the MLP blocks instead of SiLU, following recent research showing improved training efficiency with squared activations. This architecture is designed for efficient training and inference while maintaining the proven stability of the Llama design.
The Arcee model is architecturally similar to Llama but uses x * relu(x) in MLP layers for improved gradient flow and is optimized for efficiency in both training and inference scenarios.
Read more about the model in the documentation.
ColQwen2 is a variant of the ColPali model designed to retrieve documents by analyzing their visual features. Unlike traditional systems that rely heavily on text extraction and OCR, ColQwen2 treats each page as an image. It uses the Qwen2-VL backbone to capture not only text, but also the layout, tables, charts, and other visual elements to create detailed multi-vector embeddings that can be used for retrieval by computing pairwise late interaction similarity scores. This offers a more comprehensive understanding of documents and enables more efficient and accurate retrieval.
Read more about the model in the documentation.
MiniMax is a powerful language model with 456 billion total parameters, of which 45.9 billion are activated per token. To better unlock the long context capabilities of the model, MiniMax adopts a hybrid architecture that combines Lightning Attention, Softmax Attention and Mixture-of-Experts (MoE). Leveraging advanced parallel strategies and innovative compute-communication overlap methods—such as Linear Attention Sequence Parallelism Plus (LASP+), varlen ring attention, Expert Tensor Parallel (ETP), etc., MiniMax's training context length is extended to 1 million tokens, and it can handle a context of up to 4 million tokens during the inference. On various academic benchmarks, MiniMax also demonstrates the performance of a top-tier model.
The architecture of MiniMax is briefly described as follows:
For more details refer to the release blog post.
Read more about the model in the documentation.
T5Gemma (aka encoder-decoder Gemma) was proposed in a research paper by Google. It is a family of encoder-decoder large langauge models, developed by adapting pretrained decoder-only models into encoder-decoder. T5Gemma includes pretrained and instruction-tuned variants. The architecture is based on transformer encoder-decoder design following T5, with improvements from Gemma 2: GQA, RoPE, GeGLU activation, RMSNorm, and interleaved local/global attention.
T5Gemma has two groups of model sizes: 1) Gemma 2 sizes (2B-2B, 9B-2B, and 9B-9B), which are based on the offical Gemma 2 models (2B and 9B); and 2) T5 sizes (Small, Base, Large, and XL), where are pretrained under the Gemma 2 framework following T5 configuration. In addition, we also provide a model at ML size (medium large, ~2B in total), which is in-between T5 Large and T5 XL.
The pretrained varaints are trained with two objectives: prefix language modeling with knowledge distillation (PrefixLM) and UL2, separately. We release both variants for each model size. The instruction-turned varaints was post-trained with supervised fine-tuning and reinforcement learning.
Read more about the model in the documentation.
The GLM-4.1V model architecture is added to transformers; no models have yet been released with that architecture. Stay tuned for the GLM team upcoming releases!
Read more about the model in the documentation.
The FalconH1 model was developed by the TII Pretraining team. A comprehensive research paper covering the architecture, pretraining dynamics, experimental results, and conclusions is forthcoming. You can read more about this series in this website.
Read more about the model in the documentation.
The LightGlue model was proposed in LightGlue: Local Feature Matching at Light Speed by Philipp Lindenberger, Paul-Edouard Sarlin and Marc Pollefeys.
Similar to SuperGlue, this model consists of matching two sets of local features extracted from two images, its goal is to be faster than SuperGlue. Paired with the SuperPoint model, it can be used to match two images and estimate the pose between them. This model is useful for tasks such as image matching, homography estimation, etc.
The abstract from the paper is the following:
We introduce LightGlue, a deep neural network that learns to match local features across images. We revisit multiple design decisions of SuperGlue, the state of the art in sparse matching, and derive simple but effective improvements. Cumulatively, they make LightGlue more efficient - in terms of both memory and computation, more accurate, and much easier to train. One key property is that LightGlue is adaptive to the difficulty of the problem: the inference is much faster on image pairs that are intuitively easy to match, for example because of a larger visual overlap or limited appearance change. This opens up exciting prospects for deploying deep matchers in latency-sensitive applications like 3D reconstruction. The code and trained models are publicly available at this https URL
Read more about the model in the documentation.
The abstract from the report is the following:
Mixture of Experts (MoE) models have emerged as a promising paradigm for scaling language models efficiently by activating only a subset of parameters for each input token. In this report, we present dots.llm1, a large-scale MoE model that activates 14B parameters out of a total of 142B parameters, delivering performance on par with state-of-the-art models while reducing training and inference costs. Leveraging our meticulously crafted and efficient data processing pipeline, dots.llm1 achieves performance comparable to Qwen2.5-72B after pretraining on high-quality corpus and post-training to fully unlock its capabilities. Notably, no synthetic data is used during pretraining. To foster further research, we open-source intermediate training checkpoints spanning the entire training process, providing valuable insights into the learning dynamics of large language models.
Read more about the model in the documentation.
SmolLM3 is a fully open, compact language model designed for efficient deployment while maintaining strong performance. It uses a Transformer decoder architecture with Grouped Query Attention (GQA) to reduce the kv cache, and no RoPE, enabling improved performance on long-context tasks. It is trained using a multi-stage training approach on high-quality public datasets across web, code, and math domains. The model is multilingual and supports very large context lengths. The instruct variant is optimized for reasoning and tool use.
Read more about the model in the documentation.
In previous versions, installing the kernels library would automatically activate the custom kernels added to transformers, because the @use_kernel_forward_from_the_hub decorator directly swapped out the model’s forward method. This implicit behavior caused several issues for users — including problems with torch.compile, non-determinism, and inconsistent outputs.
To address this, we've introduced a new opt-in mechanism called kernelize. You can now enable kernel usage explicitly by passing use_kernels=True to from_pretrained. The use_kernel_forward_from_the_hub decorator now simply stores the kernel name that the user wants to use — and kernelize handles the rest under the hood.
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-3.2-1B-Instruct",
torch_dtype=torch.bfloat16,
device_map="cuda",
use_kernels=True
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-1B-Instruct")
input = "Hello"
input_ids = tokenizer(input, return_tensors="pt").to(model.device).input_ids
output = model.generate(input_ids, max_new_tokens=100)
print(tokenizer.decode(output[0], skip_special_tokens=True))
More kernels will be added over time — this will be a collaborative, community-driven effort to make transformers lighter and faster 🤗
Support for Flash Attention 3 is added across the most popular models.
Several efforts refactoring the repository are happening in parallel. The direction is to greatly simplify the library, removing unnecessary codepaths. Whilst the efforts are spread across the library, they're particularly visible in each individual models; where non-modeling-specific code will be simplified and eventually removed.
We take the assumption that model-agnostic utilities shouldn't be in the modeling code. Things like the output of attentions, hidden states, router logits, are important for end-users but don't need to be explicitely displayed in the modeling code.
Several minimal breaking changes aiming to bring clearer defaults while greatly simplifying the library have been merged.
dtype for pipelines to auto by @Vaibhavs10 in #38882output_attentions=True and the attn implementation is wrong by @ArthurZucker in #38288Attention] Refactor Attention Interface for Bart-based Models by @vasqu in #38108Attention] Attention refactor for Whisper-based models by @vasqu in #38235compile] re-enable for Qwen-VL models by @zucchini-nlp in #38127forced_decoder_ids by @gante in #38232liger-kernel to docker file by @ydshieh in #38292transformers env output by @yao-matrix in #38274forced_decoder_ids deletion by @gante in #38316beam_indices by @gante in #38259custom_generate and trust_remote_code by @gante in #38304vasqu to self-comment-ci.yml by @ydshieh in #38324FlexAttention] Reenable flex for encoder-decoder and make the test more robust by @vasqu in #38321kernels for AMD docker images by @ydshieh in #38354OPT] Fix attention scaling by @vasqu in #38290get_default_device for torch<2.3 by @Cyrilvallez in #38376utils/notification_service.py by @ydshieh in #38379initialize_weights by @Cyrilvallez in #38382tokenizer -> tokenize by @foldl in #38357generation_config.json as base parameterization by @gante in #38330test_offloaded_cache_implementation) by @gante in #37896pixel_values with inputs_embeds by @dxoigmn in #38334CsmForConditionalGenerationIntegrationTest by @ydshieh in #38424huggingface/transformers by @ydshieh in #38413from_pretrained by @pstjohn in #38155from_args_and_dict ProcessorMixin by @yonigozlan in #38296microsoft/python-type-stubs (post dropping support for Python 3.8) by @Avasam in #38335BatchFeature and BatchEncoding by @lgeiger in #38459Gemma3IntegrationTest by @ydshieh in #38471SinkCache to a custom_generate repo by @gante in #38399Gemma2IntegrationTest by @ydshieh in #38492av by @ydshieh in #38548python3 by @S1ro1 in #38555utils/notification_service.py by @ydshieh in #38556chameleon tests by @ydshieh in #38565utils/notification_service.py for AMD vs Nvidia by @ydshieh in #38563deepseekv3 by @ydshieh in #38562FlexAttn] Fix models with unique characteristics by @vasqu in #38433repository field to benchmarks table by @McPatate in #38582mlm_probability to be set to None when mlm=False in DataCollatorForLanguageModeling by @KameniAlexNea in #38522)isort from dependencies by @Sai-Suraj-27 in #38616return_dict=False giving errors in a few VLM models by @ydshieh in #38519MiniMax (docs and integration tests checkpoint) by @geetu040 in #38575test_initialization by @ydshieh in #38607ColQwen2ModelIntegrationTest by @ydshieh in #38583test_initialization for SwiftFormer by @ydshieh in #38636AriaForConditionalGenerationModelTest on CircleCI by @ydshieh in #38615InternVL integration test by @ydshieh in #38612aya_vision test by @ydshieh in #38674is_bitsandbytes_available() by @ved1beta in #38528llava tests by @ydshieh in #38722None instead of try/except by @zucchini-nlp in #38561average_tokens_across_devices=True and world size = 1 by @qgallouedec in #38785qwen_2_5 omni by @ydshieh in #38658llava_onevision tests by @ydshieh in #38791mllama by @ydshieh in #38704low_cpu_mem_usage by @Cyrilvallez in #38792llava_next tests by @ydshieh in #38813wandb.run.url instead of wandb.run.get_url() (deprecated) by @qgallouedec in #38817align_to_words=True in QuestionAnsweringPipeline can lead to duplicate answers by @yushi2006 in #38761qwen2_5_vl tests by @ydshieh in #38845auxiliary_in_channels default behavior in UperNet by @simonreise in #37540qwen3 tests by @ydshieh in #38862phi4_multimodal tests by @ydshieh in #38816qwen3_moe tests by @ydshieh in #38865raise from e in hub.py utility by @Wauplin in #37241fsmt tests by @ydshieh in #38904FalconMambaIntegrationTests by @ydshieh in #38566ALL_LAYERNORM_LAYERS by @Cyrilvallez in #38922test_initialization by @ydshieh in #38932mistral and mistral3 tests by @ydshieh in #38978is_split_into_words in the TokenClassificationPipeline. by @yushi2006 in #38818rag by @ydshieh in #38585Attention] Small fix on output attentions by @vasqu in #38948require_tf) by @gante in #38944The following contributors have made significant changes to the library over the last release:
liger-kernel to docker file (#38292)vasqu to self-comment-ci.yml (#38324)kernels for AMD docker images (#38354)utils/notification_service.py (#38379)CsmForConditionalGenerationIntegrationTest (#38424)huggingface/transformers (#38413)Gemma3IntegrationTest (#38471)Gemma2IntegrationTest (#38492)av (#38548)utils/notification_service.py (#38556)chameleon tests (#38565)utils/notification_service.py for AMD vs Nvidia (#38563)deepseekv3 (#38562)return_dict=False giving errors in a few VLM models (#38519)test_initialization (#38607)ColQwen2ModelIntegrationTest (#38583)test_initialization for SwiftFormer (#38636)AriaForConditionalGenerationModelTest on CircleCI (#38615)InternVL integration test (#38612)aya_vision test (#38674)llava tests (#38722)qwen_2_5 omni (#38658)llava_onevision tests (#38791)mllama (#38704)llava_next tests (#38813)qwen2_5_vl tests (#38845)qwen3 tests (#38862)phi4_multimodal tests (#38816)qwen3_moe tests (#38865)fsmt tests (#38904)FalconMambaIntegrationTests (#38566)test_initialization (#38932)mistral and mistral3 tests (#38978)rag (#38585)output_attentions=True and the attn implementation is wrong (#38288)transformers env output (#38274)Attention] Refactor Attention Interface for Bart-based Models (#38108)FlexAttention] Reenable flex for encoder-decoder and make the test more robust (#38321)OPT] Fix attention scaling (#38290)Attention] Attention refactor for Whisper-based models (#38235)FlexAttn] Fix models with unique characteristics (#38433)Attention] Small fix on output attentions (#38948)microsoft/python-type-stubs (post dropping support for Python 3.8) (#38335)MiniMax (docs and integration tests checkpoint) (#38575)The following commits are included in that patch release:
The following commits are included in that patch release:
We had to protect the imports again, a series of bad events. Here are the two prs for the patch:
We had to protect the imports again, a series of bad events. Here are the two prs for the patch:
We had to revert #37877 because of a missing flag that was overriding the device map. We re-introduced the changes because they allow native 3D parall
We had to revert #37877 because of a missing flag that was overriding the device map. We re-introduced the changes because they allow native 3D parallel training in Transformers. Sorry everyone for the troubles! 🤗
One column per quarter.