NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
PyPI, #269 most downloaded on PyPI
Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.
Latest stable
5.24.0
Last release today
21 Nov 2026
Ships fairly regularly
a new release about every 2 weeks
Nearly every release is documented
notes for 60 of the last 60 stable releases
6 versions withdrawn
withdrawn after publishing
10 years old
254 releases · first in 2016
Drag across the ruler to compare, or tap a release to open it.
Deprecate old input names ( #49424 ) by @zucchini-nlp in [ #49424 ]
MiniCPM-OCR is a document-understanding model from OpenBMB's MiniCPM-V family. It keeps the MiniCPM-V 4.6 architecture, which pairs a SigLIP vision encoder and a window-attention merger with a Qwen3.5 language model. It is specialized for reading text-dense images such as scanned pages, screenshots, receipts and tables.
Links: Documentation | Paper
Flint is Recursive's mixture-of-experts language model. Compared to a standard SwiGLU MoE transformer such as Qwen3-MoE, it uses hybrid attention (sliding-window attention with rotary embeddings over the current token and the previous 512 tokens in most layers, and full attention without positional encoding in the remaining layers). It also has learned attention sinks, as in GPT-OSS, with per-head RMS-normalized queries and keys. Each sparse layer routes a token to 6 of 256 experts by sigmoid score and adds an ungated shared expert, and the first two layers are dense.
Links: Documentation
UltraVBert extends ModernVBert by pairing the UltraBERT text encoder with a SigLIP vision encoder for visual document understanding and retrieval. Its image processor splits each page into 512-pixel tiles plus a global view. The connector merges each 4 x 4 block of vision patches into one token, so a tile becomes 64 image tokens, and RMS-normalizes them to the scale of UltraBERT's token embeddings. The max_image_tiles setting (27 by default) caps the number of tiles per page.
Links: Documentation
Gradium Phonon TTS is Gradium's family of on-device text-to-speech models, trained with the Pocket TTS recipe. A causal transformer reads a voice prompt and the text, then emits one continuous 32-dimensional audio latent per step at 12.5 Hz through a one-step flow head, and an end-of-speech head decides when the utterance ends. The latents are decoded to 24 kHz audio by a Pocket TTS audio tokenizer. Unlike Pocket TTS, voice conditioning uses 512-dimensional latents from a separate, unreleased speaker codec, so voices come as precomputed latents and cannot be cloned from a recording.
Links: Documentation
Pocket TTS is Kyutai's 100M-parameter text-to-speech model, built as a Continuous Audio Language Model that runs faster than real time on CPU. A causal transformer emits one continuous audio latent per frame (12.5 Hz) through a one-step flow head, and an end-of-speech head decides when to stop. A Mimi-based audio tokenizer encodes voice prompts and decodes the generated latents to 24 kHz audio. Two flow heads are released, Lagrangian self-distillation (default) and drifting, and voices can be cloned from a recording, passed as precomputed latents, or loaded from cached KV states.
Links: Documentation | [Paper](https://huggingface.co/papers/2509.06926 (the Continuous Audio Language Models paper, arXiv 2509.06926). It is the only paper the documentation links, and it describes the architecture family Pocket TTS belongs to. The docs do not cite a separate Pocket TTS technical report.) | [Blog Post](NONE. The documentation links no official blog post. It points only to the original code repository at https://github.com/kyutai-labs/pocket-tts.)
The Pocket TTS audio tokenizer is the codec used by Pocket TTS. It has Mimi's architecture (SEANet convolutions plus a small causal transformer on each side), but its residual vector quantizer is replaced by a continuous 32-dimensional bottleneck, the VAE-GAN from Continuous Audio Language Models. It converts 24 kHz audio into latents at 12.5 Hz and decodes them back to audio. It serves as the audio_tokenizer of PocketTTSProcessor, encoding voice prompts and decoding generated latents, and its encoder and decoder can also be loaded on their own.
Links: Documentation | Paper
Fixed Gemma4 processor imports (the Gemma4, Gemma4 unified and EmbeddingGemma2 processors) so from transformers import * no longer fails in environments that have PIL but not torch; no user action is needed.
torch but with PIL (#49748) by @Sai-Suraj-27Added an "fp8" backend to QuantizedCache, alongside quanto and hqq, that stores KV states in torch.float8_e4m3fn with a single scale per tensor; enable it with cache_implementation="quantized" and cache_config={"backend": "fp8"}.
QuantizedCache (#48677) by @jiqing-fengTensor parallelism now has a dedicated embedding style, shards every model's input embeddings by default, and supports widths that aren't divisible by the TP size, so ColwiseParallel and RowwiseParallel now apply only to linear layers; custom TP plans that applied them to embeddings must switch to the new embedding style.
Every MoE TP and EP plan now shards its experts, so models no longer keep all experts on every rank, and a new test_moe_parallel_plans_shard_experts test enforces this; custom or downstream MoE parallel plans must add expert sharding to pass it and to avoid wasted memory.
This release improves cache handling across the library. Flash Attention now works with static caches, BLOOM no longer crashes with static cache or compiled generate, chunked prefill supports cache continuation, and the Qwen4-Exp indexer's position history stays aligned with its cached keys. Linear-attention caches also gain new getters (is_recording_past, get_recurrent_state, conv-state access) that stop exposing cache internals and prepare for paged-c
Vision image processors now composite PNG transparency onto white instead of exposing black pixels, and background_color is preserved correctly across the DeepSeek-VL and Janus processors. Model fixes cover SmolVLM's inputs_merger when image tokens aren't a multiple of the patch size, HunYuan-VL image assignment during batched beam search, and Ernie4.5-VL-MoE expert sharding under TP/EP. The DETR
This release fixes a range of generation bugs: BLOOM crashes with static cache or compiled generate, generate dropping the type of dict-like inputs such as ModelOutput, and incorrect image assignment or placeholder handling in batched or beam-search generation for HunYuan-VL and Idefics2/3. It also fixes Qwen3-Omni audio generation mode handling, SAM mask-generation multi-zoom crops, empty frame s
generate (#49731) by @Cyrilvallez in [#49731]This release extends parallelism support and fixes several related bugs. It adds a tensor-parallel plan for Nemotron and makes Ernie4.5-VL-MoE shard its text and vision experts under TP and EP. It also restores expert-parallel token-dispatch gradients on torch 2.7–2.10, fixes a Molmo2 model-parallel device mismatch, and corrects base_model_prefix in several model families (PP-OCR detectors
base_model_prefix in the PP-OCR detectors, CLIPSeg, TimesFM, CLIP and VideoPrism heads so base checkpoints load (#48871) by @blipbyte in [#48871]Quantization now handles DeepSeek-V4.1's FP4 experts correctly: FineGrainedFP8Config gains an expert_dtype option ("fp8" by default, or "fp4") that the FP8 experts read instead of allocating them as FP8. TimesFM 2.5 no longer casts float inputs to an integer weight dtype when loaded with quantization such as bitsandbytes 4/8-bit. A broader cleanup
Note truncated.
Your coding agent can read these notes before it upgrades. Set up the MCP server →
This patch notably solves an issue where some inputs could loose their original type inside prepare_inputs_for_generation in generate, leading to prob
This patch notably solves an issue where some inputs could loose their original type inside prepare_inputs_for_generation in generate, leading to problems with multimodal models. It also fixes some other issues listed on the below commits.
It contains the following commits:
Special surprise including the new model Beam from Reflection AI :partying_face:
Special surprise including the new model Beam from Reflection AI :partying_face:
<img width="2912" height="1440" alt="Gemini_Generated_Image_ox4hjlox4hjlox4h" src="https://github.com/user-attachments/assets/bfc7ff16-4340-490c-925c-b1512e4ceb32" />
Beam is a token-efficient, frontier-scale model focused on coding and agentic use cases. It is built for teams that want to run a powerful model on their own hardware, infrastructure, or cloud, and customize it to their specific needs.
Links: Documentation
We broke loading models on older torch versions :cry: should be fixed now. But here is the full list of fixes
Full Changelog: https://github.com/huggingface/transformers/compare/v5.20.0...v5.20.1
Deprecated checks and methods have been removed from cache_utils.py (affecting models such as minimax_m3_vl and pi0), so any code still calling those…
<img width="2400" height="1296" alt="image" src="https://github.com/user-attachments/assets/bf34ea27-0edf-4ded-ba6f-905b04096126" />
Muse Spark Open is a multimodal mixture-of-experts model from Meta that takes text, images, audio, and video as input and generates text. It has 816B total parameters (42B active per token) and a context window of up to 1,048,576 tokens shared across modalities. It is trained for agentic coding tasks such as multi-file editing, debugging, and tool use. Checkpoints are available in BF16 and NVFP4 formats.
Links: Documentation | Blog Post
Ling 3.0 Flash is a hybrid Mixture-of-Experts language model from InclusionAI. Its decoder groups five Kimi Delta Attention (KDA) layers with one Multi-head Latent Attention (MLA) layer. The MLA blocks use interleaved rotary embeddings and a head-wise output gate, and the feed-forward blocks combine routed and shared experts. It is natively supported in Transformers as BailingHybridForCausalLM, so it loads without trust_remote_code=True.
Links: Documentation
Kolibri 1 is Aleph Alpha's mixture-of-experts reasoning model, focused on German and English. Each of its 50 layers routes every token to 6 of 384 experts using sigmoid routing, and adds one shared expert. Four sliding-window attention layers alternate with one full-attention layer, and RoPE is applied only in the sliding-window layers.
Links: Documentation
MiniCPM-V is a series of efficient multimodal large language models developed by OpenBMB. Like MiniCPM-V 4.6, the MiniCPM-V 4.7 architecture pairs a SigLIP vision encoder that has a window-attention merger with a Qwen3.5 language model backbone, and supports both 4x and 16x visual downsampling modes.
The main addition over 4.6 is canvas M-RoPE: instead of numbering visual tokens along a single 1-D sequence, the model lays every image out on a 2-D canvas and assigns each visual token a (temporal, height, width) position, so slices of the same image keep their spatial relationship and video frames keep their temporal order.
UnlimitedOcr is a single 3B-parameter OCR model from Baidu with a standard 32,768-token context length. It builds on DeepSeek OCR and replaces all decoder attention layers with Reference Sliding Window Attention (R-SWA). This keeps the KV cache constant during decoding, so the model can transcribe dozens of pages in a single forward pass. It supports "gundam" mode (640x640 tiles with dynamic cropping) and "base" mode (a single 1024x1024 view), and it can output region detections with bounding boxes.
Links: Documentation | Paper
UltraBERT is a family of bidirectional text encoders trained with a combination of masking ratios, pruning, and distillation for classification and retrieval tasks. The released encoders support sequences of up to 32,768 tokens. They use rotary position embeddings, RMSNorm, SwiGLU feed-forward layers, and alternating local and global attention (every third layer is global). Checkpoints include UltraBERT-150M and UltraBERT-380M for masked language modeling and fine-tuning, and UltraBERT-Embed-150M for text embeddings.
Links: Documentation | Paper
MOSS-Transcribe-Diarize 0.9B is an end-to-end audio understanding model for long-form multi-speaker transcription, speaker diarization, and timestamps. It combines a Whisper-style encoder, a 4× frame merge step, a multi-modal projector, and a Qwen3 language model. It produces joint transcription and diarization in a single pass, with segments formatted as [start][S01]text[end]. It supports long-form audio through Whisper-window chunking in the processor, promptable transcription with hotword biasing, and 50+ languages.
Links: Documentation
🚨 General breaking notice that inference with EP has regressions in speed; we are working on fixing this ASAP! 🚨
Deprecated checks and methods have been removed from cache_utils.py (affecting models such as minimax_m3_vl and pi0), so any code still calling those deprecated cache methods must switch to the current cache API.
Accidental imports from other models were removed from several model files (e.g. embedding_gemma2, ijepa, lightglue), and the modular converter now avoids adding such cross-model imports. If you imported these names through those model modules, import them from their original modules instead.
The minimum kernels version is now 0.17.2, which allows use_fallback=KernelizeFallback.ALL so that a failed kernel load falls back to the original implementation instead of raising an error. Upgrade kernels to 0.17.2 or later.
A new SubConfigSpec centralizes the initialization of sub-configs, so default sub-configs are collected from dataclass fields instead of by instantiating the class, and this touches the configuration utilities and many model configs. Custom or existing nested configs may need to be updated to declare their sub-configs through this mechanism.
The LUKE and mLUKE tokenizers now treat entity special tokens such as <ent> as special tokens (not split, honored by skip_special_tokens, normalized) while keeping the same spacing as regular vocabulary tokens when decoding. Tokenization and decoding output involving these tokens may therefore differ from before, so check any code or saved outputs that depend on the old behavior.
Transformers exporters now support quantization directly during export across Dynamo, ONNX, OpenVINO, and ExecuTorch. The new unified quantization API integrates PT2E, ONNX Runtime, and NNCF quantizers, enabling static and dynamic quantization as well as weight-only compression (INT8/INT4), with configurable calibration workflows. This makes it easier to export optimized models for efficient inference across different backends.
For more information see the docs: https://huggingface.co/docs/transformers/exporters_quantization
This release improves distributed training and inference. FSDP × TP model and optimizer checkpoints can now be saved and loaded across different parallel configurations, and expert-parallel dispatch, tensor-parallel plans, static-cache generate(), and the caching allocator's byte count were fixed. Tied LM heads in composite models now get their own TP plan entry, enable_expert_parallel applies the EP plan alone again, TPU loading uses the `tor
This release fixes static-cache generate() bugs with host-side inputs and under tensor parallelism. It also speeds up Whisper generation on TPU by cutting device syncs and recompilation, and fixes Whisper batched timestamp decoding for tensor and NumPy inputs. Pipelines get a cleanup of generation-argument handling that removes spurious warnings.
Vision processing was updated in several places. Image decoding now uses torchcodec, Molmo2's image and video kwargs follow the library's standard conventions, and Pixtral Flash Attention no longer detects false image boundaries after the axial RoPE refactor, which had caused CUDA illegal memory access errors and incorrect output. Several CI and Docker changes also shipped: GPU images moved to Python 3.11, a numpy/pyarrow conflict
This release improves kernel support across hardware. flash_attention_2 now falls back to the AITER kernel on ROCm when flash-attn isn't installed, and T5-style models get a contiguous SDPA position-bias mask so they use fused kernels again. The Hub rotary kernel is now limited to CUDA inference because its backward pass gave incorrect gradients. The Atlas Inference Gated DeltaNet kernels for Strix Halo and
This release fixes several cache-related bugs: static-cache generate() now handles host-side nested inputs and tensor parallelism correctly, the offloaded cache prefetch no longer races with the offloading copy, and Phi3 LongRoPE cache rebuilds at the context boundary now receive the full prefix. It also counts the query token in MiMo-V2-Flash's sliding window and corrects the quantized cache (hqq/quanto) documentation exam
torch.mm(..., out_dtype=...) to ONNX (#49635) by @IlyasMoutawwakil in [#49635]MtpModel helpers for frameworks that own the drafting loop (#49544) by @hmellor in [#49544]create_chunked_causal_mask() crashing with block_sequence_ids (#49457) by @kaixuanliu in [#49457]ep_size != tp_size (#49368) by @qgallouedec in [#49368]The following contributors have made significant changes to the library over the last release:
The "paged|" prefix for SDPA and flash attention implementations is deprecated, so users should set the regular attention implementation (e.g. sdpa or…
<img width="2716" height="2308" alt="image" src="https://github.com/user-attachments/assets/84734e74-163d-4d12-b166-ffcf6749d563" />
EmbeddingGemma 2 is a multimodal embedding model from Google built on the Gemma 4 architecture. It encodes text, images, audio, and video, individually or combined in one input, into a shared 768-dimensional vector space for cross-modal retrieval, semantic similarity, clustering, and classification. It uses Matryoshka Representation Learning, so embeddings can be truncated to 512, 256, or 128 dimensions. It also offers configurable visual and video token budgets, and unused vision or audio towers can be disabled at load time to save memory.
Links: Documentation
All MoE models whose routers compute logits now return them when output_router_logits=True, following the Qwen3-MoE pattern (a router_logits recorder on the base model, MoeModelOutputWithPast from the backbone, and a MoE causal LM output from the head), so code that relied on the previous outputs or their absence should read the router logits from these output classes.
Owlv2ForObjectDetection.embed_image_query now selects the query box with the highest objectness score, as in the original OWLv2 notebook, instead of the OWL-ViT heuristic, so image-guided query embeddings and detections may differ from earlier releases.
The "paged|" prefix for SDPA and flash attention implementations is deprecated, so users should set the regular attention implementation (e.g. sdpa or flash_attention_2) for continuous batching instead of paged|sdpa or paged|flash_attention_2.
The regular flash and SDPA attention functions (flash_attention.py, sdpa_attention.py) now support continuous batching directly, and "paged|..." implementations for these are redirected to them, while eager still requires the "paged|eager" prefix.
In continuous batching, the cache update for the index-based and block-table paths is now fused into a single call, which slightly changes the cache update function's behavior and affects any custom code that calls the separate update paths.
Continuous batching internals changed in preparation for removing "paged": `max
Expert parallelism gains a token-dispatch implementation, selected via the new ep_dispatch_experts plan rule and now the default for Qwen3 MoE and Mellum, which removes the requirement that EP size equal TP size. The Trainer was also adapted to work with expert parallelism, and the docs now note that PEFT adapters support tensor parallelism. A CI-related fix for pipeline-parallel chart2table inference was also included.
distributed]: adapt Trainer to work Expert Parallel (#48873) by @3outeille in [#48873]distributed] Add expert-parallel token dispatch, default for Qwen3 MoE (#48865) by @3outeille in [#48865]This release fixes quantized cache handling: generate no longer mutates the user's cache_config, and QuantizedLayer.reorder_cache is repaired. It also adds per-layer cache configuration, so DynamicCache and StaticCache initialize each layer from its own config (sliding window, attention chunk size, conv states, and attention head counts) to better support heterogeneous models. Separately, the deprecation cycle on mask and cache
distributed] Default MoE ep_plans to token dispatch (#49160) by @3outeille in [#49160].python-version (#49311) by @tarekziade in [#49311]distributed] decoupled tp_plan and ep_plan (#48859) by @3outeille in [#48859]distributed] Add dense and expert device mesh (#48857) by @3outeille in [#48857]labels docstring of CLIPSegForImageSegmentation (#49001) by @qgallouedec in [#49001]Mixin (#49105) by @zucchini-nlp in [#49105]torch_tpu backend (#49207) by @tengomucho in [#49207]GTE] Add memory mixin (#49234) by @vasqu in [#49234]The following contributors have made significant changes to the library over the last release:
GTE] Add memory mixin (#49234)One column per quarter.