NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
PyPI · #225 most downloaded on PyPI
Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.
Last release today
30 Sep 2026
Ships on a steady schedule
a new release about every 2 weeks
Nearly every release is documented
notes for 60 of the last 60 stable releases
6 versions withdrawn
withdrawn after publishing
10 years old
243 releases · first in 2016
…compatibility for the tensor-parallel API with a deprecation cycle for tp_plan in from_pretrained() . Additionally fixed a model parallel bug in the B…
Qwen4-Exp builds on Qwen3.5's hybrid text and multimodal architecture with three key components: GatedResidual (GR), Qwen Sparse Attention (QSA), and Per-Layer Embedding (PLE).
GR is a Qwen-developed residual architecture that combines Hyper-Connection with GatedNorm. It mixes multiple residual streams with fine-grained elementwise gating before each attention and Mixture-of-Experts (MoE) block, then controls how much of the block output is injected back into each stream.
QSA uses multiple query heads to score compressed key blocks, selects the most relevant contiguous token blocks, and keeps the incomplete trailing block uncompressed. This block-level selection reduces indexing overhead and improves memory locality for long sequences. Combined with Gated DeltaNet, QSA makes Qwen4-Exp the first hybrid architecture to integrate linear and sparse attention, substantially improving inference efficiency for long-context workloads.
PLE enriches selected decoder layers with layer-specific lexical features derived from hashed token n-grams and a dilated depthwise convolution.
Links: Documentation
Granite Speech 5.0 Turbo CTC is a lightweight (~470M parameters) conformer encoder for automatic speech recognition, trained with Connectionist Temporal Classification (CTC) on BPE targets. It is a fast, encoder-only member of the Granite Speech family: transcription requires a single forward pass followed by greedy CTC decoding, with no autoregressive decoder.
Architecturally, it extends the Granite Speech conformer CTC encoder with:
Frame stacking + block-wise time subsampling: the feature extractor stacks pairs of log-mel(+delta) frames (2x), and the first two conformer blocks each subsample time by 2 through a stride-2 depthwise convolution (with a mean-pooled residual), for a total 8x time reduction at 10 ms mel hop.
Block attention with Shaw's relative positional embeddings: attention is computed over fixed-size blocks (the sequence is right-padded to a whole number of blocks, with padded frames masked out), using separate bias-free query/key/value projections.
Self-conditioned CTC: the CTC posteriors of the middle layer are projected and fed back into the hidden states, and the CTC head is shared between this mid-layer self-conditioning and the final prediction.
Links: Documentation
Step-3.7-Flash was proposed in Step 3.7 Flash by StepFun. It is a 198B-parameter sparse Mixture-of-Experts vision-language model, pairing a 196B-parameter MoE language backbone with a 1.8B-parameter vision encoder for native image understanding.
StepFun hasn't published a technical report for Step-3.7-Flash, so the details below are drawn from the released checkpoint's configuration rather than a paper.
~GenerationMixin.generate] can use for speculative decoding via use_mtp=True.Links: Documentation
CohereCompass is the base architecture for small, specialized (vision-)language models trained by Cohere.
Links: Documentation
ESMC and ESMFold2 are new state-of-the-art protein language and folding models from BioHub. ESMC is trained with a masked language modeling objective, and it can be easily transferred to sequence and token classification tasks for proteins. Checkpoints exist in various sizes, from 300M parameters up to 6B parameters. It works as a drop-in replacement for older ESM-2 and ESM-3 models, with significantly higher accuracy.
ESMFold2 is a state-of-the-art protein folding model which produces high accuracy predictions. It uses an iterated diffusion approach that is significantly different from the original ESMFold, offering huge improvements in accuracy for more complex structures.
Links: Documentation ESMC, Documentation ESMFold2
The legacy tensor-parallel implementation has been replaced with a DTensor-native backend, so users relying on the previous TP API for inference or training must migrate to the new DTensor-based interface.
attn_implementation="sdpa" dispatch is now properly supported for wav2vec2_conformer, wav2vec2-bert, and SeamlessM4T/v2 models, which may change initialization behavior for users who previously worked around this limitation.
FuyuProcessor no longer returns the image_patch_indices output, so any code that depends on this field must be updated to remove references to it.
Several cache-related bugs were fixed in this release, including an off-by-one error in the sliding window cache, Whisper speculative decoding cache corruption, CpmAnt use-cache failures, Qwen2.5-Omni/Qwen3-Omni-MoE generation with compilable caches, and compressed-tensors loading for KV-cache-only quantized models. Documentation was also added for cache token removal using negative values, and per-layer cache configuration support (allowing models to use different cache settings per layer) was introduced.
This release fixes several generation bugs across multiple models, including Whisper speculative decoding issues (UnboundLocalError, cache corruption, speed regression, and left-padded batch position IDs), broken image generation in Emu3, garbage output in OLMo/GPTNeoX, and Qwen2.5-Omni/Qwen3-Omni-MoE generation with compilable caches. Additionally, logit distributions for candidate generators using sampling are now aligned by returning logits after applying logit processors.
generation failing with import_or_config (other (2)) (#48061) by @sergereview[bot] in [#48061]Several attention-related bug fixes were made in this release, including correcting a SigLIP2 documentation typo, fixing Flash/SDPA attention dispatch tests for xcodec2 and ROCm RDNA GPUs, resolving a GPT2 cross-attention mask being silently discarded, and enabling SDPA support declaration in TimmWrapper. Per-layer cache configuration and attention-mask selection support was also introduced, allowing models with heterogeneous layer configurations to use distinct sliding_window, attention_chunk_size, and number_of_conv_states values per layer.
TimmWrapper (#47939) by @jiqing-feng in [#47939]Quantization improvements include adding NVFP4 quantization support via HF kernels (enabling on-the-fly BF16 weight quantization with ~50% memory reduction), and fixing several bugs: reverting a regression in is_quantization_compressed that caused incorrect module layouts for packed-format checkpoints, fixing CLIP weight initialization failures with quantized checkpoints, and restoring KV-cache quantization setup for KV-cache-only quantized models.
Introduced a naive pipeline parallel inference engine supporting tied/untied weight embeddings with seamless generate() integration, while restoring backward compatibility for the tensor-parallel API with a deprecation cycle for tp_plan in from_pretrained(). Additionally fixed a model parallel bug in the BLT model affecting beam search.
Kernel support was improved with documentation updates highlighting supported models, a fix for export crashes on kernel-decorated functions by adding a is_torchdynamo_exporting guard, and the default Flash Attention 2 hub kernel version was bumped to v3 to resolve compatibility issues with newer PyTorch versions.
scores type in stopping criteria docstrings (#47676) by @qgallouedec in [#47676]BayesianDetectorModel.from_pretrained() by calling post_init() (#48254) by @woojinpaik in [#48254]GDN] Fix recurrent FLA fallback (#48266) by @vasqu in [#48266]tie_word_embeddings not lifted from text_config for some VLM configs (BC regression) (#45857) by @qgallouedec in [#45857]force_accelerate_hooks should not hide the signature it wraps (#48156) by @SunMarc in [#48156]Note truncated.
One column per quarter.
This patch most notably solves a few issues with DFlash and MTP candidate generators, as well as an issue where images could sometimes not be processe
This patch most notably solves a few issues with DFlash and MTP candidate generators, as well as an issue where images could sometimes not be processed on accelerator if using Lanczos filter.
It contains the following commits:
…during decode, and patches a potential ReDoS vulnerability caused by unescaped tokenizer filenames being used as regex patterns in from_pretrained .
Muse Glimmer, released today, is Meta’s new multimodal model, especially designed for agentic use cases. Distilled from Muse to 30B parameters, and released under the Apache 2.0 license, it can be deployed to local setups for privacy-aware applications such as coding, document analysis, personal assistants, Claw- or Hermes-like setups.
Muse Glimmer is a dense 30B parameter model consisting of:
We're covering it in the following blogpost: http://hf.co/blog/muse-glimmer
Links: Documentation
Links: Documentation
Links: Documentation
Links: Documentation
Links: Documentation
Kernels are now opt-in rather than mandatory for linear attention models (Mamba, GDN, Conv-only, etc.), so users who relied on automatic kernel selection must explicitly enable kernels to maintain previous behavior.
The cache cropping API now only accepts negative values (relative offsets) instead of absolute sizes, so users calling crop methods directly must update their code to pass negative values accordingly.
T5 and its model family (MT5, LongT5, etc.) now support SDPA and other attention backends via ALL_ATTENTION_FUNCTIONS, meaning the default attention implementation may change and users relying on the previous eager-only path should explicitly set attn_implementation="eager" if needed.
Several small private helper functions (e.g., _is_url, _build_image_tokens) have been removed from multimodal processor files, so users or downstream libraries that imported these private functions directly must remove or replace those references.
This release includes several attention fixes and improvements, including correcting Multi-Head Latent Attention (MLA) cache compression, optimizing Flash Attention max sequence length computation in vision models, and fixing bugs in CTRL flex-attention and SDPA prefill with position bias. Additional changes refactor linear attention models for better maintainability, make Gemma 4's heterogeneous attention config explicit, and improve MPS support via metal-flash-sdpa integration.
per_layer_config for Gemma 4 so that heterogeneous attention config is explicit (#47384) by @hmellor in [#47384]value padding into the attention interfaces that need it (#47451) by @hmellor in [#47451]BlockMask crash in CTRL flex-attention generation (#46854) by @jiqing-feng in [#46854]Vision improvements in this release include performance optimizations such as faster image preprocessing for vision-language models (GLM4V, MiniMaxM3-VL, and others) by eliminating redundant tensor copies, and more efficient Flash Attention variable-length paths by precomputing maximum sequence lengths once per forward pass. Several bug fixes were also applied, including correcting dtype alignment in Kosmos2/Kosmos2_5 embedding merges, fixing a position-embedding initialization fallback in Phi4Multimodal, resolving PIL resize parity in Hunyuan-VL, and patching stop-sequence handling in the image-text-to-text pipeline.
Several generation improvements and bug fixes were made, including enabling batched audio generation for Qwen2.5/3-Omni, allowing sliding window cache layers to work with speculative decoding, and fixing memory overhead from static cache persistence across generate() calls. Multiple model-specific bugs were also resolved, including crashes in KyutaiSpeechToText, MusicgenForCausalLM, CTRL flex-attention, and assisted decoding for EncoderDecoder cache and OlmoHybrid models.
generate() last window (#46952) by @jiqing-feng in [#46952]MusicgenForCausalLM.generate() (#46974) by @jiqing-feng in [#46974]Several cache-related bugs were fixed, including correcting NemotronH's missing "mlp" layer-type mapping, resolving recurrent-layer padding masks being skipped during chunked prefill and cache continuation for hybrid models, and fixing assisted decoding for models with EncoderDecoderCache and OlmoHybrid. Additional improvements include aligning OlmoHybrid to use a native cache, enabling sliding window layers to support speculative decoding rollback, and stopping the static cache from being stored as a model attribute to reduce unexpected memory overhead.
"mlp" in the cache layer-type mappings (#47535) by @qgallouedec in [#47535]⚠️ The kernels python package will very likely be a required dependency for transformers[torch] in the near future. This will help us deliver maximum performance to all users; kernels will only be downloaded from trusted publishers manually approved by the HF team. Please let us know of any issues you're facing beforehands so that we may solidify our integration.
Improved robustness of the kernels integration by refactoring function handling to use layer repos, fixing CI EROFS fallback patches for kernel downloads via HfApi, resolving a positional argument collision in causal_conv1d_fn, and bumping the FP8 kernels version to prevent NaNs.
Kernels] Refactor function handling (#46883) by @vasqu in [#46883]causal_conv1d_fn positional activation colliding with hub kernel's seq_idx (#47527) by @qgallouedec in [#47527]FP8] Bump kernels version (#47344) by @vasqu in [#47344]Quantization support was expanded with FP8 kernels for compressed-tensors models, fixes for FP8 module normalization and format-based compression detection, and a multi-device MXFP4 dequantization race condition fix. GPTQ and MXFP4 tests were also extended to cover Intel XPU devices.
_convert_moe_packed_tensors (#47423) by @kaixuanliu in [#47423]Batched audio generation is now supported for Qwen2.5/3-Omni, and several bug fixes were applied across audio models, including a dtype mismatch in Gemma4 audio feature merging, a bfloat16 positional embedding error in AudioFlamingo3, and missing backend requirement guards for Voxtral. The VibeVoice ASR processor was also updated to make audio input optional and support multiple audios per prompt.
Expanded FSDP support across 94 ForCausalLM model classes with auto-generated FSDP plans, added end-to-end FSDP tests including distributed checkpoint save/load and generation, and introduced a dedicated FSDP CI job. Additionally, fixed a device mismatch bug in create_bidirectional_sliding_window_mask under model parallelism and resolved a tensor parallel inference issue for models with tied embeddings.
create_bidirectional_sliding_window_mask (#47560) by @abcgco in [#47560]This release adds native support for Mistral's "tekken" tokenizer format via AutoTokenizer, fixes a CodeLlama tokenizer bug where leading whitespace was incorrectly dropped during decode, and patches a potential ReDoS vulnerability caused by unescaped tokenizer filenames being used as regex patterns in from_pretrained.
Improved the serve chat parsing to unify streaming and non-streaming paths under a single response parser that handles tool calls, reasoning, and content, simplifying the addition of new model support. Additionally, hardened daily CI reporting by fixing GitHub API diagnostic output being captured in Slack payloads and adding rate-limit resilience to prevent report failures when paginating large job matrices.
Note truncated.
This patch solves a few issues which appeared when integrating Inkling model, most notably an issue affecting models using EncoderDecoderCache during
This patch solves a few issues which appeared when integrating Inkling model, most notably an issue affecting models using EncoderDecoderCache during assisted generation. It also fixes an issue that could appear during prefill with StaticCache and sdpa without padding for Inkling which uses a position_bias. It contains the following commits:
Your coding agent can read these notes before it upgrades. Set up the MCP server →