NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
PyPI · #1284 most downloaded on PyPI
Parameter-Efficient Fine-Tuning (PEFT)
Last release 3 days ago
01 Oct 2026
Ships fairly regularly
a new release about every 2 months
Nearly every release is documented
notes for 36 of 38 stable releases
Nothing withdrawn
no release was ever pulled
4 years old
39 releases · first in 2023
One column per quarter.
This is a PEFT release fixes an issue that prevented encoder-decoder models to work when using Transformers ≥ 5.18.0.
This is a PEFT release fixes an issue that prevented encoder-decoder models to work when using Transformers ≥ 5.18.0.
Changes:
This is a small PEFT release to enable Tensor Parallel (TP) to work properly with PEFT. It requires Transformers ≥ 5.17.0 to work.
Riemannian-preconditional LoRA optimizer
Riemannian-preconditioned LoRA optimizer (#3382), contributed by @smellslikeml, is a new LoRA-specific optimizer similar to LoRA-FA or LoRA+ that applies a Riemannian pre-conditioning on the gradients to improve feature learning and avoiding separate tuning of learning rates for LoRA's A and B matrices.
KaSA aims at identifying knowledge aware singular values from the base weight. It then removes noisy knowledge from those weights and specifically trains the singular values most relevant to the task. Implemented in #3446 by @robbiebusinessacc and @iambogeumkim.
Super-Tuning uses calibration-free magnitude scoring to allocate a sparse tuning budget. With BottomK selection, Super adapts low-magnitude weights, motivated by the idea that they provide degrees of freedom for learning a new task while leaving larger-magnitude pretrained weights untouched. Supra adds LoRA on top of this sparse update. It is a good fit for exploring sparsity alongside higher-rank adaptation, particularly on reasoning-heavy fine-tuning tasks. Thanks to @smellslikeml.
ShadowPEFT runs a small shadow network alongside the frozen base model. That shadow state gets updated at every decoder layer, so adaptation is shared across the stack instead of a separate low-rank update on each targeted linear layer (like LoRA). It’s a good choice when you care about downstream or OOD performance and want to stay in the same trainable-parameter range as LoRA, or when you want the small shadow on the edge and the frozen base in the cloud. This works best if the shadow model starts from a small, pretrained model. Contributed by @SeanLee97.
Speed up MoE target parameter computation (#3577): For the common use case of one active adapter, weight and MoE LoRA delta computation is folded into a single baddbmm operation. Expect slight numerical differences in this case.
Support torchao v0.18.0 by supporting the newer Int8Tensor, Float8Tensor, etc. tensor subclasses.
Orthogonal Subspace Fine-tuning received a significant update in #3524 reducing the memory footprint by 22% and the training time by 46% according to the MetaMathQA benchmark while keeping task accuracy and forgetting stable. Expect slight numerical differences, especially in fp16, bf16.
#3639: Thanks to @amogh-nagri-11, prompt learning methods now allow to delete an adapter. For adaption prompt, adding or loading a second adapter no longer automatically activates it, in line with how other PEFT methods handle this.
An internal change moved state dict loading/saving method-specific code paths to the respective tuners. This should be transparent to the end-user and was extensively tested and should work exactly the same as previously, but if you suspect that the model behaves differently, let us know.
Previously, when loading or adding a second, third etc. adapter, this could affect the requires_grad of the existing adapter (or base model), depending on the value of is_trainable. This should no longer be the case thanks to @DaoyuanLi2816.
Combining LoRA adapters using add_weighted_adapter had a bug in the SVD path which led to the LoRA scaling being applied twice. This is now fixed.
We moved OFT from a method-specific quantization backend to the generic quantization backend. This affects bitsandbytes, GPT-Q, etc. This should not affect existing checkpoints, but please let us know if you run into issues.
#3667: inject_adapter_in_model now rejects PEFT methods that have shared state that would live on the parent model (like VeRA), as no such parent model exists. Contributed by @CoralGarden52.
#3534: Thanks to @LK-maker-007, the PEFT now throws an error when adding a second adapter to the PEFT model using add_adapter that targets non-existing layers. Before, the adapter would just not adapt anything but be silent about it. peft_model.targeted_module_names and peft_model.targeted_parameter_names no longer contain duplicates. They are still sorted by insertion order (not sets). To suppress potential errors, catch peft.PeftError.
#3504: When deleting an adapter, there is now a check if the adapter is merged. If it's merged, an error is raised, since deleting it would not remove its effect. This was contributed by @DaoyuanLi2816.
baddbmm by @qgallouedec in #3577prepare_model_for_kbit_training memory leak regression t… by @sywangyi in #3627Note truncated.
Following the switch to GPT-QModel as the backend in the previous release, AutoGPTQ is now formally deprecated ( #3190 ) thanks to @Qubitium .
This release adds no less than nine new PEFT methods and puts a lot of work into the surrounding infrastructure, for example adding a new image generation benchmark for the method comparison suite and greatly improving the documentation structure.
@hqsiswiliam added "HiRA: Parameter-Efficient Hadamard High-Rank Adaptation for Large Language Models" to PEFT (#2668). Instead of adding the low-rank product BA to the base weight, HiRA multiplies it elementwise (Hadamard product) with the frozen base weight. Because the base weight itself is full rank, the resulting update is no longer constrained to be low rank, while the trainable parameter count stays the same as LoRA's.
@not-lain contributed GLoRA: "One-for-All: Generalized LoRA for Parameter-Efficient Fine-Tuning" in #3098. It is a flexible PEFT method that extends LoRA with configurable weight, activation, and bias adaptation, delivering richer fine-tuning with no extra inference cost. Use it when you need per-layer flexibility or stronger adaptation than vanilla LoRA. Skip it for non-Linear layers (e.g. Conv/Embedding) or when standard LoRA is already sufficient and simplicity matters.
@whubaichuan added "BEFT: Bias-Efficient Fine-Tuning of Language Models" in #3195. BEFT builds on the observation that fine-tuning bias terms alone can be competitive in low-data regimes, but goes further: rather than training all biases, it targets the value projection by default, as the authors found this to be most efficient. This brings the trainable parameter count down to roughly 0.01% of the total parameters.
@victor7246 integrated MonteCLoRA, "Robust and Efficient Fine-tuning of LLMs with Bayesian Reparameterization of Low-Rank Adaptation" in #2943. LoRA is known to be sensitive to hyperparameters like learning rate and batch size. This new LoRA variant addresses this by treating the low-rank parameters as a distribution rather than a point estimate, using Monte Carlo estimation to obtain a low-variance posterior estimate. This should make training noticeably more robust, at the cost of only O(rank) additional parameters.
@roymiles added VeLoRA: "Memory Efficient Training using Rank-1 Sub-Token Projections" in #3159. Unlike most PEFT methods, this LoRA variant targets activation memory rather than parameter count: intermediate activations are split into sub-tokens and compressed with a fixed rank-1 projection before being cached for the backward pass, then reconstructed during backpropagation. Conceptually, it is similar to gradient checkpointing, with a lower memory saving but running faster.
@KaiyangLi1992 contributed Uni-LoRA: "One Vector is All You Need" in #3257. The paper shows that parameter-efficient LoRA variants like VeRA and VB-LoRA can all be described as projecting the full LoRA parameter space down to a much smaller subspace, differing only in the choice of projection. Uni-LoRA uses a single global projection across the whole model instead of layer-wise ones, which allows cross-layer parameter sharing and thus very low parameter counts.
@Bane-Elvin added FRoD, "Full-Rank Efficient Fine-Tuning with Rotational Degrees for Fast Convergence", in #3270. Instead of adding low-rank deltas like LoRA, it reconstructs selected weights with shared rotational subspaces and sparse trainable coefficients. It is especially useful when fast convergence and a higher full-rank capacity ceiling are important, and its large sparse rotational subspace may also be promising for model merging. The main tradeoffs are the costly joint-decomposition initialization and slightly slower forward/backward passes than LoRA, so it may be less attractive for a one-off single-task fine-tune.
@sr-networks contributed MiCA, "MiCA Learns More Knowledge Than LoRA and Full Fine-Tuning", in #3260. MiCA is a LoRA variant that initializes B from the SVD of the base weight, taking the left singular vectors belonging to the smallest singular values, and then trains only A. The idea is that adapting these underused directions integrates new knowledge more effectively than adapting the dominant subspace, with less interference with existing capabilities.
@MAXNORM8650 added DEFT, "Decompositional Efficient Fine-Tuning for Text-to-Image Models", in #3342. This PEFT method splits a weight update into two learned low-rank parts: a projection that removes a sub-space of the frozen weight, and a low-rank update that injects new content in its place. Use DEFT for personalizing a text-to-image model from a few images while retaining the base model's instruction-following/editability with minimal forgetting. It is less of a fit if you don't need to preserve the base model's other capabilities (a plain additive adapter is simpler) or for layers beyond Linear/Conv1D. The PaRa method is also supported as a special case of DEFT.
@oswaldoludwig added KappaTuneSelector in #3106 based on "The Condition Number as a Scale-Invariant Proxy for Information Encoding in Neural Units". The PR adds a new function, find_kappa_target_modules, which automatically identifies which modules best to target based on the condition number of the base weight matrices. This removes some of the guesswork from choosing target_modules when configuring LoRA or other PEFT methods.
#3117 adds generic quantization support to BOFT, MiSS, and VeRA, and #3321 does the same for SHiRA. Previously, quantization support had to be implemented per method and per backend; more methods now work with the common quantization backends out of the box. Note that as a consequence, VeRA now uses the generic quantization backend instead of its previous bitsandbytes quantization backend. If you encounter issues because of that, please let us know.
target_parameterstarget_parameters, which allows targeting nn.Parameters directly (useful for MoE models that pack experts into a single parameter), now supports adding and loading multiple adapters (#3350).
@Anai-Guo added SVDConv2d in #3196, so AdaLoRA can now target torch.nn.Conv2d layers.
The PEFT documentation was reorganized in #3300 and #3325, which should make it considerably easier to navigate now that PEFT supports so many methods. Each PEFT method also provides an overview of how well it fares in the PEFT benchmarks. There is also a long tail of documentation improvements from many contributors this release, thanks to everyone who worked on those.
Following the switch to GPT-QModel as the backend in the previous release, AutoGPTQ is now formally deprecated (#3190) thanks to @Qubitium.
The method comparison suite now has an image generation benchmark (#3082). This complements the existing MetaMathQA benchmark and allows to compare PEFT methods on a completely different task. This gives us a more complete picture of the capabilities of different PEFT methods. To check the results, visit the Gradio Space and on the top left, select the task "image-gen".
A look at the results from the image generation benchmark
Since the last release, we also made several improvements to the MetaMathQA benchmark. You should check the updated results in the Gradio Space. With the help of the community, we now cover a much broader range of experiments. Moreover, we discovered a bug in the forgetting metric -- the new numbers now accurately reflect how much fine-tuning retains of the original knowledge of the base model.
#3317 adds a new Gradio Space with the goal of making it very easy to compare different PEFT methods not only with regard to performance but also capabilities (e.g. if quantization is supported). Give this Space a visit to find out which PEFT method is the best fit for your use case.
Browse the PEFT shop for the best PEFT method for your use case
test_lora_seq2seq_lm_multi_gpu_inference by @kaixuanliu in #3226Note truncated.
A small patch release containing these fixes:
Fully deprecate AutoGPTQ and AutoAWQ for GPT-QModel by @ZX-ModelCloud in #2932
This PEFT release contains no less than nine new PEFT methods, described below. It also contains numerous enhancements that should make PEFT more useful to many users.
@yeonjoon-jung01 added "GraLoRA: Granular Low-Rank Adaptation for Parameter-Efficient Fine-Tuning" to PEFT (#2851). This method subdivides the base weight into smaller blocks and applies LoRA to those. This more granular adaptation promises to increase expressiveness and improve performance, especially at higher ranks (64+), closing the gap to full fine-tuning.
@Conzel contributed BD-LoRA: "Block-Diagonal LoRA for Eliminating Communication Overhead in Tensor Parallel LoRA Serving" (#2895). With BD-LoRA, the LoRA weights are implemented in a block-diagonal way. This allows to reduce communication overhead when using tensor parallelism (TP) and thus faster serving.
There is an experiment branch for BD-LoRA support in vLLM: vllm-project/vllm#28136.
Thanks to @kashif, PEFT now also supports Cartridges (#2953). The main purpose of this method is to train a prefix to compress a long context to a short size and thus save on tokens. On a low level, this is similar to prefix tuning. The PR also added an example recipe to quickly get started.
"PVeRA: Probabilistic Vector-Based Random Matrix Adaptation" was added to PEFT by @leofillioux in #2952. It is an extension of VeRA, a PEFT method that uses weight sharing between layers to be especially parameter efficient. PVeRA builds on top of that by adding a probabilistic element, sampling from the shared parameters and promising better performance overall.
@fei407 added PSOFT, "Efficient Orthogonal Fine-Tuning with Principal Subspace Adaptation", to PEFT in #3037. Orthogonal fine-tuning techniques like OFT and BOFT are good at preserving the structure and thus capabilities of the underlying base model. PSOFT improves efficiency of this technique by constraining the adaptation to low-rank principal subspace.
@yibozhong added Lily: "Low-Rank Interconnected Adaptation across Layers" to PEFT in #2563. Lily is on the surface similar to LoRA but has a sophisticated parameter sharing scheme. The A parameters are shared blockwise (e.g. 4 consecutive q_proj layers share the same A). There is a pool of B parameters that is shared globally, the actual B's are chosen in a data-dependent way through a router. This allows Lily to use higher ranks than LoRA while maintaining a low trainable parameter count.
In #3084, "PEANuT: Parameter-Efficient Adaptation with Weight-aware Neural Tweakers" was added to PEFT, again by @yibozhong. PEANuT adds a small, neural net (so called weight-aware neural tweakers) to the base model. Compared to LoRA, this increases expressivity for the same trainable parameter count or allows to greatly lower the parameter count without sacrificing expressivity. This comes at the expensive of a higher memory requirement for the same parameter count and decreased speed.
We have another serial contributor in @kashif, who also contributed TinyLoRA: "Learning to Reason in 13 Parameters" in #3024. This is a PEFT method that allows to train an extremely small number of parameters, much lower than what could be achieved even with LoRA rank 1. The paper shows that in particular with reinforcement learning, it can often be enough to train just a few parameters to achieve good results.
@LonglongaaaGo added "AdaMSS: Adaptive Multi-Subspace Approach for Parameter-Efficient Fine-Tuning" to PEFT. This method segments the base weights of the model into smaller subspaces that are targeted for fine-tuning. Moreover, it's possible to dynamically assign a lower parameter budget to less important subspaces during training, similar to what AdaLoRA does. This promises to provide higher expressiveness and better generalization than similar PEFT methods.
In #2939, we added functions to PEFT to allow converting checkpoints of many non-LoRA methods into LoRA checkpoints. This can be useful because many other packages support only LoRA but not other PEFT methods, e.g. Diffusers and vLLM. With the new conversions tools, more PEFT methods than just LoRA can thus be used with those packages. Conversion is lossy but empirical testing showed that with a sufficiently high LoRA rank, the error can be quite low.
@sambhavnoobcoder added a new way to initialize LoRA weights with "LoRA-GA: Low-Rank Adaptation with Gradient Approximation" (#2926). This allows you to initialize the LoRA weights in a way that aligns the gradients with full fine-tuning and should lead to faster training convergence.
In "LoRA vs Full Fine-tuning: An Illusion of Equivalence", the authors showed that LoRA fine-tuning can introduce so-called "intruder dimensions" which contribute to forgetting. We now have a utility function to remove intruder dimension in PEFT, reduce_intruder_dimension. When calling this on a fine-tuned LoRA model, forgetting should be reduced while the fine-tuned task performance should remain almost the same.
In #3048, @balvisio added support for Transformer Engine, a quantization method by NVIDIA, to PEFT.
In a series of PRs (#3079, #3091, #3096), @michaelbenayoun added support for Tensor Parallelism to LoRA.
In many LLMs, the embedding and the LM head have tied weights to save on parameter count. This can, however, lead to tricky situations when trying to fine-tune those layers. Through a series of PRs (#2803, #2922, #2870, #2879, #3126), we improved the user experience when doing so. Most notably, users can now pass ensure_weight_tying=True to their PEFT config to force weight tying to be upheld. Please check the PEFT weight tying docs for how weight tying is now being handled. Thanks to @romitjain, @sambhavnoobcoder, and @Cursx for their contributions.
#3055 makes LoRA work with base models that use very low precision floats like torch.float8_e4m3fn. An example of that would be MiniMax-M2.5.
#3128 introduces zero init to Prefix Tuning which, according to our benchmarks, reduced the result variance significantly and yielded good task accuracy without the need for prompt engineering.
With #3088 the LoftQ implementation now supports correcting errors for int8 quantization without utilizing activation thresholding alongside the already existing nf4 quantization.
The Bone PEFT method was removed in #3115. Users are directed to use MiSS instead, which is the improved replacement for Bone. Use this Bone-to-MiSS conversion script if you want to port old Bone checkpoints.
These two quantization methods now use GPTQModel as their backend (#2932) thanks to @ZX-ModelCloud.
requires_grad in modules_to_savePreviously, PEFT would enable requires_grad on the original module if the corresponding modules_to_save was disabled. This is almost never desirable and was thus fixed. Although this change is technically backwards-incompatible, it's an extreme niche case, so we don't expect any user to be negatively affected by it.
no_split_modules now captures values recursively by @githubnemo in #3032Note truncated.
Small patch release containing the following changes:
Small patch release containing the following changes:
FIX: DynamicCache key_cache attribute deprecation by @BenjaminBossan in https://github.com/huggingface/peft/pull/2737
<img width="1248" height="560" alt="peft-v0 18 0" src="https://github.com/user-attachments/assets/5f1f58d8-351a-456d-a491-1d6b6f1e4590" />
FIXME update list of all changes, so some more commits were added
@ppetrushkov added RoAd: 2D Rotary Adaptation to PEFT in #2678. RoAd learns 2D rotation matrices that are applied using only element-wise multiplication, thus promising very fast inference with adapters in unmerged state.
Remarkably, besides LoRA, RoAd is the only PEFT method that supports mixed adapter batches. This means that when you have loaded a model with multiple RoAd adapters, you can use all of them for different samples in the same batch, which is much more efficient than switching adapters between batches:
model = PeftModel.from_pretrained(base_model, <path-to-road-adapter-A>, adapter_name="adapter-A")
model.add_adapter("adapter-B", <path-to-road-adapter-B>)
inputs = ... # input with 3 samples
# apply adapter A to sample 0, adapter B to sample 1, and use the base model for sample 2:
adapter_names = ["adapter-A", "adapter-B", "__base__"]
output_mixed = model(**inputs, adapter_names=adapter_names)
gen_mixed = model.generate(**inputs, adapter_names=adapter_names)
Activated LoRA is a technique added by @kgreenewald in #2609 for causal language models, allowing to selectively enable LoRA adapters depending on a specific token invocation sequence in the input. This has the major benefit of being able to re-use most of the KV cache during inference when the adapter is only used to generate part of the response, after which the base model takes over again.
@TheTahaaa contributed not only support for Arrow, a dynamic routing algorithm between multiple loaded LoRAs in #2644, but also GenKnowSub, a technique built upon Arrow where the 'library' of LoRAs available to Arrow is first modified by subtracting general knowledge adapters (e.g., trained on subsets of Wikipedia) to enhance task-specific performance.
Thanks to @Bilican, Wavelet Fine-Tuning (WaveFT) was added to PEFT in #2560. This method trains sparse updates in the wavelet domain of residual matrices, which is especially parameter efficient. It is very interesting for image generation, as it promises to generate diverse outputs while preserving subject fidelity.
Decoupled Low-rank Adaptation (DeLoRA) was added by @mwbini in #2780. This new PEFT method is similar to DoRA in so far as it decouples the angle and magnitude of the learned adapter weights. However, DeLoRA implements this in a way that promises to better prevent divergence. Moreover, it constrains the deviation of the learned weight by imposing an upper limit of the norm, which can be adjusted via the delora_lambda parameter.
Orthogonal Fine-Tuning (OSF) was added by @NikhilNayak-debug in #2685. By freezing the high-rank subspace of the targeted weight matrices and projecting gradient updates to a low-rank subspace, OSF achieves good performance on continual learning tasks. While it is a bit memory intensive for standard fine-tuning processes, it is definitely worth checking out on tasks where performance degradation of previously learned tasks is a concern.
In #2525, @ved1beta added the text generation benchmark to PEFT. This is a framework to determine and compare metrics with regard to text generation of different PEFT methods, e.g. runtime and memory usage. Right now, this benchmark is still lacking experimental settings and a visualization, analogous to what we have in the MetaMathQA benchmark. If this is something that interests you, we encourage you to let us know or, even better, contribute to this benchmark.
PEFT has integrations with other libraries like Transformers and Diffusers. To facilitate this integration, PEFT now provides a stable interface of functions that should be used if applicable. For example, the set_adapter function can be used to switch between PEFT adapters on the model, even if the model is not a PeftModel instance. We commit to keeping these functions backwards compatible, so it's safe for other libraries to build on top of those.
Some Transformers models can have tied weights. This is especially prevalent when it comes to the embedding and the LM head. Currently, the way that this is handled in PEFT is not obvious. We thus drafted an issue to illustrate the intended behavior in #2864. This shows what our goal is, although not everything is implemented yet.
In #2803, @romitjain added the ensure_weight_tying argument to LoraConfig. This argument, if set to True, enforces weight tying of the modules targeted with modules_to_save. Thus, if embedding and LM head are tied, they will share weights, which is important to allow, for instance, weight merging. Therefore, for most users, we recommend to enable this setting if they want to fully fine-tune the embedding and LM head. For backwards compatability, the setting is off by default though.
Note that in accordance with #2864, the functionality of ensure_weight_tying=True will be expanded to also include trainable tokens (#2870) and LoRA (tbd.) in the future.
@grewalsk extended LoHa and LoKr to support nn.Conv1d layers, as well as nn.Conv2d with 1x1 kernels, in #2515.
Thanks to @macmacmacmac, we now have a new initialization option for prompt tuning, random discrete initialization (#2815). This option should generally work better than random initialization, as corroborated on our PEFT method comparison suite. Give it a try if you use prompt tuning.
If you use multiple LoRA adapters, you can merge them into a single adapter using model.add_weighted_adapter. However, so far, this only worked with positive weights per adapter. Thanks to @sambhavnoobcoder and @valteu, it is now possible to pass negative weights too.
At the time of writing, the Transformers v5 release is imminent. This Transformers version will be incomptabile with PEFT < 0.18.0. If you plan to use Transformers v5 with PEFT, please upgrade PEFT to 0.18.0+.
This PEFT version no longer supports Python 3.9, which has reached its end of life. Please use Python 3.10+.
The OFT method has been updated to make it slightly faster and to stabilize the numerics in #2805. This means, however, that existing checkpoints may give slightly different results after upgrading to PEFT 0.18.0. Therefore, if you use OFT, we recommend to retrain the adapter.
hub_online_once in trainable token tests by @githubnemo in https://github.com/huggingface/peft/pull/2701to issue for 8-bit model by @yao-matrix in https://github.com/huggingface/peft/pull/2797trainable_token_indices for lm_head by @aflueckiger in https://github.com/huggingface/peft/pull/2863max_length to replace max_seq_length; correct README for by @kaixuanliu in https://github.com/huggingface/peft/pull/2862Full Changelog: https://github.com/huggingface/peft/compare/v0.17.1...v0.18.0
FIXME update list of all changes, so some more commits were added
@ppetrushkov added RoAd: 2D Rotary Adaptation to PEFT in #2678. RoAd learns 2D rotation matrices that are applied using only element-wise multiplication, thus promising very fast inference with adapters in unmerged state.
Remarkably, besides LoRA, RoAd is the only PEFT method that supports mixed adapter batches. This means that when you have loaded a model with multiple RoAd adapters, you can use all of them for different samples in the same batch, which is much more efficient than switching adapters between batches:
model = PeftModel.from_pretrained(base_model, <path-to-road-adapter-A>, adapter_name="adapter-A")
model.add_adapter("adapter-B", <path-to-road-adapter-B>)
inputs = ... # input with 3 samples
# apply adapter A to sample 0, adapter B to sample 1, and use the base model for sample 2:
adapter_names = ["adapter-A", "adapter-B", "__base__"]
output_mixed = model(**inputs, adapter_names=adapter_names)
gen_mixed = model.generate(**inputs, adapter_names=adapter_names)Activated LoRA is a technique added by @kgreenewald in #2609 for causal language models, allowing to selectively enable LoRA adapters depending on a specific token invocation sequence in the input. This has the major benefit of being able to re-use most of the KV cache during inference when the adapter is only used to generate part of the response, after which the base model takes over again.
@TheTahaaa contributed not only support for Arrow, a dynamic routing algorithm between multiple loaded LoRAs in #2644, but also GenKnowSub, a technique built upon Arrow where the 'library' of LoRAs available to Arrow is first modified by subtracting general knowledge adapters (e.g., trained on subsets of Wikipedia) to enhance task-specific performance.
Thanks to @Bilican, Wavelet Fine-Tuning (WaveFT) was added to PEFT in #2560. This method trains sparse updates in the wavelet domain of residual matrices, which is especially parameter efficient. It is very interesting for image generation, as it promises to generate diverse outputs while preserving subject fidelity.
Decoupled Low-rank Adaptation (DeLoRA) was added by @mwbini in #2780. This new PEFT method is similar to DoRA in so far as it decouples the angle and magnitude of the learned adapter weights. However, DeLoRA implements this in a way that promises to better prevent divergence. Moreover, it constrains the deviation of the learned weight by imposing an upper limit of the norm, which can be adjusted via the delora_lambda parameter.
Orthogonal Fine-Tuning (OSF) was added by @NikhilNayak-debug in #2685. By freezing the high-rank subspace of the targeted weight matrices and projecting gradient updates to a low-rank subspace, OSF achieves good performance on continual learning tasks. While it is a bit memory intensive for standard fine-tuning processes, it is definitely worth checking out on tasks where performance degradation of previously learned tasks is a concern.
In #2525, @ved1beta added the text generation benchmark to PEFT. This is a framework to determine and compare metrics with regard to text generation of different PEFT methods, e.g. runtime and memory usage. Right now, this benchmark is still lacking experimental settings and a visualization, analogous to what we have in the MetaMathQA benchmark. If this is something that interests you, we encourage you to let us know or, even better, contribute to this benchmark.
PEFT has integrations with other libraries like Transformers and Diffusers. To facilitate this integration, PEFT now provides a stable interface of functions that should be used if applicable. For example, the set_adapter function can be used to switch between PEFT adapters on the model, even if the model is not a PeftModel instance. We commit to keeping these functions backwards compatible, so it's safe for other libraries to build on top of those.
Some Transformers models can have tied weights. This is especially prevalent when it comes to the embedding and the LM head. Currently, the way that this is handled in PEFT is not obvious. We thus drafted an issue to illustrate the intended behavior in #2864. This shows what our goal is, although not everything is implemented yet.
In #2803, @romitjain added the ensure_weight_tying argument to LoraConfig. This argument, if set to True, enforces weight tying of the modules targeted with modules_to_save. Thus, if embedding and LM head are tied, they will share weights, which is important to allow, for instance, weight merging. Therefore, for most users, we recommend to enable this setting if they want to fully fine-tune the embedding and LM head. For backwards compatability, the setting is off by default though.
Note that in accordance with #2864, the functionality of ensure_weight_tying=True will be expanded to also include trainable tokens (#2870) and LoRA (tbd.) in the future.
@grewalsk extended LoHa and LoKr to support nn.Conv1d layers, as well as nn.Conv2d with 1x1 kernels, in #2515.
Thanks to @macmacmacmac, we now have a new initialization option for prompt tuning, random discrete initialization (#2815). This option should generally work better than random initialization, as corroborated on our PEFT method comparison suite. Give it a try if you use prompt tuning.
If you use multiple LoRA adapters, you can merge them into a single adapter using model.add_weighted_adapter. However, so far, this only worked with positive weights per adapter. Thanks to @sambhavnoobcoder and @valteu, it is now possible to pass negative weights too.
At the time of writing, the Transformers v5 release is imminent. This Transformers version will be incomptabile with PEFT < 0.18.0. If you plan to use Transformers v5 with PEFT, please upgrade PEFT to 0.18.0+.
This PEFT version no longer supports Python 3.9, which has reached its end of life. Please use Python 3.10+.
The OFT method has been updated to make it slightly faster and to stabilize the numerics in #2805. This means, however, that existing checkpoints may give slightly different results after upgrading to PEFT 0.18.0. Therefore, if you use OFT, we recommend to retrain the adapter.
hub_online_once in trainable token tests by @githubnemo in #2701to issue for 8-bit model by @yao-matrix in #2797Note truncated.
Adds tag v0.18.0.rc0 for pypi
Adds tag v0.18.0.rc0 for pypi
This patch release contains a few fixes (via #2710) for the newly introduced `target_parameters` feature, which allows LoRA to target nn.Parameters di
This patch release contains a few fixes (via #2710) for the newly introduced target_parameters feature, which allows LoRA to target nn.Parameters directly (useful for mixture of expert layers). Most notably:
model.add_adapter or model.load_adapter) did not work correctly. Since a solution is not trivial, PEFT now raises an error to prevent this situation.At the same time, Bone will be deprecated in favor of MiSS and will be removed in PEFT v0.19.0. If you already have a Bone checkpoint, you can use `sc…
<img width="1248" height="560" alt="peft-v0 17 0" src="https://github.com/user-attachments/assets/a206c099-10ee-4c13-80c1-0de7ed1df5cf" />
@kkb-code contributed Sparse High Rank Adapters (SHiRA, paper) which promise to offer a potential gain in performance over LoRAs - especially the concept loss when using multiple adapters is improved. Since the adapters only train on 1-2% of the weights and are inherently sparse, switching between adapters may be cheaper than with LoRAs. (#2584)
@JL-er added a new PEFT method, MiSS (Matrix Shard Sharing) in #2604. This method is an evolution of Bone, which, according to our PEFT method comparison benchmark, gives excellent results when it comes to performance and memory efficiency. If you haven't tried it, you should do so now.
At the same time, Bone will be deprecated in favor of MiSS and will be removed in PEFT v0.19.0. If you already have a Bone checkpoint, you can use scripts/convert-bone-to-miss.py to convert it into a MiSS checkpoint and proceed with training using MiSS.
nn.ParameterLoRA is now able to target nn.Parameter directly (#2638, #2665)! Ever had this complicated nn.Module with promising parameters inside but it was too custom to be supported by your favorite fine-tuning library? No worries, now you can target nn.Parameters directly using the target_parameters config attribute which works similarly to target_modules.
This option can be especially useful for models with Mixture of Expert (MoE) layers, as those often use nn.Parameters directly and cannot be targeted with target_modules. For example, for the Llama4 family of models, use the following config to target the MoE weights:
config = LoraConfig(
...,
target_modules=[], # <= prevent targeting any modules
target_parameters=["feed_forward.experts.down_proj", "feed_forward.experts.gate_up_proj"],
)
Note that this feature is still experimental as it comes with a few caveats and therefore might change in the future. Also, MoE weights with many experts can be quite huge, so expect a higher memory usage than compared to targeting normal nn.Linear layers.
state_dictSometimes, it is possible that there is a PEFT adapter checkpoint but the corresponding PEFT config is not known for whatever reason. To inject the PEFT layers for this checkpoint, you would usually have to reverse-engineer the corresponding PEFT config, most notably the target_modules argument, based on the state_dict from the checkpoint. This can be cumbersome and error prone. To avoid this, it is also possible to call inject_adapter_in_model and pass the loaded state_dict as an argument:
from safetensors.torch import load_file
from peft import LoraConfig, inject_adapter_in_model
model = ...
state_dict = load_file(<path-to-safetensors-file>)
lora_config = LoraConfig() # <= no need to specify further
model = inject_adapter_in_model(lora_config, model, state_dict=state_dict)
Find more on state_dict based injection in the docs.
A bug in prompt learning methods caused modules_to_save to be ignored. Especially classification tasks are affected since they usually add the classification/score layer to modules_to_save. In consequence, these layers were neither trained nor stored after training. This has been corrected now. (#2646)
Full Changelog: https://github.com/huggingface/peft/compare/v0.16.0...v0.17.0
@kkb-code contributed Sparse High Rank Adapters (SHiRA, paper) which promise to offer a potential gain in performance over LoRAs - especially the concept loss when using multiple adapters is improved. Since the adapters only train on 1-2% of the weights and are inherently sparse, switching between adapters may be cheaper than with LoRAs. (#2584)
@JL-er added a new PEFT method, MiSS (Matrix Shard Sharing) in #2604. This method is an evolution of Bone, which, according to our PEFT method comparison benchmark, gives excellent results when it comes to performance and memory efficiency. If you haven't tried it, you should do so now.
At the same time, Bone will be deprecated in favor of MiSS and will be removed in PEFT v0.19.0. If you already have a Bone checkpoint, you can use scripts/convert-bone-to-miss.py to convert it into a MiSS checkpoint and proceed with training using MiSS.
nn.ParameterLoRA is now able to target nn.Parameter directly (#2638, #2665)! Ever had this complicated nn.Module with promising parameters inside but it was too custom to be supported by your favorite fine-tuning library? No worries, now you can target nn.Parameters directly using the target_parameters config attribute which works similarly to target_modules.
This option can be especially useful for models with Mixture of Expert (MoE) layers, as those often use nn.Parameters directly and cannot be targeted with target_modules. For example, for the Llama4 family of models, use the following config to target the MoE weights:
config = LoraConfig(
...,
target_modules=[], # <= prevent targeting any modules
target_parameters=["feed_forward.experts.down_proj", "feed_forward.experts.gate_up_proj"],
)Note that this feature is still experimental as it comes with a few caveats and therefore might change in the future. Also, MoE weights with many experts can be quite huge, so expect a higher memory usage than compared to targeting normal nn.Linear layers.
state_dictSometimes, it is possible that there is a PEFT adapter checkpoint but the corresponding PEFT config is not known for whatever reason. To inject the PEFT layers for this checkpoint, you would usually have to reverse-engineer the corresponding PEFT config, most notably the target_modules argument, based on the state_dict from the checkpoint. This can be cumbersome and error prone. To avoid this, it is also possible to call inject_adapter_in_model and pass the loaded state_dict as an argument:
from safetensors.torch import load_file
from peft import LoraConfig, inject_adapter_in_model
model = ...
state_dict = load_file(<path-to-safetensors-file>)
lora_config = LoraConfig() # <= no need to specify further
model = inject_adapter_in_model(lora_config, model, state_dict=state_dict)Find more on state_dict based injection in the docs.
A bug in prompt learning methods caused modules_to_save to be ignored. Especially classification tasks are affected since they usually add the classification/score layer to modules_to_save. In consequence, these layers were neither trained nor stored after training. This has been corrected now. (#2646)
Full Changelog: v0.16.0...v0.17.0
Your coding agent can read these notes before it upgrades. Set up the MCP server →