NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
PyPI, #775 most downloaded on PyPI
Embeddings, Retrieval, and Reranking
Latest stable
6.2.0
Last release 26 days ago
21 Oct 2026
Ships fairly regularly
a new release about every 3 weeks
Nearly every release is documented
notes for 55 of the last 60 stable releases
1 version withdrawn
withdrawn after publishing
7 years old
85 releases · first in 2019
Drag across the ruler to compare, or tap a release to open it.
This patch release introduces some improvements for the SentenceTransformerTrainer, as well as some updates for the automatic model card generation. I
This patch release introduces some improvements for the SentenceTransformerTrainer, as well as some updates for the automatic model card generation. It also patches some minor evaluator bugs and a bug with MatryoshkaLoss. Lastly, every single Sentence Transformer model can now be saved and loaded with the safer model.safetensors files.
Install this version with
# Full installation:
pip install sentence-transformers[train]==3.0.1
# Inference only:
pip install sentence-transformers==3.0.1
push_to_hub=True Training Argument, also implement trainer.push_to_hub(...) (#2718)This patch release improves on the automatically generated model cards in several ways:
generated_from_trainer tag is now also added (#2710)...
# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
...
This now gets replaced with your new model ID on Hugging Face (#2714)
SequentialEvaluator would be ignored in the scores calculation (#2700)print_wrong_matches=True (#1894)primary_metric in InformationRetrievalEvaluator (#2701)SequentialEvaluator (#2717)MatryoshkaLoss crash if the first dimension is not the biggest (#2719)pytorch_model.bin anymore (#2722)fix] Always override the originally saved version in the ST config by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2709model cards] Also include HF datasets in the model card metadata by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2711model cards] Improve the widget example selection: not based on embeddings, better for QA by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2713model cards] Replace 'sentence_transformers_model_id' from reused model if possible by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2714feat] Allow passing a list of evaluators to the Trainer by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2716fix] Fix gradient checkpointing to allow for much lower memory usage by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2717fix] Implement create_model_card on the Trainer, allowing args.push_to_hub=True by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2718fix] Fix MatryoshkaLoss crash if the first dimension is not the biggest by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2719typing] Improve typing for many functions & add py.typed to satisfy mypy by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2724fix] Fix edge case with evaluator being None by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2726simplify] Set can_return_loss=True globally, instead of via the data collator by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2727feat] Integrate safetensors with Dense, etc. modules too. by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2722model cards] Specify the exact dataset size as a tag, will be bucketized by HF by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2728Full Changelog: https://github.com/UKPLab/sentence-transformers/compare/v3.0.0...v3.0.1
Your coding agent can read these notes before it upgrades. Set up the MCP server →
…`SentenceTransformer.fit` method. Rather than deprecating this method, starting from v3.0, this method will use the `SentenceTransformerTrainer` behin…
This release consists of a major refactor that overhauls the training approach (introducing multi-gpu training, bf16, loss logging, callbacks, and much more), adds convenient similarity and similarity_pairwise methods, adds extra keyword arguments, introduces Hyperparameter Optimization, and includes a massive reformatting and release of 50+ datasets for training embedding models. In total, this is the largest Sentence Transformers update since the project was first created.
Install this version with
# Full installation:
pip install sentence-transformers[train]==3.0.0
# Inference only:
pip install sentence-transformers==3.0.0
The v3.0 release centers around this huge modernization of the training approach for SentenceTransformer models. Whereas training before v3.0 used to be all about InputExample, DataLoader and model.fit, the new training approach relies on 5 new components. You can learn more about these components in our Training and Finetuning Embedding Models with Sentence Transformers v3 blogpost. Additionally, you can read the new Training Overview, check out the Training Examples, or read this summary:
Dataset or DatasetDict. This class is much more suited for sharing & efficient modifications than lists/DataLoaders of InputExample instances. A Dataset can contain multiple text columns that will be fed in order to the corresponding loss function. So, if the loss expects (anchor, positive, negative) triplets, then your dataset should also have 3 columns. The names of these columns are irrelevant. If there is a "label" or "score" column, it is treated separately, and used as the labels during training.
A DatasetDict can be used to train with multiple datasets at once, e.g.:DatasetDict({
multi_nli: Dataset({
features: ['premise', 'hypothesis', 'label'],
num_rows: 392702
})
snli: Dataset({
features: ['snli_premise', 'hypothesis', 'label'],
num_rows: 549367
})
stsb: Dataset({
features: ['sentence1', 'sentence2', 'label'],
num_rows: 5749
})
})
When a DatasetDict is used, the loss parameter to the SentenceTransformerTrainer must also be a dictionary with these dataset keys, e.g.:{
'multi_nli': SoftmaxLoss(...),
'snli': SoftmaxLoss(...),
'stsb': CosineSimilarityLoss(...),
}
SentenceEvaluator instance. Unlike before, models can now be evaluated both on an evaluation dataset with some loss function and/or a SentenceEvaluator instance.SentenceTransformersTrainer instance based on the transformers Trainer. This instance is provided with a SentenceTransformer model, a SentenceTransformerTrainingArguments class, a SentenceEvaluator, a training and evaluation Dataset/DatasetDict and a loss function/dict of loss functions. Most of these parameters are optional. Once provided, all you have to do is call trainer.train().Some of the major features that are now implemented include:
This script is a minimal example (no evaluator, no training arguments) of training mpnet-base on a part of the all-nli dataset using MultipleNegativesRankingLoss:
from datasets import load_dataset
from sentence_transformers import SentenceTransformer, SentenceTransformerTrainer
from sentence_transformers.losses import MultipleNegativesRankingLoss
# 1. Load a model to finetune
model = SentenceTransformer("microsoft/mpnet-base")
# 2. Load a dataset to finetune on
dataset = load_dataset("sentence-transformers/all-nli", "triplet")
train_dataset = dataset["train"].select(range(10_000))
eval_dataset = dataset["dev"].select(range(1_000))
# 3. Define a loss function
loss = MultipleNegativesRankingLoss(model)
# 4. Create a trainer & train
trainer = SentenceTransformerTrainer(
model=model,
train_dataset=train_dataset,
eval_dataset=eval_dataset,
loss=loss,
)
trainer.train()
# 5. Save the trained model
model.save_pretrained("models/mpnet-base-all-nli")
Additionally, trained models now automatically produce extensive model cards. Each of the following models were trained using some script from the Training Examples, and the model cards were not edited manually whatsoever:
Prior to the Sentence Transformer v3 release, all models would be trained using the SentenceTransformer.fit method. Rather than deprecating this method, starting from v3.0, this method will use the SentenceTransformerTrainer behind the scenes. This means that your old training code should still work, and should even be upgraded with the new features such as multi-gpu training, loss logging, etc. That said, the new training approach is much more powerful, so it is recommended to write new training scripts using the new approach.
Many of the old training scripts were updated to use the new Trainer-based approach, but not all have been updated yet. We accept help via Pull Requests to assist in updating the scripts.
Sentence Transformers v3.0 introduces two new useful methods:
and one property:
These can be used to calculate the similarity between embeddings, and to specify which similarity function should be used, for example:
>>> from sentence_transformers import SentenceTransformer
>>> model = SentenceTransformer("all-mpnet-base-v2")
>>> sentences = [
... "The weather is so nice!",
... "It's so sunny outside.",
... "He's driving to the movie theater.",
... "She's going to the cinema.",
... ]
>>> embeddings = model.encode(sentences, normalize_embeddings=True)
>>> model.similarity(embeddings, embeddings)
tensor([[1.0000, 0.7235, 0.0290, 0.1309],
[0.7235, 1.0000, 0.0613, 0.1129],
[0.0290, 0.0613, 1.0000, 0.5027],
[0.1309, 0.1129, 0.5027, 1.0000]])
>>> model.similarity_fn_name
"cosine"
>>> model.similarity_fn_name = "euclidean"
>>> model.similarity(embeddings, embeddings)
tensor([[-0.0000, -0.7437, -1.3935, -1.3184],
[-0.7437, -0.0000, -1.3702, -1.3320],
[-1.3935, -1.3702, -0.0000, -0.9973],
[-1.3184, -1.3320, -0.9973, -0.0000]])
Additionally, you can compute the similarity between pairs of embeddings, resulting in a 1-dimensional vector of similarities rather than a 2-dimensional matrix:
>>> model = SentenceTransformer("all-mpnet-base-v2")
>>> sentences = [
... "The weather is so nice!",
... "It's so sunny outside.",
... "He's driving to the movie theater.",
... "She's going to the cinema.",
... ]
>>> embeddings = model.encode(sentences, normalize_embeddings=True)
>>> model.similarity_pairwise(embeddings[::2], embeddings[1::2])
tensor([0.7235, 0.5027])
>>> model.similarity_fn_name
"cosine"
>>> model.similarity_fn_name = "euclidean"
>>> model.similarity_pairwise(embeddings[::2], embeddings[1::2])
tensor([-0.7437, -0.9973])
The similarity_fn_name can now be specified via the SentenceTransformer like so:
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("sentence-transformers/multi-qa-mpnet-base-dot-v1", similarity_fn_name="dot")
Valid options include "cosine" (default), "dot", "euclidean", "manhattan". The chosen similarity_fn_name will also be saved into the model configuration, and loaded automatically. For example, the msmarco-distilbert-dot-v5 model was trained to work best with dot, so we've configured it to use that similarity_fn_name in its configuration:
>>> from sentence_transformers import SentenceTransformer
>>> model = SentenceTransformer("sentence-transformers/msmarco-distilbert-dot-v5")
>>> model.similarity_fn_name
'dot'
Big thanks to @ir2718 for helping set up this major feature.
model_kwargs, tokenizer_kwargs, and config_kwargs to SentenceTransformer (#2578)To those familiar with the internals of Sentence Transformers, you might know that internally, we call AutoModel.from_pretrained, AutoTokenizer.from_pretrained and AutoConfig.from_pretrained from transformers.
Each of these are rather powerful, and they are constantly improved with new features. For example, the AutoModel keyword arguments include:
torch_dtype - this allows you to immediately load a model in bfloat16 or float16 (or "auto", i.e. whatever the model was stored in), which can speed up inference a lot.quantization_configattn_implementation - all models support "eager", but some also support the much faster "fa2" (Flash Attention 2) and "sdpa" (Scaled Dot Product Attention).These options allow for speeding up the model inference. Additionally, via AutoConfig you can update the model configuration, e.g. updating the dropout probability during training, and with AutoTokenizer you can disable the fast Rust-based tokenizer if you're having issues with it via use_fast=False.
Due to how useful these options can be, the following arguments are added to SentenceTransformer:
model_kwargs for AutoModel.from_pretrained keyword argumentstokenizer_kwargs for AutoTokenizer.from_pretrained keyword argumentsconfig_kwargs for AutoConfig.from_pretrained keyword argumentsYou can use it like so:
from sentence_transformers import SentenceTransformer
import torch
model = SentenceTransformer(
"mixedbread-ai/mxbai-embed-large-v1",
model_kwargs={"torch_dtype": torch.bfloat16, "attn_implementation": "sdpa"},
config_kwargs={"hidden_dropout_prob": 0.3},
)
embeddings = model.encode(["He drove his yellow car to the beach.", "He played football with his friends."])
print(embeddings.shape)
Big thanks to @satyamk7054 for starting this work.
Sentence Transformers v3.0 introduces Hyperparameter Optimization (HPO) by extending the transformers HPO support. We recommend reading the all new Hyperparameter Optimization for many more details.
Alongside Sentence Transformers v3.0, we reformat and release 50+ useful datasets in our Embedding Model Datasets Collection on Hugging Face. These can be used with at least one loss function in Sentence Transformers v3.0 out of the box. We recommend browsing through these to see if there are datasets akin to your use cases - training a model on them might just produce large gains on your task(s).
The MSELoss now accepts multiple text columns for each label (where each label is a target/gold embedding), rather than only accepting one text column. This is extremely powerful for following the excellent Multilingual Models strategy to convert a monolingual model into a multilingual one. You can now conveniently train both English and (identical but translated) non-English texts to represent the same embedding (that was generated by a powerful English embedding model).
local_files_only argument to SentenceTransformer & CrossEncoder (#2603)You can now initialize a SentenceTransformer and CrossEncoder with local_files_only. If True, then it will not try and download a model from Hugging Face, it will only look in the local filesystem for the model or try and load it from a cache.
Thanks @debanjum for this change.
v3] Training refactor - MultiGPU, loss logging, bf16, etc. by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2449v3] Add similarity and similarity_pairwise methods to Sentence Transformers by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2615v3] Fix various model card errors by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2616v3] Fix trainer compute_loss when evaluating/predicting if the loss updated the inputs in-place by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2617v3] Never return None in infer_datasets, could result in crash by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2620v3] Trainer: Implement resume from checkpoint support by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2621trust_remote_code to CrossEncoder.tokenizer by @michaelfeil in https://github.com/UKPLab/sentence-transformers/pull/2623v3] Update example scripts to the new v3 training format by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2622v3] Remove "return_outputs" as it's not strictly necessary. Avoids OOM & speeds up training by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2633v3] Fix crash from inferring the dataset_id from a local dataset by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2636v3] Fix multilingual conversion script; extend MSELoss to multi-column by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2641v3] Update evaluation scripts to use HF Datasets by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2642b1 quantization for USearch by @ashvardanian in https://github.com/UKPLab/sentence-transformers/pull/2644v3] Fix resume_from_checkpoint by also updating the loss model by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2648v3] Fix backwards pass on MSELoss due to in-place update by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2647v3] Simplify load_from_checkpoint using load_state_dict by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2650v3] Use torch.arange instead of torch.tensor(range(...)) by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2651v3] Resolve inplace modification error in DDP by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2654v3] Add hyperparameter optimization support by letting loss be a Callable that accepts a model by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2655v3] Add tag hinting at the number of training samples by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2660v3] For the Cached losses; ignore gradients if grad is disabled (e.g. eval) by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2668docs] Rewrite the https://sbert.net documentation by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2632v3] Chore - include import sorting in ruff by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2672v3] Prevent warning with 'model.fit' with transformers >= 4.41.0 due to evaluation_strategy by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2673v3] Add various useful Sphinx packages (copy code, link to code, nicer tabs) by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2674v3] Make the "primary_metric" for evaluators a bit more robust by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2675v3] Set broadcast_buffers = False when training with DDP by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2663v3] Warn about using DP instead of DDP + set dataloader_drop_last with DDP by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2677v3] Add warning that Evaluators only run on 1 GPU when multi-GPU training by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2678v3] Move training dependencies into a "train" extra by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2676v3] Docs: update references to the API reference by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2679v3] Add "dataset_size:" to the tag denoting the number of training samples by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2680A special shoutout to @Jakobhenningjensen, @smerrill, @b5y, @ScottishFold007, @pszemraj, @bwanglzu, @igorkurinnyi, for experimenting with the v3.0 release prior to release and @matthewfranglen for the initial work on the training refactor back in October of 2022 in #1733.
cc @AlexJonesNLP as I know you are interested in this release!
Full Changelog: https://github.com/UKPLab/sentence-transformers/compare/v2.7.0...v3.0.0
This release introduces a new promising loss function, easier inference for Matryoshka models, new functionality for CrossEncoders and Inference on In
This release introduces a new promising loss function, easier inference for Matryoshka models, new functionality for CrossEncoders and Inference on Intel Gaudi2, along much more.
Install this version with
pip install sentence-transformers==2.7.0
For a number of years, MultipleNegativesRankingLoss (also known as SimCSE, InfoNCE, in-batch negatives loss) has been the state of the art in embedding model training. Notably, this loss function performs better with a larger batch size.
Recently, various improvements have been introduced:
CachedMultipleNegativesRankingLoss was introduced, which allows you to pick much higher batch sizes (e.g. 65536) with constant memory.GISTEmbedLoss takes a guide model to guide the in-batch negative sample selection. This prevents false negatives, resulting in a stronger training signal.Now, @JacksonCakes has combined these two approaches to produce the best of both worlds: CachedGISTEmbedLoss. This loss function allows for high batch sizes with constant memory usage, while also using a guide model to assist with the in-batch negative sample selection.
As can be seen in our Loss Overview, this model should be used with (anchor, positive) pairs or (anchor, positive, negative) triplets, much like MultipleNegativesRankingLoss, CachedMultipleNegativesRankingLoss, and GISTEmbedLoss. In short, any example using those loss functions can be updated to use CachedGISTEmbedLoss! Feel free to experiment, e.g. with this training script.
Sentence Transformers v2.4.0 introduced Matryoshka models: models whose embeddings are still useful after truncation. Since then, many useful Matryoshka models have been trained.
As of this release, the truncation for these Matryoshka embedding models can be done automatically via a new truncate_dim constructor argument:
from sentence_transformers import SentenceTransformer
from sentence_transformers.util import cos_sim
matryoshka_dim = 64
model = SentenceTransformer("nomic-ai/nomic-embed-text-v1.5", trust_remote_code=True, truncate_dim=matryoshka_dim)
embeddings = model.encode(
[
"search_query: What is TSNE?",
"search_document: t-distributed stochastic neighbor embedding (t-SNE) is a statistical method for visualizing high-dimensional data by giving each datapoint a location in a two or three-dimensional map.",
"search_document: Amelia Mary Earhart was an American aviation pioneer and writer.",
]
)
print(embeddings.shape)
# => [3, 64]
similarities = cos_sim(embeddings[0], embeddings[1:])
# => tensor([[0.7839, 0.4933]])
Extra information:
Alongside easier inference with Matryoshka models, evaluating them is now also much easier. You can also pass truncate_dim to any Evaluator. This way you can easily check the performance of any Sentence Transformer model at various truncated dimensions (even if the model was not trained with MatryoshkaLoss!)
from sentence_transformers.evaluation import EmbeddingSimilarityEvaluator
from sentence_transformers import SentenceTransformer
import datasets
model = SentenceTransformer("tomaarsen/mpnet-base-nli-matryoshka")
stsb = datasets.load_dataset("mteb/stsbenchmark-sts", split="test")
for dim in [768, 512, 256, 128, 64, 32, 16, 8, 4]:
evaluator = EmbeddingSimilarityEvaluator(
stsb["sentence1"],
stsb["sentence2"],
[score / 5 for score in stsb["score"]],
name=f"sts-test-{dim}",
truncate_dim=dim,
)
print(f"dim={dim:<3}: {evaluator(model) * 100:.2f} Spearman Correlation")
dim=768: 86.81 Spearman Correlation
dim=512: 86.76 Spearman Correlation
dim=256: 86.66 Spearman Correlation
dim=128: 86.20 Spearman Correlation
dim=64 : 85.40 Spearman Correlation
dim=32 : 82.42 Spearman Correlation
dim=16 : 79.31 Spearman Correlation
dim=8 : 72.82 Spearman Correlation
dim=4 : 63.44 Spearman Correlation
Here are some example training scripts that use this new truncate_dim option to assist with training Matryoshka models:
This release improves the support for CrossEncoder reranker models.
push_to_hub (#2524)You can now push trained CrossEncoder models to the 🤗 Hugging Face Hub!
from sentence_transformers import CrossEncoder
...
model = CrossEncoder("distilroberta-base")
# Train the model
model.fit(
train_dataloader=train_dataloader,
evaluator=evaluator,
epochs=num_epochs,
warmup_steps=warmup_steps,
)
model.push_to_hub("tomaarsen/distilroberta-base-stsb-cross-encoder")
CrossEncoder.push_to_hubtrust_remote_code for custom models (#2595)You can now load custom models from the Hugging Face Hub, i.e. models that have custom modelling code that require trust_remote_code to load.
from sentence_transformers import CrossEncoder
# Note: this model does not require `trust_remote_code=True` - there are currently no models that require it yet.
model = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2", trust_remote_code=True)
# We want to compute the similarity between the query sentence
query = "A man is eating pasta."
# With all sentences in the corpus
corpus = [
"A man is eating food.",
"A man is eating a piece of bread.",
"The girl is carrying a baby.",
"A man is riding a horse.",
"A woman is playing violin.",
"Two men pushed carts through the woods.",
"A man is riding a white horse on an enclosed ground.",
"A monkey is playing drums.",
"A cheetah is running behind its prey.",
]
# We rank all sentences in the corpus for the query
ranks = model.rank(query, corpus)
# Print the scores
print("Query:", query)
for rank in ranks:
print(f"{rank['score']:.2f}\t{corpus[rank['corpus_id']]}")
CrossEncoderFrom this release onwards, you will be able to perform inference on Intel Gaudi2 accelerators. No modifications are needed, as the library will automatically detect the hpu device and configure the model accordingly. Thanks to Intel Habana for the support here.
docs] Add simple Makefile for building docs by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2566examples] Add Matryoshka evaluation plot by @kddubey in https://github.com/UKPLab/sentence-transformers/pull/2564push_to_hub to CrossEncoder by @imvladikon in https://github.com/UKPLab/sentence-transformers/pull/2524requirements] Set minimum transformers version to 4.34.0 for is_nltk_available by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2574docs] Update link: retrieve_rerank_simple_wikipedia.py -> .ipynb by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2580feat] Add truncation support by @kddubey in https://github.com/UKPLab/sentence-transformers/pull/2573examples] Add model upload for training_nli_v3 with GISTEmbedLoss by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2584fix] Matryoshka training always patch original forward, and check matryoshka_dims by @kddubey in https://github.com/UKPLab/sentence-transformers/pull/2593docs] Fix search bar on sbert.net by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2597clip] Prevent warning with padding when tokenizing for CLIP by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2599I especially want to thank @JacksonCakes for their excellent CachedGISTEmbedLoss PR and @kddubey for their wonderful PRs surrounding Matryoshka models and general repository housekeeping.
Full Changelog: https://github.com/UKPLab/sentence-transformers/compare/v2.6.1...v2.7.0
This is a patch release to fix a bug in `semantic_search_faiss` and `semantic_search_usearch` that caused the scores to not correspond to the returned
This is a patch release to fix a bug in semantic_search_faiss and semantic_search_usearch that caused the scores to not correspond to the returned corpus indices. Additionally, you can now evaluate embedding models after quantizing their embeddings.
You can now pass precision to the EmbeddingSimilarityEvaluator to evaluate the performance after quantization:
from sentence_transformers import SentenceTransformer
from sentence_transformers.evaluation import EmbeddingSimilarityEvaluator, SimilarityFunction
import datasets
model = SentenceTransformer("all-mpnet-base-v2")
stsb = datasets.load_dataset("mteb/stsbenchmark-sts", split="test")
print("Spearman correlation based on Cosine Similarity on the STS Benchmark test set:")
for precision in ["float32", "uint8", "int8", "ubinary", "binary"]:
evaluator = EmbeddingSimilarityEvaluator(
stsb["sentence1"],
stsb["sentence2"],
[score / 5 for score in stsb["score"]],
main_similarity=SimilarityFunction.COSINE,
name="sts-test",
precision=precision,
)
print(precision, evaluator(model))
Spearman correlation based on Cosine Similarity on the STS Benchmark test set:
float32 0.8342190421330611
uint8 0.8260094846238505
int8 0.8312754408857808
ubinary 0.8244338431442343
binary 0.8244338431442343
Full Changelog: https://github.com/UKPLab/sentence-transformers/compare/v2.6.0...v2.6.1
[deprecation] Deprecate save_to_hub in favor of push_to_hub; add safe_serialization support to push_to_hub by @tomaarsen in https://github.com/UKPLab/…
This release brings embedding quantization: a way to heavily speed up retrieval & other tasks, and a new powerful loss function: GISTEmbedLoss.
Install this version with
pip install sentence-transformers==2.6.0
Embeddings may be challenging to scale up, which leads to expensive solutions and high latencies. However, there is a new approach to counter this problem; it entails reducing the size of each of the individual values in the embedding: Quantization. Experiments on quantization have shown that we can maintain a large amount of performance while significantly speeding up computation and saving on memory, storage, and costs.
To be specific, using binary quantization may result in retaining 96% of the retrieval performance, while speeding up retrieval by 25x and saving on memory & disk space with 32x. Do not underestimate this approach! Read more about Embedding Quantization in our extensive blogpost.
Two forms of quantization exist at this time: binary and scalar (int8). These quantize embedding values from float32 into binary and int8, respectively. For Binary quantization, you can use the following snippet:
from sentence_transformers import SentenceTransformer
from sentence_transformers.quantization import quantize_embeddings
# 1. Load an embedding model
model = SentenceTransformer("mixedbread-ai/mxbai-embed-large-v1")
# 2a. Encode some text using "binary" quantization
binary_embeddings = model.encode(
["I am driving to the lake.", "It is a beautiful day."],
precision="binary",
)
# 2b. or, encode some text without quantization & apply quantization afterwards
embeddings = model.encode(["I am driving to the lake.", "It is a beautiful day."])
binary_embeddings = quantize_embeddings(embeddings, precision="binary")
References:
GISTEmbedLoss, as introduced in Solatorio (2024), is a guided variant of the more standard in-batch negatives (MultipleNegativesRankingLoss) loss. Both loss functions are provided with a list of (anchor, positive) pairs, but while MultipleNegativesRankingLoss uses anchor_i and positive_i as positive pair and all positive_j with i != j as negative pairs, GISTEmbedLoss uses a second model to guide the in-batch negative sample selection.
This can be very useful, because it is plausible that anchor_i and positive_j are actually quite semantically similar. In this case, GISTEmbedLoss would not consider them a negative pair, while MultipleNegativesRankingLoss would. When finetuning MPNet-base on the AllNLI dataset, these are the Spearman correlation based on cosine similarity using the STS Benchmark dev set (higher is better):
The blue line is MultipleNegativesRankingLoss, whereas the grey line is GISTEmbedLoss with the small all-MiniLM-L6-v2 as the guide model. Note that all-MiniLM-L6-v2 by itself does not reach 88 Spearman correlation on this dataset, so this is really the effect of two models (mpnet-base and all-MiniLM-L6-v2) reaching a performance that they could not reach separately.
save_to_hub DeprecationMost codebases that allow for pushing models to the Hugging Face Hub adopt a push_to_hub method instead of a save_to_hub method, and now Sentence Transformers will follow that convention. The push_to_hub method will now be the recommended approach, although save_to_hub will continue to exist for the time being: it will simply call push_to_hub internally.
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("all-mpnet-base-v2")
...
# Train the model
model.fit(
train_objectives=[(train_dataloader, train_loss)],
evaluator=dev_evaluator,
epochs=num_epochs,
evaluation_steps=1000,
warmup_steps=warmup_steps,
)
# Push the model to Hugging Face
model.push_to_hub("tomaarsen/mpnet-base-nli-stsb")
feat] Add 'get_config_dict' method to GISTEmbedLoss for better model cards by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2543deprecation] Deprecate save_to_hub in favor of push_to_hub; add safe_serialization support to push_to_hub by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2544docs] Update return docstring of encode_multi_process by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2548feat] Add binary & scalar embedding quantization support to Sentence Transformers by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2549Full Changelog: https://github.com/UKPLab/sentence-transformers/compare/v2.5.1...v2.6.0
This is a patch release to fix a bug in CrossEncoder.rank that caused the last value to be discarded when using the default top_k=-1.
This is a patch release to fix a bug in CrossEncoder.rank that caused the last value to be discarded when using the default top_k=-1.
CrossEncoder.rank patch:from sentence_transformers.cross_encoder import CrossEncoder
# Pre-trained cross encoder
model = CrossEncoder("cross-encoder/stsb-distilroberta-base")
# We want to compute the similarity between the query sentence
query = "A man is eating pasta."
# With all sentences in the corpus
corpus = [
"A man is eating food.",
"A man is eating a piece of bread.",
"The girl is carrying a baby.",
"A man is riding a horse.",
"A woman is playing violin.",
"Two men pushed carts through the woods.",
"A man is riding a white horse on an enclosed ground.",
"A monkey is playing drums.",
"A cheetah is running behind its prey.",
]
# We rank all sentences in the corpus for the query
ranks = model.rank(query, corpus)
# Print the scores
print("Query:", query)
for rank in ranks:
print(f"{rank['score']:.2f}\t{corpus[rank['corpus_id']]}")
Query: A man is eating pasta.
0.67 A man is eating food.
0.34 A man is eating a piece of bread.
0.08 A man is riding a horse.
0.07 A man is riding a white horse on an enclosed ground.
0.01 The girl is carrying a baby.
0.01 Two men pushed carts through the woods.
0.01 A monkey is playing drums.
0.01 A woman is playing violin.
0.01 A cheetah is running behind its prey.
Previously, the lowest score document would be removed from the output.
examples] Update model repo_id in 2dMatryoshka example by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2515feat] Add get_config_dict to new Matryoshka2dLoss & AdaptiveLayerLoss by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2516chore] Update to ruff 0.3.0; update ruff.toml by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2517example] Don't always normalize the embeddings in clustering example by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2520top_k by @xenova in https://github.com/UKPLab/sentence-transformers/pull/2518Full Changelog: https://github.com/UKPLab/sentence-transformers/compare/v2.5.0...v2.5.1
This release brings two new loss functions, a new way to (re)rank with CrossEncoder models, and more fixes
This release brings two new loss functions, a new way to (re)rank with CrossEncoder models, and more fixes
Install this version with
pip install sentence-transformers==2.5.0
Embedding models are often encoder models with numerous layers, such as 12 (e.g. all-mpnet-base-v2) or 6 (e.g. all-MiniLM-L6-v2). To get embeddings, every single one of these layers must be traversed. 2D Matryoshka Sentence Embeddings (2DMSE) revisits this concept by proposing an approach to train embedding models that will perform well when only using a selection of all layers. This results in faster inference speeds at relatively low performance costs.
For example, using Sentence Transformers, you can train an Adaptive Layer model that can be sped up by 2x at a 15% reduction in performance, or 5x on GPU & 10x on CPU for a 20% reduction in performance. The 2DMSE paper highlights scenarios where this is superior to using a smaller model.
Training with Adaptive Layer support is quite elementary: rather than applying some loss function on only the last layer, we also apply that same loss function on the pooled embeddings from previous layers. Additionally, we employ a KL-divergence loss that aims to make the embeddings of the non-last layers match that of the last layer. This can be seen as a fascinating approach of knowledge distillation, but with the last layer as the teacher model and the prior layers as the student models.
For example, with the 12-layer microsoft/mpnet-base, it will now be trained such that the model produces meaningful embeddings after each of the 12 layers.
from sentence_transformers import SentenceTransformer
from sentence_transformers.losses import CoSENTLoss, AdaptiveLayerLoss
model = SentenceTransformer("microsoft/mpnet-base")
base_loss = CoSENTLoss(model=model)
loss = AdaptiveLayerLoss(model=model, loss=base_loss)
Additionally, this can be combined with the MatryoshkaLoss such that the resulting model can be reduced both in the number of layers, but also in the size of the output dimensions. See also the Matryoshka Embeddings for more information on reducing output dimensions. In Sentence Transformers, the combination of these two losses is called Matryoshka2dLoss, and a shorthand is provided for simpler training.
from sentence_transformers import SentenceTransformer
from sentence_transformers.losses import CoSENTLoss, Matryoshka2dLoss
model = SentenceTransformer("microsoft/mpnet-base")
base_loss = CoSENTLoss(model=model)
loss = Matryoshka2dLoss(model=model, loss=base_loss, matryoshka_dims=[768, 512, 256, 128, 64])
<details><summary>Performance Results</summary>
Let's look at the performance that we may be able to expect from an Adaptive Layer embedding model versus a regular embedding model. For this experiment, I have trained two models:
MultipleNegativesRankingLoss rather than AdaptiveLayerLoss on top of MultipleNegativesRankingLoss. I also use microsoft/mpnet-base as the base model.Both of these models were trained on the AllNLI dataset, which is a concatenation of the SNLI and MultiNLI datasets. I have evaluated these models on the STSBenchmark test set using multiple different embedding dimensions. The results are plotted in the following figure:
The first figure shows that the Adaptive Layer model stays much more performant when reducing the number of layers in the model. This is also clearly shown in the second figure, which displays that 80% of the performance is preserved when the number of layers is reduced all the way to 1.
Lastly, the third figure shows the expected speedup ratio for GPU & CPU devices in my tests. As you can see, removing half of the layers results in roughly a 2x speedup, at a cost of ~15% performance on STSB (~86 -> ~75 Spearman correlation). When removing even more layers, the performance benefit gets larger for CPUs, and between 5x and 10x speedups are very feasible with a 20% loss in performance.
</details>
<details><summary>Inference</summary>
After a model has been trained using the Adaptive Layer loss, you can then truncate the model layers to your desired layer count. Note that this requires doing a bit of surgery on the model itself, and each model is structured a bit differently, so the steps are slightly different depending on the model.
First of all, we will load the model & access the underlying transformers model like so:
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("tomaarsen/mpnet-base-nli-adaptive-layer")
# We can access the underlying model with `model[0].auto_model`
print(model[0].auto_model)
MPNetModel(
(embeddings): MPNetEmbeddings(
(word_embeddings): Embedding(30527, 768, padding_idx=1)
(position_embeddings): Embedding(514, 768, padding_idx=1)
(LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
(dropout): Dropout(p=0.1, inplace=False)
)
(encoder): MPNetEncoder(
(layer): ModuleList(
(0-11): 12 x MPNetLayer(
(attention): MPNetAttention(
(attn): MPNetSelfAttention(
(q): Linear(in_features=768, out_features=768, bias=True)
(k): Linear(in_features=768, out_features=768, bias=True)
(v): Linear(in_features=768, out_features=768, bias=True)
(o): Linear(in_features=768, out_features=768, bias=True)
(dropout): Dropout(p=0.1, inplace=False)
)
(LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
(dropout): Dropout(p=0.1, inplace=False)
)
(intermediate): MPNetIntermediate(
(dense): Linear(in_features=768, out_features=3072, bias=True)
(intermediate_act_fn): GELUActivation()
)
(output): MPNetOutput(
(dense): Linear(in_features=3072, out_features=768, bias=True)
(LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
(dropout): Dropout(p=0.1, inplace=False)
)
)
)
(relative_attention_bias): Embedding(32, 12)
)
(pooler): MPNetPooler(
(dense): Linear(in_features=768, out_features=768, bias=True)
(activation): Tanh()
)
)
This output will differ depending on the model. We will look for the repeated layers in the encoder. For this MPNet model, this is stored under model[0].auto_model.encoder.layer. Then we can slice the model to only keep the first few layers to speed up the model:
new_num_layers = 3
model[0].auto_model.encoder.layer = model[0].auto_model.encoder.layer[:new_num_layers]
Then we can run inference with it using <a href="https://sbert.net/docs/package_reference/SentenceTransformer.html#sentence_transformers.SentenceTransformer.encode"><code>SentenceTransformers.encode</code></a>.
from sentence_transformers import SentenceTransformer
from sentence_transformers.util import cos_sim
model = SentenceTransformer("tomaarsen/mpnet-base-nli-adaptive-layer")
new_num_layers = 3
model[0].auto_model.encoder.layer = model[0].auto_model.encoder.layer[:new_num_layers]
embeddings = model.encode(
[
"The weather is so nice!",
"It's so sunny outside!",
"He drove to the stadium.",
]
)
# Similarity of the first sentence with the other two
similarities = cos_sim(embeddings[0], embeddings[1:])
# => tensor([[0.7761, 0.1655]])
# compared to tensor([[ 0.7547, -0.0162]]) for the full model
As you can see, the similarity between the related sentences is much higher than the unrelated sentence, despite only using 3 layers. Feel free to copy this script locally, modify the new_num_layers, and observe the difference in similarities.
</details>
Extra information:
Example training scripts:
CrossEncoder models are often even better than biencoder (SentenceTransformer) models, as the model can compare two texts using the attention mechanism, unlike biencoders. However, they are more computationally expensive as well. They are commonly used for reranking the top retrieval results of a biencoder model. As of this release, that should now be more convenient!
We now support a rank method, which allows you to rank a bunch of documents given a query:
from sentence_transformers.cross_encoder import CrossEncoder
# Pre-trained cross encoder
model = CrossEncoder("cross-encoder/stsb-distilroberta-base")
# We want to compute the similarity between the query sentence
query = "A man is eating pasta."
# With all sentences in the corpus
corpus = [
"A man is eating food.",
"A man is eating a piece of bread.",
"The girl is carrying a baby.",
"A man is riding a horse.",
"A woman is playing violin.",
"Two men pushed carts through the woods.",
"A man is riding a white horse on an enclosed ground.",
"A monkey is playing drums.",
"A cheetah is running behind its prey.",
]
# We rank all sentences in the corpus for the query
ranks = model.rank(query, corpus)
# Print the scores
print("Query:", query)
for rank in ranks:
print(f"{rank['score']:.2f}\t{corpus[rank['corpus_id']]}")
0.67 A man is eating food.
0.34 A man is eating a piece of bread.
0.08 A man is riding a horse.
0.07 A man is riding a white horse on an enclosed ground.
0.01 The girl is carrying a baby.
0.01 Two men pushed carts through the woods.
0.01 A monkey is playing drums.
0.01 A woman is playing violin.
Extra information:
Semantic Textual Similarity example by @alvarobartt in https://github.com/UKPLab/sentence-transformers/pull/2511loss] Add AdaptiveLayerLoss; 2d Matryoshka loss modifiers by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2506I especially want to thank @SeanLee97 and @fkdosilovic for their valuable contributions in this release.
Full Changelog: https://github.com/UKPLab/sentence-transformers/compare/v2.4.0...v2.5.0
This release introduces numerous notable features that are well worth learning about!
This release introduces numerous notable features that are well worth learning about!
Install this version with
pip install sentence-transformers==2.4.0
Dense embedding models typically produce embeddings with a fixed size, such as 768 or 1024. All further computations (clustering, classification, semantic search, retrieval, reranking, etc.) must then be done on these full embeddings. Matryoshka Representation Learning revisits this idea, and proposes a solution to train embedding models whose embeddings are still useful after truncation to much smaller sizes. This allows for considerably faster (bulk) processing.
Training using Matryoshka Representation Learning (MRL) is quite elementary: rather than applying some loss function on only the full-size embeddings, we also apply that same loss function on truncated portions of the embeddings. For example, if a model has an embedding dimension of 768 by default, it can now be trained on 768, 512, 256, 128, 64 and 32. Each of these losses will be added together, optionally with some weight:
from sentence_transformers import SentenceTransformer
from sentence_transformers.losses import CoSENTLoss, MatryoshkaLoss
model = SentenceTransformer("microsoft/mpnet-base")
base_loss = CoSENTLoss(model=model)
loss = MatryoshkaLoss(model=model, loss=base_loss, matryoshka_dims=[768, 512, 256, 128, 64])
<details><summary>Inference</summary>
After a model has been trained using a Matryoshka loss, you can then run inference with it using <a href="https://sbert.net/docs/package_reference/SentenceTransformer.html#sentence_transformers.SentenceTransformer.encode"><code>SentenceTransformers.encode</code></a>. You must then truncate the resulting embeddings, and it is recommended to renormalize the embeddings.
from sentence_transformers import SentenceTransformer
from sentence_transformers.util import cos_sim
import torch.nn.functional as F
model = SentenceTransformer("nomic-ai/nomic-embed-text-v1.5", trust_remote_code=True)
matryoshka_dim = 64
embeddings = model.encode(
[
"search_query: What is TSNE?",
"search_document: t-distributed stochastic neighbor embedding (t-SNE) is a statistical method for visualizing high-dimensional data by giving each datapoint a location in a two or three-dimensional map.",
"search_document: Amelia Mary Earhart was an American aviation pioneer and writer.",
]
)
embeddings = embeddings[..., :matryoshka_dim] # Shrink the embedding dimensions
similarities = cos_sim(embeddings[0], embeddings[1:])
# => tensor([[0.7839, 0.4933]])
As you can see, the similarity between the search query and the correct document is much higher than that of an unrelated document, despite the very small matryoshka dimension applied. Feel free to copy this script locally, modify the matryoshka_dim, and observe the difference in similarities.
Note: Despite the embeddings being smaller, training and inference of a Matryoshka model is not faster, not more memory-efficient, and not smaller. Only the processing and storage of the resulting embeddings will be faster and cheaper.
</details>
Extra information:
Example training scripts:
CoSENTLoss was introduced by Jianlin Su, 2022 as a drop-in replacement of CosineSimilarityLoss. Experiments have shown that it produces a stronger learning signal than CosineSimilarityLoss.
from sentence_transformers import SentenceTransformer, losses
from sentence_transformers.readers import InputExample
model = SentenceTransformer('bert-base-uncased')
train_examples = [
InputExample(texts=['My first sentence', 'My second sentence'], label=1.0),
InputExample(texts=['My third sentence', 'Unrelated sentence'], label=0.3)
]
train_dataloader = DataLoader(train_examples, shuffle=True, batch_size=train_batch_size)
train_loss = losses.CoSENTLoss(model=model)
You can update training_stsbenchmark.py by replacing CosineSimilarityLoss with CoSENTLoss & you can observe the improved performance.
AnglELoss was introduced in Li and Li, 2023. It is an adaptation of the CoSENTLoss, and also acts as a strong drop-in replacement of CosineSimilarityLoss. Compared to CoSENTLoss, AnglELoss uses a different similarity function which aims to avoid vanishing gradients.
Like with CoSENTLoss, you can use it just like CosineSimilarityLoss.
from sentence_transformers import SentenceTransformer, losses
from sentence_transformers.readers import InputExample
model = SentenceTransformer('bert-base-uncased')
train_examples = [
InputExample(texts=['My first sentence', 'My second sentence'], label=1.0),
InputExample(texts=['My third sentence', 'Unrelated sentence'], label=0.3)
]
train_dataloader = DataLoader(train_examples, shuffle=True, batch_size=train_batch_size)
train_loss = losses.AnglELoss(model=model)
You can update training_stsbenchmark.py by replacing CosineSimilarityLoss with AnglELoss & you can observe the improved performance.
Some models require using specific text prompts to achieve optimal performance. For example, with intfloat/multilingual-e5-large you should prefix all queries with query: and all passages with passage: . Another example is BAAI/bge-large-en-v1.5, which performs best for retrieval when the input texts are prefixed with Represent this sentence for searching relevant passages: .
Sentence Transformer models can now be initialized with prompts and default_prompt_name parameters:
prompts is an optional argument that accepts a dictionary of prompts with prompt names to prompt texts. The prompt will be prepended to the input text during inference. For example,model = SentenceTransformer(
"intfloat/multilingual-e5-large",
prompts={
"classification": "Classify the following text: ",
"retrieval": "Retrieve semantically similar text: ",
"clustering": "Identify the topic or theme based on the text: ",
},
)
# or
model.prompts = {
"classification": "Classify the following text: ",
"retrieval": "Retrieve semantically similar text: ",
"clustering": "Identify the topic or theme based on the text: ",
}
default_prompt_name is an optional argument that determines the default prompt to be used. It has to correspond with a prompt name from prompts. If None, then no prompt is used by default. For example,model = SentenceTransformer(
"intfloat/multilingual-e5-large",
prompts={
"classification": "Classify the following text: ",
"retrieval": "Retrieve semantically similar text: ",
"clustering": "Identify the topic or theme based on the text: ",
},
default_prompt_name="retrieval",
)
# or
model.default_prompt_name="retrieval"
Both of these parameters can also be specified in the config_sentence_transformers.json file of a saved model. That way, you won't have to specify these options manually when loading. When you save a Sentence Transformer model, these options will be automatically saved as well.
During inference, prompts can be applied in a few different ways. All of these scenarios result in identical texts being embedded:
prompt option in SentenceTransformer.encode:embeddings = model.encode("How to bake a strawberry cake", prompt="Retrieve semantically similar text: ")
prompt_name option in SentenceTransformer.encode by relying on the prompts loaded from a) initialization or b) the model config.embeddings = model.encode("How to bake a strawberry cake", prompt_name="retrieval")
prompt nor prompt_name are specified in SentenceTransformer.encode, then the prompt specified by default_prompt_name will be applied. If it is None, then no prompt will be applied.embeddings = model.encode("How to bake a strawberry cake")
Some INSTRUCTOR models, such as hkunlp/instructor-large, are natively supported in Sentence Transformers. These models are special, as they are trained with instructions in mind. Notably, the primary difference between normal Sentence Transformer models and Instructor models is that the latter do not include the instructions themselves in the pooling step.
The following models work out of the box:
You can use these models like so:
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("hkunlp/instructor-large")
embeddings = model.encode(
[
"Dynamical Scalar Degree of Freedom in Horava-Lifshitz Gravity",
"Comparison of Atmospheric Neutrino Flux Calculations at Low Energies",
"Fermion Bags in the Massive Gross-Neveu Model",
"QCD corrections to Associated t-tbar-H production at the Tevatron",
],
prompt="Represent the Medicine sentence for clustering: ",
)
print(embeddings.shape)
# => (4, 768)
<details><summary>Information Retrieval usage</summary>
from sentence_transformers import SentenceTransformer
from sentence_transformers.util import cos_sim
model = SentenceTransformer("hkunlp/instructor-large")
query = "where is the food stored in a yam plant"
query_instruction = (
"Represent the Wikipedia question for retrieving supporting documents: "
)
corpus = [
'Yams are perennial herbaceous vines native to Africa, Asia, and the Americas and cultivated for the consumption of their starchy tubers in many temperate and tropical regions. The tubers themselves, also called "yams", come in a variety of forms owing to numerous cultivars and related species.',
"The disparate impact theory is especially controversial under the Fair Housing Act because the Act regulates many activities relating to housing, insurance, and mortgage loans—and some scholars have argued that the theory's use under the Fair Housing Act, combined with extensions of the Community Reinvestment Act, contributed to rise of sub-prime lending and the crash of the U.S. housing market and ensuing global economic recession",
"Disparate impact in United States labor law refers to practices in employment, housing, and other areas that adversely affect one group of people of a protected characteristic more than another, even though rules applied by employers or landlords are formally neutral. Although the protected classes vary by statute, most federal civil rights laws protect based on race, color, religion, national origin, and sex as protected traits, and some laws include disability status and other traits as well.",
]
corpus_instruction = "Represent the Wikipedia document for retrieval: "
query_embedding = model.encode(query, prompt=query_instruction)
corpus_embeddings = model.encode(corpus, prompt=corpus_instruction)
similarities = cos_sim(query_embedding, corpus_embeddings)
print(similarities)
# => tensor([[0.8835, 0.7037, 0.6970]])
</details>
All other Instructor models either 1) will not load as they refer to InstructorEmbedding in their modules.json or 2) require calling model.set_pooling_include_prompt(include_prompt=False) after loading.
Sentence Transformers now no longer uses sentencepiece or nltk as mandatory dependencies. This should make Sentence Transformers 1) lighter to install, 2) quicker to import and 3) less likely to result in dependency issues.
The documentation has been upgraded with a Loss Overview: https://sbert.net/docs/training/loss_overview.html This section contains tables with loss functions and their required data formats. This should help you narrow down which loss functions might suit your use cases:
Additionally, each loss function now has extended documentation, including references, requirements, relations to other loss functions, inputs, a code snippet with example usage, and/or links to other documentation/examples using that loss function.
hotfix] Don't require loading files for Normalize by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2460fix] Avoid sets and dicts in BinaryClassificationEvaluator when sentences are not hashable by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2462docs] Refactor & improve the loss documentation by @ir2718 in https://github.com/UKPLab/sentence-transformers/pull/2447deps] Remove the sentencepiece & nltk dependencies by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2476ci] On Ubuntu CI runner, use temporary directories as cache folders for some models by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2481docs] Slight improvements to docs phrasing by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2486feat] Add Matryoshka loss + examples + docs by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2485feat] Add prompt templates by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2477docs] Move loss overview to "main" documentation by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2496feat] Allow saving a model to the Hub without providing a user + Upload Matryoshka models after training by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2497docs] Address some small mistakes by @tomaarsen in https://github.com/UKPLab/sentence-transformers/pull/2498I especially want to thank @ir2718, @johneckberg & @SeanLee97 for their valuable contributions in this release, and @fkdosilovic and @milistu for their valuable improvements to the CrossEncoder.
Full Changelog: https://github.com/UKPLab/sentence-transformers/compare/v2.3.1...v2.4.0
This releases patches a niche bug when loading a Sentence Transformer model which:
This releases patches a niche bug when loading a Sentence Transformer model which:
Normalize module as specified in modules.jsonThis only occurs when a model with Normalize is downloaded from the Hugging Face hub and then later used locally.
See #2458 and #2459 for more details.
Full Changelog: https://github.com/UKPLab/sentence-transformers/compare/v2.3.0...v2.3.1
Sentence Transformers has deprecated Python 3.7 following its end of security support. Additionally, various dependencies have been updated to prevent…
This release focuses on various bug fixes & improvements to keep up with adjacent works like transformers and huggingface_hub. These are the key changes in the release:
Prior to Sentence Transformers v2.3.0, saving models to the Hugging Face Hub may have resulted in various errors depending on the versions of the dependencies. Sentence Transformers v2.3.0 introduces a refactor to save_to_hub to resolve these issues.
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
...
model.save_to_hub("tomaarsen/all-MiniLM-L6-v2-quora")
pytorch_model.bin: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 90.9M/90.9M [00:06<00:00, 13.7MB/s]
Upload 1 LFS files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:07<00:00, 7.11s/it]
Recently, transformers has shifted towards using safetensors files as their primary model file formats. Additionally, various other file formats are commonly used, such as PyTorch (pytorch_model.bin), Rust (rust_model.ot), Tensorflow (tf_model.h5) and ONNX (model.onnx).
Prior to Sentence Transformers v2.3.0, almost all files of a repository would be downloaded, even if theye are not strictly required. Since v2.3.0, only the strictly required files will be downloaded. For example, when loading sentence-transformers/all-MiniLM-L6-v2 which has its model weights in three formats (pytorch_model.bin, rust_model.ot, tf_model.h5), only pytorch_model.bin will be downloaded. Additionally, when downloading intfloat/multilingual-e5-small with two formats (model.safetensors, pytorch_model.bin), only model.safetensors will be downloaded.
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
Downloading modules.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████| 349/349 [00:00<?, ?B/s]
Downloading (…)ce_transformers.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████| 116/116 [00:00<?, ?B/s]
Downloading README.md: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████| 10.6k/10.6k [00:00<?, ?B/s]
Downloading (…)nce_bert_config.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 53.0/53.0 [00:00<?, ?B/s]
Downloading config.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████| 612/612 [00:00<?, ?B/s]
Downloading pytorch_model.bin: 100%|█████████████████████████████████████████████████████████████████████████████████████| 90.9M/90.9M [00:06<00:00, 15.0MB/s]
Downloading tokenizer_config.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████| 350/350 [00:00<?, ?B/s]
Downloading vocab.txt: 100%|███████████████████████████████████████████████████████████████████████████████████████████████| 232k/232k [00:00<00:00, 1.37MB/s]
Downloading tokenizer.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████| 466k/466k [00:00<00:00, 4.61MB/s]
Downloading (…)cial_tokens_map.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████| 112/112 [00:00<?, ?B/s]
Downloading 1_Pooling/config.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████| 190/190 [00:00<?, ?B/s]
[!NOTE]
This release updates the default cache location from~/.cache/torch/sentence_transformersto the default cache location oftransformers, i.e.~/.cache/huggingface. You can still specify custom cache locations via theSENTENCE_TRANSFORMERS_HOMEenvironment variable or thecache_folderargument. Additionally, by supporting newer versions of various dependencies (e.g.huggingface_hub), the cache format changed. A consequence is that the old cached models cannot be used in v2.3.0 onwards, and those models need to be redownloaded. Once redownloaded, an airgapped machine can load the model like normal despite having no internet access.
This release brings models with custom code to Sentence Transformers through trust_remote_code, such as jinaai/jina-embeddings-v2-base-en.
from sentence_transformers import SentenceTransformer
from sentence_transformers.util import cos_sim
model = SentenceTransformer("jinaai/jina-embeddings-v2-base-en", trust_remote_code=True)
embeddings = model.encode(['How is the weather today?', 'What is the current weather like today?'])
print(cos_sim(embeddings[0], embeddings[1]))
# => tensor([[0.9341]])
If an embedding model is ever updated, it would invalidate all of the embeddings that you have created with the prior version of that model. We promise to never update the weights of any sentence-transformers/... model, but we cannot offer this guarantee for models by the community.
That is why this version introduces a revision keyword, allowing you to specify exactly which revision or branch you'd like to load:
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("BAAI/bge-small-en-v1.5", revision="982532469af0dff5df8e70b38075b0940e863662")
# or a branch:
model = SentenceTransformer("BAAI/bge-small-en-v1.5", revision="main")
use_auth_token, use token instead (#2376)Following updates from transformers & huggingface_hub, Sentence Transformers now recommends that you use the token argument to provide your Hugging Face authentication token to download private models.
from sentence_transformers import SentenceTransformer
# new:
model = SentenceTransformer("tomaarsen/all-mpnet-base-v2", token="hf_...")
# old, still works, but throws a warning to upgrade to "token"
model = SentenceTransformer("tomaarsen/all-mpnet-base-v2", use_auth_token="hf_...")
[!NOTE] The recommended way to include your Hugging Face authentication token is to run
huggingface-cli login& paste your User Access Token from your Hugging Face Settings. See these docs for more information. Then, you don't have to include thetokenargument at all; it'll be automatically read from your filesystem.
Prior to this release, SentenceTransformers.device would not always correspond to the device on which embeddings were computed, or on which a model gets trained. This release brings a few fixes:
SentenceTransformers.device now always corresponds to the device that the model is on, and on which it will do its computations.SentenceTransformers.to(...), SentenceTransformers.cpu(), SentenceTransformers.cuda(), etc. will now work as expected, rather than being ignored.MultipleNegativesRankingLoss (MNRL) is a powerful loss function that is commonly applied to train embedding models. It uses in-batch negative sampling to produce a large number of negative pairs, allowing the model to receive a training signal to push the embeddings of this pair apart. It is commonly shown that a larger batch size results in better performing models (Qu et al., 2021, Li et al., 2023), but a larger batch size requires more VRAM in practice.
To counteract that, @kwang2049 has implemented a slightly modified GradCache technique that is able to separate the batch computation into mini-batches without any reduction in training quality. This allows the common practitioner to train with competitive batch sizes, e.g. 65536! The downside is that training with Cached MNRL (CMNRL) is roughly 2 to 2.4 times slower than using normal MNRL.
CachedMultipleNegativesRankingLoss is a drop-in replacement for MultipleNegativesRankingLoss, but with a new mini_batch_size argument. I recommend trying out CMNRL with a large batch size and a fairly small mini_batch_size - the larger mini batch size that will fit into memory.
from sentence_transformers import SentenceTransformer, losses, InputExample
from torch.utils.data import DataLoader
model = SentenceTransformer("distilbert-base-uncased")
train_examples = [
InputExample(texts=['Anchor 1', 'Positive 1']),
InputExample(texts=['Anchor 2', 'Positive 2']),
]
train_dataloader = DataLoader(train_examples, shuffle=True, batch_size=1024) # Here we can try much larger batch sizes!
train_loss = losses.CachedMultipleNegativesRankingLoss(model=model, mini_batch_size = 32)
model.fit([(train_dataloader, train_loss)], ...)
This release updates the community_detection function in various ways. Notably:
show_progress_bar option has been added (#1879)In the below graph, master refers to Sentence Transformers v2.2.2 and refactor refers to v2.3.0. On GPU, the computation time was heavily reduced.
Sentence Transformers has deprecated Python 3.7 following its end of security support. Additionally, various dependencies have been updated to prevent functionality from breaking. In particular:
torch >= 1.11.0transformers>= 4.32.0huggingface_hub>=0.15.1Lastly, torchvision has been removed as a dependency.
See the following for a list of release highlights:
community_detection from running forever by @nreimers (d8982c9f0d44f8a3c41579fa64c603eca029649b)to from getting ignored, replace ._target_device with .device by @tomaarsen (#2351)normalize_embeddings support to multi-process encoding by @tomaarsen (#2377)save_to_hub, remote git dependency, add token argument by @tomaarsen (#2376)transformers>=4.32.0 and huggingface_hub>=0.15.1 by @tomaarsen (#2376)community_detection by @dyaaalbakour (#2277)token and trust_remote_code to tokenizer_args too by @tomaarsen (#2411)cache_folder nor SENTENCE_TRANSFORMERS_HOME are set, use HF default cache by @tomaarsen (#2412)@k at the end of csv file name for RerankingEvaluator by @milistu (#2427)huggingface_hub dropped support in version 0.5.0 for Python 3.6
huggingface_hub dropped support in version 0.5.0 for Python 3.6
This release fixes the issue so that huggingface_hub with version 0.4.0 and Python 3.6 can still be used.
Version 0.8.1 of huggingface_hub introduces several changes that resulted in errors and warnings. This version of sentence-transformers fixes these is
Version 0.8.1 of huggingface_hub introduces several changes that resulted in errors and warnings. This version of sentence-transformers fixes these issues.
Further, several improvements have been added / merged:
util.community_detection was improved: 1) It works in a batched mode to save memory, 2) Overlapping clusters are no longer dropped but removed by overlapping items, 3) The parameter init_max_size was removed and replaced by a heuristic to estimate the max size of clustersYou can now use the encoder from T5 to learn text embeddings. You can use it like any other transformer model: `python from sentence_transformers impo
You can now use the encoder from T5 to learn text embeddings. You can use it like any other transformer model:
from sentence_transformers import SentenceTransformer, models
word_embedding_model = models.Transformer('t5-base', max_seq_length=256)
pooling_model = models.Pooling(word_embedding_model.get_word_embedding_dimension())
model = SentenceTransformer(modules=[word_embedding_model, pooling_model])
See T5-Benchmark results - the T5 encoder is not the best model for learning text embeddings models. It requires quite a lot of training data and training steps. Other models perform much better, at least in the given experiment with 560k training triplets.
The models from the papers Sentence-T5: Scalable sentence encoders from pre-trained text-to-text models and Large Dual Encoders Are Generalizable Retrievers have been added:
For benchmark results, see https://seb.sbert.net
Thanks to #1406 you can now load private models from the hub:
model = SentenceTransformer("your-username/your-model", use_auth_token=True)
This is a smaller release with some new features
This is a smaller release with some new features
MarginMSELoss is a great method to train embeddings model with the help of a cross-encoder model. The details are explained here: MSMARCO - MarginMSE Training
You pass your training data in the format:
InputExample(texts=[query, positive, negative], label=cross_encoder.predict([query, positive])-cross_encoder.predict([query, negative])
MultipleNegativesRankingLoss computes the loss just in one way: Find the correct answer for a given question.
MultipleNegativesSymmetricRankingLoss also computes the loss in the other direction: Find the correct question for a given answer.
The CLIPModel is now based on the transformers model.
You can still load it like this:
model = SentenceTransformer('clip-ViT-B-32')
Older SentenceTransformers versions are now longer able to load and use the 'clip-ViT-B-32' model.
PR #1116 checks if you have all files in your local cache or if there are added files on the hub. If this is the case, it will automatically download them.
SentenceTransformers.encode() can return all valuesWhen you set output_value=None for the encode method, all values (token_ids, token_embeddings, sentence_embedding) will be returned.
There should be no breaking changes. Old models can still be loaded from disc. However, if you use one of the provided pre-trained models, it will be…
All pre-trained models are now hosted on the Huggingface Models hub.
Our pre-trained models can be found here: https://huggingface.co/sentence-transformers
But you can easily share your own sentence-transformer model on the hub and have other people easily access it. Simple upload the folder and have people load it via:
model = SentenceTransformer('[your_username]/[model_name]')
For more information, see: Sentence Transformers in the Hugging Face Hub
There should be no breaking changes. Old models can still be loaded from disc. However, if you use one of the provided pre-trained models, it will be downloaded again in version 2 of sentence transformers as the cache path has slightly changed.
You can filter the hub for sentence-transformers models: https://huggingface.co/models?filter=sentence-transformers
Add the sentence-transformers tag to you model card so that others can find your model.
A widget was added to sentence-transformers models on the hub that lets you interact directly on the models website: https://huggingface.co/sentence-transformers/paraphrase-MiniLM-L6-v2
Further, models can now be used with the Accelerated Inference API: Send you sentences to the API and get back the embeddings from the respective model.
A new method was added to the SentenceTransformer class: save_to_hub.
Provide the model name and the model is saved on the hub.
Here you find the explanation from transformers how the hub works: Model sharing and uploading
When you save a model with save or save_to_hub, a README.md (also known as model card) is automatically generated with basic information about the respective SentenceTransformer model.
Final release of version 1: Makes v1 of sentence-transformers forward compatible with models from version 2 of sentence-transformers.
Final release of version 1: Makes v1 of sentence-transformers forward compatible with models from version 2 of sentence-transformers.
New methods integrated to train sentence embedding models without labeled data. See Unsupervised Learning for an overview of all existent methods.
New methods integrated to train sentence embedding models without labeled data. See Unsupervised Learning for an overview of all existent methods.
New methods:
New NLI & STS models: Following the Paraphrase Data training example we published new models trained on NLI and NLI+STS data. Training code is available: training_nli_v2.py.
| Model-Name | STSb-test performance |
|---|---|
| Previous best models | |
| nli-bert-large | 79.19 |
| stsb-roberta-large | 86.39 |
| New v2 models | |
| nli-mpnet-base-v2 | 86.53 |
| stsb-mpnet-base-v2 | 88.57 |
New MS MARCO model for Semantic Search: Hofstätter et al. optimized the training procedure on the MS MARCO dataset. The resulting model is integrated as msmarco-distilbert-base-tas-b and improves the performance on the MS MARCO dataset from 33.13 to 34.43 MRR@10
SentenceTransformer.fit() Checkpoints: The fit() method now allows to save checkpoints during the training at a fixed number of steps. More infomodels.Pooling() as string:pooling_model = models.Pooling(word_embedding_model.get_word_embedding_dimension(), pooling_mode='mean')
Valid values are mean/max/cls.Nothing published for this version
This release integrates methods that allows to learn sentence embeddings without having labeled data:
This release integrates methods that allows to learn sentence embeddings without having labeled data:
default_activation_function, that is applied on-top of the output logits generated by the class.It was not possible to fine-tune and save the CLIPModel. This release fixes it. CLIPModel can now be saved like any other model by calling model.save(
It was not possible to fine-tune and save the CLIPModel. This release fixes it. CLIPModel can now be saved like any other model by calling model.save(path)
v1.0.3 - Patch for util.paraphrase_mining method
v1.0.3 - Patch for util.paraphrase_mining method
v1.0.2 - Patch for CLIPModel, new Image Examples
v1.0.2 - Patch for CLIPModel, new Image Examples
Nothing published for this version
This release brings many new improvements and new features. Also, the version number scheme is updated. Now we use the format x.y.z with x: for major
This release brings many new improvements and new features. Also, the version number scheme is updated. Now we use the format x.y.z with x: for major releases, y: smaller releases with new features, z: bugfixes
You can now encode text and images in the same vector space using the OpenAI CLIP Model. You can use the model like this:
from sentence_transformers import SentenceTransformer, util
from PIL import Image
#Load CLIP model
model = SentenceTransformer('clip-ViT-B-32')
#Encode an image:
img_emb = model.encode(Image.open('two_dogs_in_snow.jpg'))
#Encode text descriptions
text_emb = model.encode(['Two dogs in the snow', 'A cat on a table', 'A picture of London at night'])
#Compute cosine similarities
cos_scores = util.cos_sim(img_emb, text_emb)
print(cos_scores)
More Information IPython Demo Colab Demo
Examples how to train the CLIP model on your data will be added soon.
util.dot_score computes the dot product of two embedding matrices. util.normalize_embeddings will normalize embeddings to unit lengthSentenceTransformer.encode method: normalize_embeddings if set to true, it will normalize embeddings to unit length. In that case the faster util.dot_score can be used instead of util.cos_sim to compute cosine similarity scores.models.Transformer(do_lower_case=True) when creating a new SentenceTransformer, then all input will be lower cased.output_value='token_embeddings' is definedLabelAccuracyEvaluatorencode(sent, convert_to_tensor=True). They now stay on the GPUNothing published for this version
Nothing published for this version
Faster tokenization speed: Using batched tokenization for training & inference - Now, all sentences in a batch are tokenized simoultanously.
Refactored Tokenization
SentencesDataset no longer needed for training. You can pass your train examples directly to the DataLoader:train_examples = [InputExample(texts=['My first sentence', 'My second sentence'], label=0.8),
InputExample(texts=['Another pair', 'Unrelated sentence'], label=0.3)]
train_dataloader = DataLoader(train_examples, shuffle=True, batch_size=16)
InputExample objects instead of tokenized textsSentenceLabelDataset has been updated to new tokenization flow: It returns always two or more InputExamples with the same labelAsymmetric Models
Add new models.Asym class that allows different encoding of sentences based on some tag (e.g. query vs paragraph). Minimal example:
word_embedding_model = models.Transformer(base_model, max_seq_length=250)
pooling_model = models.Pooling(word_embedding_model.get_word_embedding_dimension())
d1 = models.Dense(word_embedding_model.get_word_embedding_dimension(), 256, bias=False, activation_function=nn.Identity())
d2 = models.Dense(word_embedding_model.get_word_embedding_dimension(), 256, bias=False, activation_function=nn.Identity())
asym_model = models.Asym({'QRY': [d1], 'DOC': [d2]})
model = SentenceTransformer(modules=[word_embedding_model, pooling_model, asym_model])
##Your input examples have to look like this:
inp_example = InputExample(texts=[{'QRY': 'your query'}, {'DOC': 'your document text'}], label=1)
##Encoding (Note: Mixed inputs are not allowed)
model.encode([{'QRY': 'your query1'}, {'QRY': 'your query2'}])
Inputs that have the key 'QRY' will be passed through the d1 dense layer, while inputs with they key 'DOC' through the d2 dense layer.
More documentation on how to design asymmetric models will follow soon.
New Namespace & Models for Cross-Encoder Cross-Encoder are now hosted at https://huggingface.co/cross-encoder. Also, new pre-trained models have been added for: NLI & QNLI.
Logging
Log messages now use a custom logger from logging thanks to PR #623. This allows you which log messages you want to see from which components.
Unit tests A lot more unit tests have been added, which test the different components of the framework.
Updated the dependencies so that it works with Huggingface Transformers version 4. Sentence-Transformers still works with huggingface transformers ver
This release only include some smaller updates:
This release only include some smaller updates:
SentenceTransformer.fit() method - Parameter output_path_ignore_not_empty deprecated. No longer checks that target folder must be empty
Smaller changes:
Nothing published for this version
Nothing published for this version
Upgrade transformers dependency, transformers 3.1.0, 3.2.0 and 3.3.1 are working
Minor changes:
Hugginface Transformers version 3.1.0 had a breaking change with previous version 3.0.2
Hugginface Transformers version 3.1.0 had a breaking change with previous version 3.0.2
This release fixes the issue so that Sentence-Transformers is compatible with Huggingface Transformers 3.1.0. Note, that this and future version will not be compatible with transformers < 3.1.0.
Nothing published for this version
The old FP16 training code in model.fit() was replaced by using Pytorch 1.6.0 automatic mixed precision (AMP). When setting model.fit(use_amp=True), A
model.fit(use_amp=True), AMP will be used. On suitable GPUs, this leads to a significant speed-up while requiring less memory.The documentation is substantially improved and can be found at: www.SBERT.net - Feedback welcome
num_workers to a positive integer in your DataLoader, tokenization will happen in a background thread. This substantially increases the start-up time for training.model.encode() uses also a PyTorch DataSet + DataLoader. If you set num_workers to a positive integer, tokenization will happen in the background leading to faster encoding speed for large corpora.Breaking changes:
Multi-process tokenization (Linux only) for the model encode function. Significant speed-up when encoding large sets
This is a minor release. There should be no breaking changes.
This is a minor release. There should be no breaking changes.
evaluation/MSEEvaluator.py - breaking change. Now, this class expects lists of strings with parallel (translated) sentences. The old class has been re…
This is a minor update that changes some classes for training & evaluating multilingual sentence embedding methods.
The examples for training multi-lingual sentence embeddings models have been significantly extended. See docs/training/multilingual-models.md for details. An automatic script that downloads suitable data and extends sentence embeddings to multiple languages has been added.
The following classes/files have been changed:
New evaluation files:
Bugfixes:
This release updates HuggingFace transformers to v3.0.2. Transformers did some breaking changes to the tokenization API. This (and future) versions wi…
This release updates HuggingFace transformers to v3.0.2. Transformers did some breaking changes to the tokenization API. This (and future) versions will not be compatible with HuggingFace transfomers v2.
There are no known breaking changes for existent models or existent code. Models trained with version 2 can be loaded without issues.
Thanks to PR #299 and #176 several new loss functions: Different triplet loss functions and ContrastiveLoss
Nothing published for this version
Nothing published for this version
The release update huggingface/transformers to the release v2.8.0.
The release update huggingface/transformers to the release v2.8.0.
Nothing published for this version
huggingface/transformers was updated to version 2.3.0
huggingface/transformers was updated to version 2.3.0
Changes:
Nothing published for this version
This version update the underlying HuggingFace Transformer package to v2.2.1.
This version update the underlying HuggingFace Transformer package to v2.2.1.
Changes:
No breaking changes. Just update with `pip install -U sentence-transformers`
No breaking changes. Just update with pip install -U sentence-transformers
Bugfixes:
Improvements:
Updated pytorch-transformers to v1.1.0. Adding support for RoBERTa model.
Updated pytorch-transformers to v1.1.0. Adding support for RoBERTa model.
Bugfixes:
This is a minor fix: Packages were not correctly defined for pypi
This is a minor fix: Packages were not correctly defined for pypi
This release has many breaking changes with the previous release. If you need help with the migration, open a new issue.
v0.2.0 completely changes the architecture of sentence transformers.
The new architecture is based on a sequential architecture: You define individual models that transform step-by-step a sentence to a fixed sized sentence embedding.
The modular architecture allows to easily swap different components. You can choose between different embedding methods (BERT, XLNet, word embeddings), transformations (LSTM, CNN), weighting & pooling methods as well as adding deep averaging networks.
New models in this release:
This release has many breaking changes with the previous release. If you need help with the migration, open a new issue.
New model storing procedure: Each sub-module is stored in its own subfolder. If you need to migrate old models, it is best to create the subfolder structure by the system (model.save()) and then to copy the pytorch_model.bin into the correct subfolder.
First release of sentence transformers framework
First release of sentence transformers framework
One column per quarter.