NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
PyPI
PyPI, #4828 most downloaded on PyPI
Massive Text Embedding Benchmark
Latest stable
2.27.8
Last release 12 days ago
10 Nov 2026
Ships on a steady schedule
a new release about every 8 days
Nearly every release is documented
notes for 60 of the last 60 stable releases
Nothing withdrawn
no release was ever pulled
4 years old
875 releases · first in 2022
Drag across the ruler to compare, or tap a release to open it.
fix: delegate pairwise similarity to the sentence-transformers model
SentenceTransformerEncoderWrapper delegated similarity() to the wrapped
model but not similarity_pairwise(). For a model whose similarity_fn_name
is euclidean or manhattan, AbsEncoder.similarity_pairwise raised
"Similarity function not specified.", so STS and pair classification
tasks crashed. Pairwise scores now use the same function as similarity(). (f459fcb)
Signed-off-by: Ting Zhang <181050363+GPT-AGI@users.noreply.github.com>
Co-authored-by: Ting Zhang <181050363+GPT-AGI@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com> (2029daa)
fix: exit with usage when mteb has no command
Your coding agent can read these notes before it upgrades. Set up the MCP server →
mteb has no command (#5860)fix: exit with usage when mteb has no command
Require the CLI subcommand destination so argparse exits with a named command error instead of dispatching an incomplete namespace.
Co-authored-by: Ting Zhang <181050363+GPT-AGI@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com> (0bec7fb)
fix: accept device strings in mteb run
mteb run (#5859)fix: accept device strings in mteb run
The flag was typed as an int, so cpu and cuda:1 were rejected and a bare GPU index reached loaders that call str.startswith.
Co-authored-by: Ting Zhang <181050363+GPT-AGI@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com> (db653d1)
ci: report cited arXiv preprints that have since been published
ci: report cited arXiv preprints that have since been published (#5722)
ci: report cited arXiv preprints that have since been published
update to use SEMANTIC_SCHOLAR_API_KEY
update the script to get links of published version
combine workflows, update allowlist and moved to scripts/tools
add OpenAlex fallback and move other file under scripts/tools
added tests
add citation group
update lock file
skip test
remove test
cite only confirmed published versions, and open a PR with them
reference links to the published version. Only
entries citing the same arXiv paper are replaced, as keys are sometimes reused.Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> (5c94f55)
fix: drop undecodable video from StanfordI2V tasks (#5799)
fix: drop undecodable video from StanfordI2V tasks
simplify StanfordI2V broken video filter (35847e5)
fix: keep top_k results per query when ignore_identical_ids is set
fix: keep top_k results per query when ignore_identical_ids is set (#5844)
fix: keep top_k results per query when ignore_identical_ids is set
With ignore_identical_ids, the query's own document is dropped only at evaluation time, after search has already kept exactly top_k documents. A query that retrieved itself (usual for ArguAna, Quora and similar tasks) was scored on top_k - 1 documents, which lowered every metric at the largest cutoff.
Search one extra document for these tasks, and after dropping the identical id trim each query back to max(k_values), breaking ties like trec_eval. Tasks without ignore_identical_ids are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
update test
simplify comment
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Roman Solomatin <36135455+Samoed@users.noreply.github.com> (bde81c7)
fix: video frame timing (#5652)
fix: OmniVinci and Gemini get video files that play at the real frame spacing
Both receive the sampled frames as a video file: OmniVinci's was written at a fixed 24 fps (32 frames of any clip spanned 1.3 s, and its time embeddings read these times) and Gemini's at 1 fps (a long clip capped at 32 frames became a 32 s video). The video collators can now return each clip's frame timing (video_metadata=True), and both files are written at the frames' real spacing, as when the models get the original video.
The Qwen3-VL reranker is left as is (sentence-transformers' cross-encoder does not forward video_metadata), and so is ColQwen2.5-Omni (colpali-engine has the same mismatch).
sample_fps = sampled frames / source frames * source fps, instead of the frames' spacing.
Apply review suggestion: VideoInput.video is a tensor or a VideoInputItem
Always return frame timing as batch["video_metadata"]
Review feedback: the video collators always add each video's timing (VideoMetadata) next to the frames, which stay a plain tensor, instead of an opt-in flag that changes the frames' format. Wrappers that don't use it are unchanged; GVE, OmniVinci and Qwen-Omni read the new key, and Qwen3-VL-Embedding and BidirLM-Omni join it with the frames for sentence-transformers.
48c1172)fix: disable KV cache for stella_en_1.5B_v5 so it runs on current transformers
fix: disable KV cache for stella_en_1.5B_v5 so it runs on current transformers (#5850)
fix: disable KV cache for stella_en_1.5B_v5 so it runs on current transformers
The remote modeling_qwen.py at the pinned revision calls DynamicCache.get_usable_length(), which was removed in transformers 4.54. With the mteb entry's dtype kwarg (transformers >= 4.56), the model loads but fails on the first forward pass, so it could not run on any version.
Encoding never uses the KV cache, so pass config_kwargs={"use_cache": False} to skip that code path, as already done for gte-Qwen2-7B-instruct (#4935). Outputs are unchanged: on transformers 4.53.3 hidden states with the cache on and off are identical, and match transformers 4.57.6 with the cache off.
Fixes #5832
Co-authored-by: Aarush Shetty <182738217+AarushShetty346@users.noreply.github.com>
Co-authored-by: Roman Solomatin <36135455+Samoed@users.noreply.github.com> (34207fc)
model: add BrunoHays/embeddinggemma-2-nullspace-audio (#5775)
model: add BrunoHays/embeddinggemma-2-nullspace-audio
chore(models): rename file to embeddinggemma2_nullspace.py
refactor(models): inject null-space audio features inside model.forward
Co-authored-by: Cursor Agent <cursoragent@cursor.com> (e47213f)
fix: add GoogleSVQ v2 with one text per sentence
fix: add GoogleSVQ v2 with one text per sentence (#5790)
fix: add GoogleSVQ v2 with one text per sentence
normalize SVQ transcriptions before grouping sentences
regenerate SVQ v2 stats after normalization
Use the same folding as the near-duplicate analysis for SVQ sentences
Build GoogleSVQ v2 splits in self.dataset (010d62f)
model: add Reza2kn/RaahNaame-Koochik-v1 (Persian embedding model) (#5851)
model: add Reza2kn/RaahNaame-300m
model: use SentenceTransformerEncoderWrapper and google_gemini import (current main)
model: rename to Reza2kn/RaahNaame-Koochik-v1
model: drop non-paper citation (review) (9242b36)
model: add opensearch-neural-embed-v1 and opensearch-neural-sparse-encoding-asym-v1 (#5852)
model: add opensearch-neural-embed-v1 and opensearch-neural-sparse-encoding-asym-v1
fix: avoid importing torch at module import in opensearch_neural_embed_models (50c3917)
fix: Clarify documentation on output_dtypes and update in ModelMeta
fix: Clarify documentation on output_dtypes and update in ModelMeta (#5723)
Clarify documentation on output_dtypes and update in ModelMeta
change from review (1842b7e)
Co-authored-by: Mahmoud Zahran <200594164+mahmoudzahran01@users.noreply.github.com> (b9ccd7a)
feat: Add feature to remove unclear labels in mteb.data_cleaning
feat: Add feature to remove unclear labels in mteb.data_cleaning (#5696)
feat: Add feature to remove unclear labels in mteb.data_cleaning
add tests
update tests
split conflicting and unlabel filter (f126291)
feat: Add feature to remove unclear labels in mteb.data_cleaning (#5696)
feat: Add feature to remove unclear labels in mteb.data_cleaning
add tests
update tests
split conflicting and unlabel filter (d10c228)
This update adds utilities to mteb.data_cleaning to remove the
samples whose content carries more than one label, and those that carry none:
from mteb.data_cleaning import remove_conflicting_labels, remove_unlabeled
cleaned = remove_conflicting_labels(task) # content carrying more than one label
cleaned = remove_unlabeled(task) # samples carrying no label
See cleaning task data for what each of them removes.
fix: match only the requested script for language-script codes in get_tasks
fix: match only the requested script for language-script codes in get_tasks (#5826)
fix: match only the requested script for language-script codes in get_tasks
Fixes #5768. Follows up on #5483 and #5720.
languages=["zho-Hant"] also matched zho-Hans, because filter_tasks used contains_language, which falls back to the bare language. Added LanguageScripts.contains_langscript (strict for script-qualified codes, lenient for bare ones) and use it in filter_tasks, AbsTask.filter_languages and TaskResult.get_score. contains_language is unchanged.script= now raises a ValueError. It is validated up front in get_tasks, so the tasks= branch does not swallow it as a skipped task.filter_tasks now requires language and script to match on the same code, as filter_languages does per subset. get_tasks(languages=["eng", "rus"], script=["Cyrl"]) no longer raises.get_tasks without tasks= now passes exclusive_language_filter and skips tasks with no matching subset (with the existing warning), like the tasks= branch.AbsTask.language_scripts and language type hints.Tests added in tests/test_get_tasks.py, tests/test_filter_tasks.py and tests/test_languages.py.
Co-authored-by: Spicy Shark <321699413+spicysharkco@users.noreply.github.com>
Co-authored-by: Spicy Shark <321699413+spicysharkco@users.noreply.github.com> (01fd73c)
Add cohere embed v5 (#5838)
Add Cohere embed-v5.0-pro and embed-v5.0-fast models
fix: score every reranking candidate in BM25 search
With top_ranked, BM25Search and BB25Search took the full-corpus top-k
from bm25s and then kept only the hits that were in the query's candidate
list. Candidates outside the full-corpus top-k were silently dropped, so
reranking runs (and HybridSearch with a BM25 sub-model) returned fewer
than the candidates, often none.
Score each query's candidates directly with get_scores instead.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> (9e3c1c7)
fix citations (#5836)
fix citations 2
move json ignore (17b0382)
Update citations to fix citations CI (#5833)
update citations (e9c128f)
model: add RuneLab Turkish models (Vor-LateEmbed-TR, Kenning) (#5831)
model: add RuneLab Turkish models (Vor-LateEmbed-TR, Kenning)
Allowlist RuneLab model-card citations
Update mteb/models/model_implementations/runelab_models.py
Co-authored-by: Roman Solomatin <samoed.roman@gmail.com>
Co-authored-by: Roman Solomatin <samoed.roman@gmail.com>
Co-authored-by: Roman Solomatin <samoed.roman@gmail.com>
Co-authored-by: Roman Solomatin <samoed.roman@gmail.com> (1085234)
fix: handle unequal FAISS reranking candidate lengths
fix: handle unequal FAISS reranking candidate lengths (#5811)
fix: handle unequal FAISS reranking candidate lengths
simplify test
Co-authored-by: Roman Solomatin <36135455+Samoed@users.noreply.github.com> (fa81073)
08c4d98)add citations (89acedb)
fix: retain active API cache locks across LRU eviction
fix: retain active API cache locks across LRU eviction (#5809)
fix(api): retain active locks across LRU eviction
Delete tests/test_api/test_cache.py
Co-authored-by: Roman Solomatin <samoed.roman@gmail.com> (5e90e37)
Add KaLM-Reranker-V1-R2 models and update metadata (75674f4)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> (a7fd9d1)
fix: Clotho v2 captions glued together
fix: Clotho v2 captions glued together (#5803)
fix: rebuild Clotho v2 retrieval data with the official captions
add script used to rebuild the Clotho v2 data
simplify Clotho data script (031b5d3)
fix: repair two truncated images in EncyclopediaVQAIT2ITRetrieval (#5793)
fix: use mteb copy of EncyclopediaVQA with the two truncated images repaired
add script used to repair the EncyclopediaVQA images
simplify EncyclopediaVQA data script (a86ce28)
fix: truncate long texts for MuQ-MuLan (#5805) (ab31a6f)
fix: FSD50K labels were used as characters (#5807)
fix: add FSD50K v2 with the labels split into a list (148acb8)
fix gemma (667d77f)
fix: VideoConPairClassification column names were tuples (#5801) (`8d18e7b`)
fix: VideoConPairClassification column names were tuples (#5801) (8d18e7b)
fix: BeehiveStatesClassification failed to load (#5802) (6ff3af7)
fix: retry corrupt trailing video frames in resample_video (#5795)
fix: retry trailing frames that fail with invalid data in resample_video (61750f8)
Add model: lakinekaki/all-MiniLM-L6-v2-genbais (#5776)
Create genbais_models.py
Update genbais_models.py
added citation
Add training datasets to genbais_models.py
rm citation
Apply suggestion
Co-authored-by: Roman Solomatin <samoed.roman@gmail.com> (d87b873)
papr-embed-code-1: max_seq_length 32768 for long-document tasks (e82fb01)
fix: cite the published versions of preprints that have been published
fix: cite the published versions of preprints that have been published (#5724)
fix: cite the published versions of preprints that have since been published
update tests
add some missed citations
add cache file
chnage reference which pointing to arxiv to point to actual links (de720d5)
fix: add HatefulMemes retrieval v2 with labels pointing at the kept memes
fix: add HatefulMemes retrieval v2 with labels pointing at the kept memes (#5787)
fix: add HatefulMemes retrieval v2 with labels pointing at the kept memes
Make the v2 task inherit from AbsTaskRetrieval (7c54da7)
fix: KoVidore2 tasks keep only the columns the model reads (#5791) (`f95b31a`)
f95b31a)test: make the cross-encoder tests pass on the lowest pinned sentence-transformers and datasets
test_output_folder_model_meta read CrossEncoder.model_card_data, which sentence-transformers 3.0.0 does not have. Use config._commit_hash first and fall back to model_card_data, the same order ModelMeta uses.
test_search_cross_encoder imported datasets.Video at module level, which
datasets 2.19.0 does not have, so the whole module failed to collect.
Only the video test needs it, so it now skips via importorskip. (d8c4fd7)
test: drop temporary MTEB(slk, v1) skip in reference model coverage
Results for the gated Slovak datasets are now in the results repo.
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> (e1986fe)
papr-embed-code-1: prompts for CodeEditSearch/CodeSearchNet, declare both as seen training data (17e91a4)
fix: BidirLM-Omni uses the full audio clip, as its processor does (#5778) (`bdfe4fb`)
bdfe4fb)fix: rerankers crash on video QA tasks
fix: rerankers crash on video QA tasks (#5777)
fix: cross-encoders prepare each query once, so video queries don't overflow Arrow
update implementation
Co-authored-by: Roman Solomatin <36135455+Samoed@users.noreply.github.com> (8d9a84c)
upd revision handling (`acbcb19`)
ci: Update lock with uv 0.13.3 (#5731)
upd lock with uv 0.13.3
upd revision handling (acbcb19)
fix: Qwen3-VL embedders, reranker and UEmbed size video frames like qwen-vl-utils (#5766)
fix: Qwen3-VL embedders, reranker and UEmbed size video frames like qwen-vl-utils
Add qwen-vl-utils to the multimodal-sbert extra for the Qwen3-VL video resize (5610d84)
fix: Qwen3-VL-Reranker puts media before text, as the official script does
sentence-transformers keeps the order of the input's keys, and mteb's rows put
text before video, so a video+text query reached the model as text, then video.
The official format_mm_content puts video, then image, then text. (2b1533a)
The model's internal split of long inputs mixes the chunks of batched clips and
zero-pads the last one (DashengEncoder.forward). Split the features into
1008-frame chunks here instead, dropping frames that don't fill a patch, which
matches the authors' dasheng package. (a52da0a)
Its batched and single-clip audio embeddings differ (cosine down to 0.954 on a
1.4 s clip), through sentence-transformers directly as well, so audio follows
the batch-independence rule other audio models use. (32582d6)
The wrapper capped every input at its MIEB / MAEB lengths (380 / 1024 tokens),
shorter than one video, and set no frame rate, so every video task failed with a
video token-count mismatch. Video now uses the model card's defaults: 1 fps
(140 tokens per frame), at most 56 frames, and the full 8192-token context. (83f1922)
fix: VoxPopuli accent tasks load with current datasets (#5765)
fix: VoxPopuli accent tasks load with current datasets
VoxPopuliAccentClustering.dataset_transform didn't accept the num_proc that load_data passes, and VoxPopuliAccentID read audio with dict .get(), which torchcodec's AudioDecoder doesn't have. Empty clips raise when decoded, so the empty-audio filters treat a decode error as empty.
Use the standard dataset_transform signature and pass num_proc to filter (e6dc6d4)
fix: AVE classification trains on all classes (#5769) (bbb60c9)
fix: Ops-MM-embedding samples 8 video frames (#5770) (80eb2c5)
papr-embed-code-1: drop wrong bfloat16 output_dtypes (models run and score in float32) (428e3ca)
Add papr-embed-code-1-4b and papr-embed-code-1-0.6b (#5752)
Add papr-embed-code-1-4b and papr-embed-code-1-0.6b
Re-trigger CI (transient apt 403 on packages.microsoft.com)
papr_models: use SentenceTransformerEncoderWrapper, drop Qwen citation
papr_models: use InstructSentenceTransformerModel
Co-authored-by: Shawkat Kabbara <> (b23789a)
fix size of ColNanoVDR (#5759) (2ad7782)
model: add 5 sparse (SPLADE-style) models (#5754)
model: add 5 sparse (SPLADE-style) models
Add ModelMeta for five SparseEncoder models evaluated on MTEB(kor, v2):
All use SparseEncoderWrapper with the published Sentence Transformers configuration (no extra pruning, prompts or max-length overrides). The mock-run report for all five models is included.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Roman Solomatin <samoed.roman@gmail.com> (e924258)
fix: include image metadata in content hashes
fix: include image metadata in content hashes (#5755)
fix: include image metadata in content hashes
update also video
Co-authored-by: Roman Solomatin <36135455+Samoed@users.noreply.github.com> (bff7df1)
fix: cache BF16 and FP8 tensor embeddings
fix: cache BF16 and FP8 tensor embeddings (#5756)
fix: cache BF16 and FP8 tensor embeddings
simplify test
Co-authored-by: Roman Solomatin <36135455+Samoed@users.noreply.github.com> (17af96c)
fix: memory caps so Qwen3-Omni fit one 80 GB GPU
fix: memory caps so Qwen3-Omni fit one 80 GB GPU (#5743)
fix: memory caps so PE-AV large and Qwen3-Omni fit one 80 GB GPU
Measured on the longest clip of each task on one H100:
drop the PE-AV cap: its OOM came from the full-length VATEX v1 videos, which VATEX v2 cuts to 10 s
cap Qwen3-Omni at 32 frames too: 64 frames with 60 s audio still runs out on MusicAVQA and WorldSense
keep the default audio length: with 32 frames, 300 s of audio still fits (710c86f)
fix lower dependecy test (#5758) (3954800)
model: Add Bekko Static Embedding v1 (#5741)
feat(models): register Bekko Static Embedding v1
Delete mteb_mock_run_results.md
fix(models): complete Bekko Static training disclosures and citation
Co-authored-by: Roman Solomatin <samoed.roman@gmail.com> (91e8a72)
model: point KURE-Reranker-base at its retrained weights (#5753) (3e32c85)
dataset: VATEX and DiDeMo v2 (#5746)
dataset: VATEX and DiDeMo v2
VATEX stored the full YouTube videos instead of the 10 s clips its captions describe, and DiDeMo stored full videos while its descriptions cover only the first 30 s. The v2 tasks cut each video and its audio to the annotated window and drop clips whose audio is missing or silent. Scripts to rebuild the data are in scripts/data.
dataset: note where the VATEX and DiDeMo v1 data comes from
dataset: DiDeMo v2 keeps one row per video uploaded twice under different ids
dataset: describe the VATEX and DiDeMo v2 drop rules instead of counts, fix DiDeMo V2A/A2V reference (77b3dcd)
filter to actual splits and subsets from metadata (#5751) (1df984a)
model: Add divakantpandey/code-embed-v2 (#5749)
model: Add divakantpandey/code-embed-v2
fix: restore v1 metadata and strictly add code_embed_v2
Update mteb/models/model_implementations/code_embed_models.py
Co-authored-by: Roman Solomatin <samoed.roman@gmail.com>
Co-authored-by: Roman Solomatin <samoed.roman@gmail.com> (c365d2b)
d66bbd5)fix: add Nemotron 3.5 8B training code links
fix: add Nemotron 3.5 8B training code links (#5737)
fix: add Nemotron 3.5 training code links
Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>
Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>
Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com> (848a0b6)
fix: BidirLM-Omni repeats the last frame of 1-frame videos
The Qwen3-VL video processor needs at least temporal_patch_size frames and fails on
1-frame videos (AVMeme). Repeat the last frame, as the processor already does for an
odd frame count. (9a7aab1)
Add SlideOn and mSlideOn model definitions (#5736)
Add SlideOn and mSlideOn model definitions
Update mteb/models/model_implementations/slideon_models.py
Co-authored-by: Roman Solomatin <samoed.roman@gmail.com>
Co-authored-by: Roman Solomatin <samoed.roman@gmail.com>
Co-authored-by: Roman Solomatin <samoed.roman@gmail.com> (873f502)
pad short audio (#5721)
fix: pad audio shorter than the model's first window
wav2vec2, hubert, data2vec, wavlm and sew-d crash on clips shorter than one output frame of their conv feature encoder (400 samples at 16 kHz; sew-d needs 720 because it squeezes frames by squeeze_factor), LanguageBind on clips shorter than one 25 ms fbank window, and Qwen3-Voice below 5 mel hops (1280 samples at 24 kHz). Pad such clips with silence, as the ast, imagebind, ebind and encodec wrappers already do; longer clips are unchanged. MridinghamTonic has a 23 ms clip that crashed all of these models.
All registered wav2vec2/hubert/data2vec/wavlm checkpoints need 400 samples and all sew-d checkpoints 720, so use the numbers directly instead of computing the receptive field from the config.
test: drop pad_to_min_samples unit test
fix: also pad short audio for unispeech-sat, speecht5, dasheng and voiceclap
unispeech-sat and speecht5 use the same 400-sample conv feature encoder; dasheng and voiceclap-small need at least 480 samples (30 ms at 16 kHz).
Move the padding into AudioCollator next to max_samples, so each wrapper passes
min_samples instead of padding in its own loop. speecht5 now uses AudioCollator too
(same torchaudio Resample it did by hand). (b203d7a)
fix: bump EmotionAnalysisPlus dataset revision to drop empty val set
fix: bump EmotionAnalysisPlus dataset revision to drop empty val set (#5739)
fix: bump EmotionAnalysisPlus dataset revision to drop empty validation split
test: remove stale EmotionAnalysisPlus KNOWN_ISSUES entries
Apply batched suggestions from code review
Co-authored-by: Roman Solomatin <samoed.roman@gmail.com>
Co-authored-by: Nicolas Helmeyer <helmeyen@login-2.server.mila.quebec>
Co-authored-by: Roman Solomatin <samoed.roman@gmail.com> (415994d)
17 audio retrieval tasks repeat the same query text (or image) with a different relevant item each time, e.g. AudioSetStrong "The sound of Speech" x8, CASTELLA "Music plays" x38, the same CMU Arctic sentence read by two speakers. Identical queries can't be told apart, so all but one copy were scored as wrong.
The .v2 tasks merge rows with identical content: a repeated query becomes one query with every matching item relevant, and a repeated document becomes one document. Texts of 3 characters or less are dropped, as the task quality test requires. Kept rows are byte-identical to v1; the v1 tasks are unchanged and point to the v2 via superseded_by.
AudioSetStrong, CMUArctic, EmoVDB, HiFiTTS, MACS, LibriTTS: new revision of the same repo.
LASS: mteb/lass-synth-*.v2. CASTELLA, SoundingEarth: mteb/ copies of the dukesun99 repos. (ec2b404)
dataset: GigaSpeech v2 (#5710)
data: GigaSpeech v2, MridinghamTonic short clip, UrbanSound8K retrieval superseded
Drop the 151 transcripts that are 3 characters or less or occur more than once (fillers like "YEAH." x32), which the task quality test rejects and which retrieval cannot tell apart. 5564 pairs; new HF revisions of the same repos.
Same source recordings as MridinghamTonic: 228622__akshaylaya__thom-d-089.wav is 23 ms (368 samples at 16 kHz), shorter than the 25 ms window of wav2vec2-style encoders and LanguageBind's fbank. Revision bump, same task.
The 23 ms clip crashes models whose first layer needs a 25 ms window; that is a
model-side issue, fixed by padding short audio in those wrappers instead. (88d3837)
fix: add --num-proc to mteb run
fix: add --num-proc to mteb run (#5719)
feat: add --num-proc to mteb run
Pass the value through to mteb.evaluate(num_proc=...) so CLI runs can
load and transform datasets with more than one process. The default stays
None, so existing runs and CI behave as before.
Closes #5697
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> (a962bc5)
fix: add log message for num_proc
add log message for num_proc (ec10491)
fix: encode classification labels with one label set across splits
stratified_subsampling used DatasetDict.class_encode_column, which encodes each split on its own. When a label is missing from one split, every label after it alphabetically gets a different id in train and test, so the classifier is scored against shifted labels (e.g. CommonLanguageAgeDetection: 'nineties' only in train, so 'twenties' is 7 in train and 6 in test).
Build the label names once from all splits. Names are sorted as strings like
class_encode_column, so tasks whose splits share labels keep the same ids. (f32667b)
add model meta for divakantpandey/code-embed (ef6709e)
fix: make instruction prompt priority configurable via prompt_priority
fix: make instruction prompt priority configurable via prompt_priority (#5695)
feat: make instruction prompt priority configurable via prompt_priority
change naming to make them clear
make choices enum (3471c0c)
model: Add BiUltraVBERT (#5713)
model: Add BiUltraVBERT
model: Update ColUltraVBERT revision (f8aac21)
fix: run cross-encoder scoring head in fp32
fix: run cross-encoder scoring head in fp32 (#5679)
run cross-encoder scoring head in fp32 and clarify output_dtypes
fix failing CI
adapt implementation for LogitScore rerankers
change from review
remove clarification lines (af7735a)
fix: preserve FAISS cache vectors when appending after reload (#5707) (`849dc51`)
849dc51)fix: citation check only looks up new citations on PRs
fix: citation check only looks up new citations on PRs (#5701)
fix: citation check only looks up new citations on PRs
add citation_cache json (161131c)
model: EmbeddingGemma 2 (#5588)
Add EmbeddingGemma 2 ModelMeta for text, text-vision, and text-audio
This PR registers three modular entries for the EmbeddingGemma 2 model family in mteb/models/model_implementations/google_embeddinggemma.py:
google/embeddinggemma-2-text (271.0M parameters): Text backbone for MTEB (Multilingual v2, English v2, Code v1).google/embeddinggemma-2-text-vision (438.8M parameters): Text-vision model for MIEB (lite).google/embeddinggemma-2-text-audio (576.6M parameters): Text-audio model for MAEB.914f7f89142e33e77833254d9c9b90c3cef7303bmteb.get_model(model_name, revision) andmteb.get_model_meta(model_name, revision)Add canonical google/embeddinggemma-2 ModelMeta and loader wrapper
Add EmbeddingGemma 2 per-task evaluation recipe and reference wrapper
The model-card numbers for MTEB(Multilingual, v2), MTEB(eng, v2), MTEB(Code, v1) and MIEB(lite) were produced with one instruction and one max sequence length per task, so a task-type prompt dictionary cannot reproduce them. This commit makes the exact setup part of the reference implementation:
Reproduced with public weights/datasets on mteb 2.20.3 (768d): MTEB(Multilingual, v2) 61.05 (card 61.36), MTEB(eng, v2) 67.66 (68.46), MTEB(Code, v1) 77.48 (78.68), MIEB(lite) 65.22 Mean(TaskType) (64.64, clean images; the card's MIEB run JPEG re-encoded every image).
Per review (one model entry, settings as experiments): replace the -text / -vision-text / -audio-text ModelMetas with a single google/embeddinggemma-2 entry whose loader is EmbeddingGemma2Wrapper.
use_image / use_audio are the wrapper's experiment kwargs; they
select which towers are loaded (text is always on) and are recorded as
experiment_kwargs, so MIEB results live under
google__embeddinggemma-2/<rev>/experiments/use_image=True/.
A mteb_model_meta setter (same pattern as uembed_models.py) reports the
modalities, n_parameters and memory_usage_mb of the towers actually
loaded: text 271M, text+image 439M, text+audio 577M, all 744M.
embed_dim lists the supported MRL sizes [768, 512, 256, 128] so
embed_dim= can be passed as an experiment.
Simplify EmbeddingGemma 2 wrapper and fix text-less batches
One flat EMBEDDING_GEMMA_2_TASK_RECIPE table instead of four per-benchmark
tables and a recipe kwarg: mteb keeps one result per task, and for the
tasks shared between benchmarks the entry kept is the one the reported
score was computed with (the alternative gives the same score for most).
Drop default_max_seq_length and the config/model kwargs merging; tower
bookkeeping is a small table; no torch/logging imports.
Fix encode() for batches without text: the batch size is now taken from whichever column is present (image, audio or video) instead of assuming "image", which raised KeyError on audio/video tasks and produced empty document texts for image-corpus retrieval tasks.
Add MAEB recipe, 16kHz audio resampling, prompt-first ordering, and simplified task types
Make media/text ordering explicit and keep image blocks within max_seq_length
media_first kwarg (default False = text before image/audio, as in the
previous commit). media_first=True reproduces the model-card MIEB(lite)
run: the internal pipeline inserted images at text position 0, and the
image-first scores match the internal per-task numbers more closely
(mean |delta| 1.56 vs 2.15 over the order-sensitive tasks; e.g.
VidoreShiftProject internal 54.85 -> image-first 55.74 / text-first 65.31).
Overall text-first is ~+0.4 Mean(TaskType).
With text before an image, long captions pushed the 256-token image block past max_seq_length and the processor raised "Mismatch in image token count" (WebQAT2ITRetrieval). Text preceding an image is now truncated so the full image block always fits.
removed max-len tuning and unreasonable prompt selection
also extended the defaults to be slightly more appropricate. E.g. use fact checking for task of the subtask claim verification and code retrieval for retrieval tasks containing code.
Co-authored-by: Sara Smoot <sarasmoot@google.com>
Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsen@gmail.com> (1e8ccf6)
fix remaining av wrappers (#5651)
fix: AuroLA-Omni keeps the checkpoint's image size
The model card caps pixels for video only. The processor-level max_pixels also capped images, shrinking a 1080p image from a 78x138 to a 12x20 patch grid. Video is unchanged (its cap is set per call).
custom_st's _align_eval_processor only sets the video size when it is a dict; on transformers 5 it is a SizeDict, so the eval limits (longest 12845056, shortest 262144) were skipped. 32 frames at 1080p now give a 16x28x52 grid instead of 16x40x72.
min_pixels/max_pixels passed to AutoVideoProcessor.from_pretrained are dropped, so videos kept the processor defaults (16 frames at 1080p gave about 12k tokens). UEmbed sizes a frame list with qwen-vl-utils (total_pixels=1076832*32) and calls the processor with do_resize=False; the wrapper now does the same, giving the reference's grids and pixels.
The reference sets only a 360420 video max_pixels, so qwen-vl-utils' VIDEO_MIN_PIXELS (12828*28) is the floor. Only longest_edge was set, so small clips stayed at the processor floor of 3136 px. Grids now match qwen-vl-utils on transformers 4.57.
video_plus_text is trained only against audio and audio_plus_text only against video (PeAudioVideoModel.forward). They were also used when the other side was text or the same media (EgoSchema, OmniVideoBench, CoVR, CLDAT2A, ClothoMoment, SpeechEdit, XModBench VT2T/AT2T/AT2I), where matching pairs scored about 0.15. Those tasks use the summed embeddings again.
The official video eval (video.yaml) uses num_frames: 8 for every video set; the wrapper sampled at 2 fps up to 64 frames.
msclap pads text to text_len but does not truncate, so the GPT-2 encoder always pools on the appended <|endoftext|>. Truncating at 77 tokens cut it off for longer captions. Text is now cut only at the encoder's position limit; embeddings match msclap's preprocess_text for short and long captions (2022 and 2023).
Mixed-modality datasets can hold None for one modality; the collator tried to resample it and crashed.
The authors' chat template ends each item with 'Summarize the above <modality>', preferring image, then video, then audio, then text. mteb chose one word per batch, preferring audio, so video+audio inputs said 'audio' and mixed batches gave items the wrong word.
The model card sets only max_pixels=2002828 for video, so qwen-vl-utils' VIDEO_MIN_PIXELS (1282828) is the floor, not 42828. Small clips now get the card's grids.
docs: note per-clip forwards for encoders whose pooling does not mask padding
fix: install extras for AuroLA-Omni, PE-AV and VideoPrism
AuroLA loads through sentence-transformers' multimodal path (the card asks for sentence-transformers and transformers 5), not qwen-vl-utils; PeAudioVideoModel and VideoPrism ship with transformers 5.
Its video processor configs have do_sample_frames=True with num_frames 16 or 8, so any other frame count was re-sampled (large: 16 -> 8) or crashed (fewer frames than num_frames). Default runs were unaffected.
MMEB-V2 puts a dataset instruction on the video side of video classification (e.g. UCF101: 'What activities or sports are being performed by the person in the video?'). Classification passes no prompt_type, so the video got none. Kinetics-700, SSv2, UCF101, HMDB51 and Breakfast now get MMEB's instructions.
torch.autocast takes a device type; str(self.device) failed for an integer device such as 0.
Its remote processor's pad_fn cuts the padding to the sequence's own length, so any batch with a sequence under half the longest crashed ("Sizes of tensors must match"), e.g. MSR-VTT queries and video documents; batched audio embeddings also changed with their batch-mates. Every input now goes one at a time. H.264 also rejects odd frame sizes (NExT-QA has 500x375 clips), so frames are padded to even sizes before writing the mp4.
colpali's process_videos lets the Qwen2.5-Omni processor defaults (up to 602112 px per frame) override the checkpoint's 200704 px; video is now processed with colpali's prompt plus the checkpoint's size (a 64-frame 480p clip: 12566 -> 7542 tokens). Scoring used colpali's default batch of 128 documents, which needs ~70 GiB for video documents; it now uses 16. DiDeMo ran out of memory before (207 GiB) and now completes (R@1 0.517).
chore: lock qwen-vl-utils for the uembed extra
fix: VideoPrism requires transformers 5.13
VideoPrismVisionModel and VideoPrismClipModel first ship in transformers 5.13.0 (not in any 5.12.x); checked the base checkpoint loads and runs on 5.13.0. Adds a videoprism extra (in the conflict group, like the other transformers-pinning extras) and relocks.
fix: MuQ-MuLan similarity accepts tensors (reranking tasks crashed)
Jina v5 omni: pass the eval video size through processor_kwargs
Review feedback: build the video processor with the eval sizes and pass it in processor_kwargs instead of setting the size after loading. A processor-level size kwarg doesn't work: custom_st's min/max pixels for images override it.
Review feedback: apply qwen-vl-utils' total-pixel resize to the frame tensor. The released qwen-vl-utils (0.0.14) rounds frame-list inputs to 64 pixels first (fixed upstream in QwenLM/Qwen3-VL@fe12058, not yet released); the direct resize follows the fixed code.
Fixes the wrapper tests, which pass an already loaded model, and the
lowest-dependency run, whose transformers has no AutoVideoProcessor. (ac38ebd)
5382a46)fix: update vidore reranking task revisions
fix: update vidore reranking task revisions (#5692)
Update private ViDoRe Nemotron reranking revisions
fix: update vidore reranking task revisions
fixes https://github.com/embeddings-benchmark/mteb/issues/5690 (586e77c)
dataset: Add Spatial Arrangement AbsTask and Stakeholder Grounding Task (#5656)
dataset: Add Spatial Arrangement AbsTask and Stakeholder Grounding Task
fix: refactor to STS (8f2be3b)
fix: make the multilabel classification evaluation cap work
fix: make the multilabel classification evaluation cap work (#5514)
fix: make the multilabel classification evaluation cap work
add capped .v2 tasks for the multilabel tasks that have results
raise the multilabel eval cap to 3000 and drop the v2 tasks it no longer affects
define the v2 multilabel tasks as standalone classes
Tasks shouldn't inherit from other tasks, so each .v2 task now has its own TaskMetadata instead of copying the original's. (3e4aa0e)
fix: make nAUC independent of the order of equally confident queries
fix: make nAUC independent of the order of equally confident queries (#5660)
fix: make nAUC independent of the order of equally confident queries
nauc sorts the confidences with a stable argsort and keeps the suffix of round(rate * n) entries. When a cutoff lands inside a group of equal confidences, which members survive depends on their input order, so the same data in a different order gives a different nAUC. With four equal confidences and metrics [0, 0, 1, 1] one order gives 1.0 and the reverse gives -1.0.
Give every member of an exactly equal confidence group that group's mean metric, so the retained suffix is the score expected under random tie-breaking. Cutoffs, rates and the rounded retention counts are unchanged, and the transformation is the identity when there are no exact ties.
fix: keep nauc returning nan for an empty input
test: reduce the nAUC tie-order regression to one case
The five cases asserted the same property from different angles. One test on a constant confidence vector covers it: before the fix the same two vectors scored 1.0 and -1.0000000000000002 depending on which queries came first, and after it both are 0.0.
Reviewed-by: KennethEnevoldsen
abstention_curve is declared to return an array of length len(abstention_rates) and its result is passed straight to sklearn.metrics.auc, so returning a float made nauc raise InvalidParameterError for an empty input instead of returning nan as before. (37b801f)
fix: add VideoFramesWrapper to run image models on video tasks (#5537)
feat: add VideoFramesWrapper to run image models on video tasks
Image-text models like CLIP are usually reported on video benchmarks by sampling frames uniformly, encoding each frame as an image and mean-pooling. The wrapper makes that protocol explicit and records num_frames and pooling in experiment_kwargs, so results stay distinguishable from native video models.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
mteb.evaluate now wraps a model in VideoFramesWrapper per task when the
task needs video and the model only supports images, so no explicit wrapper
or ModelMeta change is needed. The frame count is exposed as
video_frames on evaluate and --video-frames on the CLI; when unset, the
default of 8 is used with a warning so it is always a deliberate choice.
The wrapper now keeps its own ModelMeta copy instead of mutating the
wrapped model, which keeps image-task results in the same run unaffected.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The module imported datasets.Video at collection time, which does not exist on the lowest-pin CI (datasets<4.0). Guard with importorskip like the other video tests and import Video and torchvision inside the test that uses them.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Review feedback: the name now says what the wrapper does, frame sampling supports the same fixed-count or fps modes as the native video models (recorded in experiment_kwargs), and tqdm comes from tqdm.auto.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
evaluate.py now only calls wrap_image_model_for_video once per task; the modality check, default-frames warning, model loading and wrapping live next to Video2ImagesWrapper. Behaviour is unchanged.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Results with the default 8 uniform frames are now saved next to the model's other results so they reach the leaderboard; any other sampling is recorded in experiment_kwargs and stored under experiments/, matching how loader overrides on native video models are handled.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
wrap_image_model_for_video loaded the model itself, and ran before
_check_cache, so an image model was downloaded and loaded for every
video task even when results were already cached — and even under
overwrite_strategy="only-cache", which is documented not to run the
task at all.
Split it into video2images_frames_for_task (pure, resolves the frame
count) and wrap_image_model_for_video (wraps an already-loaded model).
evaluate now derives the recorded meta with video2images_model_meta
before the cache check and wraps only after the existing lazy load.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The wrapper and the relaxed _check_model_modalities keyed off
modalities alone, so every image model without video was now accepted
on video tasks — 51 of them are not dense: 43 late-interaction, 6
cross-encoder, 2 sparse. Before this branch they were rejected with a
clear modality error.
They cannot work: cross-encoders (the LightOn-rerank-* family) expose
predict but no encode/similarity, and late-interaction models emit
per-token embeddings padded per batch, so torch.stack raises across
batches and silently mean-pools padding within a single batch.
Gate the wrapping, the modality relaxation and Video2ImagesWrapper
itself on "dense" in meta.model_type. Every image model in the
registry declares a model type, so this excludes exactly those 51.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
evaluate never wraps a native video model, but __init__ only checked
for image and dense, so one could still be wrapped by hand. That would
replace the model's native video path with mean-pooled frames and, at
the default frame count, store the result as its regular video results.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
_check_model_modalities added video to the model's modalities for
every task, so tasks that mix video with other inputs on one retrieval
side passed the upfront check and then hit the wrapper's
NotImplementedError. XModBenchVT2TReranking (query video+text) went
from a partial-overlap warning to a crash; CoVRRVT2VRetrieval and
VCDBCoreAudioVideoRetrieval went from being rejected outright to
loading the model first.
Video now counts as covered only when a retrieval side is video-only,
and video2images_frames_for_task applies the same rule so these tasks
are not wrapped. Non-retrieval tasks keep the old check: they encode
each modality in its own dataloader (zeroshot classification encodes
text labels separately from the videos), so their metadata is a union
across calls rather than one dataloader's features.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ibm-granite/granite-vision-3.3-2b-embedding declared
model_type=["dense"] while padding multi-vector output with
pad_sequence and scoring it with score_multi_vector, so it passed the
dense gate added for video frame pooling and would have failed the same
way colpali does.
It is the only model in the registry that scored MaxSim while claiming
to be dense; test_max_sim_models_are_not_registered_as_dense keeps it
that way, since the mismatch breaks anything keying off model_type,
not just the video wrapper.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
mteb.evaluate no longer wraps image models for video tasks, and the --video-frames flag and video_frames argument are removed. Evaluating an image model on a video task now fails with an error pointing to Video2ImagesWrapper; tasks that only partially overlap still warn, so image-only evaluation stays possible.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
show_progress_bar parameter, forwarded to the wrapped modelDEFAULT_NUM_FRAMESCo-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
_has_mixed_video_side with the upfront rejectionmteb[image,video]Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
mteb.get_model already requires the image extra for image models, so the check never helps real models. Reword the docs to say so.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
validate_tasks hook; evaluate only calls the hook if the model has onevalidate_tasks, so cache-wrapped models
are also rejected upfront instead of failing mid-runCo-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
CachedEmbeddingWrapper cannot hash video items, so forwarding validate_tasks through it only served an ordering that cannot run video. Restore the cache wrapper, test and document Video2ImagesWrapper(CachedEmbeddingWrapper(model)).
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Drop _as_task_list and with_video_hint in favour of inline expressions, tighten the docs intro and opt-in wording, and remove the CLI note, the partial-overlap paragraph and the Limitations section.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The check only looked at retrieval query/document sides. It now derives the modalities that share one input from the task category for other task types too, so video with audio in classification or pair classification is caught by validate_tasks. Also link the related docs pages.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Replace the duck-typed validate_tasks hook in evaluate with a non-serialized ModelMeta.video_only_inputs flag set by Video2ImagesWrapper. The existing modality check now rejects tasks that mix video with another modality in one input, from the model and task metadata alone, before any data or model work.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The wrapper already records video_frame_pooling in experiment_kwargs, so _check_model_modalities recognises wrapped models from that instead of a new ModelMeta attribute that only this wrapper would use.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> (fe6b2f1)
fix: write the citation cache as indented JSON
refaudit writes the cache on one line, so every cache update was a single unreadable line in the diff. Re-dump it after each flush with the same layout as the pretty-format-json hook.
Closes #5664 (ce894d7)
benchmark: Extend vidore reranking benchmark (#5680)
Add nuclear nemtron task + fixes
add nuclear nemotron stats + Vidore3TelecomNemotron35Embed8BRerankingv2
add desc stats
fix tests (634d8ce)
fix: require PyTorch 2.13 for Nemotron 3.5
Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com> (5b9337d)
fix: add missing imports for vidore reranking tasks (#5683) (`fd46df5`)
fix: Record retrieval search phases in TimingStack
fix: Record retrieval search phases in TimingStack (#5578)
Record retrieval search phases in TimingStack
Pass the timer to search() instead of storing it on the wrapper
Record full corpus search as one phase and log per-chunk timings
Document the index_backend phases and test them and plain search models
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> (937acf3)
fix: make hybrid fusion ties deterministic (#5665)
fix: make hybrid fusion ties deterministic
test: simplify hybrid fusion regression (78d61b1)
dataset: Added Vidore V3.1 Reranking tasks for nvidia/Nemotron-3.5-Embed-8B-BF16 (#5674)
Added reranking tasks for nvidia/Nemotron-3.5-Embed-8B-BF16
Fix linting issues
fix tests
Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsen@gmail.com> (edf8019)
OLMo-3-7B LoRA fine-tune (merged) for retrieval: last-token pooling, instruction-prefixed queries, loaded with InstructSentenceTransformerModel.
Claude-Session: https://claude.ai/code/session_01AvAXHqrxm8pyzF85vqSzFh
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> (85542f1)
Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com> (100a7a2)
fix: decouple baseline model revision
These can change independently and should be decoupled.
discovered when adding no-reranking model. I will keep revision at 2 (bumping it down will look wierd in the diff) (58d062b)
benchmark: add vidore reranking (#5671)
add vidore reranking
Apply suggestion from @KennethEnevoldsen (7ce4466)
dataset: Add SIB200Classification.v3 (#5501)
Add SIB200Classification v3
Mark SIB200Classification.v3 train-test leakage as known
Keep existing benchmarks unchanged
Use canonical SIB200 citation for v3
Clarify new SIB200 v3 configurations
Update mteb/tasks/classification/multilingual/sib200_classification.py
Co-authored-by: Kenneth Enevoldsen <kenevoldsen@pm.me>
Co-authored-by: Kenneth Enevoldsen <kenevoldsen@pm.me> (5b43459)
Add Nemotron 3.5 Rerank 8B BF16
Register text and image reranking with an FP32 scoring head and training datasets for zero-shot accounting. Default to Flash Attention 2 with runtime overrides and include its dependency in the Nemotron extra.
Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com> (ee2dbaf)
fixes #5655 (ec7fddb)
docs: show how to set corpus_chunk_size (#5545) (`e55b875`)
e55b875)feat: Add mteb submit-results (#5632)
Add --submit to mteb run to submit results after evaluation
fix bug in submit_results
separaet 2 CLI commands
simplify by removing create-pr from mteb run
add early PR check requirements
remove unnecessary test (e2551a4)
dataset: Add datasets for vidore reranking and outline the benchmark (#5635)
dataset: Adding Vidore V3 ranking tasks
Moving reranking tasks to the right module
Adding missing script
Fixed linting issue
Fixing wrong dataset revision
Updating tasks metadata
dataset: Adding Vidore V3.1 re-ranking benchmark
Continued from #5584
The output table should look roughly at follows:
| Rank (Mean Task) | Model | Mean (Task) | Mean (Public) | Mean (Private) | First-stage retriever: BM25s | First-stage retriever: BGE-M3 | First-stage retriever: Qwen3-VL-Embedding-2B |
|---|---|---|---|---|---|---|---|
| 1 | reranker A | 0.500 | 0.500 | 0.500 | 0.30 | 0.50 | 0.70 |
| 2 | reranker B | 0.483 | 0.483 | 0.483 | 0.40 | 0.45 | 0.60 |
First stage retriever will be one cell above the three
Merge remote-tracking branch 'origin/main' into add-vidore-reranking
fix: batch Multimodal cross-encoders pairs to prevent OOM
fixes #5638
clean
hiding leaderboard for now
skip mteb(slk) for now
added missing desc stats
ignroe quality issues of existing datassets
Delete table.md
Co-authored-by: Gabriel Moreira <gmoreira@nvidia.com> (62d76f8)
dataset: add TurkishRAGEvalRetrieval (Turkish passage retrieval, health) (#5615)
Add TurkishRAGEvalRetrieval, a Turkish passage retrieval task over Wikipedia health articles
Allowlist our TurkishRAGEval citation
The citation check could not find our own dataset card, so the key failed on the PR. Added it to the allowlist with the reason, same as the other unindexed entries.
The dataset now loads from mteb/turkish-rag-eval so it can be maintained
there, and the bibtex entry is gone until there is a paper; the repo link
stays in the reference field. (dea64e4)
This update adds utilities to mteb.data_cleaning to allow you to
prevent train leakage from other splits:
from mteb.data_cleaning import remove_train_leakage
cleaned = remove_train_leakage(task)
See cleaning task data for what counts as a
leak and for removing duplicates across splits with remove_duplicates.
chore: update the citation cache [skip ci] (`4e0a01b`)
4e0a01b)feat: Add feature to remove train leakage in mteb.data_cleaning (#5607)
feat: Add feature to remove train leakage in mteb.data_cleaning
used reference mechanism with leakage as one use
changes in docs from review
docs changes and better error messages (c5adf68)
dataset: add Spanish Wiki clustering v2 task (#5319)
dataset: add Spanish Wiki clustering v2 task
fix: avoid typo false positive in Spanish Wiki prompt
dataset: add Spanish Wiki clustering v2 statistics
Add SpanishWikiClustering v2 transformation script
Make SpanishWiki transformation publicly reproducible
Format SpanishWiki transformation script
Rebuild SpanishWiki data from Wikimedia source
Rename Spanish Wiki clustering task
Allowlist Spanish Wiki dataset citation
Rename Spanish Wiki descriptive statistics
Apply batched suggestions from code review
Co-authored-by: Kenneth Enevoldsen <kenevoldsen@pm.me>
Co-authored-by: Kenneth Enevoldsen <kenevoldsen@pm.me> (b1ddfa6)
model: add UltraBERT-Embed-150M (#5653)
model: add UltraBERT-Embed-150M
Delete mteb_mock_run_results.md (2596dbd)
Audio and video models previously shared fixed defaults, most notably a 30 second audio cap. Each model now uses the limits it declares itself, with the source cited next to the value in its implementation:
batch_size if a batch runs out of memory. Random crops (CLAP fusion, LanguageBind audio) are seeded per clip.facebook/wav2vec2-xls-r-2b-21-to-en loaded none of its pretrained weights; SeamlessM4T failed on every input with transformers 5; RzenEmbed never applied its frame sampling; UME-R1-7B sampled its reasoning text at random; ViCLIP looked for its vocabulary file relative to the working directory; Cosmos-Embed1 336p and 448p failed on a dtype mismatch; AuroLA decoded every video frame and ignored its pixel limit for video; LanguageBind flipped videos at random; CLAP fusion (clap-htsat-fused) encoded short clips through its long-clip path (the feature extractor marks a random clip as long, a training trick); ColQwen2.5-Omni decoded every frame; the Qwen3-VL reranker resampled already-sampled frames down to about 6; omni-embed-nemotron ran causal attention whenever a batch was padded (on transformers 5 its bidirectional mask override is never called), which also affected its text results.Scores for the affected audio and video models change. Results computed with earlier versions are not comparable; rerun them with overwrite_strategy="always".
The max_audio_length_s argument of the CNN14, MS-CLAP, MuQ-MuLan, SpeechT5 and Wav2CLIP wrappers is renamed to max_audio_length_seconds, matching the other audio wrappers. Audio length caps must be positive; zero or negative values now raise an error instead of being ignored or producing empty audio.
fix: shared video collators — keep audio/video order, enough frames for short clips
fix: shared video collators — keep audio/video order, enough frames for short clips (#5624)
fix: keep the dataset's column order in the video collators
In fps mode a clip under 2 s at 1 fps got a single frame, which every
Qwen3-VL video processor rejects (t:1 must be larger than temporal_factor:2). Use at least 4 frames, rounded down to a multiple of
2, like qwen-vl-utils. Wrappers whose processor takes any frame count
(BidirLM-Omni, PE-AV) set frame_factor=1; API wrappers that re-encode at a
fixed rate keep min_frames=1, frame_factor=1.
Clips with fewer frames than requested now repeat each frame in place (0, 0, 1, 1, ...) instead of looping back to the start.
test: skip the audio collator tests when torchaudio is not installed
refactor: fixed modality order in VideoCollator; make the frame minimum opt-in
Review feedback: put the row's modalities in one fixed order (text, image, video, audio) instead of only moving audio after video, and keep the old one-frame default so other models' scores don't change. min_frames is now an opt-in for wrappers whose processor needs more frames; frame_factor and the per-wrapper settings are dropped.
Review feedback: Qwen3-VL-Embedding, DNA-VL-STEER, Qwen3-VL-Reranker, UEmbed
and BidirLM-Omni set min_frames=4 like their reference code (qwen-vl-utils
FPS_MIN_FRAMES, BidirLM's video config), since their processor fails on a
single frame. InstructSentenceTransformerModel and CrossEncoderWrapper pass
min_frames through to the collator; the default stays 1. (5fb18ea)
fix: VLM embedders — WeMM tokenizer, VLM2Vec-V2 / UniME-V2 / UME-R1 image settings, Fusion text cap (#5628)
fix: WeMM-Embedding restores the checkpoint's tokenizer normalizer
482ebe3)fix: Update random models to use _content_hashes
fix: Update random models to use _content_hashes (#5644)
update random to use _content_hashes
update scores
upd revision and socres (de586a5)
The potion base and multilingual model cards list the Tokenlearn C4 datasets they were trained on. Set public_training_data to them.
Refs #5646 (04b9b75)
fix: LM head scores only the last position; cross-encoder search passes encode_kwargs
fix: LM head scores only the last position; cross-encoder search passes encode_kwargs (#5629)
fix: embedding wrappers score only the last position with the LM head
fix: Ops-MM uses logits_to_keep=1; pass encode_kwargs to cross-encoders unchanged
Audio Flamingo 3, Eagerworks and UniME-V2 compute logits for the last position only
Review feedback: pass logits_to_keep=1 in the other wrappers that only use
hidden states and whose model class supports it. (5438503)
chore: update the citation cache [skip ci] (`ecbf20e`)
ecbf20e)fix: audio/video support in cache wrapper (#5427)
feat(cache): add audio/video hashing and shared cross-task embedding cache
simplify tests
fix imports
don't recast dataset
fix test & typing
Co-authored-by: Rakshitha Ireddi <rakshithaireddi@Rakshithas-MacBook-Pro.local>
Co-authored-by: Roman Solomatin <36135455+Samoed@users.noreply.github.com> (760affd)
benchmarks: promote MAEB and MVEB to v1 (#5640)
benchmarks: promote MAEB and MVEB to v1
Rename the MAEB/MVEB benchmarks from (beta) to (v1), keeping the old names as aliases, and drop is_beta from their tasks. Add the MVEB citation.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> (50631ac)
fix: video encoders — VideoMAE, V-JEPA2 classifiers, ViCLIP, VideoPrism, PE-AV, LanguageBind, EBind
fix: video encoders — VideoMAE, V-JEPA2 classifiers, ViCLIP, VideoPrism, PE-AV, LanguageBind, EBind (#5627)
fix: VideoMAE fine-tuned checkpoints apply fc_norm after mean pooling
Review feedback: infer the whole PE-AV model in fp32 (as Meta's eval) instead of bf16 autocast with an fp32 DAC patch, choose the text head from get_modalities of the side the text is compared with, and choose the fused head from the batch keys. The grayscale image fix in LanguageBind is better done in the dataset.
PE-AV: keep output_dtypes from main (2683a41)
fix: batch Multimodal cross-encoders pairs to prevent OOM (#5641)
fixes #5638 (4b0a935)
fix: update revision for Qwen/Qwen3-VL-Reranker-8B
fix: forward cross-encoder batch size and progress options
fix: forward cross-encoder batch size and progress options (#5595)
fix: forward batch size to cross-encoder prediction
Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>
Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>
Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>
pass all encode kwargs
simplify tests
lint
Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>
Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsen@gmail.com> (ce68dc6)
fix: ModelMeta numbers, install extras and revision pins for audio/video models
fix: ModelMeta numbers, install extras and revision pins for audio/video models (#5631)
fix: embed_dim of audio models to the width they actually return
chore: update uv.lock for the new extras
upd lock
Co-authored-by: Roman Solomatin <36135455+Samoed@users.noreply.github.com> (743e930)
join_revisions rebuilt the results without passing the benchmark along, so
results loaded for a benchmark lost it after joining. get_benchmark_result()
then raised "No benchmark associated with these results", and
get_aggregated_scores() quietly fell back to the plain task mean. I now pass
self.benchmark in both return paths, same as select_models does since #5507. (133a5d6)
model: add DataScience-UIBK/uibk-embedding-v1 (#5623)
model: add DataScience-UIBK/uibk-embedding-v1
Address review: name and revision inline, OutputDType.BF16
Update mteb/models/model_implementations/uibk_models.py
Co-authored-by: Roman Solomatin <samoed.roman@gmail.com>
Co-authored-by: Roman Solomatin <samoed.roman@gmail.com> (02810eb)
fix API models — Amazon Nova MME, Gemini Embedding 2, video extra (#5630)
fix: Nova MME loading, embedding purpose and request limits
video extra for every video modelchore: update uv.lock for the new extras (642ff47)
Build API image with mteb-core (#5634)
api mteb core (dd4e109)
fix classic audio models — VGGish, YAMNet, CNN14, Encodec, msclap-2023, LAION CLAP, ImageBind, multi-modal-embed-small (#5626)
fix: VGGish uses the official embedding layer
fix: VGGish computes log-mel like the official vggish_input
c2b65ea)fix: speech audio models — SEW-D, WavLM-large, HeAR, voiceclap-large
fix: speech audio models — SEW-D, WavLM-large, HeAR, voiceclap-large (#5625)
fix: SEW-D pools the base model's final output, not the half-rate encoder
2b96d9d)fix: Fill output_dtypes in ModelMeta of models (#5620)
Fill output_dtypes in ModelMeta of models
test: drop implementation-level _upcast_low_precision tests
Behaviour is covered by the wrapper and AbsEncoder delegation tests.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> (276fbe6)
docs: Update documentation about running benchmark
docs: Update documentation about running benchmark (#5621)
update docs
update doc (2763bc7)
fix: split the package into mteb and mteb-core (#5532)
feat: split the package into mteb and mteb-core
make changes in scripts
guards for not installing mteb and mteb-core simultaneously
update tests
add messaging at all places where torch is required
remove mteb-core[run], reverted docs changes, changes in .yml file and added tests
update CI, remoce unnecessary tests and changes from review
add full_install to test requiring torch
remove benchmark file
changes from review
marked core to test and test modules which can be run on mteb-core
added mteb.api.app, mteb.data_cleaning and mteb.leaderboard so they will not import top-level torch in future
fx failures
fix for loweset dependencies CI
update test docstring (4caaa9d)
clean: Move scripts to subfolder (#5576)
move scripts
upd lint
remove str fallback
move to scripts/tools
fix folder path (fcd9435)
fix lower dependency test (`b4d6993`)
fix lower dependency test (b4d6993)
fix: audio video input config (#5454)
fix: correct four audio input configs that did not match their checkpoint
padding=True. ClapFeatureExtractor resolves it as
padding if padding else self.padding, so it replaced the checkpoint's
"repeatpad" and zero-padded sub-10 s clips. Measured: BeijingOpera +5.9,
GunshotTriangulation +3.4, FSDD +5.2.MS-CLAP's 48 kHz is also wrong but is fixed upstream in #5404, so it is only annotated there, not duplicated.
Each wrapper now states, in a line at the value, where its sampling rate, audio length or frame budget comes from, so "native, from the checkpoint" is distinguishable from "our cost cap" without re-deriving it.
Four kinds of note: native (Whisper/Qwen2-Audio chunk_length=30, AST's 1024 frames, HeAR's 2 s, Cosmos/X-CLIP 8 frames, V-JEPA's per-checkpoint 64/32/16, VideoPrism 16); reference-matching (ImageBind mirrors its own data.py defaults); mteb-side cap (the speech SSL family, the fps=2/max_frames video budget, Gemini's cost cap); and unverified (SeamlessM4T's 5 s, traced to #2751 where it was fit to BeijingOpera's 1.7 s clips).
Also records the values deliberately left alone: CNN14, MuQ-MuLan and Wav2CLIP are each trained on 10 s clips, but lowering a cap removes signal and is untested, so they stay at 30 s; and the omni audio caps differ legitimately (Qwen2.5-Omni 300 s vs BidirLM's own 30 s) rather than being inconsistent.
moeb_run.py -- runs one model's share of the pool. Work is derived rather than
assigned: eligibility is task.modalities subset of model.modalities, minus what
already exists in the results repo at the pinned mteb version, sharded by content
hash so parallel runners never collide and no spreadsheet is needed. Refuses to
run on a different mteb version, since results across versions cannot be pooled.
--dry-run reports the work without a GPU; --submit opens a results PR.
The eligibility rule also does the tiering for free: Qwen2.5-Omni-3B resolves to 685 tasks, Cosmos-Embed1 (text+video) to 153, with no hand-maintained matrix.
moeb_smoke.py -- loads a model and encodes a tiny synthetic batch per declared modality, failing on crashes, wrong shapes, NaN, all-zero embeddings, or identical embeddings for different inputs. Exit code gates a campaign.
This already found a real defect: the AST wrapper calls .numpy() on the numpy
array AudioCollator hands it (ast_model.py:66) and torch.cats it with a torch
tensor, so it raises before inference. Same class as the MuQ buffer bug in #5404;
both are likely broken since #4069 moved the collator to numpy, and both have
published results predating that.
run_audio_eval.sbatch -- one GPU per model, expandable_segments, logs per job.
Each value below was checked against the model's own source -- its HF config, its upstream repo, or its paper -- rather than inferred. Wrappers now either match a declared native limit, or carry no cap where the architecture declares none, or state explicitly that the cap is an mteb memory guard.
Uncapped, because the architecture declares no limit and pools internally:
chunk_length_s: null -- a fully
convolutional streaming codec.These four had a 30 s cap that discarded signal for no architectural reason, on models whose own benchmarks (HEAR, X-ARES) pass the full clip.
Corrected to declared native values:
Three wrappers were broken before this change, all by the same root cause -- PR #4069 made the collator yield numpy arrays while these still expected torch:
All three now pass the wrapper smoke test. This is the same family as the MuQ buffer bug being fixed in #5404.
Verified: 2622 model-meta tests pass; ruff clean; EnCodec, AST and CNN14 smoke -tested end to end on CPU.
docs: reduce every config note to a single line naming its source
feat: window long audio instead of truncating it, for encoders with no declared limit
wav2vec2, HuBERT, WavLM, data2vec, UniSpeech, MMS and SEW-D declare no maximum input length: their positional encoding is convolutional, so the architecture is length-agnostic and only memory bounds the input. mteb previously truncated them at a uniform 30 s, which discards most of a long clip and matches no framework that evaluates these models.
Every reference implementation windows and pools instead:
duration = duration or training_duration) and slides it.New mteb/models/audio_windowing.py implements split-into-windows and mean-pool. Following pyannote's rule, each wrapper's window is the duration the model was actually trained on, with the reference on the parameter:
wav2vec2 15.6 s pretraining crop, arXiv:2006.11477 S4.2
SEW-D 20 s declared: max_position_embeddings=512 at 25 frames/s
data2vec 10 s X-ARES reference encoder
HuBERT, WavLM, UniSpeech, MMS 20 s s3prl UnfoldChunkByFrame
truncation=True / max_length are dropped from these feature-extractor calls,
since windowing now bounds length; the feature extractor no longer cuts.
Verified: a 90 s clip now pools over 5 windows and its embedding differs from the truncated first-20 s embedding at cosine 0.733 -- i.e. truncation had been discarding most of the signal. All seven wrappers pass the smoke test; 2622 model-meta tests pass; ruff clean.
Going through all 45 audio wrappers one by one surfaced two values still not matching their checkpoint, and three wrappers with no source note:
max_speech_positions: 4000 at 50 fps
(conv_stride 320x). The max_length: 450 in the same config is the text side
(max_text_positions: 450), not the audio input.max_position_embeddings: 920 at 30 ms per
frame (10 ms hop with conv_stride 3).Also simplifies audio_windowing: drops the unused normalize parameter, replaces
the manual accumulation loop with np.add.at/bincount, and trims the module
docstring to the four references. 112 -> 76 lines, same behaviour (re-verified:
a 50 s clip still splits 20/20/10 and pools to the same values).
Every audio wrapper now carries a single-line note naming the source of its setting; total added comment lines across the branch is 72.
Verified: 2622 model-meta tests pass; ruff clean; SpeechT5, HuBERT, EnCodec, CNN14 and CLAP pass the smoke test.
Checking the three wrappers that had no source note found two of them documented against the wrong upstream:
Values that are genuinely ours are now derived rather than picked:
Verified: 2622 model-meta tests pass; ruff clean; HuBERT, SpeechT5, EnCodec, CLAP and CNN14 pass the smoke test.
Two values were still ours by construction. Both turned out to have a model-specific source once looked for properly.
Because these are windows rather than cuts, the window size only affects compute and context per forward pass, not coverage -- so taking the model's own documented number costs nothing.
Every value in the branch now traces to the checkpoint's config, the model's own repo or paper, or a named cross-framework convention. None is derived from mteb's own task statistics.
Verified: 2622 model-meta tests pass; ruff clean.
docs: reduce every note to the source of the value, with no commentary
feat: window strict-cap encoders too, using the declared cap as the window
A declared limit says how large each window may be, not how much of the clip to discard. X-ARES applies exactly this: its reference encoder windows Whisper at 10 s despite Whisper declaring 30 s natively.
Truncating at the declared cap also creates a measurement artefact rather than a capability difference. On a 600 s clip, CLAP covers the whole span via its fusion mechanism while AST sees 1.7% of it -- a gap produced by how each upstream handles overflow, not by embedding quality.
Windowed here, each at its own declared size:
Left alone, because they already cover the whole clip themselves:
Verified: a 90 s clip through Whisper now becomes 3 windows and its embedding differs from the truncated 30 s one at cosine 0.945. 2622 model-meta tests pass; ruff clean.
Checking each remaining model's own source rather than assuming:
audio_chunk_length = 120 and its
model card exposes that value as a user-settable knob, so it chunks internally.
Our 30 s cut it to a quarter of its own chunk and bypassed that mechanism.Left alone, because they already cover the whole clip: LAION-CLAP (fusion, verified), ImageBind (3 clips across the full duration), VGGish, YAMNet, CNN14, Dasheng, EnCodec (internal pooling), and now OmniVinci.
Verified: AST on a 60 s clip now uses 6 windows and differs from the truncated 10.24 s embedding at cosine 0.949. 2622 model-meta tests pass; ruff clean; AST, Whisper and HuBERT pass the smoke test.
Each note now points at the exact file it cites -- the checkpoint's config on the Hub, the reference implementation on GitHub, or the paper on arXiv -- so a value can be checked without hunting for its source.
All 41 distinct links were fetched and return 200. Examples:
huggingface.co/speechbrain/cnn14-esc50/blob/main/hyperparams.yaml huggingface.co/Qwen/Qwen2.5-Omni-7B/blob/main/preprocessor_config.json github.com/jimbozhang/xares/blob/main/example/data2vec/data2vec_encoder.py github.com/s3prl/s3prl/blob/main/s3prl/dataset/chunking.py github.com/facebookresearch/ImageBind/blob/main/imagebind/data.py ai.google.dev/gemini-api/docs/embeddings
The only notes left without a link are the three that cite nothing external: the random baseline (never encodes its input) and a one-line note that AudioCollator yields numpy.
Verified: all links resolve; 2622 model-meta tests pass; ruff clean; Whisper, AST, EnCodec, CLAP, CNN14 and HuBERT pass the smoke test.
scripts/moeb_run.py, scripts/moeb_smoke.py and the slurm launchers are MOEB campaign tooling, not library code, and do not belong in a wrapper-config fix. Removed here so this PR contains only changes under mteb/.
They stay in the working tree for local use and can be proposed separately if they are wanted upstream -- the smoke test in particular found four wrappers that raise before producing an embedding (AST, EnCodec, SeamlessM4T, and the MuQ bug in #5404).
docs: put each source link on its own line, to keep comment lines short
docs: link the CLAP padding note to ClapFeatureExtractor
fix: cap Nova 2 audio at its documented limit; document the remaining video wrappers
Extending the audio audit to video and omni wrappers found one real defect and four undocumented values.
Amazon Nova 2 documents "8K tokens or 30s of video and 30s of audio". The wrapper capped video at 30 s via NOVA_MAX_VIDEO_SECONDS but applied no cap to audio at all, so a long clip was sent whole to an endpoint that accepts 30 s. Audio is now capped the same way, with the limit cited.
Documented, each against its own source:
Also checked and found clean: no video wrapper has the positional-collator or numpy/torch defects that broke four audio wrappers -- every FramesCollator and VideoCollator call passes keyword arguments.
Verified: 2622 model-meta tests pass; ruff clean.
Checking each video value against its source rather than only documenting it found one that did not match.
UNITE's model card runs video with fps: 1 and max_frames: 32. The wrapper
instead took a fixed 32 frames, which over-samples short clips: a 10 s video got
32 frames where UNITE's own config would take 10. The wrapper already accepted
fps and max_frames, so this just sets them.
Also tightens the ViCLIP citation to the line that declares the value (video_input_num_frames = 8 in Data/InternVid/viclip/viclip.py) rather than the repository root.
Verified against source, unchanged: Cosmos-Embed1 num_video_frames=8, X-CLIP num_frames=8, V-JEPA frames_per_clip 64/32/16, VideoPrism f16 in the checkpoint name, InternVideo2 f8 in the checkpoint name, LanguageBind vision_config.num_frames=8, VideoMAE reads num_frames from the checkpoint, UEmbed uses the clip's own length.
Verified: 2622 model-meta tests pass; ruff clean.
docs: cite the exact line declaring ViCLIP's frame count
fix: remove the last arbitrary audio cap, correct two wrong source notes
A full census of the registry (912 models, 143 audio/video-capable across 60 files) turned up three wrappers whose config did not match their own source:
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both are plain encoders, the class where window-plus-mean is measured to beat truncation (10/10 on long-audio retrieval, 4/4 on classification):
Left alone deliberately: BioVITA truncates via the authors' own energy-onset crop, wav2clip already pads to the batch maximum without truncating, and the LLM and API wrappers (Qwen2-Audio, bidirlm, jina, Nova, Gemini) are not the class this was measured on -- windowing them means N language-model passes over a joint audio-text sequence, which is a different operation.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both values were right but unfindable from the link. BioVITA's card only says it uses CLAP-HTSAT-unfused; nb_max_samples=480000 is in CLAP's own preprocessor_config.json. UNITE's fps=1/max_frames=32 are in its README.
Every cited key=value in the audio and video wrappers now re-derives from the linked source (42/42, checked by fetching each one).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A pass over the frame settings of all 77 video models, which the earlier key=value check could not reach because these are written as prose:
Verified unchanged: V-JEPA (8 checkpoints) and VideoPrism (4) each match the frame count in their own checkpoint name, and InternVideo2's f8 likewise.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
HeAR here is Google Health Acoustic Representations, not the HEAR benchmark at hearbenchmark.com -- similar name, different project, so the hear-eval-kit link was wrong. The checkpoint's own card declares the protocol directly: window 2.0 s, hop 2.0 s, pooling over windows mean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Replaces the uniform 30 s cap that had spread by copying and matched no model's configuration. Each model now uses the length its own config declares:
Verified by encoding a clip that changes tone partway through: capped models return an identical embedding for the clip and its prefix, uncapped and windowing models do not. 10/10 as expected.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
wav2vec2, MMS, HuBERT, data2vec and UniSpeech default to no cap, but the parameter is kept so a length can be set deliberately -- to ablate it, or to fit a smaller card -- rather than being unreachable. It flows to AudioCollator, which already truncates when max_samples is given.
WavLM keeps its 90 s default, with the reason stated where the value is set: transformers has no SDPA or flash kernel for it, so attention is quadratic and 90 s is the most that fits batch 32 in 80 GB.
Verified per model: the default sees audio past the ablation point, and passing the kwarg makes the clip and its prefix encode identically.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Found by checking that every source note sits directly above the value it documents and that the number in the note appears in that line:
The three remaining max_audio_length / max_samples parameters are in samples, not seconds, so they keep a distinct name.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The window/hop/mean I cited sits under the card's "MAEB(beta, audio-only) evaluation" heading, and the package ships maeb_run_config.json, described as the saved MAEB inference settings for that run. MAEB is our own benchmark, so that is not the authors prescribing a protocol -- it is our settings written back into the card, and citing it was circular.
What the card does declare is architectural: mel+PCEN spectrograms of shape [B, 1, 192, 128] built from 2-second clips, and clip_seconds=2.0 in config. So HeAR takes its declared 2 s, like every other model here, using the checkpoint's existing centre crop.
No wrapper windows audio any more.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Re-verified the values that are prose rather than key=value: Audio Flamingo 600 s ("Max Audio Length: 10 Minutes"), PE-AV 400 s (max_position_embeddings 10000 at 48 kHz / 1920 = 25 Hz), SeamlessM4T 10 s (MAX_INPUT_AUDIO_LENGTH = 10), OmniVinci 120 s (audio_chunk_length = 120), Nova 30 s (AWS: "8K tokens or 30s of video and 30s of audio").
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A pass over every audio and video wrapper for values that are declared but never reach the model:
Also checked every hardcoded sampling rate against the model's own config. CNN14 44.1 kHz, BioVITA 48 kHz, MuQ 24 kHz and VoiceCLAP 16 kHz all agree with their sources. MSCLAP's 48 kHz remains wrong (both its configs declare 44,100); that is PR #5404's fix, not duplicated here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both verified end to end: MuQ returns (2, 512) and MSCLAP (2, 1024), finite and distinct, with MSCLAP reporting 44100 Hz for 2022 and 2023.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
For any clip longer than its configured duration, msclap takes a random window:
start_index = random.randrange(
audio_time_series.shape[0] - audio_duration * sample_rate)
so the same model on the same data returned different embeddings each run. The wrapper's max_audio_length_seconds was declared but never applied, so nothing prevented it.
It now defaults to the duration msclap itself declares (5 s for 2022, 7 s for 2023) and is passed to the collator. Cutting at exactly that length sends msclap down its deterministic repeat-or-trim branch instead.
Verified: encoding one 30 s clip twice now returns identical embeddings.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Each note gives the value and its source; the reasoning belongs in the PR, not in every constructor. Trims MSCLAP's two notes and VideoMAE's.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The comment claiming every checkpoint keeps "repeatpad" + "fusion" was wrong; only clap-htsat-fused declares fusion.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
AST: ASTFeatureExtractor.call takes (raw_speech, sampling_rate, return_tensors, **kwargs), so truncation and padding were swallowed. Output is (1, 1024, 128) with or without them -- the 10.24 s cap is unconditional. The two arguments are removed and the note no longer credits them.
Citations, all same value, corrected provenance:
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Found by an independent review and reproduced here:
SeamlessM4T and wav2clip verified end to end.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
At 18 sites the value sat directly under a link to qwen-omni-utils or qwen-vl-utils, which documents the fps above it. Both ship FPS_MAX_FRAMES = 768, so the adjacency read as if 64 were sourced when it is a cap mteb chose for cost. Each site now says so.
UEmbed keeps its plain note: 64 there is the checkpoint's own MAX_FRAMES.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The encoder's constraint is a position count, not a duration, so 27.6 was a derived constant that looked arbitrary in review. It is now computed at load from max_position_embeddings and conv_stride, and the number no longer appears in the source. mteb already reads text limits off the model the same way (max_tokens from max_seq_length).
The attribute is max_feature_frames, not max_frames: these are filterbank frames, one per hop_length ms, and max_frames means video frames everywhere else in the repo.
The value is logged at load, since a silent cap is how the original 30 s default went unnoticed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
max_position_embeddings x conv_stride x squeeze_factor = 512 x 320 x 2 = 327,680 samples, identical across all three checkpoints, so 20.5 no longer appears in the source. Logged at load like M-CTC-T.
These two numbers are the span of the models' learned relative-position embeddings rather than hard limits: both run past them without complaining, with zero positional bias beyond the trained range. Neither supports SDPA, so batch 32 needs a cap regardless (244 s and 188 s respectively), and the declared spans sit well inside that.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
max_speech_positions x conv_stride = 4000 x 320 = 1,280,000 samples, so 80.0 no longer appears in the source. Logged at load, like M-CTC-T and SEW-D.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The step-based sampler kept only the head of the clip whenever it had fewer than twice the target frames (15 frames -> 8 gave 0-7) and always dropped the tail. Use np.linspace, as qwen-vl-utils, UNITE, Jina, Rzen, OmniRetriever, LanguageBind and PE-AV do upstream.
fix: keep UEmbed at fps=1, its value on main
fix: encode audio clips one at a time where padding leaks into pooling
MuQ-MuLan, EnCodec, wav2clip, Qwen3-Voice and CNN14 zero-padded each batch to its longest clip and pooled over the padding unmasked, so a clip's embedding depended on its batch-mates (3 s clip alone vs batched with a 40 s clip: cos 0.33 to 0.91). Now 1.0 for all five.
CNN14 also uses the checkpoint's classify_batch front end (compute_stft -> spectral_magnitude -> compute_fbank) instead of compute_features.
wav2vec2, HuBERT, data2vec, MMS, UniSpeech, SEW-D, WavLM, SpeechT5, M-CTC-T and SeamlessM4T padded each batch to its longest clip. Padding reaches feature normalisation and group norm, so a clip's embedding depended on its batch-mates (3 s clip alone vs batched with a 40 s clip: cos 0.15 to 0.81, also on main). Now 1.0 for every measured model.
fix: typecheck the evenly spaced frame indices
fix: batch layer-norm speech encoders, share audio cap helpers
docs: what's new entry for the audio and video input changes
test: update random-baseline video scores for evenly spaced frames
The mock videos have 24 frames and the random baselines sample 10. The old sampler picked frames 0-18; the evenly spaced one picks 0-23, so the hashed embeddings and the six Linux/Windows expected scores change.
The processor's audio keyword is audios before transformers 4.57 and
audio from v5, which rejects audios. Call the feature extractor
directly, as the processor does internally on every version; features
are identical to the v5 audio= path.
Padding changes group-norm embeddings (min cosine 0.12 batched vs alone on real wav2vec2-base clips, with or without an attention mask), and a padded batch of long uncapped clips runs out of memory on 80 GB. All wav2vec2, HuBERT, MMS, UniSpeech, SEW-D, SpeechT5, data2vec, WavLM, M-CTC-T and SeamlessM4T wrappers now encode one clip per forward; the reason is logged once per run.
Real clips up to 1272 s, batch 32 vs 8 vs 1. Each wrapper below now gives identical embeddings at every batch size and fits on 80 GB.
The mteb default fps=2 with a 64-frame cap stays the fallback for models that declare no frame setting.
Most audio wrappers already reject a cap <= 0 at encode time through AudioCollator or seconds_to_samples, so the extra check in their init was redundant. Keep it only in the five wrappers that pass the cap straight to the feature extractor, where a 0 otherwise produced empty audio (SEW-D, SpeechT5, M-CTC-T) or an unclear crash (WavLM, Qwen2-Audio).
All of these were already broken on main; each was checked on real videos.
Measured on real clips and videos (batched vs single, min cosine):
Memory alone no longer forces batch_size=1; lower batch_size for tasks with very long clips.
The lowest-dependencies CI job has no Qwen2AudioForConditionalGeneration, so Qwen2-Audio and WavLM failed on the import before reaching the cap check. Run the check first, as SEW-D, SpeechT5 and M-CTC-T already do.
No model sets it any more: omni-embed-nemotron batches after its attention fix, and e5-omni, Tevatron and AuroLA batch within 0.99.
Co-authored-by: Roman Solomatin <samoed.roman@gmail.com>
Co-authored-by: AdnanElAssadi56 <adnan@tyce.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Roman Solomatin <samoed.roman@gmail.com> (5621901)
refactor: migrate XQuAD retrieval task to v2 (#5617) (6645cf3)
refactor: migrate WMT19 retrieval task to v2 (#5616) (eb3cdd1)
Add JMullings/spectral-embed-v1-140m model (#5601)
Add JMullings/spectral-embed-v1-140m model
Address review comments: simplify model class, fix metadata, remove test file
Lint Completed
Apply suggestion from @Samoed
lint
Add n_embedding_parameters to spectral model metas
MTEB requires every non-historic model to declare n_embedding_parameters. The spectral embedding metas left this as None, failing test_n_embedding_parameters for two registry entries.
Fixes: test_n_embedding_parameters[model_meta118] Fixes: test_n_embedding_parameters[model_meta705]
Co-authored-by: Roman Solomatin <samoed.roman@gmail.com>
Co-authored-by: Roman Solomatin <36135455+Samoed@users.noreply.github.com> (8f26694)
fix: apply script filter in exclusive_language_filter branch of filter_languages
fix: apply script filter in exclusive_language_filter branch of filter_languages (#5612)
fix: apply script filter in exclusive_language_filter branch of filter_languages
#5608 applied the script filter to the non-exclusive branch of AbsTask.filter_languages, but the exclusive_language_filter=True branch still only checked the language via contains_languages(langs) and ignored script=.
As a result, filter_languages(languages=[rus], script=[Latn], exclusive_language_filter=True) kept the pure-Cyrillic 'rus' subset (rus-Cyrl) even though the user asked for Latin script -- the script part was silently dropped.
Mirror the non-exclusive fix: under exclusive filtering a subset must be entirely within the (language, script) filter, so every langscript in the subset must also match the requested script(s). script=None keeps the previous behaviour unchanged.
Co-authored-by: Roman Solomatin <36135455+Samoed@users.noreply.github.com> (4d5f6cc)
pplx-embed-v1 (#5610)add requirement group pplx (930659e)
0e233f7)fix: apply script filter in AbsTask.filter_languages
fix: apply script filter in AbsTask.filter_languages (#5608)
fix: apply script filter in AbsTask.filter_languages
script= was ignored when filtering subsets: contains_script was fed a full language-script code (e.g. "rus-Cyrl") which never matched bare script codes (e.g. "Cyrl"), and with languages=None contains_language short-circuited to True for every subset. Mirror the fix in TaskResult.get_score (#5490): keep a subset only if it matches the language AND, when a script filter is given, the extracted script code. Fixes #5594.
script= was ignored when filtering subsets: contains_script was fed a full language-script code (e.g. "rus-Cyrl") which never matched bare script codes (e.g. "Cyrl"), and with languages=None contains_language short-circuited to True for every subset. Mirror the fix in TaskResult.get_score (#5490): keep a subset only if it matches the language AND, when a script filter is given, the extracted script code. Fixes #5594.
Co-authored-by: Roman Solomatin <36135455+Samoed@users.noreply.github.com> (2d24bfe)
fix: handle lowercase maxsim in ModelMeta.from_hub (#5605) (`86e2051`)
docs: Update zensical. (activate social, llmstxt)
feat: Add feature to remove empty and tiny samples in mteb.data_cleaning
feat: Add feature to remove empty and tiny samples in mteb.data_cleaning (#5493)
mteb.data_cleaninglintter
changes from review
renaming, docstring changes, and other minor fixes
update tests
update docs for task based threshold
update docs
made text wrapping in docs
changes in docs
updates test naming, rename Filter and other changes from review
refactor and move filter to individual files
modified each filter to act both as lower and upper bound
added quantile feature
removed custom quantile class
minor update in doc (756d399)
Add ModelMeta for alphabrothers/alpha-sts-v0 (#5592)
Add ModelMeta for alphabrothers/alpha-sts-v0
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Roman Solomatin <samoed.roman@gmail.com> (131ab46)
mteb.data_cleaning gains four filters that remove samples whose
content is too small, or too large, to be worth scoring: remove_by_text_length, remove_by_image_size,
remove_by_audio_duration and remove_by_video_duration. Each takes a lower bound, an upper bound, or both, and
returns a cleaned copy named after the filters applied to it:
from mteb.data_cleaning import remove_duplicates, remove_by_text_length
cleaned = remove_by_text_length(remove_duplicates(task), min_length=1)
print(
cleaned.metadata.name
) # MassiveIntentClassification (remove_duplicates, remove_by_text_length)
A bound can also be given as a quantile of the task's own sizes, e.g. min_quantile=0.05 for about the shortest 5%,
taken per split and per subset. That needs no threshold of its own, which helps for a multilingual task, where a length
in characters is not comparable between languages.
Each filter measures its content the way the task's descriptive statistics do, except that texts are measured without
their surrounding whitespace, so you can judge a threshold before downloading any data: a task whose min_text_length
is 0 contains empty texts, which min_length=1 removes. The thresholds have no default, as what counts as too small
depends on the task.
remove_duplicates now compares retrieval documents and queries the way the model reads them. Queries that share their
text but differ in their instruction or conversation are no longer merged, which previously halved the queries of
instruction-following tasks such as Core17InstructionRetrieval.
One column per quarter.