open-clip-torch

What's Changed

• Initial work on adding local-dir: schema for model & tokenizer loading from local folder by @rwightman in #1069
• Add --force-context-length argument by @rwightman in #1080
• fix CustomTextCLIP.no_weight_decay by @thelaao in #1082
• Fix an issue where CustomText models not gettin proper pos embed interpolation by @rwightman in #1085
• Removing redundent checks by @shreyaskamathkm in #1096
• PE Core by @rwightman in #1097
• Alternative tokenizer support for CLIPS by @rwightman in #1081
• Wire up custom attention block via config by @rwightman in #1086
• An alternative text masking helper fn by @rwightman in #1084
• Update min reqs, test on python 3.10 by @rwightman in #1098
• Resize timm image encoders by @rwightman in #1099

New Contributors

• @thelaao made their first contribution in #1082
• @shreyaskamathkm made their first contribution in #1096

Full Changelog: v2.32.0...v3.0.0

What's Changed

• API for getting intermediate image and text features, forward_intermediates() by @rwightman in #1035
• Updated PyTorch code in README by @Ja-Tink in #1050

New Contributors

• @Ja-Tink made their first contribution in #1050

Full Changelog: v2.31.0...v2.32.0

What's Changed

• Add SigLIP2 models by @rwightman in #1033

Full Changelog: v2.30.0...v2.31.0

What's Changed

• Support using timm optimizers for alternative to adamw default by @rwightman in #979
• add missing ViTamin configs by @xywei00 in #978
• Experimenting with alternative siglip loss impl for better dist scaling by @rwightman in #971

New Contributors

• @xywei00 made their first contribution in #978

Full Changelog: v2.29.0...v2.30.0

What's Changed

• All default pretrained weights pushed to HF hub by @rwightman in #970

Full Changelog: v2.28.0...v2.29.0

• Fix missing space in error message
• use model flag for normalizing embeddings
• init logit_bias for non siglip pretrained models
• Fix logit_bias load_checkpoint addition
• Make CoCa model match CLIP models for logit scale/bias init
• Fix missing return of "logit_bias" in CoCa.forward
• Add NLLB-CLIP with SigLIP models
• Add get_logits method and NLLB tokenizer
• Remove the empty file src/open_clip/generation_utils.py
• Update params.py: "BatchNorm" -> "LayerNorm" in the description string for "--lock-text-freeze-layer-norm"

• Add CLIPA-v2 models
• Add SigLIP models
• Add MetaCLIP models
• Add NLLB-CLIP models
• CLIPA train code
• Minor changes/fixes
  • Remove protobuf version limit
  • Stop checking model name when loading CoCa models
  • Log native wandb step
  • Use bool instead of long masks

• Add EVA models
• Support serial worker training
• Fix Python 3.7 compatibility

• Add DataComp models

• Enable int8 inference without .weight attribute

• Update push_to_hf_hub

• Fixes for context_length and vocab_size attributes

• Fixes for context_length and vocab_size attributes
• Fix --train-num-samples logic
• Add HF BERT configs for PubMed CLIP model

• Add improved g-14 weights
• Update protobuf version

• Add convnext_xxlarge weights
• Fixed import in readme
• Add samples per second per gpu logging
• Fix slurm example

• Move dataset mixtures logic to shard level
• Fix CoCa accum-grad training
• Safer transformers import guard
• get_labels refactoring

• Add support for dataset mixtures with different sampling weights
• Make transformers optional again

• Updated convnext configs for consistency
• Added input_patchnorm option
• Clean and improve CoCa generation
• Support model distillation
• Add ConvNeXt-Large 320x320 fine-tune weights

• Make transformers optional
• Add MSCOCO CoCa finetunes to pretrained models

• coca support and weights
• ConvNeXt-Large weights

• hf-hub:org/model_id support for loading models w/ config and weights in Hugging Face Hub

• Added a ViT-bigG-14 model.
• Added an up-to-date example slurm script for large training jobs.
• Added a option to sync logs and checkpoints to S3 during training.
• New options for LR schedulers, constant and constant with cooldown
• Fix wandb autoresuming when resume is not set
• ConvNeXt base & base_w pretrained models added
• timm- model prefix removed from configs
• timm augmentation + regularization (dropout / drop-path) supported

• Fix wandb collapsing multiple parallel runs into a single one

• Fix braceexpand memory explosion for complex webdataset urls

• Fix release

• wrapped patchdropout in a torch.nn.Module

• relax protobuf dependency
• override the default patch dropout value in 'vision_cfg'

• better support for HF models
• add support for gradient accumulation
• CI fixes
• add support for patch dropout
• add convnext configs

• add multilingual H/14 xlm roberta large

• fix setup.py _read_reqs

• pretrained B/32 xlm roberta base: first multilingual clip trained on laion5B
• pretrained B/32 roberta base: first clip trained using an HF text encoder

• Add missing hf_tokenizer_name in CLIPTextCfg.

• Fix #211, missing RN50x64 config. Fix type of dropout param for ResNet models
• Bring back LayerNorm impl that casts to input for non bf16/fp16
• zero_shot.py: set correct tokenizer based on args
• training/params.py: remove hf params and get them from model config

• Implement grad checkpointing for hf model.
• custom_text: True if hf_model_name is set
• Disable hf tokenizer parallelism

• Generalizable Text Transformer with HuggingFace Models (@iejMac)

• Support for custom text tower
• Add checksum verification for pretrained model weights

• ViT-B/32 trained on Laion2B-en
• add missing openai RN50x64 model

• ViT-B/16+
• Add grad checkpointing support
• more robust data loader