Source-linked AI summary
SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features
Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier Hénaff, Jeremiah Harmsen, Andreas Steiner, Xiaohua Zhai
TL;DR
SigLIP-style encoders lacked a single open model incorporating recent advances while improving localization and dense representations. SigLIP 2 unifies decoder-based pretraining, self-supervised losses, and data curation, achieving improvements across core, localization, and dense prediction tasks, with multilingual and aspect-ratio variants. Its reported fairness gains are uneven across evaluations, with only minor or no benefits on some geographic and income-level comparisons.
Problem
Existing open releases did not combine the full breadth of recent vision-language training improvements into one model while addressing localization and dense-representation weaknesses.
Method
SigLIP 2 combines sigmoid-loss training with decoder-based pretraining, self-distillation, masked prediction, multilingual data, and de-biasing techniques.
Results
SigLIP 2 improves zero-shot classification, VLM transfer, localization, and dense prediction, while attaining more balanced quality across culturally diverse data.
Takeaways & Limitations
NaFlex supports multiple resolutions and native aspect ratios, while backward compatibility allows SigLIP users to adopt improvements through new weights and a multilingual tokenizer.
Takeaways & Limitations
On Dollar Street results by income level and GeoDE results by geographic region, matching SigLIP 2 models show only minor or no benefits over SigLIP.
Abstract
from arXiv · showhide
We introduce SigLIP 2, a family of new multilingual vision-language encoders that build on the success of the original SigLIP. In this second iteration, we extend the original image-text training objective with several prior, independently developed techniques into a unified recipe -- this includes captioning-based pretraining, self-supervised losses (self-distillation, masked prediction) and online data curation. With these changes, SigLIP 2 models outperform their SigLIP counterparts at all model scales in core capabilities, including zero-shot classification, image-text retrieval, and transfer performance when extracting visual representations for Vision-Language Models (VLMs). Furthermore, the new training recipe leads to significant improvements on localization and dense prediction tasks. We also train variants which support multiple resolutions and preserve the input's native aspect ratio. Finally, we train on a more diverse data-mixture that includes de-biasing techniques, leading to much better multilingual understanding and improved fairness. To allow users to trade off inference cost with performance, we release model checkpoints at four sizes: ViT-B (86M), L (303M), So400m (400M), and g (1B).
1. Introduction
SigLIP 2 unifies several improvements beyond the original CLIP-style recipe in one open multilingual model family. It retains strong core vision-language capabilities while addressing localization, dense features, multilingual use, and aspect-ratio handling.
- SigLIP 2 combines prior improvements into a single model family that excels at zero-shot classification, retrieval, and VLM feature extraction.The unified recipe builds on SigLIP while targeting weaknesses of vanilla CLIP-style models.
- SigLIP 2 provides strong multilingual vision-language performance with a single model across languages and cultural contexts.
- Self-supervised and decoder-based losses improve dense features for segmentation and depth estimation, while also improving localization tasks.Referring expression comprehension is given as an example of localization.
- The NaFlex variant supports multiple resolutions while preserving native image aspect ratio for aspect-sensitive applications such as document understanding.
- SigLIP 2 preserves the original SigLIP architecture, allowing existing users to swap model weights and the multilingual tokenizer.
2. Training recipe
SigLIP 2 combines SigLIP’s sigmoid-loss training with decoder-based pretraining, self-supervised objectives, curated-data fine-tuning, and NaFlex variants. The recipe preserves SigLIP’s architecture while targeting semantic, localization, dense-feature, multilingual, and aspect-ratio capabilities.
- Core recipe: SigLIP 2 combines the original SigLIP objective with decoder-based pretraining, self-distillation, and masked prediction.The decoder-based stage uses captioning and referring-expression objectives, while the self-supervised losses target local semantics and dense representations.
- Architecture and data: SigLIP 2 retains SigLIP’s architecture for weight swapping, uses a multilingual Gemma tokenizer, and trains on WebLI data spanning 109 languages with bias filtering.The mixture contains 90% English web pairs and 10% non-English web pairs; the smallest models additionally use active-sample-selection fine-tuning.
- Decoder-based pretraining: The decoder predicts captions, bounding-box coordinates for region descriptions, and region-specific captions conditioned on bounding boxes.These three targets are trained through separate decoder forward passes, with region-caption pairs automatically annotated from alt-texts and object categories.
- Self-supervised objectives: Self-distillation trains local student views to match a teacher representation from the full image, while masked prediction matches teacher features at masked patch locations.The teacher parameters are obtained as an exponential moving average of the student parameters; masked prediction operates on per-patch rather than pooled features.
- Training schedule: The additional self-supervised losses begin at 80% of training, using loss weights of 1 and 0.25 for local-to-global consistency and masked prediction.The original image remains used for SigLIP and LocCa losses, while the additional objectives use augmented views to protect image-text alignment.
- NaFlex variants: NaFlex combines multiple predefined sequence lengths with native-aspect-ratio processing, enabling resolution choices suited to image types while reducing aspect-ratio distortion.The variant is based on ideas from FlexiViT and NaViT and is intended for aspect-sensitive applications such as OCR.
3. Experiments and results
SigLIP 2 outperforms SigLIP across core retrieval, multilingual retrieval, and vision-language transfer evaluations. Its NaFlex variant supports native aspect ratios and variable resolutions, with benefits that depend on benchmark content and model size.
- Core evaluations: SigLIP 2 outperforms SigLIP and other open-weight baselines across zero-shot classification and image-text retrieval benchmarks.The comparison covers ImageNet, ObjectNet, ImageNet-v2, ImageNet ReaL, and retrieval benchmarks, despite SigLIP 2 supporting many languages.
- Core evaluations: SigLIP 2 substantially improves multilingual retrieval recall over SigLIP on XM3600, which covers 36 languages.It slightly trails mSigLIP on this multilingual benchmark, while mSigLIP performs substantially worse on English-focused benchmarks.
- Native aspect ratio and resolution: NaFlex outperforms the standard variant on most OCR, document, and screen-focused retrieval benchmarks, especially at small sequence lengths.Small sequence lengths correspond to lower resolutions, where aspect-ratio distortion is more damaging.
- Native aspect ratio and resolution: On natural-image benchmarks, the standard B-sized variant outperforms NaFlex, while the two So400m variants perform similarly.The standard B-sized advantage is attributed in the passage to its self-distillation step.
- Vision-language transfer: SigLIP 2 is evaluated at 224 or 256 and 384 pixels, with the 384-pixel setup repeating stage 1 rather than starting from the 224-pixel variant.These experiments test how input resolution affects transfer performance.
- Vision-language transfer: SigLIP 2 improves vision-language transfer after combining vision encoders with Gemma 2 and fine-tuning across individual datasets.The setup trains the LLM on 50M examples with the vision encoder frozen before downstream fine-tuning.
3.3. Dense prediction tasks
SigLIP 2 transfers effectively to dense prediction, open-vocabulary segmentation, referring expression comprehension, and open-vocabulary detection. Improvements extend across model sizes and resolutions, with decoder-based pretraining particularly relevant to localization.
- Dense prediction: SigLIP 2 frozen representations outperform several popular open-weight models across semantic segmentation, monocular depth, and surface normal estimation.The evaluation uses mIoU for segmentation, RMSE for depth, and angular RMSE for surface normals.
- Open-vocabulary segmentation: SigLIP 2 L/16 improves over SigLIP and surpasses the much larger OpenCLIP G/14 model on open-vocabulary segmentation.Cat-Seg is trained on COCO-Stuff-164k and tested across datasets with different class vocabularies.
- Referring expression comprehension: SigLIP 2 outperforms SigLIP, CLIP, and captioning-pretrained models by a large margin on referring expression comprehension across resolutions and model sizes.The evaluation attaches a six-layer transformer decoder to frozen, unpooled vision representations.
- Referring expression comprehension: SigLIP 2 is outperformed only by LocCa on referring expression comprehension, and the passage hypothesizes multilingual pretraining as one possible reason.LocCa uses text only from English websites, and the authors expect further gains from using the decoder during pretraining.
- Open-vocabulary detection: SigLIP 2 achieves better open-vocabulary detection performance than SigLIP on COCO and LVIS, with the largest relative improvement on LVIS rare categories.The evaluation adapts the OWL-ViT approach to SigLIP-style models.
3.5. Cultural diversity and fairness
SigLIP 2 combines multilingual and de-biasing training data with techniques targeting cultural diversity and fairness. It improves geolocalization and representation bias relative to matched SigLIP models, while benefits by income level or region remain limited.
- SigLIP 2 combines English and multilingual data with de-biasing techniques addressing gender representation and gender–occupation associations.
- Cultural Diversity: 44.4% GeoDE region 10-shot geolocalization accuracy for SigLIP 2 L/16 at 256px, versus 36.2% for matched SigLIP.
- Cultural Diversity: 55.2% Dollar Street 0-shot accuracy for SigLIP 2 L/16 at 256px, versus 52.1% for matched SigLIP.
- Fairness: 7.3% representation bias for SigLIP 2 L/16 at 256px, compared with about 35.5% for matched SigLIP.The metric measures associations of random objects with particular gender groups.
- Cultural Diversity: Dollar Street results by income level and GeoDE results by geographic region show only minor or no benefits over matched SigLIP models.
4. Related work
Related work centers on contrastive vision-language pretraining, which supports semantic representations and open-vocabulary tasks. Subsequent research improves data quality and training objectives, while many open-weight models remain close to the original CLIP approach.
- CLIP- and ALIGN-style contrastive pretraining became dominant for semantic visual representations supporting classification, retrieval, VLMs, detection, and segmentation.
- Open-weight contrastive models generally remain relatively close to CLIP’s original training method, with SigLIP modifying loss functions and others targeting data quality.
- Prior work improves contrastive training through data filtering, VLM-based image recaptioning, and combinations of contrastive and self-supervised losses.
- SigLIP 2 consistently outperforms SigLIP on the reported GeoDE country/region and GLDv2 landmark localization tasks.
5. Conclusion
SigLIP 2 is an open-weight multilingual encoder family that combines decoder pretraining, self-supervised losses, and active data curation. The resulting models improve core, localization, and dense-prediction capabilities, while NaFlex supports variable resolutions and native aspect ratios.
- SigLIP 2 combines decoder-based pretraining, self-supervised losses, and active data curation in an open-weight multilingual encoder family.
- SigLIP 2 achieves improvements in zero-shot classification, VLM transfer performance, localization, and dense prediction tasks.
- Multilingual training and de-biasing filters produce more balanced quality across culturally diverse data.
- The NaFlex variant supports multiple resolutions with one checkpoint while preserving the native image aspect ratio.
A. Full PaliGemma results
Table 6 compares Large-sized and So400M-sized SigLIP models under specified token, patch-size, and resolution configurations. The table uses the same data as Figure 4.
- Table 6 compares Large-sized models using 256 tokens each, with AIMv2 at 224px and SigLIP models at 256px.
- The final four columns compare So400M-sized SigLIP models with patch size 14 at two resolutions and corresponding token counts.
- Table 6 uses the same data as Figure 4.
B. Full NaFlex results
Table 7 compares NaFlex variants, which support native aspect ratios and variable sequence lengths, with standard square-input SigLIP variants using separate checkpoints per sequence length.
- NaFlex supports native aspect ratios and variable sequence lengths, whereas standard square-input SigLIP uses a separate checkpoint for each sequence length.
- The comparison covers TextCaps, HierText, SciCap, and Screen2Words.
- Table 7 reports numerical data corresponding to the plots in Figure 3.
C. Full cultural diversity and fairness results
SigLIP 2 is evaluated on geographically diverse object classification, geolocalization, landmark localization, and fairness measures. It consistently outperforms SigLIP on most listed benchmarks, while de-biased training reduces representation bias.
- SigLIP 2 consistently outperforms SigLIP on most geographically diverse classification, geolocalization, and landmark localization benchmarks.The tasks include Dollar Street, GeoDE object classification and country/region geolocalization, and GLDv2 landmark localization.
- The evaluation reports both 10-shot and 0-shot accuracy for geographically diverse object classification tasks.
- SigLIP 2 slightly reduces performance disparity on Dollar Street when accuracy is disaggregated by income level.Disparity is defined as the maximum difference in 0-shot accuracy across income levels.
- SigLIP 2 exhibits significantly reduced representation bias compared with its predecessor after training on de-biased data.Lower representation-bias values are better.
- Larger models perform better than smaller models, consistent with earlier findings.