Source-linked AI summary
Meta CLIP 2: A Worldwide Scaling Recipe
Yung-Sung Chuang, Yang Li, Dong Wang, Ching-Feng Yeh, Kehan Lyu, Ramya Raghavendra, James Glass, Lifei Huang, Jason Weston, Luke Zettlemoyer, Xinlei Chen, Zhuang Liu, Saining Xie, Wen-tau Yih, Shang-Wen Li, Hu Xu
TL;DR
CLIP scaling beyond English is limited by missing worldwide-data curation and degraded English performance in multilingual models. Meta CLIP 2 trains from scratch on native worldwide image-text pairs using scaled metadata, per-language curation, and a worldwide training framework. Its single model improves over its English-only counterpart and achieves state-of-the-art multilingual benchmark results.
Problem
CLIP pipelines lack a scalable curation method for non-English data, while multilingual models can perform worse in English than English-only counterparts.
Method
Meta CLIP 2 scales metadata to 300+ languages, uses per-language substring matching and balancing, and jointly scales training pairs and model capacity.
Results
Meta CLIP 2 ViT-H/14 improves zero-shot ImageNet from 80.5% to 81.3% and sets new state-of-the-art on XM3600, Babel-ImageNet, and CVQA with one model.
Takeaways & Limitations
English and non-English data become mutually beneficial, supporting worldwide CLIP scaling without filtering image-text pairs solely by language.
Takeaways & Limitations
The worldwide curation thresholds assume a shared 6% tail-match proportion across languages.
Abstract
from arXiv · showhide
Contrastive Language-Image Pretraining (CLIP) is a popular foundation model, supporting from zero-shot classification, retrieval to encoders for multimodal large language models (MLLMs). Although CLIP is successfully trained on billion-scale image-text pairs from the English world, scaling CLIP's training further to learning from the worldwide web data is still challenging: (1) no curation method is available to handle data points from non-English world; (2) the English performance from existing multilingual CLIP is worse than its English-only counterpart, i.e., "curse of multilinguality" that is common in LLMs. Here, we present Meta CLIP 2, the first recipe training CLIP from scratch on worldwide web-scale image-text pairs. To generalize our findings, we conduct rigorous ablations with minimal changes that are necessary to address the above challenges and present a recipe enabling mutual benefits from English and non-English world data. In zero-shot ImageNet classification, Meta CLIP 2 ViT-H/14 surpasses its English-only counterpart by 0.8% and mSigLIP by 0.7%, and surprisingly sets new state-of-the-art without system-level confounding factors (e.g., translation, bespoke architecture changes) on multilingual benchmarks, such as CVQA with 57.4%, Babel-ImageNet with 50.2% and XM3600 with 64.3% on image-to-text retrieval.
1 Introduction
Meta CLIP 2 addresses the challenges of scaling CLIP from English-only data to worldwide image-text pairs. Its recipe combines worldwide curation, multilingual metadata, and jointly scaled model training to make English and non-English data mutually beneficial.
- 50.9% of worldwide web data is non-English and is discarded by English-only CLIP curation.
- Worldwide scaling lacks a fundamental curation method for non-English data and risks worse English performance than English-only CLIP.
- Meta CLIP 2 trains CLIP from scratch on native worldwide image-text pairs without private data, machine translation, or distillation.
- Scaling metadata, data curation, model capacity, and training jointly makes English and non-English data mutually beneficial.
- The recipe scales metadata to 300+ languages, applies per-language substring matching and balancing, and increases seen training pairs with added non-English data.
- Meta CLIP 2 achieves 64.3% on XM3600 image-to-text retrieval, 50.2% on Babel-ImageNet, and 57.4% on CVQA.
- The recipe retains worldwide image-text pairs and supports native-language supervision and cultural diversity without filtering by language.
2 Related Work
Prior CLIP work established scalable curation and versatile multimodal representations, but multilingual systems often depend on external resources or machine translation. These systems also face an English-performance trade-off when incorporating non-English data.
- 2.1 Evolution of CLIP and its Data Processing: CLIP learns versatile image and text representations used for downstream tasks and as components in vision and multimodal research.
- 2.1 Evolution of CLIP and its Data Processing: Meta CLIP formalizes OpenAI CLIP’s curation guidance into a model-free, transparent algorithm scaled to 2.5B image-text pairs.
- 2.1 Evolution of CLIP and its Data Processing: Distillation-based CLIP methods learn from external teacher models, which can introduce untractable bias and save compute.
- 2.2 Vision Encoding: CLIP-style models use language supervision for compact, semantic-rich visual representations, unlike purely visual self-supervised learning methods.
- 2.2 Vision Encoding: In this paper, curation means selecting and aligning the training-data distribution with human-defined structure from a raw data source.
- 2.3 Multilingual CLIP Models: Early multilingual CLIP attempts mainly distill from English-only CLIP or use machine translation to incorporate non-English data.
3 The Meta CLIP 2 Recipe
Meta CLIP 2 scales CLIP to native worldwide image-text data through worldwide metadata, language-specific curation, and a training framework that adds multilingual tokenization, more seen pairs, and sufficient model capacity. Its curation balances concepts across languages using language-dependent thresholds, while ViT-H/14 is identified as the capacity inflection point for breaking the multilinguality curse.
- Recipe overview: Meta CLIP 2’s worldwide recipe has three steps: construct worldwide metadata, implement a worldwide curation algorithm, and build a worldwide model-training framework.The design maximizes overlap with OpenAI CLIP and Meta CLIP while adding only changes needed for worldwide data.
- Worldwide metadata: Worldwide metadata uses multilingual WordNet, Wikipedia unigrams and bigrams, and traffic-ranked Wikipedia titles, with independent metadata maintained per language.The metadata covers 31-language WordNet synsets and Wikipedia titles from 40 random snapshot dates.
- Worldwide curation: Language-dependent thresholds preserve cross-language balance: a single threshold can overrepresent head concepts and dilute tail concepts in languages with fewer pairs.The method transfers a 6% English tail-match proportion to derive each language’s threshold from the English threshold.
- Worldwide curation: The curation algorithm identifies each alt-text’s language, matches it against language-specific metadata, counts concepts, computes thresholds, and samples balanced training pairs.Algorithm 1 outputs a curated dataset D* through language-specific matching, threshold computation, and sampling stages.
- Worldwide training: The training framework adds a multilingual tokenizer, scales seen pairs to avoid reducing English exposure, and studies the model capacity needed for extra pairs.Keeping worldwide training to the English CLIP number of seen pairs would downsample English pairs and harm English performance.
- Worldwide training: ViT-H/14 is the inflection point for breaking the multilinguality curse, whereas ViT-L/14 still suffers from deficient capacity.The larger model shows strong performance improvement on both English and non-English tasks.
4 Experiment
The experiments test how worldwide data, scaling, curation, metadata, tokenization, and model capacity affect English and multilingual CLIP performance. Scaling worldwide training with ViT-H/14 breaks the curse of multilinguality, while targeted ablations identify important curation and tokenizer choices.
- 4.2.1 Main Ablation: ViT-H/14 trained on worldwide data with scaled seen pairs outperforms English-only and non-English counterparts on both English and multilingual tasks.The curse remains with non-scaled worldwide training or the smaller ViT-L/14 model.
- 4.2.1 Main Ablation: 2.3× scaling of seen pairs with added non-English data enables Meta CLIP 2 to break the curse of multilinguality using ViT-H/14.The main ablation compares worldwide and language-specific data across ViT-L/14 and ViT-H/14 training runs.
- 4.2.1 Main Ablation: Meta CLIP 2 surpasses mSigLIP on ImageNet, SLIP 26, and DataComp 37, and surpasses SigLIP 2 on the latter two with fewer seen pairs and lower resolution.It reports 72% of the SigLIP-series seen pairs and 224px resolution versus mSigLIP’s 256px.
- 4.2.2 Ablation on Metadata, Curation, and Tokenizer: On ViT-B/32, removing English filtering lowers ImageNet accuracy by 0.6%, while merged metadata initially harms English performance but begins building multilingual capability.Language-isolated curation and per-language thresholds improve English and non-English performance, although the curse remains unresolved at this scale.
- 4.2.2 Ablation on Metadata, Curation, and Tokenizer: The XLM-V vocabulary yields the strongest performance in both English and non-English evaluations among the tested multilingual tokenizers.The tokenizer comparison uses ViT-B/32 with Worldwide 1.0× training.
- 4.2.3 Cultural Diversity: Cultural-diversity evaluations include zero-shot classification, few-shot geo-localization, and embedding alignment and uniformity on a 5k holdout set.The benchmark suite covers Dollar Street, GeoDE, GLDv2, and XM3600, while Figure 4 measures alignment and uniformity for worldwide-data models.
5 Conclusion
Meta CLIP 2 is presented as the first CLIP trained from scratch on worldwide image-text pairs. The study concludes that scaling metadata, curation, and training capacity jointly can remove English–non-English trade-offs and support strong multilingual performance with one model.
- 5 Conclusion: Meta CLIP 2 is the first CLIP trained from scratch on worldwide image-text pairs.The recipe avoids relying on outsourced private data, machine translation, or distillation.
- 5 Conclusion: Scaling metadata, curation, and training capacity jointly can break the curse of multilinguality, allowing English and non-English data to benefit each other.The conclusion frames this as the explanation supported by the study’s careful experiments.
- 5 Conclusion: Meta CLIP 2 ViT-H/14 improves zero-shot ImageNet accuracy from 80.5% to 81.3% over its English-only counterpart and sets multilingual state-of-the-art with one model.The cited multilingual benchmarks are XM3600, Babel-IN, and CVQA.
A.1 Unigram and Bigram Tokenizer for Special Languages
The appendix addresses languages whose writing systems do not consistently separate words with spaces. It uses open-source tokenizers developed by local communities for processing corresponding Wikipedia dumps.
- A.1 Unigram and Bigram Tokenizer for Special Languages: Most languages use spaces between words, but some Asian languages use scriptio continua instead.The appendix identifies this writing-system distinction as relevant to tokenization.
- A.1 Unigram and Bigram Tokenizer for Special Languages: Open-source tokenizers developed by local communities are used to split special-language text while preserving semantic integrity.These tokenizers process Wikipedia dumps labeled with the listed language codes.
A.2 Scaling Curation
Worldwide curation increases computational demands because metadata spans hundreds of languages. The implementation combines efficient matching, lazy loading, memory mapping, safety filtering, and benchmark deduplication to support billion-scale processing.
- A.2 Scaling Curation: Worldwide data curation increases time and space complexity because metadata must be stored across hundreds of languages.The implementation is designed to address this additional complexity.
- A.2 Scaling Curation: Aho-Corasick string matching is about 2k times faster than Meta CLIP’s brute-force implementation, enabling million-scale metadata matching.The method uses prefix trees, or tries, for rapid substring matching.
- A.2 Scaling Curation: Language-specific metadata automata are loaded lazily when new languages appear during shard processing, reducing recompilation overhead.This minimizes the total languages encountered for each data shard.
- A.2 Scaling Curation: Memory-mapped files provide access to per-entry counts across languages without loading all counts simultaneously, preventing sampling-related out-of-memory errors.The approach supports memory-constrained balancing across languages.
- A.2 Scaling Curation: These implementation choices make worldwide curation computationally feasible for billions of image-text pairs from hundreds of languages.The curation pipeline is explicitly intended to scale to worldwide data.
- A.2 Scaling Curation: Safety filtering removes NSFW content and human biometrics or personally identifiable information from the training data.A state-of-the-art safety classifier and face detector are used.
- A.2 Scaling Curation: ImageNet benchmark leakage is reduced by removing overlaps using 64-bit hashes derived from projected and sign-quantized feature embeddings.The hashes reduce embeddings to 64 dimensions before sign-based quantization.
B Training Setup
Meta CLIP 2 follows the OpenAI CLIP and Meta CLIP training setup, with targeted changes for worldwide scaling. Its curation algorithm runs in parallel at substantial computational scale.
- Meta CLIP 2 follows OpenAI CLIP and Meta CLIP training setups while introducing changes for worldwide scaling.The specific changes are detailed in Table 6.
- 800 parallel jobs, each with 40GB of CPU memory, run the data curation algorithm.
- 1 hour is required to substring-match and count all alt-text pairs.
C Limitation on Benchmark
Worldwide benchmark construction remains difficult because evaluation must cover diverse concepts across locations, cultures, and languages. Existing datasets and collection methods also have documented geographic, demographic, or quality limitations.
- Many established benchmarks mainly contain North American and Western European content and focus on English.
- Worldwide benchmark construction is long and resource-intensive because it must cover diverse concepts across geo-locations, cultures, and languages.
- Open Images-based geographic selection can remain biased toward Western images or specific activities such as tourism.
- Crowdsourced geographically diverse images may have uncertain worker demographics, proficiency, and data quality.