PaperlayerView exact revision on arXiv

Source-linked AI summary

Demystifying CLIP Data

Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer, Christoph Feichtenhofer

arXiv:2309.16671v6cs.CVcs.CL

TL;DR

CLIP’s data curation process is poorly documented, motivating a transparent alternative to reproductions based on blackbox model filtering. MetaCLIP balances metadata-derived subsets from raw data while isolating data effects, and it outperforms CLIP’s data across standard benchmarks and model scales.

  • Problem

    CLIP provides limited information about how its training data is collected and curated, leaving important design choices for direct reproduction unspecified.

  • Method

    MetaCLIP applies metadata derived from CLIP concepts to a raw data pool and yields a balanced subset while keeping model architecture and training schedule fixed.

  • Results

    MetaCLIP outperforms CLIP across ImageNet and average accuracy over 26 tasks for three model scales, including +2.1% ImageNet and +1.6% average accuracy on ViT-B/32.

  • Takeaways & Limitations

    Metadata curation and balancing are essential for good data quality, and MetaCLIP’s publicly available pipeline supports transparent data ownership without external models.

  • Takeaways & Limitations

    The paper’s curation choices may differ from CLIP’s because CLIP’s original design choices are not publicly known.

Abstract

from arXiv · show

Contrastive Language-Image Pre-training (CLIP) is an approach that has advanced research and applications in computer vision, fueling modern recognition systems and generative models. We believe that the main ingredient to the success of CLIP is its data and not the model architecture or pre-training objective. However, CLIP only provides very limited information about its data and how it has been collected, leading to works that aim to reproduce CLIP's data by filtering with its model parameters. In this work, we intend to reveal CLIP's data curation approach and in our pursuit of making it open to the community introduce Metadata-Curated Language-Image Pre-training (MetaCLIP). MetaCLIP takes a raw data pool and metadata (derived from CLIP's concepts) and yields a balanced subset over the metadata distribution. Our experimental study rigorously isolates the model and training settings, concentrating solely on data. MetaCLIP applied to CommonCrawl with 400M image-text data pairs outperforms CLIP's data on multiple standard benchmarks. In zero-shot ImageNet classification, MetaCLIP achieves 70.8% accuracy, surpassing CLIP's 68.3% on ViT-B models. Scaling to 1B data, while maintaining the same training budget, attains 72.4%. Our observations hold across various model sizes, exemplified by ViT-H achieving 80.5%, without any bells-and-whistles. Curation code and training data distribution on metadata is made available at https://github.com/facebookresearch/MetaCLIP.

1 INTRODUCTION

The paper argues that CLIP’s success depends substantially on its web-curated data, whose collection process remains opaque. MetaCLIP makes curation transparent by balancing metadata-derived subsets while holding model and training settings fixed.

  • CLIP’s WIT400M dataset is widely believed to contribute to the dominance of CLIP models, but its curation process remains unknown.
  • Existing reproductions use CLIP as a blackbox filter, effectively distilling information from WIT400M.
  • The study freezes model architecture and training schedule to isolate the impact of training data and provides an adaptable curation algorithm.
  • MetaCLIP takes raw data and metadata derived from CLIP concepts, producing a balanced subset over the metadata distribution.
  • 70.8% versus CLIP’s 68.3% is achieved by MetaCLIP on ViT-B zero-shot ImageNet classification, while scaling to 2.5B data reaches 79.2% on ViT-L, 80.5% on ViT-H, and 82.1% on ViT-bigG.
  • 65.5% accuracy is achieved by MetaCLIP after metadata curation and balancing, compared with 57.4% for Raw English, 60.8% without balancing, 63.4% for WIT400M, and 60.0% for LAION 400M.

2 RELATED WORK

Related work addresses dataset pruning, noisy Internet data, and replication of CLIP’s training data. The paper emphasizes that existing CLIP-data reproductions rely on post-hoc model filtering and often target dataset size rather than distribution.

  • CLIP training uses large-scale mixed-quality image-text pairs rather than categorized images with human-annotated labels.
  • Data pruning methods select subsets from established datasets, often using pre-trained models to preserve performance relative to full-dataset training.
  • Internet-data cleaning commonly relies on human-designed filters, dataset cleaning, and outlier removal to discard potentially biasing samples.
  • LAION and DataComp use vanilla CLIP as a teacher-model filter and labor-intensive filtering pipelines, making their resulting data distributions difficult to understand.
  • 72.77% accuracy is reported for LAION400M versus 75.5% for vanilla CLIP on ViT-L/14 ImageNet, indicating sub-optimal 400M-scale replication performance.
  • Understanding CLIP’s curation process can help researchers identify factors associated with its success and develop future vision-language pre-training algorithms.

3 METACLIP

MetaCLIP formalizes CLIP-style curation as metadata construction, substring matching, and balancing of web image-text data. The process preserves signal while mitigating noise and flattens the highly long-tailed distribution by limiting head-entry counts.

  • Curation overview: The curation process reconstructs 500,000 metadata queries, aligns texts through substring matching, and organizes matched texts by metadata entry.Substring matching associates unstructured text with structured metadata and retains about 50% of English image-text pairs.
  • Data pool: The CommonCrawl pool contains 1.6B image-text pairs and 5.6B substring-match counts, with 3.5 matches per text on average.The source is adopted to approximate CLIP’s unknown data source.
  • Distribution analysis: Only 16k of 500k entries exceed 20k matches, but they account for 94.5% of all match counts, revealing a strongly long-tailed distribution.The remaining entries include 114k entries with no matches.
  • Balancing: Entries below t = 20k retain all pairs, whereas head entries are subsampled to 20k, prioritizing texts with denser metadata matches.The threshold marks the transition between tail and head entries and reduces cumulative-count growth to linear.
  • Balancing: Balancing reduces dominance from common head terms, diversifies tail and head entries, and supports more task-agnostic foundation data.For example, only 20k texts containing “photo” are kept from 54M pool instances.

4 EXPERIMENTS

The experiments evaluate MetaCLIP under controlled CLIP training settings across data pools, model scales, data sizes, and balancing thresholds. MetaCLIP generally improves benchmark performance, while balancing and data-pool quality materially affect results.

  • Zero-shot evaluation: MetaCLIP-400M outperforms OpenAI CLIP across ImageNet and average accuracy over 26 tasks at three model scales.The comparison uses identical training settings and 400M training data points.
  • Zero-shot evaluation: +2.1% on ImageNet and +1.6% average accuracy are achieved by MetaCLIP over CLIP with ViT-B/32 and 400M training data.For ViT-B/16, the corresponding gains are +2.5% and +1.5%; for ViT-L/14, they are +0.7% and +1.4%.
  • Scaling data: Scaling from 400M to 1B or 2.5B data produces a large gain with the same number of training iterations.The 1B configuration adds data mainly to tail entries, whereas 2.5B adds data to both head and tail entries.
  • Scaling data: The 1B configuration is more balanced and benefits specific datasets, while 2.5B data with more head entries improves more datasets by smaller amounts.Average accuracies for ViT-L are similar: 70.2% for 1B versus 69.8% for 2.5B.
  • Ablation study: t = 20k gives the best ImageNet and averaged accuracy at the 400M scale, while t = 15k and t = 35k are slightly worse.The ablation uses ViT-B/32 and compares balancing thresholds.
  • Ablation study: Training on the whole 1.6B matched pool hurts ImageNet and average accuracy compared with the balanced 400M set.ImageNet accuracy is 61.9 versus 65.5, and average accuracy across 26 tasks is 56.6 versus 58.2.
  • Ablation study: Online balancing slightly improves average accuracy from 58.2 to 58.5 by down-sampling head entries during data loading.The authors explain the improvement by the larger diversity in head data.

5 CONCLUSION

The paper presents MetaCLIP as a transparent metadata-based approach to curating and balancing raw web data. Across CommonCrawl scales, it reports strong performance relative to CLIP’s proprietary data and releases the data-generation pipeline.

  • Conclusion: MetaCLIP uses metadata to curate and balance raw web data, with experiments isolating data effects under fixed model and training settings.The approach is evaluated across different CommonCrawl data scales without relying on an external model.
  • Conclusion: MetaCLIP performs well across CommonCrawl scales and outperforms CLIP’s proprietary data source.The paper states that these results hold without reliance on any external model.
  • Conclusion: The authors make the pipeline for generating the data publicly available.The conclusion identifies transparency and accessibility of the curation process as goals.

A.1 ADDITIONAL RESULTS

Additional experiments examine MetaCLIP on DataComp’s 12.8B pool and a 38-task benchmark. They find that curated DataComp data has lower quality than the authors’ pools, while MetaCLIP improves over CLIP and OpenCLIP across model sizes.

  • DataComp curation: DataComp’s 12.8B unfiltered pool contains implicit biases, including exclusion of relative image URLs from quality websites.The authors curate this pool with t=60k, matching the tail-count proportion of t=20k on their 400M setting.
  • DataComp curation: MetaCLIP curation is effective across the tested pools, but the unfiltered DataComp-12.8B pool lacks quality.The comparison uses ViT-B/32 and a fixed 12.8B seen-pair training budget.
  • DataComp curation: A quality drop occurs when using 1B pairs curated from DataComp compared with data curated from the authors’ pools.The authors’ 400M set is slightly better than DataComp-1B, while their 1B and 2.5B sets are significantly better.
  • Evaluation caveat: Prompt and class-name differences between the DataComp evaluation and OpenAI CLIP may affect direct comparisons.The paper notes that Gadre et al.’s prompts and class names could differ from those used by OpenAI CLIP.
  • DataComp benchmark: MetaCLIP outperforms CLIP and OpenCLIP across various model sizes on the 38-task DataComp benchmark.The benchmark includes classification, retrieval, and VTAB-related tasks, with scores averaged by category.
  • DataComp benchmark: +1.4% for ViT-B/16 and +2.5% for ViT-L/14 are MetaCLIP’s gains over OpenCLIP at the same 400M data scale.These are average-accuracy gains across the 38 tasks.
  • DataComp benchmark: Average accuracy rises from 62.2% to 65.0% when MetaCLIP data scales to 1B, then reaches 65.5% with 2.5B and more head entries.The reported increase is especially pronounced for the larger model.

A.2 DETAILS ON EFFICIENT CURATION

MetaCLIP’s curation pipeline uses modular parsing, filtering, deduplication, and metadata-based balancing to reduce large web-scale pools before image downloading. The approach supports offline and data-loader-based curation while substantially reducing storage and transfer requirements.

  • Pipeline components: The pipeline combines HTML parsing, language identification, URL/text deduplication, image downloading, NSFW filtering, image deduplication, and packaging.The components are described as five major pipeline stages, with downloading and filtering included in the broader implementation.
  • Pipeline placement: MetaCLIP curation operates on raw data and metadata without requiring image access, enabling placement before image downloading.Its modular design allows different algorithmic parts to run at different pipeline stages.
  • Early reduction: Balancing can be applied before image downloading, while substring matching can follow HTML parsing to reduce English-only pairs by ∼50%.The passage reports balancing as an earlier reduction stage and gives the substring-matching reduction explicitly.
  • Scale and efficiency: The staged approach achieves a total reduction of ∼90%, supporting curation of 300B+ URL-text pairs without storing or transferring all data points.The reported MetaCLIP keeping rate is ∼0.1, corresponding to 0.5 × 0.23.
  • Online curation: The balancing and sampling component can run in the data loader, preserving a similar distribution while diversifying pairs from head entries.Tail entries are always sampled, whereas diverse pairs matching head entries are sub-sampled from a larger pool.

A.3 HUMAN STUDY ON THE EFFECTS OF CURATION

The human study evaluates how balancing affects image-text data quality, including informativeness, aesthetics, and alignment. Balanced data receives higher ratings and improves visual-content alignment through text-side curation effects.

  • Study design: The human evaluation assesses image-text data along informativeness, aesthetics, and alignment dimensions.Annotators evaluate image and text informativeness and aesthetics using criteria covering visual composition and textual quality.
  • Human ratings: Balanced data is rated of higher quality across the evaluated dimensions.Table 10 reports average human ratings with confidence intervals, where higher ratings indicate better quality.
  • Noise mitigation: MetaCLIP uses substring matching and balancing rather than specific hard filters such as removing shorter text or dates.Balancing allows signal and noise to coexist when human-designed filters cannot reliably separate them.
  • Noise mitigation: Balancing limits the contribution of generic entries such as “image” or “photo” to 0.005% of 400M data.The example assumes an entry is capped at t = 20k.
  • Visual content alignment: Text-side curation improves visual-content alignment by increasing entity mentions and favoring long-tailed entries with potentially more diverse visual content.The passage contrasts long-tailed entries with head entries such as “1”.

A.4 MEASURING TASK-ALIGNMENT

MetaCLIP measures task alignment by comparing metadata coverage and distribution between pre-training data and downstream benchmarks. Balanced data shows improved similarity with most task distributions under KL-divergence.

  • Metadata coverage: Metadata substring matching measures how many downstream class names are represented in the pre-training metadata.The analysis reports class coverage for each downstream dataset alongside MetaCLIP accuracy.
  • Distributional alignment: Lower KL-divergence indicates greater similarity between pre-training and benchmark task distributions.The table caption explicitly defines KL-divergence as a lower-is-better measure.
  • Distributional alignment: KL-divergence compares the benchmark task distribution T(m) with the pre-training distribution P(m) over metadata M.For ImageNet, T(m) uses 998 uniformly distributed entries, while P(m) uses normalized counts over the pre-training entries.
  • Distributional alignment: Balanced data points improve similarity with most task distributions compared with unbalanced data points.The comparison is made using KL-divergence for each evaluated task.

A.5 TRAINING SETUP OF OPENAI CLIP VS OPENCLIP

The paper emphasizes a controlled comparison of data curation by following CLIP’s setup and distinguishing it from alternative training configurations. OpenCLIP and DataComp differ from the original setup in several ways.

  • Controlled comparison: MetaCLIP follows CLIP’s setup to isolate the effects of data curation and data quality.The authors identify model-training differences in other systems rather than changing the controlled comparison described here.
  • Setup differences: OpenCLIP varies from CLIP in factors including global batch size and learning schedule.The passage specifically discusses the LAION-400M comparison as closer to the CLIP setup.
  • Setup differences: DataComp differs even more, including by curating images close to ImageNet training data.This distinction is presented as a training-setup and curation difference from CLIP.

A.6 BENCHMARK DEDUPLICATION

The appendix documents data-pool preparation and explores how metadata matching, balancing, and related curation choices affect the resulting examples. It also records abandoned approaches, sampling variability, and known matching noise.

  • Deduplication: 400M and 2.5B pools were deduplicated from benchmark/ImageNet data using a 64-bit PCA hash; DataComp-12.8B was already deduplicated.The hash uses similarity-search feature embeddings, PCA reduction to 64 dimensions, and sign quantization.
  • Abandoned approaches: Self-curated metadata built from frequent terms in raw captions appeared worse because its long-tailed entries differed substantially from CLIP’s metadata.The attempted scheme ranked terms by count and retained the top 500,000 entries.
  • Abandoned approaches: Stopword and useless-entry removal produced almost no difference because balancing reduced each entry’s contribution to 0.0002% of total data points.The authors therefore avoided adding further artificial filters.
  • Additional analyses: The appendix examines five examples per metadata-count group and measures algorithmic randomness across three runs for two pools.Standard deviations were 4035 examples for Pool 1 and 2104 examples for Pool 2.
  • Example analysis: Substring matching and balancing favor longer text and long-tail entities, while matched entries receive higher sampling probabilities.The examples use font size to indicate sampling probability, ranging from near zero to one.
Loading 2309.16671v6…