Source-linked AI summary
DataComp: In search of the next generation of multimodal datasets
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mehdi Cherti, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexander Ratner, Shuran Song, Hannaneh Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, Ludwig Schmidt
TL;DR
Multimodal dataset design lacks the systematic study applied to model architectures and training algorithms, despite its central role in recent advances. DATACOMP addresses this gap with a controlled benchmark for designing datasets and evaluating standardized CLIP models across scales and tasks, with DATACOMP-1B achieving 79.2% ImageNet zero-shot accuracy and exceeding OpenAI’s CLIP ViT-L/14 by 3.7 percentage points at the same compute. The released benchmark and infrastructure support further dataset-design experiments, while web-scale data introduce safety, privacy, and bias risks.
Problem
Multimodal dataset design is poorly understood because many datasets are proprietary and public datasets lack detailed investigation of sources and filtering choices.
Method
DATACOMP fixes training code and compute while letting participants design datasets through filtering or BYOD, then evaluates CLIP models on 38 downstream tasks across four scales.
Results
DATACOMP-1B enables CLIP ViT-L/14 to reach 79.2% ImageNet zero-shot accuracy, 3.7 percentage points above OpenAI’s CLIP ViT-L/14 at the same compute budget.
Takeaways & Limitations
DATACOMP provides a controlled empirical foundation for iterating on multimodal dataset design and building improved training datasets.
Takeaways & Limitations
COMMONPOOL’s public-internet index can contain unsafe, toxic, and sensitive content despite NSFW removal and face blurring.
Abstract
from arXiv · showhide
Multimodal datasets are a critical component in recent breakthroughs such as Stable Diffusion and GPT-4, yet their design does not receive the same research attention as model architectures or training algorithms. To address this shortcoming in the ML ecosystem, we introduce DataComp, a testbed for dataset experiments centered around a new candidate pool of 12.8 billion image-text pairs from Common Crawl. Participants in our benchmark design new filtering techniques or curate new data sources and then evaluate their new dataset by running our standardized CLIP training code and testing the resulting model on 38 downstream test sets. Our benchmark consists of multiple compute scales spanning four orders of magnitude, which enables the study of scaling trends and makes the benchmark accessible to researchers with varying resources. Our baseline experiments show that the DataComp workflow leads to better training sets. In particular, our best baseline, DataComp-1B, enables training a CLIP ViT-L/14 from scratch to 79.2% zero-shot accuracy on ImageNet, outperforming OpenAI's CLIP ViT-L/14 by 3.7 percentage points while using the same training procedure and compute. We release DataComp and all accompanying code at www.datacomp.ai.
1 Introduction
DATACOMP addresses the limited empirical understanding of multimodal dataset design by fixing training and evaluation while allowing participants to innovate on training sets. Its benchmark spans multiple scales and tracks, and baseline experiments show that careful filtering can substantially improve downstream performance.
- Large multimodal datasets underpin recent advances, but dataset design receives less systematic study than model architectures and training algorithms.
- DATACOMP fixes the training code and computational budget while evaluating participant-designed training sets on 38 classification and retrieval tasks.
- COMMONPOOL provides 12.8B Common Crawl image-text pairs for the filtering track, while BYOD permits any non-overlapping data source.
- Four scales vary training budgets and candidate-pool sizes from 12.8M to 12.8B samples, with filtering-method rankings largely consistent across scale.
- At the 12.8B scale, the best filtering baseline improves ImageNet zero-shot accuracy by 6.9 percentage points over the unfiltered pool.
- DATACOMP-1B reaches 79.2% ImageNet zero-shot accuracy with CLIP ViT-L/14, outperforming OpenAI’s model by 3.7 percentage points at the same compute budget.
- The authors release the candidate-pool index, assembly tooling, filtering baselines, and training and evaluation code to support controlled dataset experiments.
2 Related Work
Prior work established the importance of large multimodal datasets but left the properties of good image-text data poorly understood. Existing data-centric benchmarks and curation studies generally operate at smaller scales or on narrower dataset settings than DATACOMP.
- Dataset cleaning, coreset selection, pruning, deduplication, and CAT filtering address data curation, but some approaches scale poorly or target already curated datasets.
- Existing data-centric benchmarks operated at small scales compared with datasets such as LAION-2B, which contains over two billion images.
- DATACOMP aligns data-centric investigation with large-scale image-text training to study dataset design at web-scale regimes.
- Large heterogeneous datasets have supported models such as CLIP, Flamingo, Stable Diffusion, DALL-E, and GPT-4 alongside advances in modeling techniques.
- Public image-text datasets vary in their resulting model properties, but prior analysis was limited to smaller 10–15M-example regimes.
3 The DATACOMP benchmark
DATACOMP is a controlled benchmark for comparing multimodal training datasets by fixing the training procedure and evaluating models across diverse tasks. It combines a Common Crawl candidate pool, two dataset-construction tracks, multiple compute scales, and standardized zero-shot evaluation.
- DATACOMP quantifies dataset submissions by training CLIP models from scratch with fixed procedures and evaluating them on 38 image classification and retrieval tasks.
- Competition design: The filtering track fixes the candidate pool while allowing participants to control training-set size, separating reservoir size from dataset size and compute.
- Competition tracks: DATACOMP offers filtering from COMMONPOOL and Bring Your Own Data, where participants may combine multiple external data streams.
- Competition compute scales: Four compute scales increase samples seen and pool size by 10× steps, from 12.8M to 12.8B samples seen, supporting researchers with different resources.
- COMMONPOOL generation: COMMONPOOL is a Common Crawl image-text index constructed with URL extraction, downloading, safety filtering, evaluation deduplication, and face blurring.
- Training procedure: The fixed CLIP recipe trains image and text encoders contrastively, with architecture and hyperparameters fixed at each scale.
- Evaluation: DATACOMP uses zero-shot evaluation, testing models without additional training and finding a rank correlation above 0.99 with linear-probe zero-shot settings.
4 Baselines
DATACOMP evaluates simple filtering and data-source strategies under a standardized CLIP training setup. Across the benchmark, filtering generally beats using the full pool, with image-based and CLIP-score intersections performing especially well.
- The filtering track studies six simple methods, including no filtering, random subsets, basic metadata filters, CLIP-score filtering, text-based filtering, and image-based filtering.
- No filtering uses the entire pool, whereas random-subset baselines select 1%, 10%, 25%, 50%, or 75% of the pool.
- Basic filtering uses language, caption-length, image-size, and combinations of these criteria.
- CLIP-score filtering retains pairs above cosine-similarity thresholds and tests OpenAI ViT-B/32 and ViT-L/14 scoring models.
- Text-based filtering selects English captions containing ImageNet-21K or ImageNet-1K class-synset words, while image-based filtering selects visual clusters near ImageNet examples.
- Filtering strategies outperform no filtering on all scales, and the intersection of image-based and CLIP-score filtering performs well on most tasks except small-scale and retrieval settings.
- The BYOD experiments compare external sources including CC12M, YFCC15M, RedCaps, Shutterstock, and LAION-2B without additional preprocessing.
5 Results and discussion
DATACOMP results show that filtering strategy and dataset composition matter substantially: carefully filtered subsets often outperform larger unfiltered pools, with findings generally consistent across scales and training changes.
- The intersection of image-based and CLIP-score filtering performs best on most tasks, except at small scale and for retrieval datasets.
- 6.1 percentage points separate DATACOMP-1B from the LAION-2B competitor on ImageNet.
- 4.3 percentage points of ImageNet improvement result from combining CLIP-filtered COMMONPOOL data with several external sources at large scale.
- Selecting approximately 30% of the pool by highest CLIP scores is optimal, whereas random-subset performance improves with larger subsets.
- Filtering-strategy rankings remain broadly consistent across scales, with rank correlations between 0.71 and 0.90 for different scale pairs.
- DATACOMP-1B remains stronger than OpenAI’s WIT and LAION-2B when trained with ViT-B/16 or ViT-B/32, while hyperparameter changes minimally affect method ordering.
- ImageNet accuracy correlates 0.99 with average performance across all studied datasets but is not representative of every individual evaluation task.
6 Limitations and conclusion
The authors present DATACOMP as a controlled, open foundation for improving multimodal training datasets, while acknowledging societal risks from indexing public-internet data and several directions for future work.
- Public-internet image-text indexing can expose datasets to unsafe, toxic, and sensitive content despite NSFW removal and face blurring.
- Future work includes more data sources, improved filtering, additional supervision and modalities, and broader vision-language and robotics evaluations.
- DATACOMP is described as a first step toward improving training datasets through controlled experimentation and open-source resources.
- The benchmark rules prohibit modifying the training procedure, including the model architecture, optimizer, hyperparameters, compute budget, and training steps.
A.2 Bring your own data track: amendments
The BYOD amendments broaden participation by allowing external or self-curated data under evaluation-set exclusions, alongside documentation and safety requirements for submissions.
- Track 2 permits more diverse submissions than the original competition structure.
- Participants may augment COMMONPOOL with existing datasets or assemble their own, provided evaluation-task test images are not used.
- The submission checklist asks participants to disclose evaluation-image use, existing datasets, self-curated data, and safety steps.
- COMMONPOOL construction parses Common Crawl metadata, extracts image URLs and alt-text, hashes URL-text pairs into UIDs, and removes duplicate UIDs.
- The pipeline applies NSFW text and image filtering, including Detoxify caption screening and a CLIP-feature classifier validated against Google Vision SafeSearch.
F Deduplication against evaluation sets
DATACOMP removes exact and near-duplicate images matching evaluation-set images to reduce data leakage, using feature similarity and transformations representative of web duplication.
- Evaluation-set images and their exact or transformed near-duplicates are removed from COMMONPOOL.Detected transformations include aspect-ratio changes, JPEG compression, overlays, color adjustment, and artistic rendering.
- Cosine similarity between Common Crawl query-image features and evaluation-image reference features determines duplicate matches.The deduplication model is based on Yokoo’s image-similarity method.
- 0.604169 is the cosine-similarity threshold selected to maximize true duplicates while limiting false removals.At this threshold, the ISC model achieves precision 0.9 and recall 0.8.
- Approximately 2.8% of downloaded samples are flagged as evaluation-set near-duplicates.The ISC model substantially outperforms CLIP ViT-B/32 in recall at comparable precision under the reported thresholds.
G Face blurring
DATACOMP detects visible faces with SCRFD and blurs them during downloading to improve privacy, while reporting little effect on downstream discriminative performance.
- SCRFD-10G detects face bounding boxes that are stored as metadata for COMMONPOOL images.The detector uses preprocessing from its official repository.
- 3293 COMMONPOOL images were evaluated for visible-face detection, counting an image as a true positive only when all clearly visible faces were detected.Cartoon drawings of non-real human faces were excluded from the positive category.
- Face bounding boxes feed a standard blurring pipeline that creates a smooth transition between blurred and clean image regions.Images are blurred by default during the download process.
- The majority of small COMMONPOOL images contain no faces.
- Face blurring negligibly affects downstream zero-shot performance in the discriminative setting evaluated.The authors note it may be more detrimental for generative models that need to output faces.
H DATACOMP COMMONPOOL creation pipeline
COMMONPOOL is built from Common Crawl image-text pairs through downloading, metadata enrichment, safety and deduplication filtering, and scale-specific subset selection. The resulting pools are characterized by image, text, language, URL, and access statistics.
- Pool construction: The data funnel produces 13.1B suitable image-text pairs from Common Crawl, from which 12.8B are sampled for the xlarge pool.
- Pool construction: The pipeline parses URLs and alt-text, downloads images, adds metadata, and applies safety-content and evaluation-set deduplication filters.
- Pool construction: Approximately 16.8B downloaded samples are resized so their largest dimension is at most 512 pixels before richer metadata generation.Generated metadata includes CLIP image-text similarities, NSFW scores, deduplication scores, and detected-face bounding boxes.
- Pool construction: Smaller COMMONPOOL scales are telescoping subsets of larger pools, with the small pool an i.i.d. sample of the xlarge pool.Participants receive metadata and use img2dataset, which automatically blurs faces during downloading.
- Additional statistics and access: CommonPool analyses also report CLIP similarity distributions, frequent domains and suffixes, pixel occupancy, and efficient WebDataset access patterns.Filtering a subset before training requires at most twice as many reads as needed for training, avoiding the overhead of filtering during loading.
- Image statistics: The majority of downloaded images are around 224 × 224 pixels, matching the final training resolution.Image statistics also cover original aspect ratios and sizes before resizing.
- Text and URL statistics: Caption token counts are long-tailed with greater mass among short sequences, and English is the predominant language according to fasttext and cld3.
- Robustness checks: Doubling batch size preserves filtering-method rankings closely, with rank correlations of 0.96 on ImageNet and 0.98 for Average over 38 datasets.
L.3 Number of training steps
DATACOMP fixes training hyperparameters to emphasize dataset curation, then tests whether extending training by 10× changes baseline filtering rankings. The reported figure shows a generally positive correlation, despite some ordering changes.
- DATACOMP fixes model-training hyperparameters to reduce confounding differences between participants and focus comparisons on data curation.
- 10× training steps produce a generally positive correlation with baseline performance rankings, although some filtering-method orderings change substantially.The comparison covers both ImageNet accuracy and average performance.
- The extended-training experiment uses small-scale filtering-track baselines and compares ImageNet with average performance.
M Detector-based baselines
Detector-based baselines apply object detection and balancing strategies to multimodal training pools, including compositions with the strongest filtered pool. These strategies do not improve performance: balancing causes divergence, while the existing Image-based ∩ CLIP score baseline remains best.
- Detector-based strategies: Detic annotations from 1203 LVIS categories support object-existence, object-centered, class-balancing, and position-balancing strategies on the 128M-sample medium pool.Predictions are retained when their confidence exceeds 0.5.
- Experimental setup: Five detector-based strategies are also applied to the Image-based ∩CLIP score (L/14 30%) pool, producing 10 medium-scale baselines in total.The strategies can be composed with any starting pool.
- Results: All balancing strategies cause CLIP contrastive-loss divergence and poor model performance.The authors hypothesize that repeated samples from long-tailed data increase the chance of self-contrasting examples in the loss.
- Configuration: The benchmark uses scale-specific model architectures and hyperparameters, with reduced batch sizes for small, medium, and large tracks relative to the CLIP setup.The xlarge track increases batch size to accelerate training across many GPUs.
O Evaluation details
DataComp evaluates models across diverse classification, retrieval, and visual question-answering tasks using standardized zero-shot protocols and fixed prompt choices. The evaluation also compares filtering strategies and examines whether performance transfers across benchmarks.
- Evaluation suite: The evaluation suite spans 38 tasks covering multiple domains, including classification and retrieval.It includes 22 classification tasks from the CLIP test suite, six ImageNet-generalization datasets, and most of the Visual Task Adaptation Benchmark.
- Metrics: Accuracy is the primary metric for most classification tasks, with mean per-class accuracy or dataset-specific leaderboard metrics used when appropriate.Dollar Street and GeoDE use metrics designed for their diversity-focused evaluation settings.
- Evaluation protocol: Five tasks show near-random zero-shot performance without meaningful dependence on filtering, so the paper reports averages over the full 38-task suite.Averages over the remaining 33 tasks did not produce substantial qualitative differences.
- Evaluation protocol: Zero-shot evaluation avoids fine-tuning to reduce participant compute and measure out-of-the-box model generalization.Prompt templates and class-name choices are standardized where possible to limit prompt engineering and overtuning.
- VQA evaluation: VQA uses question-and-answer prompts without fine-tuning, and VQA performance correlates strongly with ImageNet accuracy at 0.877.VQA performance also correlates strongly with average performance across the full evaluation suite.
- Filtering comparisons: Qualitative samples show noisier image-text matches under No filtering and more descriptive captions under the CLIP score (L/14 30%) strategy.The visualizations compare randomly sampled examples from the corresponding filtered pools.
P.2.1 Additional results
Additional results examine BYOD models and dataset diversity across socioeconomic and geographic distributions. Performance generally improves with scale, while GeoDE evaluations show smaller bias gaps than Dollar Street.
- BYOD results: The BYOD track reports zero-shot performance for selected baselines in Table 18.BYOD results are presented separately from the filtering-track evaluations.
- Socioeconomic diversity: Dollar Street evaluates classification across household-income levels using images of household items from homes worldwide.The dataset includes homes with no Internet access and represents a wide socioeconomic range.
- Geographic diversity: GeoDE represents six world regions equally and targets geographic diversity using everyday objects in ImageNet categories.The dataset is designed to assess geographic generalization.
- Geographic diversity: GeoDE shows a smaller worst-group accuracy gap than Dollar Street, and performance across regions improves steadily with increased scale.Performance approaches that of a model fine-tuned on GeoDE.
- Dataset naming: COMMONPOOL denotes the pool filtered with the CLIP score (L/14, 30%) strategy.This naming convention identifies the filtered pool used in the benchmark analyses.
Q.2 Fairness
DataComp evaluates fairness-related behavior in models trained on its datasets, including demographic classification and harmful-label associations. Despite face blurring, models perform above chance and show substantial subgroup disparities, while the authors caution against using them for decisions about people.
- Evaluation scope: Fairness evaluations use FairFace and UTKFace to test zero-shot race, gender, and age classification in models trained on COMMONPOOL derivatives.The authors note that these evaluations are probes of model behavior, not intended benchmark end goals.
- Interpretation and scope: The authors state that race, gender, and age should not be treated as fixed categories and do not condone using COMMONPOOL-trained models for decisions involving people.They include the evaluations for continuity with prior work and as probes into model behavior, while calling for improved face-fairness evaluations.
- Observed behavior: Face-blurred filtering-track models still classify demographic traits significantly above random chance, with BYOD models performing better on gender but worse on race and age than filtering models.The authors hypothesize that residual faces, contextual cues, or skin-color signals may contribute to this behavior.
- Subgroup disparities: Gender classification accuracy varies drastically across annotated race-and-gender subgroups, including elevated misclassification of several Asian and Black male groups as female.On FairFace, the reported misclassification rates are 20.7% for Black males, 17% for Southeast Asian males, and 19.3% for East Asian males.
- Harmful associations: Models rarely classify faces as non-human, but they assign crime-related labels to a significant proportion of people, highlighting risks in zero-shot classification.Table 21 reports that this harmful misclassification is exacerbated by the BYOD model in many cases.