PaperlayerView exact revision on arXiv

Source-linked AI summary

DataComp: In search of the next generation of multimodal datasets

Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mehdi Cherti, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexander Ratner, Shuran Song, Hannaneh Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, Ludwig Schmidt

arXiv:2304.14108v5cs.CVcs.CLcs.LG

TL;DR

How image-text data sources and filtering choices affect multimodal models remains poorly understood. DataComp compares training sets using fixed training code and compute budgets across 38 classification and retrieval tasks. Smaller, more stringently filtered datasets can produce better-generalizing models, and DataComp-1B achieves 79.2% ImageNet zero-shot accuracy with CLIP ViT-L/14.

  • Problem

    How image-text data sources and filtering choices affect models remains unclear, despite these datasets' central role in multimodal learning.

    Source: §1 · p. 1

  • Method

    DataComp compares curated image-text training sets under fixed training code and compute budgets, evaluating resulting models on 38 classification and retrieval tasks.

    Source: §1 · p. 2

  • Results

    Models trained on smaller, more stringently filtered datasets can generalize better than those using larger same-pool datasets; DataComp-1B yields 79.2% ImageNet zero-shot accuracy with CLIP ViT-L/14.

    Source: §1 · p. 2

  • Takeaways & Limitations

    DataComp's multiple scales let researchers with different resources participate, with filtering approaches showing largely consistent rankings across scales.

    Source: §1 · p. 2

    CommonPool and DataComp-1B require further investigation of biases and societal risks despite measures to remove NSFW content and blur human faces.

    Source: §6 · p. 9

Abstract

from arXiv · show

Multimodal datasets are a critical component in recent breakthroughs such as Stable Diffusion and GPT-4, yet their design does not receive the same research attention as model architectures or training algorithms. To address this shortcoming in the ML ecosystem, we introduce DataComp, a testbed for dataset experiments centered around a new candidate pool of 12.8 billion image-text pairs from Common Crawl. Participants in our benchmark design new filtering techniques or curate new data sources and then evaluate their new dataset by running our standardized CLIP training code and testing the resulting model on 38 downstream test sets. Our benchmark consists of multiple compute scales spanning four orders of magnitude, which enables the study of scaling trends and makes the benchmark accessible to researchers with varying resources. Our baseline experiments show that the DataComp workflow leads to better training sets. In particular, our best baseline, DataComp-1B, enables training a CLIP ViT-L/14 from scratch to 79.2% zero-shot accuracy on ImageNet, outperforming OpenAI's CLIP ViT-L/14 by 3.7 percentage points while using the same training procedure and compute. We release DataComp and all accompanying code at www.datacomp.ai.

1 Introduction

DATACOMP benchmarks multimodal dataset design by fixing training code and compute while evaluating alternative training sets on 38 classification and retrieval tasks. Its DATACOMP-1B dataset improves ImageNet zero-shot accuracy over OpenAI’s original CLIP ViT-L/14 at the same compute budget.

1 Introduction

  • 79.2% ImageNet zero-shot accuracy with DATACOMP-1B-trained CLIP ViT-L/14 exceeds OpenAI’s original CLIP ViT-L/14 by 3.7 percentage points at the same compute budget.DATACOMP-1B combines the two most promising filtering baselines. It reduces computational cost by 9× compared with a larger CLIP ViT-g/14 trained on LAION-2B for about 3× longer.Source: Table 1 · p. 2, §1 · p. 2
  • 38 classification and retrieval tasks evaluate proposed training sets in DATACOMP while training code and computational budgets remain fixed.This controlled benchmark addresses poorly understood data-source and filtering choices and the lack of benchmark-driven dataset development that isolates data improvements from model changes.Source: §1 · pp. 1–2
  • 12.8B Common Crawl image-text pairs form COMMONPOOL for the filtering track; Bring Your Own Data permits alternative sources without evaluation-testbed overlap.COMMONPOOL is released as an image URL–text index under CC-BY-4.0, with construction-time content checks to remove unsafe or unwanted content.Source: §1 · p. 2
  • Four scales span candidate pools of 12.8M–12.8B samples and training costs of 4–40,000 A100 GPU hours per run.These scales accommodate different resource budgets, and filtering approaches have largely consistent rankings across scales.Source: §1 · p. 2
  • At the 12.8B scale, the best filtering baseline increases ImageNet zero-shot accuracy by 6.9 percentage points compared with models trained on the unfiltered data pool.Over three hundred baseline experiments examine caption keywords, image embeddings, and CLIP-score thresholds, finding that smaller, more stringently filtered datasets can generalize better than larger same-pool datasets. Initial BYOD experiments show that 109M additional data points—less than 1% of the 12.8B pool—improve CLIP-filtered COMMONPOOL subsets by up to 1.2 percentage points of ImageNet accuracy.Source: §1 · p. 2
  • DATACOMP publicly releases its candidate-pool URL index, pool-assembly tooling, filtering baselines, and model-training and evaluation code.These resources provide a shared environment for controlled dataset experiments.Source: §1 · pp. 2–3

2 Related Work

DataComp addresses a gap between small-scale data-curation research and large-scale, noisy image-text training. Related work also highlights proprietary training data, uncertainty about dataset quality, and safety risks in web-scale multimodal datasets.

The effects of data curation

  • DataComp aligns data-centric research with large-scale, noisy image-text training, whereas prior subset-selection studies and benchmarks largely address smaller or already curated datasets.Classical cleaning removes potentially biasing samples [125], while coreset selection seeks full-dataset performance from subsets [61, 7, 46, 11, 94, 145, 32] but scales poorly [51]. DCBENCH and DATAPERF [97] likewise operate at smaller scales.Source: §2 · p. 3
  • 12.8B samples give DataComp a larger-scale setting for data-centric investigation and a common experimental framework for comparing dataset creation algorithms.Earlier work studies ImageNet pruning [135], sentence deduplication and its language-modeling benefits [90], and rule-based CAT filtering [110]. SemDeDup removes semantic duplicates by clustering the CAT-filtered LAION-440M subset.Source: §2 · p. 3

Large-scale multimodal datasets

  • 400M web image-text pairs were used to train OpenAI’s CLIP [111], roughly 300× ImageNet’s size, but proprietary training datasets impede data-centric investigation.Large, heterogeneous datasets underpin multimodal model successes, and prior image-text scaling studies show promising zero-shot performance trends [73] [107].Source: §2 · p. 3
  • 10-15M examples were the scale of prior analyses linking image-text data sources to different CLIP model properties [101].Little is known about what constitutes a good image-text dataset.Source: §2 · p. 4
  • DataComp removes NSFW content and blurs detected human faces, addressing web-scale dataset risks exemplified by NSFW imagery and racial slurs in LAION-400M.These measures are applied when preprocessing the candidate pool.Source: §2 · p. 4

3 The DATACOMP benchmark

DATACOMP benchmarks dataset development through shared-pool filtering and external-data tracks, using standardized model training and evaluation. Four compute scales accommodate different computational resources while enabling studies of scaling.

3.1 Competition design

  • DATACOMP fixes candidate-pool size and total training samples seen while allowing variable training-set sizes; smaller, more stringently filtered datasets can generalize better.At the 12.8B compute scale, datasets containing 6.4B and 3.2B pairs receive two and four training passes, respectively.Source: §3.1 · p. 4

3.3 The bring your own data (BYOD) track

  • The BYOD track lets participants combine arbitrary data sources rather than filtering COMMONPOOL, enabling research on multi-source dataset curation.The public datasets explored by the authors are acquired from their respective sources and are not redistributed with DATACOMP.Source: §3.1 · p. 4, §3.3 · p. 5

3.1 Competition design

  • 10× increases in candidate-pool size and training samples seen separate four compute scales—small, medium, large, and xlarge—ranging from 12.8M to 12.8B.Small-scale runs took four hours on one A100 GPU; xlarge runs took 81 hours on 512 GPUs.Source: §3.1 · pp. 4–5

3.2 COMMONPOOL generation, for the filtering track

  • COMMONPOOL distributes Common Crawl image URL–text pairs under CC-BY-4.0, with per-sample metadata and successively nested random subsets for smaller scales.Extraction covers Common Crawl snapshots from 2014–2022; downloading approximately 40B candidates yields approximately 16.8B image-text pairs.Source: §3.2 · p. 5
  • 12.8B pairs form the xlarge pool after unsafe-content filtering, evaluation-set near-duplicate removal, and random subsampling; detected faces are blurred for privacy.Unsafe-content filtering removes around 19% of downloaded pairs, reducing approximately 16.8B to 13.6B; deduplication reduces this to approximately 13.1B before subsampling. Face blurring has little observed impact on model performance.Source: §3.2 · p. 5

3.4 Training

  • Training follows Radford et al. [111]: contrastive image–caption learning from scratch, with architecture, hyperparameters, and training steps fixed within each scale.Vision Transformer image encoders [39] are trained using OpenCLIP, maximizing matched image–text similarity relative to unaligned pairs.Source: §3.4 · pp. 5–6

3.5 Evaluation

  • 38 image classification and retrieval tasks evaluate models zero-shot, with aggregate performance defined as the average of each task’s preferred metric.The benchmark additionally studies two fairness tasks and reserves three secret test sets for release after a year to guard against overfitting.Source: §3 · p. 4, §3.5 · p. 6

4 Baselines

Filtering baselines improve zero-shot performance over using the entire unfiltered pool at every scale. The section also evaluates external datasets without additional preprocessing, both alone and in combinations.

4.1 Filtering baselines

  • Unfiltered training makes one pass over the entire pool, while random subsets isolate dataset-size effects from compute-budget effects.Random subsets contain 1%, 10%, 25%, 50%, or 75% of the pool; the full pool size equals the training sample budget.Source: §4.1 · p. 6
  • Basic filters screen language, caption length, and image size, while text-based filters select English captions overlapping ImageNet class names.Inspired by Schuhmann et al. [128] and Byeon et al., basic filtering defaults to fasttext [77] English, captions over two words and five characters, and images with smaller dimension above 200 pixels and aspect ratio below three. Text-based filtering uses fasttext English captions containing words from ImageNet-21K or ImageNet-1K class synsets.Source: §4.1 · p. 6
  • CLIP-score filtering selects image-text pairs above a cosine-similarity threshold, testing OpenAI ViT-B/32 and ViT-L/14 scoring models.The experiments vary thresholds and combine CLIP-score thresholds with cld3 English filtering to reproduce the LAION-2B filtering scheme.Source: §4.1 · p. 6
  • Filtering improves zero-shot performance over the unfiltered pool at every scale, with image-based and CLIP-score intersections performing well on most tasks and scales.The supplied table caption establishes these qualitative findings but provides no numerical performance values.Source: Table 3 · p. 7
  • Image-based filtering retains examples in embedding clusters nearest to ImageNet training images after English-language and caption-length filtering.OpenAI ViT-L/14 image embeddings are clustered into 100K groups using Faiss [75]. Selection uses either ImageNet-21K with 14M images or ImageNet-1K with 1.2M images.Source: §4.1 · p. 7

4.2 BYOD baselines

  • BYOD baselines evaluate CC12M, YFCC15M [140] [111], RedCaps, Shutterstock [101], and LAION-2B [129] without additional preprocessing, including dataset combinations.The four moderately sized datasets contain 10 to 58M samples and were studied by Nguyen et al. [101]. Combinations use external datasets alone or add data from the candidate pool.Source: §4.2 · p. 7

5 Results and discussion

DATACOMP-1B improves ImageNet accuracy over LAION-2B and OpenAI’s ViT-L/14, while filtering rankings generally persist across scales and training changes. Evaluation also reveals task-dependent performance and remaining fairness limitations.

5.1 Building better datasets

  • Intersecting image-based and CLIP score filtering excels on most tasks, except at small scale and on retrieval datasets.Basic, CLIP score, image-based, and text-based filtering also outperform no filtering. DATACOMP-1B is the xlarge-scale output of the Image-based ∩CLIP score (L/14 30%) baseline filter.Source: §5.1 · p. 7
  • Approximately 30% of the pool with the highest CLIP scores yields the optimal training subset, whereas larger random subsets perform better.Random selection substantially degrades performance when only small fractions of the pool are retained, demonstrating the importance of filtering rather than subset reduction alone.Source: Figure 2 · p. 8, §5.1 · p. 8
  • 6.1 percentage points over LAION-2B and 3.7 percentage points over OpenAI’s ViT-L/14 are DATACOMP-1B’s ImageNet accuracy gains.DATACOMP-1B contains 1.4B samples versus LAION-2B’s 2.3B, despite coming from a smaller pool. The OpenAI comparison uses the same compute budget, and even a 400M-sample DATACOMP-1B subset outperforms OpenAI’s ViT-L/14.Source: §5.1 · p. 8
  • 4.3 percentage points of ImageNet accuracy are gained at large scale by adding external data to CLIP-filtered COMMONPOOL data.The added sources are CC12M, YFCC15M [140] [111], RedCaps, and Shutterstock [101].Source: §5.1 · p. 8

5.2 DATACOMP design analyses

  • 55.3% versus 55.7% ImageNet accuracy shows comparable performance for COMMONPOOL and LAION-2B under matched filtering and sample counts.Both use 130M samples, English captions, and ViT-B/32 CLIP cosine similarity above 0.28. Average evaluation performance is 0.501 for COMMONPOOL versus 0.489 for LAION-2B.Source: §5.2 · p. 8
  • 0.71 to 0.90 performance rank correlations across scale pairs support generally consistent filtering rankings.Small- and medium-scale baselines are positively correlated; DATACOMP-1B also outperforms OpenAI’s WIT and LAION-2B when substituting ViT-B/16 or ViT-B/32. Changing training steps and batch size minimally affects curation rankings.Source: §5.2 · pp. 8–9, Figure 3 · p. 9

5.3 Evaluation trends

  • 0.99 correlation links ImageNet performance with average performance across evaluation datasets, but individual-task correlations vary substantially and can be negative.ImageNet accuracy is therefore indicative of aggregate performance, but does not represent every evaluation task.Source: §5.3 · p. 9
  • COMMONPOOL-trained CLIP models improve distribution-shift robustness and geographic diversity over ImageNet-trained models, but retain demographic biases.The robustness comparison uses Taori et al. [139]’s testbed; geographic performance remains below models fine-tuned on diverse curated datasets. Face classification analysis indicates that the examined BYOD datasets can increase misclassification risk.Source: §5.3 · p. 9

6 Limitations and conclusion … A.1 Filtering track rules

DataComp provides a controlled setting for dataset-design experiments and open-sources its code, data, and infrastructure. Its filtering track fixes training procedures and restricts evaluation-test access, while internet-derived data still raises safety, privacy, and bias concerns.

6 Limitations and conclusion

  • Internet-derived image-text pairs pose safety and privacy risks despite NSFW removal and face blurring, motivating further investigation of CommonPool and DataComp-1B biases and risks.Public internet content can include unsafe, toxic, and sensitive material.Source: §6 · p. 9
  • Future work includes new data sources, improved filtering, additional supervision signals, more input modalities, and broader evaluations.Suggested extensions include model-generated captions, video and 3D objects, and evaluations covering vision-and-language and robotics tasks.Source: §6 · p. 9
  • DataComp supports rigorous dataset-design experiments through a controlled setting and openly released code, data, and infrastructure.The authors position the benchmark as a first step toward improving multimodal training datasets.Source: §6 · p. 9

A Benchmark rules

  • DataComp defines filtering and BYOD competition tracks and provides a checklist encouraging participants to document design decisions for more granular submission comparisons.Source: A Benchmark rules · p. 24

A.1 Filtering track rules

  • Filtering-track participants must leave the entire training procedure unchanged, including hyperparameters, architecture, optimizer, compute budget, and training steps.Changes to any other training details are also prohibited.Source: §A.1 · p. 24
  • Filtering submissions select CommonPool sample identifiers at one or more scales—small, medium, large, or xlarge—to recover image-text pairs for CLIP training.Duplicate identifiers are allowed. Releasing both the identifier list and its generation code is strongly encouraged but optional.Source: §A.1 · p. 24
  • Filtering algorithms cannot run code on evaluation test images or depend on them, but may use other task images, templates, and class labels.These restrictions are intended to prevent overfitting to evaluation tasks.Source: §A.1 · p. 24

A.2 Bring your own data track: amendments … E Not safe for work (NSFW) filtering

DataComp permits external or independently curated training data while excluding evaluation images and enforcing safety standards. Its candidate-pool pipeline deduplicates Common Crawl URL–text pairs and applies text and image NSFW filters.

A.2 Bring your own data track: amendments

  • The bring-your-own-data track allows existing or independently assembled datasets, but prohibits evaluation images and permits disqualification for inadequate safety standards.Using COMMONPOOL is optional; evaluation images also cannot be used to construct a submission. Participants are encouraged to release images or URLs with text and document face blurring and other safety checks.Source: §A.2 · p. 24

Checklist.

  • The submission checklist records evaluation-image inclusion, existing dataset use, independently curated data, and safety procedures to support finer-grained comparisons.For independently curated data, it requests images or URLs, corresponding text, and safety steps including face blurring and explicit-content image and text filtering.Source: Checklist · p. 24

C Additional related work

  • Dataset-safety research motivates DataComp’s NSFW preprocessing, detected-face blurring, and preliminary fairness evaluations.Birhane et al. examine problematic LAION-400M content; Yang et al. [147] identify ImageNet curation limitations affecting fairness. Prabhu & Birhane find pornographic ImageNet content, reinforcing concerns about harms from improper curation.Source: C Additional related work · p. 26
  • DataComp emphasizes selecting training data before training, distinct from training-data attribution, while related weak-supervision methods reduce annotation burdens.Programmatic weak supervision [118] [119] [157] [158] uses labeling functions and aggregates noisy, conflicting votes [117] [47] [133]. Attribution links trained-model behavior to training subsets through influence functions or repeated-training dynamics and can subsequently refine datasets.Source: C Additional related work · p. 26
  • DataComp extends scaling investigations beyond dataset size by examining how dataset sources and filtering strategies affect scaling trends.Prior scaling research considers compute budgets, model sizes, and training-sample counts [79] [21] [28], but does not consider dataset-design effects. The benchmark seeks improved multimodal datasets rather than treating scaling analysis as its sole objective.Source: C Additional related work · p. 26

D Parsing Common Crawl; B.1 Candidate pool; B.2 Participant tooling; B.3 Baselines; B.4 Leadership and Advising

  • 88 billion URL–text pairs remain after hash-based deduplication and are randomized through a distributed shuffle.Apache Spark extracts image URLs and alt-text from 2014–November 2022 WAT files; deduplication ignores image content, retaining identical images at different URLs. Named leads are Vaishaal Shankar for the candidate pool, Gabriel Ilharco for participant tooling, Yair Carmon for baselines, and Ludwig Schmidt overall.Source: §B.1 · p. 25, §B.2 · p. 25, §B.3 · p. 25

E Not safe for work (NSFW) filtering

  • 0.1 is the cutoff for every Detoxify caption category: a sample is discarded if any category score exceeds it.The multilingual XLM-RoBERTa variant was spot-checked on one million captions without ground-truth labels. Multilingual content, acronyms, and innuendo remained difficult; lowering the threshold increased false positives while some NSFW captions still escaped filtering.Source: E Not safe for work (NSFW) filtering · p. 27
  • 97.4% held-out classification accuracy, versus 96.1% for the original LAION NSFW model, was achieved by the modified LAION-5B [129] image filter.The filter uses normalized CLIP ViT-L/14 embeddings and a four-layer perceptron trained on roughly balanced safe/NSFW data containing 682K images. Against SafeSearch labels for 40,000 candidate-pool images, threshold 0.1 yielded high recall; manual review found very few true positives.Source: E Not safe for work (NSFW) filtering · p. 27

F Deduplication against evaluation sets

DataComp removes exact and near-duplicate evaluation images from COMMONPOOL to prevent data leakage. Its chosen deduplication model achieves substantially higher recall than CLIP ViT-B/32 at the same precision.

F Deduplication against evaluation sets

  • Over 2× as much data is removed by CLIP ViT-L/14 filtering at high recall compared with Yokoo’s deduplication model [150] in transformation tests.The tests adapt Douze et al., sampling 20% of each evaluation dataset—about 140,000 reference images—and 560,000 LAION-2B distractors, a 4-to-1 ratio. Across JPEG compression, flips, rotations, aspect-ratio changes, and grayscaling, Yokoo’s model [150] performs better on almost every transformation except very aggressive aspect-ratio modification.Source: Figure 5 · p. 28, F Deduplication against evaluation sets · p. 28
  • COMMONPOOL excludes evaluation-image duplicates using cosine similarity between image features from Yokoo’s model [150], including near-duplicates altered by compression, overlays, color adjustment, or artistic rendering.A candidate is considered a duplicate when its similarity to an evaluation image exceeds a fixed threshold.Source: F Deduplication against evaluation sets · pp. 27–28, Figure 4 · p. 28
  • 0.8 recall at 0.9 precision is achieved by the ISC deduplication model, versus 0.02 recall at 0.9 precision for OpenAI’s CLIP ViT-B/32.The deduplication model uses a cosine similarity threshold of 0.604169, chosen to balance duplicate detection against false removals; CLIP uses 0.96.Source: F Deduplication against evaluation sets · p. 28
  • 2.8% of downloaded samples are approximately flagged as evaluation-set near-duplicates.These flagged matches are removed from the candidate pool to prevent evaluation leakage.Source: F Deduplication against evaluation sets · pp. 27–28

G Face blurring

DataComp blurs faces by default to prioritize privacy, with only small downstream performance differences in medium-scale discriminative experiments.

G Face blurring

  • 224 × 224 padded-and-resized images feed the pretrained SCRFD-10G face detector [53], whose bounding boxes are included in pool metadata.The reduced input size limits computation costs. Detections use a score threshold of 0.3, the repository’s default for bounding-box visualization.Source: G Face blurring · p. 29
  • 3293 COMMONPOOL images were evaluated by human inspection, requiring detection of every clearly visible real human face for an image-level true positive.Amazon Rekognition, used for ImageNet face detection [148], provides the comparison. Recall is prioritized over precision because extra detected boxes do not compromise privacy.Source: G Face blurring · p. 29
  • The blurring pipeline proposed by Yang et al. [148] creates smooth transitions between blurred faces and unblurred image regions.Most images in the small COMMONPOOL contain no faces.Source: G Face blurring · p. 29
  • Face blurring produced small zero-shot performance differences across filtering methods in medium-scale training experiments.Images are blurred by default during downloading to prioritize safety and privacy. The authors caution that blurring may be more detrimental for generative models that must output faces.Source: G Face blurring · p. 29, Table 7 · p. 30
  • The detector found the central face in every FairFace [80] validation image, regardless of annotated subgroup.This evaluation checked the detector for potential biases.Source: G Face blurring · p. 30

H DATACOMP COMMONPOOL creation pipeline … L.2 Model architecture

COMMONPOOL construction combines web harvesting, metadata generation, and safety and evaluation-set duplication filtering, with accompanying dataset statistics and tools for efficient subset training. Training-set deduplication yields results similar to CLIP filtering alone, while batch-size and architecture changes largely preserve filtering-method rankings.

H DATACOMP COMMONPOOL creation pipeline

  • ∼16.8B downloaded image-text pairs undergo resizing, metadata annotation, and filtering by NSFW and deduplication thresholds, yielding 13.1B suitable pairs and a uniformly sampled 12.8B-pair xlarge COMMONPOOL.Images have a maximum dimension of 512 pixels; metadata include CLIP features and similarities, image/text NSFW scores, image deduplication scores, and face bounding boxes. Smaller pools are nested subsets; participants receive metadata and image URLs rather than raw images, and the provided img2dataset configuration automatically blurs faces during downloads.Source: Figure 7 · p. 31, H DATACOMP COMMONPOOL creation pipeline · pp. 31–32
  • 100% of ImageNet-train images are detected as duplicates; CLIP ViT-B/32 and ViT-L/14 achieve 63.36% and 75.54% top-1 accuracy, respectively, matching CLIP [111].This pipeline unit test uses generated image features and corresponding text features from OpenAI’s 80-prompt ensemble.Source: H DATACOMP COMMONPOOL creation pipeline · p. 32

I COMMONPOOL statistics

  • COMMONPOOL images mostly measure around 224 × 224 pixels after download resizing, captions favor short sequences and English, and CLIP ViT-L/14 similarity scores have a flatter distribution than corresponding ViT-B/32 scores.Caption token counts follow a long-tailed distribution, and English predominates according to fasttext and cld3. Additional analyses examine original image geometry and common URL domains and suffixes.Source: I COMMONPOOL statistics · p. 32

J Efficient training on data subsets

  • At most twice the training-required reads are needed with pretraining subset filtering, versus 1/p-fold reads when filtering a p-fraction of the pool during training.WebDataset lacks efficient random access, so selecting an arbitrary subset requires scanning the pool. The provided resharder writes NumPy-UID-selected examples into a new WebDataset, supporting multiprocessing, distribution across computers, and cloud-storage streaming.Source: J Efficient training on data subsets · p. 35

K Effect of duplicates in the training data

  • Training-set deduplication produces results similar to CLIP filtering alone on medium COMMONPOOL.Using compressed CLIP features and approximate nearest-neighbor search from Webster et al. [144], the method retains the highest-CLIP-ViT-L/14-similarity image among duplicates found within 1000 neighbors. It is tested alone and alongside CLIP B32 score top 30% filtering.Source: Table 9 · p. 35, K Effect of duplicates in the training… · p. 35

L.1 Batch size

  • 0.96 rank correlation on ImageNet and 0.98 for Average over 38 datasets follow doubling medium-scale batch size from 4096 to 8192.All other hyperparameters remain constant, and filtering-method rankings change by at most one position.Source: Table 10 · p. 36, §L.1 · p. 36

L Hyperparameter ablations

  • DATACOMP freezes training and hyperparameters to isolate dataset curation, then probes dataset-ranking stability under changes to batch size, architecture, and training steps.The ablations ask whether datasets yielding stronger zero-shot downstream accuracy remain better when the training procedure changes.Source: L Hyperparameter ablations · p. 36

L.2 Model architecture

  • 1.0 rank correlation for ImageNet and 0.87 for the average metric persist when medium-scale ViT-B/32 is replaced by GMAC-matched ConvNeXt [93].ViT [39] was originally chosen for favorable CLIP scaling versus vanilla ResNets [62], as reported by Radford et al. [111].Source: §L.2 · p. 36

L.3 Number of training steps

Training small DataComp baselines for 10× more steps produces performance positively correlated with standard training, although some method rankings change substantially. DataComp fixes training hyperparameters to emphasize data curation and remove confounders from hyperparameter differences.

L.3 Number of training steps

  • DataComp fixes training hyperparameters, closely following prior work [111], to emphasize data curation and remove confounders arising from participants’ hyperparameter differences.The longer-training experiment ablates this design choice rather than changing the benchmark’s standard configuration.Source: §L.3 · p. 36
  • 10× more training steps produce results positively correlated with standard-training results for both ImageNet accuracy and average performance on small filtering-track baselines.This ablation tests DataComp’s fixed training configuration.Source: §L.3 · p. 36, Figure 14 · p. 37
  • Some filtering methods change ordering drastically under longer training, despite the generally positive performance correlation.The overall correlation therefore does not imply that every method retains its ranking.Source: Figure 14 · p. 37

M Detector-based baselines · N Training details

At medium scale, detector-based filtering and balancing did not outperform Image-based ∩CLIP score (L/14 30%), and all balancing strategies caused contrastive-loss divergence. Training follows OpenCLIP, with scale-dependent batch sizes and a doubled xlarge learning rate following prior work [28].

M Detector-based baselines

  • 128M medium-pool samples were annotated using Detic [160] bounding boxes and class labels for 1203 LVIS [54] categories, retaining predictions with confidence above 0.5.These annotations support filtering and balancing experiments analogous to controls used in supervised learning.Source: M Detector-based baselines · p. 39
  • Object-existence filtering requires at least one LVIS detection; object-centered filtering additionally requires a detected bounding-box center within the central cell of a 3x3 image grid.Source: M Detector-based baselines · p. 39
  • 1204 equally sampled buckets balance 1203 LVIS classes plus images without detections, repeating rare-class samples and subsampling common classes.Images enter every bucket corresponding to their detected classes, and the constructed dataset matches the sample count specified by the training scale.Source: M Detector-based baselines · p. 39
  • Position balancing equalizes 26 buckets covering a 5x5 grid and no detections; count balancing equalizes 12 buckets covering zero through ten detections and more than ten.Position buckets are assigned using detected bounding-box centers; an image can belong to multiple position buckets.Source: M Detector-based baselines · p. 39
  • Image-based ∩CLIP score (L/14 30%) remained the best-performing medium-scale baseline despite additional detector-based filtering and balancing.Five detector-based strategies were applied to both the unfiltered pool and this previously best filtered pool, yielding 10 baselines.Source: Table 13 · p. 39, M Detector-based baselines · p. 39
  • All balancing strategies caused CLIP contrastive-loss divergence and poor model performance.The authors hypothesize that balancing long-tailed data creates repeated samples, increasing the likelihood that samples are contrasted with themselves.Source: M Detector-based baselines · p. 40

N Training details

  • Training follows OpenCLIP, adjusting batch sizes by scale and doubling the xlarge learning rate following prior work [28].Small, medium, and large tracks use CLIP-paper hyperparameters except for reduced batch sizes to accommodate reasonable hardware. The xlarge track increases batch size relative to OpenAI’s CLIP to accelerate training with many highly utilized GPUs.Source: N Training details · p. 40

O Evaluation details … P.2 BYOD track

DataComp uses broad zero-shot evaluation, with ImageNet linear probes and visual question answering providing complementary checks. These sections also describe filtering mechanisms and external-data baseline construction.

O Evaluation details

  • 38 evaluation tasks cover classification, distribution-shift robustness, socioeconomic and geographic diversity, image–text retrieval, and image association.Classification generally uses accuracy or mean per-class accuracy, while WILDS uses dataset-specific primary metrics; Dollar Street uses income-group worst-group top-5 accuracy following Rojas et al., and GeoDE uses region-group worst-group accuracy following Ramaswamy et al. [114]. Retrieval averages image and text recall; WinoGAViL reports Jaccard scores on harder examples with 10 or 12 candidates [17].Source: O Evaluation details · pp. 40–41
  • 33-task averages excluding five near-random tasks produced no substantial qualitative differences, so reported averages retain the full evaluation suite.The excluded tasks were the two CLEVR tasks, two Camelyon tasks, and KITTI. To limit prompt engineering and overtuning, evaluation reuses Radford et al. [111] templates where possible, literature-based labels, and analogous-task prompts otherwise.Source: O Evaluation details · pp. 40–41
  • 0.99 and 1.0 Spearman rank correlations connect zero-shot and linear-probe ImageNet performance for filtering and BYOD models, respectively.Zero-shot evaluation reduces participants’ computational burden and measures out-of-the-box generalization. Validation probes train the final linear layer from zero-shot initialization for 40 epochs.Source: O Evaluation details · p. 41, Figure 16 · p. 43

O.1 Visual Question Answering

  • 0.877 correlation links zero-shot VQA performance with ImageNet accuracy; VQA performance also correlates strongly with the full evaluation-suite average.Following Shen et al. [132] and Ilharco et al. [70], CLIP contrasts images against question-and-candidate-answer prompts without fine-tuning. CLIP struggles with this task, potentially because these prompts differ from pretraining captions.Source: §O.1 · pp. 42–43, Figure 17 · p. 43

P Baseline details

  • Qualitative baseline examples show noisy image–caption matches without filtering and more descriptive captions after CLIP-score filtering.The visualizations compare random samples from No filtering, Basic filtering, and CLIP score (L/14 30%) subsets of small COMMONPOOL.Source: P Baseline details · p. 43, Figure 18 · p. 44, Figure 19 · p. 45

P.1 Filtering track

  • Image-based filtering retains pool images in embedding clusters reached by clean ImageNet examples, after removing non-English and very short captions.Large and xlarge pools use about 160M examples for clustering; clean reference embeddings come from ImageNet 1K or ImageNet 21K. Text-based sampling instead temperature-balances matched IN21K concepts, samples with replacement, and discards examples repeated more than 100 times.Source: §P.1 · p. 47
  • 0.2 is the best tested image-based sampling α, but sampling still underperforms image-based filtering on small and medium pools.Sampling scores clusters by assigned ImageNet examples and was omitted at large scale; removing the highest CLIP-score percentiles improved small-pool average accuracy by at most 0.5 but reduced medium-pool average accuracy by 0.4–0.8. CLIP scores use image–text cosine similarity, while ImageNet-distance filtering ranks minimum image-embedding distance to ImageNet, using pretrained OpenAI CLIP models [111].Source: §P.1 · pp. 47–48

P.2 BYOD track

  • 5M-example random subsets train CLIP models for external-source quality checks following Nguyen et al. [101], complemented by pretrained-CLIP image–text similarity averages.Checks evaluate ImageNet and derived distributions, and average cosine similarities from OpenAI CLIP ViT-B/32 and ViT-L/14. Sources include YFCC15M from YFCC100M [140], previously used by Radford et al. [111]; preferred configurations include Shutterstock’s 58M-photo subset [101] and WIT attribution captions [136].Source: §P.2 · pp. 48–49

P.2.1 Additional results · Q Fairness and biases · Q.1 Diversity

The section presents additional BYOD results and evaluates biases using diversity and demographic datasets. Dollar Street reveals income-related disparities, while GeoDE [114] shows smaller worst-group gaps and improving regional performance with scale.

Q Fairness and biases

  • Bias evaluation includes Dollar Street and GeoDE [114], plus demographic-label analyses of FairFace [80] and UTKFace [159], following Radford et al. [111].The first two datasets assess diversity; the latter two are face datasets.Source: Q Fairness and biases · p. 49

Q.1 Diversity

  • On Dollar Street, trained models show smaller worst-group performance gaps than an ImageNet-trained ResNet-50, but underperform a model fine-tuned on Dollar Street.Models with higher average accuracy show larger worst-group gaps.Source: §Q.1 · p. 49
  • Dollar Street evaluations reveal income-related bias similar to the increasing performance with household income previously reported for ImageNet-trained models.The image-classification dataset covers household items across a wide socioeconomic range, including homes without Internet access; evaluation uses income subgroups from Rojas et al..Source: §Q.1 · p. 49
  • GeoDE performance improves steadily across all regions with increased scale, approaching that of a model fine-tuned on GeoDE.GeoDE equally represents six world regions through images of everyday objects belonging to ImageNet categories [114].Source: §Q.1 · p. 49
  • Both ImageNet models and DataComp models show smaller worst-group accuracy gaps on GeoDE than on Dollar Street.Previously documented ImageNet geographic biases [114] are not apparent in these GeoDE evaluations, but geographic biases outside GeoDE remain unexamined.Source: §Q.1 · p. 49

P.2.1 Additional results

  • Table 18 presents zero-shot performance for selected baselines in the BYOD track.The supplied caption provides no numerical results or supported ranking among baselines.Source: §P.2.1 · p. 49, Table 18 · p. 50

Q.2 Fairness … Q59 Any other comments?

This section examines demographic disparities and harmful crime-related associations in face evaluations, alongside persistent privacy, toxicity, bias, and data-integrity risks in COMMONPOOL despite safeguards. It clarifies the dataset’s intended use for dataset-curation research—not production systems or decisions about people—and documents collection and consent limitations, public URL–text distribution, takedown handling, and maintenance policies.

Q.2 Fairness

  • BYOD improves annotated-gender classification but worsens annotated-race and age classification relative to the filtering-track model.The best filtering and BYOD models are evaluated on FairFace and UTKFace using zero-shot classification. Despite face-blurred training images, filtering models perform significantly above chance, potentially reflecting missed faces, contextual clues, or skin-color signals.Source: §Q.2 · pp. 50–51, Table 19 · p. 51
  • 20.7%, 17%, and 19.3% of Black, Southeast Asian, and East Asian males, respectively, are misclassified as female by the filtering model on FairFace.BYOD’s FairFace gender-classification improvements mostly benefit men, ranging from 1.7pp to 9.9pp gains. For women, changes range from a 0.6pp gain to a 6.2pp drop.Source: Table 20 · p. 51, §Q.2 · p. 51
  • 0.1% is the maximum non-human misclassification rate across subgroups, yet crime-related labeling remains substantial and BYOD exacerbates it in many cases.Following Radford et al. [111], tests contrast race-gender classes with crime-related or non-human labels on FairFace and UTKFace. The authors caution against treating race and gender as fixed categories and do not condone COMMONPOOL-based decisions involving people.Source: §Q.2 · pp. 50–51, Table 21 · p. 52

S.1 Motivation

  • 38 downstream classification and retrieval tasks anchor DataComp’s dataset research; filtering gains transfer across scales, with relatively consistent strategy rankings, especially between adjacent scales.COMMONPOOL aims to support more accurate, safer, and more equitable models. Additional figures show that CLIP filtering implicitly favors English and that ImageNet accuracy correlates with suite averages, but not every individual dataset.Source: Figure 22 · p. 53, Table 22 · p. 53, Figure 24 · p. 55

S.2 Composition

  • 12.8B COMMONPOOL instances pair downloadable image URLs with alt-text and metadata, without category labels.The funnel starts with ∼88B Common Crawl candidates, attempts ∼40B downloads, retrieves ∼16.8B, and retains ∼13.1B after NSFW filtering and evaluation-set deduplication before randomly sampling 12.8B. Data is downloaded from the public internet, and alt-text provides soft, noisy supervision.Source: Q6 How many instances are there in… · pp. 60–61, §S.2 · p. 60, §S.3 · p. 62

Q11 Are relationships between individual instances made explicit (e.g., users’ movie ratings, social

  • COMMONPOOL’s external image links can change or disappear; SHA256 hashes mitigate poisoning, but no complete archival version is provided.Instances have no explicit relationships or recommended splits; evaluation uses existing classification tasks with deduplication intended to prevent test contamination. Internet-derived captions can be unrelated to images, and duplicate data can occur.Source: Q11 Are relationships between… · p. 61
  • COMMONPOOL may contain confidential information and likely retains toxic content despite imperfect NSFW filtering.Public availability does not exclude confidential material, and filtering is not fully accurate at this scale. The authors frame COMMONPOOL as a research artifact for critically studying and improving dataset safety.Source: Q11 Are relationships between… · p. 61, Q21 Any other comments? · p. 62

Q18 Does the dataset identify any subpopulations (e.g., by age, gender)?

  • Face blurring may not prevent identification through missed detections, distinctive clothing, or caption names; captions can also reveal subpopulation information.Sensitive data is considered highly likely at COMMONPOOL’s scale. Mitigations discard samples flagged by NSFW image or text filters and automatically blur detected faces in download scripts.Source: Q18 Does the dataset identify any… · pp. 61–62

S.3 Collection Process

  • December 2022–March 2023 downloads used URLs from 2014–2022 Common Crawl dumps, so collection dates do not indicate content creation dates.Research authors received no project-specific compensation; processing parsed crawls, downloaded images, filtered NSFW content, deduplicated against downstream tests, blurred faces, and computed CLIP features. NSFW filtering was validated against commercial APIs and the LAION-5B NSFW test set, while face detection was evaluated against commercial APIs.Source: Q26 Over what timeframe was the data… · p. 62, §S.3 · p. 62

Q26 Over what timeframe was the data collected? Does this timeframe match the creation timeframe

  • No formal ethics review was conducted, individuals were not notified, and some images may have been uploaded without their consent.Collection respects robots.txt, but this does not establish consent from depicted individuals, including those appearing in medical images.Source: Q26 Over what timeframe was the data… · p. 63

Q32 If consent was obtained, were the consenting individuals provided with a mechanism to revoke

  • COMMONPOOL admits only samples passing NSFW checks and blurs detected faces during download, but harmful internet-derived content likely remains.The authors evaluate fairness in models trained on COMMONPOOL and its derivatives. They monitor takedown requests with LAION and offer to help users identify hosting URLs for sensitive content.Source: Q32 If consent was obtained, were the… · p. 63

Q36 Was the “raw” data saved in addition to the preprocessed/cleaned/labeled data (e.g., to support

  • Raw data and URLs flagged as NSFW are withheld for safety; only URLs retained in the dataset are distributed.Processing uses open-source tools including Spark, Ray, img2dataset, OpenAI CLIP, near-duplicate and face detectors, and Detoxify. The NSFW classifier is a 4-layer MLP trained using the training dataset of the listed CLIP-based NSFW detector.Source: Q36 Was the “raw” data saved in… · p. 64

S.5 Uses

  • 38 downstream classification and retrieval tasks provide zero-shot evaluation of CLIP models trained on the full dataset and subsets across scales and compute budgets.The benchmark aims to measure dataset progress across diverse tasks and support the development of more performant, safer datasets.Source: §S.5 · p. 64, Q44 Any other comments? · p. 65

Q41 What (other) tasks could the dataset be used for?

  • COMMONPOOL is a research artifact, not suitable for production products or software making decisions about people.Potential research uses include captioning, language-conditional image generation, and sociological studies; face-blurred downloads are not expected to support recognizable-face generation. The dataset reflects societal biases and stereotypes, and the authors do not condone surveillance or military applications.Source: Q41 What (other) tasks could the… · pp. 64–65

S.6 Distribution

Q55 Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete instances)?

Related papers

Loading 2304.14108v5…