DataComp: In search of the next generation of multimodal datasets

Samir Yitzhak Gadre; Gabriel Ilharco; Alex Fang; Jonathan Hayase; Georgios Smyrnis; Thao Nguyen; Ryan Marten; Mitchell Wortsman; Dhruba Ghosh; Jieyu Zhang; Eyal Orgad; Rahim Entezari; Giannis Daras; Sarah Pratt; Vivek Ramanujan; Yonatan Bitton; Kalyani Marathe; Stephen Mussmann; Richard Vencu; Mehdi Cherti; Ranjay Krishna; Pang Wei Koh; Olga Saukh; Alexander Ratner; Shuran Song; Hannaneh Hajishirzi; Ali Farhadi; Romain Beaumont; Sewoong Oh; Alex Dimakis; Jenia Jitsev; Yair Carmon; Vaishaal Shankar; Ludwig Schmidt

Source-linked AI summary

DataComp: In search of the next generation of multimodal datasets

Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mehdi Cherti, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexander Ratner, Shuran Song, Hannaneh Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, Ludwig Schmidt

arXiv:2304.14108v5Apr 27, 2023cs.CVcs.CLcs.LG

TL;DR

Multimodal datasets lack systematic, benchmark-driven study of how data sources and filtering affect learned models. DataComp addresses this gap with a controlled dataset-design benchmark, whose best baseline achieves 79.2% ImageNet zero-shot accuracy and surpasses OpenAI’s CLIP ViT-L/14 by 3.7 percentage points.

Problem
Multimodal datasets are often proprietary or treated as monolithic artifacts, leaving limited evidence about how data sources and filtering affect resulting models.
Method
DataComp holds training code and compute constant while participants design image-text datasets evaluated by CLIP training across 38 classification and retrieval tasks.
Results
79.2% ImageNet zero-shot accuracy is achieved by DataComp-1B, outperforming OpenAI’s CLIP ViT-L/14 by 3.7 percentage points at the same compute budget.
Takeaways & Limitations
DataComp provides a controlled empirical foundation for iterating on multimodal dataset design and studying scaling trends across resource levels.
Takeaways & Limitations
CommonPool and DataComp-1B may contain societal risks from unsafe, toxic, sensitive, and biased public-internet content despite content-removal and privacy measures.

Abstract

from arXiv · show

Multimodal datasets are a critical component in recent breakthroughs such as Stable Diffusion and GPT-4, yet their design does not receive the same research attention as model architectures or training algorithms. To address this shortcoming in the ML ecosystem, we introduce DataComp, a testbed for dataset experiments centered around a new candidate pool of 12.8 billion image-text pairs from Common Crawl. Participants in our benchmark design new filtering techniques or curate new data sources and then evaluate their new dataset by running our standardized CLIP training code and testing the resulting model on 38 downstream test sets. Our benchmark consists of multiple compute scales spanning four orders of magnitude, which enables the study of scaling trends and makes the benchmark accessible to researchers with varying resources. Our baseline experiments show that the DataComp workflow leads to better training sets. In particular, our best baseline, DataComp-1B, enables training a CLIP ViT-L/14 from scratch to 79.2% zero-shot accuracy on ImageNet, outperforming OpenAI's CLIP ViT-L/14 by 3.7 percentage points while using the same training procedure and compute. We release DataComp and all accompanying code at www.datacomp.ai.

1 Introduction

DataComp addresses the understudied design of large image-text datasets by fixing training and evaluation conditions while letting researchers innovate on training data. It introduces a Common Crawl candidate pool, multiple dataset-design tracks and scales, extensive baselines, and a state-of-the-art dataset.

Motivation: DataComp responds to the field’s limited understanding of dataset choices and lack of benchmark-driven development, despite multimodal advances relying on enormous paired image-text datasets.CLIP used 400M pairs and Stable Diffusion used 2B LAION-2B examples, far exceeding ImageNet’s 1.2M images.
Contributions: 79.2% ImageNet zero-shot accuracy from DataComp-1B surpasses OpenAI CLIP ViT-L/14 by 3.7 percentage points at the same compute budget, while using 9× less computation than LAION-2B CLIP ViT-g/14.DataComp-1B combines the two most promising filtering baselines; Table 1 compares zero-shot performance across datasets.
Contributions: DataComp fixes training code and computational budgets, then evaluates proposed training sets with models tested on 38 classification and retrieval tasks.The testbed includes ImageNet, ImageNetV2, DTD, EuroSAT, SUN-397, and MSCOCO.
Contributions: CommonPool provides 12.8B Common Crawl image-text pairs for filtering, while the BYOD track permits any non-overlapping data source.CommonPool is released as a CC-BY-4.0 index after content checks remove unsafe or unwanted content.
Contributions: DataComp spans candidate pools and training budgets from 12.8M to 12.8B samples and 4 to 40,000 GPU hours, with filtering rankings largely consistent across scales.The four scales make participation possible for researchers with different resources.
Contributions: At the 12.8B scale, the best filtering baseline improves ImageNet zero-shot accuracy by 6.9 percentage points over the unfiltered pool, while 109M BYOD samples add up to 1.2 points.The baseline findings also show that smaller, more stringently filtered datasets can generalize better than larger datasets from the same pool.

2 Related Work

Prior data-curation and multimodal-dataset research largely studies smaller or already curated datasets, while many leading training datasets are proprietary. DataComp addresses this gap with large-scale image-text data and a common setting for comparing dataset-creation methods, alongside safety preprocessing for web-scale data.

The effects of data curation: DataComp targets the gap between data-centric investigation and modern training on noisy image-text pairs at billion-sample scales.Existing subset-selection benchmarks often use curated datasets or smaller regimes, while prior benchmarks operate at smaller scales than datasets such as LAION-2B [129].
The effects of data curation: DataComp provides a common experimental setting at the 12.8B-sample scale for fair comparison among pruning, deduplication, filtering, and other dataset-creation algorithms.Prior work includes dataset pruning, language-data deduplication, CAT filtering, and SemDeDup, but addresses smaller or different settings [90] [110] [135].
Large-scale multimodal datasets: Large heterogeneous datasets enabled multimodal models including CLIP [111], Flamingo [8], Stable Diffusion [123], DALL-E [115], and GPT-4 [103], yet proprietary training data limits data-centric study.OpenAI’s CLIP uses 400M web image-text pairs, roughly 300× ImageNet, while prior scaling work reports promising zero-shot trends [73] [107].
Large-scale multimodal datasets: Public image-text datasets still leave dataset quality poorly understood, while web-scale collections can contain NSFW imagery and racial slurs; DataComp removes NSFW content and blurs detected human faces.Preliminary analysis finds that image-text sources produce CLIP models with different properties [101], but prior analysis was limited to 10–15M examples.

3 The DATACOMP benchmark

DataComp is a data-centric benchmark that evaluates image-text dataset development by training CLIP models from scratch and testing them across 38 downstream tasks. It standardizes candidate pools, competition tracks, compute scales, training, and zero-shot evaluation to enable comparable dataset experiments.

3 The DATACOMP benchmark: DataComp evaluates dataset submissions by training CLIP from scratch and measuring performance on 38 downstream image classification and retrieval tasks, with three secret test sets to guard against overfitting.The benchmark provides an uncurated public-internet candidate pool, filtering and external-data tracks, and four compute scales for researchers with different resources.
3.1 Competition design: DataComp fixes the candidate pool in its filtering track while allowing participants to choose training-set size, and equalizes datasets by fixing the total number of training samples seen.It includes filtering and Bring Your Own Data tracks, with small, medium, large, and xlarge scales; each scale increases samples seen from 12.8M to 12.8B by 10×, with xlarge runs taking 81 hours on 512 GPUs.
3.2 COMMONPOOL generation, for the filtering track: COMMONPOOL is a 12.8B image-text-pair index from Common Crawl, constructed through URL extraction and downloading, safety filtering, evaluation-set deduplication, and face blurring.The pipeline begins with ∼88B possible samples, downloads ∼16.8B pairs, removes around 19% as NSFW, deduplicates ∼3% of images, and samples the final 12.8B pool.
3.3 The bring your own data (BYOD) track: The BYOD track lets participants combine multiple data streams, including public datasets such as CC12M and YFCC100M [140], rather than restricting them to COMMONPOOL.These datasets are acquired from their original sources and are not re-released as part of DataComp.
3.4 Training: DataComp fixes the CLIP training procedure across scales: models are trained from scratch with a contrastive image-caption objective, fixed architectures and hyperparameters, and Vision Transformer image encoders [39].Training follows Radford et al. [111] and uses the OpenCLIP repository.
3.5 Evaluation: Evaluation uses zero-shot testing across 38 image classification and retrieval tasks, aggregates each task’s preferred metric by averaging, and removes test-set images from DataComp to avoid contamination.The suite includes datasets spanning satellite imagery, metastatic-tissue classification, distribution shifts, VTAB, WILDS, Flickr30k, MSCOCO, and WinoGAViL [17].

4 Baselines

Section 4 establishes filtering and external-data baselines for evaluating multimodal dataset construction. Filtering strategies outperform using the entire unfiltered pool, while intersections of image-based and CLIP-score filters perform well across tasks and scales.

4.1 Filtering baselines: The study evaluates six simple filtering methods, including no filtering, random subsets, basic language/caption-length/image-size filtering, CLIP-score/LAION filtering, and text-based ImageNet class-name filtering.Random subsets use 1%, 10%, 25%, 50% and 75% of the pool, chosen at random.
4.1 Filtering baselines: CLIP-score filtering retains examples above cosine-similarity thresholds computed with OpenAI CLIP ViT-B/32 or ViT-L/14, and can be combined with cld3 English filtering.These configurations reproduce the LAION-2B filtering scheme and are summarized in Appendix Table 16.
4.1 Filtering baselines: Filtering strategies outperform the entire unfiltered pool at all scales, with image-based ∩ CLIP-score filtering performing well on most tasks and scales.Table 3 reports zero-shot performance for selected filtering baselines; higher values are better.
4.2 BYOD baselines: The BYOD baselines evaluate CC12M, YFCC15M, RedCaps, Shutterstock, and LAION-2B, both individually and in combinations with one another or the candidate pool.The external datasets are used as provided without additional preprocessing; the first four contain 10–58M samples, while LAION-2B is larger.

5 Results and discussion

DataComp produces stronger multimodal datasets through filtering, with DATACOMP-1B outperforming LAION-2B and OpenAI’s equivalent model on ImageNet. Analyses show filtering choices transfer across scales and training settings, while evaluation reveals strong robustness but important task-specific and fairness limitations.

5.1 Building better datasets: The intersection of image-based and CLIP-score filtering performs best on most tasks, except at small scale and on retrieval datasets.Basic, CLIP-score, image-based, and text-based filters all outperform no filtering on downstream performance.
5.1 Building better datasets: DATACOMP-1B outperforms LAION-2B by 6.1 percentage points on ImageNet and OpenAI’s equally compute-budgeted ViT-L/14 by 3.7 points.DATACOMP-1B uses the Image-based ∩ CLIP score (L/14 30%) baseline filter at xlarge scale; a 400M-sample subset also outperforms OpenAI’s model.
5.1 Building better datasets: External data can improve performance: combining CLIP-filtered COMMONPOOL data with CC12M, YFCC15M [140] [111], RedCaps, and Shutterstock [101] raises large-scale ImageNet accuracy by 4.3 points.The comparison is against using only COMMONPOOL data in the BYOD track.
5.1 Building better datasets: Selecting approximately 30% of the pool by highest CLIP scores outperforms random subsets, showing that filtering balances diversity and repetition.Random subsets improve with size, whereas CLIP-filtered subsets perform best at an intermediate size.
5.2 DATACOMP design analyses: COMMONPOOL and LAION are comparable under identical filtering, reaching 55.3% versus 55.7% ImageNet accuracy and 0.501 versus 0.489 average evaluation performance.The comparison uses 130M samples at the large scale and applies LAION-2B’s English-caption and CLIP-threshold filtering to COMMONPOOL.
5.2 DATACOMP design analyses: Filtering-strategy rankings remain consistent across scales, with rank correlations of 0.71–0.90 across scale pairs.Small and medium scale baselines are positively correlated, indicating that smaller scales can guide larger-scale experiments.
5.2 DATACOMP design analyses: DATACOMP-1B remains superior to OpenAI’s WIT and LAION-2B when trained with ViT-B/16 or ViT-B/32, while training-step and batch-size changes minimally alter method rankings.These results test transfer beyond DATACOMP’s fixed ViT-L/14 procedure.
5.3 Evaluation trends: ImageNet accuracy correlates 0.99 with average performance across all evaluated datasets, but correlations with individual datasets vary substantially and can be negative.Thus, ImageNet is informative but does not represent every evaluation task.

6 Limitations and conclusion … A.1 Filtering track rules

DataComp is presented as a first step toward improving multimodal training datasets through a controlled, rigorous benchmark and open-source resources. The paper also highlights societal risks, future research directions, and rules designed to make filtering submissions comparable and prevent evaluation leakage.

6 Limitations and conclusion: Public-internet image-text indexes may contain unsafe, toxic, sensitive, and privacy-threatening content despite NSFW removal and face blurring.The authors call for further study of biases and risks in COMMONPOOL and DATACOMP-1B.
6 Limitations and conclusion: Future work includes curating more data sources, improving filtering, adding supervision such as model-generated captions, incorporating video and 3D modalities, and broadening vision-language and robotics evaluations.
6 Limitations and conclusion: DataComp is a first step toward improving training datasets, providing a controlled setting for rigorous dataset-design experiments and open-source code, data, and infrastructure.The authors hope these resources support development of the next generation of multimodal datasets.
A Benchmark rules: DataComp defines filtering and BYOD competition tracks with concrete rules and a checklist that records design decisions for more granular submission comparisons.
A.1 Filtering track rules: Filtering submissions may target small, medium, large, or xlarge scales by selecting CommonPool uids, including duplicate uids, while leaving the standardized training procedure unchanged.Participants are encouraged, but not required, to open-source the selected uids and generation code.
A.1 Filtering track rules: Filtering algorithms cannot depend on evaluation test images or their predictions, although downstream class labels, templates, and other associated images may be used.For example, “triceratops” is permitted, whereas examining individual or aggregate evaluation-set predictions is forbidden.

A.2 Bring your own data track: amendments … E Not safe for work (NSFW) filtering

The paper broadens participation through a bring-your-own-data track, documents dataset provenance and safety, and constructs COMMONPOOL from deduplicated Common Crawl pairs with layered NSFW filtering.

A.2 Bring your own data track: amendments: Track 2 permits augmenting COMMONPOOL or assembling a dataset from other sources, provided evaluation images are excluded and safety documentation is supplied.Organizers may rerun safety checks and disqualify submissions that fail adequate standards.
Checklist.: The submission checklist asks participants to disclose evaluation-image inclusion, existing datasets such as YFCC100M [140], curated image-text data, and safety steps including face blurring and explicit-content filtering.These requirements apply especially to the bring-your-own-data track.
B Contributions: The project credits separate leads for the candidate pool, participant tooling, baselines, and overall leadership, with contributions spanning collection, preprocessing, training, evaluation, and infrastructure.Contributors are ordered alphabetically within the contribution section.
C Additional related work: Related work motivates safety checks, fairness evaluation, data curation, attribution, sample difficulty, and scaling analysis, while the benchmark measures dataset sources and filtering across 38 diverse tasks.The paper distinguishes selecting training data before model training from post hoc training-data attribution methods.
D Parsing Common Crawl: 88 billion URL–text pairs resulted from parsing Common Crawl WAT files from 2014 through November 2022, hashing URL–text pairs, deduplicating them, and shuffling them.Deduplication did not inspect image content, so identical images with different URLs and the same caption remained.
E Not safe for work (NSFW) filtering: A 0.1 Detoxify threshold filters captions across seven toxicity-related categories, balancing NSFW coverage against false positives despite multilingual and implicit-content limitations.Manual spot checks used a random one-million-caption subset of COMMONPOOL.
E Not safe for work (NSFW) filtering: 97.4% classification accuracy was achieved by the retrained CLIP-based NSFW image filter, versus 96.1% for LAION’s original model on a held-out test set.The model uses CLIP ViT-L/14 embeddings, an initial normalization layer, and a 4-layer multilayer perceptron.
E Not safe for work (NSFW) filtering: Thresholding the image filter at 0.1 achieved high recall relative to Google Vision SafeSearch, with very few manually confirmed true positives on a 40,000-image subset.Images labeled likely or very likely racy by SafeSearch were generally benign, suggestive but non-explicit, or already labeled adult.

F Deduplication against evaluation sets

The deduplication pipeline removes exact and near-duplicate evaluation images from COMMONPOOL using feature cosine similarity and a Yokoo [150] model. With threshold 0.604169, it achieves high duplicate-detection quality while flagging approximately 2.8% of downloaded samples as evaluation near-duplicates.

Deduplication procedure: The pipeline removes exact and near-duplicate evaluation images, including changes in aspect ratio, JPEG compression, overlays, color, and artistic rendering.Images are treated as duplicates when query-reference feature cosine similarity exceeds a fixed threshold.
Deduplication model: The deduplication model proposed by Yokoo [150], which won first place in the Facebook AI Image Similarity Challenge (ISC), is used for filtering.The evaluation compares this model with OpenAI’s CLIP ViT-B/32 baseline on ISC.
Deduplication threshold and performance: Precision 0.9 and recall 0.8 are achieved at threshold 0.604169, while CLIP at threshold 0.96 has precision 0.9 but recall 0.02; approximately 2.8% of samples are flagged.The threshold is selected to maximize true duplicates detected without removing too many false duplicates.
Transformation robustness: Across naturally occurring transformations, CLIP filtering removes over 2× as much data as the Yokoo [150] deduplication model at high recall values.The analysis evaluates JPEG compression, flips, rotations, aspect-ratio changes, and grayscaling using reference images from evaluation datasets and distractors from LAION-2B.

G Face blurring

DataComp incorporates face detection and blurring into pool creation and download tooling to improve privacy. On discriminative downstream tasks, blurring negligibly affects performance, though it may be more detrimental for generative models that output faces.

Detection and blurring: SCRFD-10G detects faces on padded-and-resized 224 × 224 images, retaining bounding boxes whose detection scores exceed 0.3.The resulting bounding boxes are stored as image metadata and used for the blurring pipeline.
Detector evaluation: The detector is evaluated on 3293 COMMONPOOL images using human-judged detection of all clearly visible faces, with recall prioritized over precision and Amazon Rekognition as a comparison.Cartoon drawings of non-real human faces are excluded from the visible-face positives.
Blurring pipeline: The standard blurring pipeline produces smoothly transitioning blurred face regions, and most images in the small COMMONPOOL contain no faces.Face distributions are shown in Figure 6.
Privacy-performance tradeoff: Face blurring negligibly affects downstream performance while improving dataset privacy, supporting its default use for this discriminative benchmark.The authors caution that blurring may be more detrimental in generative settings, especially when models need to output faces.
Bias evaluation: The detector found the central face in every FairFace validation image across annotated subgroups, providing a limited bias check.This evaluation used the FairFace validation set [80].

H DATACOMP COMMONPOOL creation pipeline … L.2 Model architecture

DataComp constructs COMMONPOOL through a multistep pipeline that harvests, annotates, filters, and packages web-scale image-text pairs, then characterizes the resulting data and tooling. Ablations show that subset resharding is efficient and that filtering rankings are largely robust to deduplication, batch size, and model architecture changes.

H DATACOMP COMMONPOOL creation pipeline: 13.1B image-text pairs suitable for COMMONPOOL were produced through harvesting, metadata generation, safety and deduplication filtering, and sampling 12.8B pairs for the xlarge pool.The pipeline downloads approximately 16.8B samples, generates CLIP features, NSFW scores, deduplication scores, and face bounding boxes, then releases metadata and image URLs rather than raw images.
I COMMONPOOL statistics: English predominates in COMMONPOOL, caption token counts are long-tailed toward short sequences, and most downloaded images are near the 224 × 224 training resolution.The statistics analyze an i.i.d. small-pool sample using language, token-length, image-size, and resizing distributions.
J Efficient training on data subsets: Resharding subsets before training performs at most twice as many reads as needed, avoiding the 1/p read overhead of filtering during training for a p-fraction subset.The resharder scans the pool, selects requested examples, and writes a new WebDataset, with multiprocessing and cloud-storage streaming support.
K Effect of duplicates in the training data: Deduplication results were similar to those from CLIP filtering alone when applied independently or together with ViT-B/32 filtering.Duplicates are identified using compressed CLIP image features and approximate nearest-neighbor search, retaining the duplicate with the highest CLIP ViT-L/14 similarity.
L.2 Model architecture: With matched GMACs, replacing ViT-B/32 with ConvNeXt produced rank correlations of 1.0 on ImageNet and 0.87 for the average metric, suggesting filtering improvements extend beyond CLIP ViTs.The broader ablation asks whether datasets that perform well for zero-shot ViT evaluation remain preferable when training procedures, architectures, or steps change.
L.1 Batch size: Rank correlations of filtering strategies after doubling batch size were 0.96 on ImageNet and 0.98 for Average over 38 datasets, with rankings changing by at most one position.The ablation doubles batch size at medium scale while holding other hyperparameters constant, testing whether dataset quality is stable under modified training choices.

L.3 Number of training steps

Training for 10× more steps generally preserves baseline quality relationships, with positive correlations on ImageNet and average performance, although some method orderings change substantially.

Number of training steps: 10× more training steps produce positively correlated ImageNet accuracy and average performance relative to the original configuration.The experiment ablates the fixed training schedule using small-scale baselines.
Number of training steps: The standard configuration fixes training hyperparameters to emphasize data curation and avoid confounding comparisons between participants.The fixed settings follow prior work [111].
Number of training steps: Some filtering-method orderings change quite drastically when training is extended 10×, despite the overall positive correlation.

M Detector-based baselines · N Training details

Detector-based filtering and balancing strategies were tested at medium scale, but the existing Image-based ∩CLIP score (L/14 30%) baseline remained best. Balanced datasets diverged during CLIP training, while scale-specific configurations largely followed OpenCLIP and CLIP settings with targeted large-scale adjustments.

M Detector-based baselines: Detic annotates the 128M-sample medium pool with bounding boxes and labels for 1203 LVIS categories, retaining predictions above 0.5 confidence.These annotations provide the basis for detector-driven filtering and balancing experiments.
M Detector-based baselines: The detector-based baselines include object existence, object centering, and balancing by class, position, or detection count.Class balancing uses 1204 buckets, position balancing uses 26 buckets, and count balancing uses 12 buckets, with equalized sampling across buckets.
M Detector-based baselines: Each detector strategy is evaluated on the medium scale both from the unfiltered pool and from the previous best Image-based ∩CLIP score (L/14 30%) pool, yielding 10 baselines.The strategies can be composed with either starting pool.
M Detector-based baselines: The Image-based ∩CLIP score (L/14 30%) baseline still performs best at medium scale, while all balancing strategies produce poor performance.Table 13 compares unfiltered and detector-based variants; the results identify the existing filtered baseline as the strongest medium-scale option.
N Training details: The experimental configuration for each scale specifies the pool size, architecture, and hyperparameters.These configurations are summarized in Table 14.
M Detector-based baselines: All balancing strategies cause the CLIP contrastive loss to diverge, likely because repeated samples increase self-contrasting under long-tailed class distributions.Rare classes require repeated samples in the balanced construction, increasing the chance that samples are contrasted with themselves.
N Training details: Small, medium, and large tracks follow CLIP-paper hyperparameters with reduced batch sizes, while xlarge increases batch size and doubles learning rate for multi-GPU training.The choices follow OpenCLIP, with the xlarge learning-rate change following prior work [28].

O Evaluation details … P.2 BYOD track

The paper evaluates models with a broad zero-shot suite spanning classification, retrieval, diversity, and VQA, while comparing filtering strategies and external data sources. Image-based filtering outperforms image-based sampling on small and medium pools, and VQA performance strongly correlates with ImageNet and overall evaluation performance.

O Evaluation details: Zero-shot evaluation covers 38 tasks, including classification, retrieval, image association, robustness to distribution shifts, and socioeconomic and geographic diversity.The suite includes 22 Radford et al. classification tasks [111], WILDS datasets [83], Dollar Street and GeoDE [114], Flickr30k, MSCOCO, and WinoGAViL [26] [17].
O Evaluation details: Zero-shot evaluation is used to reduce computational burden and measure out-of-the-box generalization, with ImageNet linear-probe rankings closely matching zero-shot rankings.Spearman rank correlation is 0.99 for the filtering track and 1.0 for BYOD; Figure 16 presents the corresponding comparison.
O.1 Visual Question Answering: VQA performance correlates strongly with ImageNet accuracy (0.877) and average performance on the full evaluation suite in zero-shot experiments.The evaluation uses VQA v1 with question-and-answer prompts, without fine-tuning; Figure 17 reports the full results.
P Baseline details: Baseline visualizations show that no filtering produces noisy image-text pairs, whereas CLIP-score filtering yields more descriptive captions.The compared strategies are No filtering, Basic filtering, and CLIP score (L/14 30%), with visualizations in Figures 18–20.
P.1 Filtering track: Filtering methods combine basic caption cleaning with CLIP-based or image-content selection, including clustering pool embeddings around ImageNet 1K or ImageNet 21K examples.Basic filtering removes non-English captions and captions shorter than two words or six characters; image-based filtering uses K = 105 clusters.
P.1 Filtering track: Image-based sampling performs best at α = 0.2 but remains worse than image-based filtering on small and medium pools, so it is omitted for the large pool.The sampling procedure weights clusters according to their ImageNet-assignment scores and evaluates α values {0, 0.2, 0.5, 1.0, 2.0}.

P.2.1 Additional results · Q Fairness and biases · Q.1 Diversity

The paper reports additional BYOD results and evaluates fairness-related biases using diversity-focused datasets and demographic face datasets. On Dollar Street and GeoDE, the models show dataset-dependent group disparities, with scale improving performance but not eliminating all potential biases.

Q Fairness and biases: The fairness analysis evaluates Dollar Street and GeoDE, and further examines FairFace and UTKFace using demographic labels.This evaluation follows the demographic-bias analysis of Radford et al. [111].
Q.1 Diversity: Dollar Street results reproduce socioeconomic bias: higher-income groups perform better, while the models have a smaller worst-group gap than ImageNet-trained ResNet-50 but underperform Dollar Street fine-tuning.Dollar Street covers household objects from homes worldwide, including homes without Internet access, across ImageNet categories and income-defined subgroups from Rojas et al..
Q.1 Diversity: Higher average accuracy on Dollar Street is associated with a larger worst-group performance gap, motivating future work on this trade-off.This relationship is reported alongside the models’ smaller gap than an ImageNet-trained ResNet-50 and weaker performance than a model fine-tuned on Dollar Street.
Q.1 Diversity: GeoDE exhibits smaller worst-group accuracy gaps than Dollar Street for both ImageNet models and the paper’s models, indicating less bias under its geographically balanced distribution.GeoDE contains everyday objects from six equally represented world regions and is designed to promote geographic diversity [114].
Q.1 Diversity: Increasing scale steadily improves performance across all GeoDE regions, approaching the performance of a model fine-tuned on GeoDE.Geographic bias is not apparent in these GeoDE evaluations, although the paper notes that biases outside this evaluation remain to be investigated.
P.2.1 Additional results: Table 18 reports zero-shot performance for selected baselines in the BYOD track.The paper defines COMMONPOOL as the pool filtered using a CLIP L/14 score threshold of 30%.

Q.2 Fairness … Q59 Any other comments?

This section examines COMMONPOOL’s fairness, privacy, safety, and harmful-content risks in the context of its web-scale, internet-sourced construction. It also describes the dataset’s research-oriented distribution, metadata, and limited updating and takedown support.

Q.2 Fairness: Face-blurred filtering models still classify race, gender, and age above chance, likely through residual faces, contextual clues, or skin-color signals; BYOD performs better overall but differs by trait.Race is binary, gender follows the annotations, and age is grouped into nine ranges; the evaluations are included for continuity and model-behavior probing, not as intended applications.
Q.2 Fairness: 20.7%, 17%, and 19.3% of Black, Southeast Asian, and East Asian men, respectively, are misclassified as female by filtering models on FairFace.BYOD improves men’s accuracy by 1.7–9.9 percentage points but changes women’s accuracy by only 0.6-point gains to 6.2-point drops; race and gender labels are annotated categories rather than fixed identities.
Q.2 Fairness: 0.1% is the maximum non-human misclassification rate, whereas a significant proportion of people receive crime-related labels, exacerbated by BYOD models.These experiments use race-gender intersection classes and harmful crime-related or non-human terms on FairFace and UTKFace, highlighting risks from zero-shot classification on web-scraped data.
Q4 Any other comments?: COMMONPOOL aims to support data-centric research toward web-scale datasets that yield higher-accuracy, safer, and more equitable models.The dataset and benchmark target broad applications, with the initial release evaluating 38 downstream image-classification and image-retrieval tasks.
Q6 How many instances are there in total (of each type, if appropriate)?: 12.8B instances comprise COMMONPOOL, with each sample containing an image URL, alt-text caption, and metadata rather than category labels.The associated text serves as a soft, noisy label, and samples are image-text pairs without explicit relationships between instances.
Q11 Are relationships between individual instances made explicit (e.g., users’ movie ratings, social: COMMONPOOL is a noisy, non-self-contained Common Crawl snapshot whose links may disappear or change, and it cannot be archived at full scale.The dataset may contain unrelated alt-text, duplicates, confidential information, offensive material, and toxic content despite NSFW filtering.
Q18 Does the dataset identify any subpopulations (e.g., by age, gender)?: Face blurring does not guarantee anonymity: detector failures, distinctive celebrity or athlete cues, and names in captions may permit identification.COMMONPOOL also likely contains sensitive data; mitigation includes NSFW image and text filtering plus automatic face blurring during downloads.
S.3 Collection Process: Data is downloaded directly from the public internet, and COMMONPOOL is presented as a research artifact for studying safer internet-scale datasets.The authors explicitly caution against using COMMONPOOL or models trained on it for decisions involving people.
Q26 Over what timeframe was the data collected? Does this timeframe match the creation timeframe: Individuals were not notified or necessarily consenting, although the process respects robots.txt, applies NSFW filtering and face blurring, and supports user takedown requests.The dataset is collected from websites, and the authors direct users to LAION’s dataset-request mechanism for takedowns.
Q32 If consent was obtained, were the consenting individuals provided with a mechanism to revoke: COMMONPOOL filters NSFW content and blurs detected faces, yet may still contain harmful data because it is collected from the internet.The authors report fairness evaluation and distinguish COMMONPOOL from LAION-400M through NSFW preprocessing and face blurring, while acknowledging residual harmful content.
Q36 Was the “raw” data saved in addition to the preprocessed/cleaned/labeled data (e.g., to support: The raw data are withheld for safety reasons; only URL-text pairs and metadata are distributed, with preprocessing supported by open-source tools including Spark, Ray, img2dataset, CLIP, and safety detectors.The released index excludes URLs for images flagged as NSFW, while the software stack includes face detection, toxicity detection, deduplication, and NSFW classification components.
S.5 Uses: The dataset has been used to train CLIP models evaluated zero-shot on 38 downstream image-classification and retrieval tasks, and could also support captioning, image generation, and sociological studies.Face blurring means generative models trained on COMMONPOOL are not expected to generate recognizable human faces.
Q41 What (other) tasks could the dataset be used for?: COMMONPOOL is not intended for production systems making decisions about people, surveillance, or military applications because it reflects societal biases, unfairness, and stereotypes.The authors describe it as a research artifact for studying multimodal dataset curation and downstream effects, while warning against deployment involving race, gender, ethnicity, and other sensitive attributes.
S.6 Distribution: The URL-text sample and metadata are distributed through HuggingFace under CC-BY-4.0, while the dataset is not scientifically intended to be updated and deprecated versions will not be maintained.The team will respond to takedown requests, maintain the hosted dataset, communicate issues through the DATACOMP website, and handle alterations case by case.