PaperlayerView exact revision on arXiv

Source-linked AI summary

No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance

Vishaal Udandarao, Ameya Prabhu, Adhiraj Ghosh, Yash Sharma, Philip H. S. Torr, Adel Bibi, Samuel Albanie, Matthias Bethge

arXiv:2404.04125v3cs.CVcs.CLcs.LG

TL;DR

The paper asks how much downstream concepts are represented in multimodal pretraining data, a gap that makes the meaning of “zero-shot” generalization unclear. It analyzes concept frequency and performance across models and tasks, finding that linear performance gains require exponentially more data. This trend persists under similarity controls and synthetic distributions, while tested models perform poorly on the long-tail “Let it Wag!” benchmark.

  • Problem

    It is unclear how often downstream evaluation concepts occur in multimodal pretraining data, leaving the extent of true zero-shot generalization uncertain.

    Source: §1 · p. 1

  • Method

    The authors compare concept frequencies in pretraining datasets with downstream performance across multimodal models and tasks, and construct the long-tail “Let it Wag!” test set.

    Source: §1 · p. 2

  • Results

    Across concepts, zero-shot performance improves linearly as pretraining frequency grows exponentially, and the trend persists under similarity controls and synthetic distributions.

    Source: §1 · p. 2, §8 · p. 10

  • Takeaways & Limitations

    The exponential data need and poor long-tail benchmark performance indicate that the key to zero-shot generalization under large-scale training remains to be found.

    Source: §1 · p. 2, §8 · p. 10

    The paper does not propose specific solutions to the issues it identifies in current multimodal pretraining strategies.

    Source: R Limitations, Open Questions and… · p. 50

Abstract

from arXiv · show

Web-crawled pretraining datasets underlie the impressive "zero-shot" evaluation performance of multimodal models, such as CLIP for classification/retrieval and Stable-Diffusion for image generation. However, it is unclear how meaningful the notion of "zero-shot" generalization is for such multimodal models, as it is not known to what extent their pretraining datasets encompass the downstream concepts targeted for during "zero-shot" evaluation. In this work, we ask: How is the performance of multimodal models on downstream concepts influenced by the frequency of these concepts in their pretraining datasets? We comprehensively investigate this question across 34 models and five standard pretraining datasets (CC-3M, CC-12M, YFCC-15M, LAION-400M, LAION-Aesthetics), generating over 300GB of data artifacts. We consistently find that, far from exhibiting "zero-shot" generalization, multimodal models require exponentially more data to achieve linear improvements in downstream "zero-shot" performance, following a sample inefficient log-linear scaling trend. This trend persists even when controlling for sample-level similarity between pretraining and downstream datasets, and testing on purely synthetic data distributions. Furthermore, upon benchmarking models on long-tailed data sampled based on our analysis, we demonstrate that multimodal models across the board perform poorly. We contribute this long-tail test set as the "Let it Wag!" benchmark to further research in this direction. Taken together, our study reveals an exponential need for training data which implies that the key to "zero-shot" generalization capabilities under large-scale training paradigms remains to be found.

1 Introduction

The paper asks whether multimodal models truly generalize to unseen concepts, given uncertainty about how often downstream concepts appear in pretraining. Across tasks and models, concept frequency predicts performance: improving performance linearly requires exponentially more data, and tested models perform poorly on a long-tail benchmark.

1 Introduction

  • The study compares downstream performance with the frequency of test concepts in pretraining, compiling 4,029 concepts from 27 evaluation datasets.The concepts span classification, retrieval, and image-generation tasks.Source: §1 · p. 1
  • Across concepts, performance improves linearly as pretraining frequency grows exponentially, a pattern robust to correlated factors and synthetic distributions.The analysis covers classification, retrieval, and generation across 34 models and five pretraining datasets.Source: §1 · p. 2
  • Across pretraining datasets, concept frequencies are long-tailed, so many concepts are rare and are not properly learned given the observed sample inefficiency.Concept distributions are strongly correlated across datasets, and concepts can appear in only one modality; the frequency-performance trend persists despite this misalignment.Source: §1 · p. 2
  • The paper introduces “Let It Wag!”, a long-tailed test set designed to benchmark multimodal generalization while controlling for pretraining concept frequency.Models trained on both open and closed-source datasets show significant performance drops on the benchmark.Source: §1 · p. 2
  • The authors conclude that the key to zero-shot generalization under large-scale training paradigms remains to be found.They situate the work as identifying the exponential data need as a fundamental issue for current multimodal foundation models.Source: §1 · p. 2

2 Concepts in Pretraining Data and Quantifying Frequency

The authors estimate downstream-concept frequencies in pretraining by defining concepts from evaluation datasets and searching text captions and images separately. They intersect the resulting hits to count examples where both image and caption match.

Defining Concepts

  • Concepts are objects or relations drawn from 27 target evaluation datasets, including class names, verbs, background nouns, and nouns in retrieval captions or generation prompts.Ambiguous or irrelevant concepts appearing in fewer than five downstream samples are filtered out.Source: §2 · pp. 2–3

Concept Frequency from Captions

  • Caption frequency is counted by indexing lemmatized nouns and intersecting unigram hit lists for multiword concepts.The inverted dictionaries support concept search in O(1) time with respect to the number of captions.Source: §2 · p. 3

Concept Frequency from Images

  • Image frequency is estimated by tagging images for the 4,029 downstream concepts with RAM++, an open-set multilabel tagging model.RAM++ uses concept descriptions to account for relationships such as synonyms and concept hierarchies.Source: §2 · p. 3

Image-Text Matched Concept Frequencies

  • Matched image-text frequency counts pretraining samples where both the image and its associated caption correspond to a concept.The count is obtained by intersecting image-search and text-search hits, addressing observed image-text misalignment.Source: §2 · p. 3

3 Comparing Pretraining Frequency & “Zero-Shot” Performance

The study evaluates image-text and text-to-image models on downstream tasks against the frequency of their concepts in pretraining. Across varied models, tasks, datasets, and metrics, performance consistently follows a log-linear relationship with concept frequency.

3.1 Experimental Setup

  • The setup covers image-text models and text-to-image models, with downstream evaluations spanning classification, retrieval, and image generation.The CLIP experiments include pretraining and downstream datasets, model architectures, prompting strategies, and accuracy and retrieval-recall metrics; T2I models use LAION-Aesthetics and eight evaluation datasets.Source: §3.1 · p. 4, §3.1.1 · p. 4, §3.1.2 · p. 4

3.1.1 Image-Text (CLIP) Models

  • The CLIP experiments use four pretraining datasets and downstream retrieval and classification datasets, with ResNet and Vision Transformer models.The tested CLIP models vary in architecture and prompting; classification uses mean accuracy and retrieval uses Recall@1, Recall@5, and Recall@10.Source: §3.1.1 · p. 4

3.1.2 Text-to-Image Models

  • The T2I experiments evaluate 24 models pretrained on LAION-Aesthetics using prompts from eight downstream datasets and four generated images per prompt.Evaluation includes image-text alignment and aesthetic scores, including automated metrics and human ratings.Source: §3.1.2 · pp. 4–5

3.2 Result: Pretraining Concept Frequency is Predictive of “Zero-Shot” Performance

  • CLIP results examine classification accuracy and text-to-image Recall@10, while T2I results use expected aesthetics scores; the paper reports similar trends for other listed metrics.The plots show classification results averaged across 17 datasets, retrieval across two, and T2I aesthetic scores across eight datasets.Source: §3.2 · p. 5, Figure 2 · p. 5, Figure 3 · p. 5
  • Across 16 plots, zero-shot performance has a clear log-linear relationship with pretraining concept frequency across model types, tasks, architectures, datasets, metrics, prompting strategies, and modality-specific frequency estimates.The trend persists across all seven analyzed dimensions, although some plots show a slight dip at high frequencies.Source: Figure 2 · p. 5, Figure 3 · p. 5, §3.2 · p. 6

4 Stress-Testing Frequency-Performance Trends with Distributional Controls

The paper tests whether the frequency-performance trend persists after controlling for sample similarity and under a synthetic data distribution. In both settings, it continues to observe log-linear scaling.

4.1 Controlling for Similar Samples in Pretraining and Downstream Data

  • The similarity control tests whether the trend is an artifact of more frequent concepts having more similar pretraining samples to evaluation data.The experiment uses LAION-200M and removes the most similar 50 million samples.Source: §4.1 · p. 6
  • After removing the 50 million samples most similar to test sets, the log-linear relationship between concept frequency and zero-shot performance persists despite substantial drops in absolute performance.The controlled dataset contains 150 million samples; the evaluation uses the same downstream classification and retrieval datasets.Source: §4.1 · pp. 6–7

4.2 Testing Generalization to Purely Synthetic Concept and Data Distributions

  • The synthetic-data experiment tests whether frequency remains predictive under a concept distribution explicitly different from real-world sampling.It assesses models pretrained on entirely synthetic images and captions.Source: §4.2 · p. 7
  • On SynthCI-30M, a purely synthetic dataset with a novel concept distribution, concept frequency still has a clear log-linear relationship with zero-shot performance.The dataset contains 30 million synthetic image-text pairs.Source: §4.2 · p. 7

5 Additional Insights from Pretraining Concept Frequencies

Pretraining concept frequencies are long-tailed, and concept frequencies are strongly correlated across datasets despite differences in scale and curation. Image-text pairs also show substantial misalignment, while CLIP performance on “Let It Wag!” remains below ImageNet.

Finding 1: Pretraining Datasets Exhibit Long-tailed Concept Distributions

  • Over two-thirds of concepts occur at almost negligible frequencies relative to dataset size, reflecting a long-tailed distribution that mirrors performance disparities.The figure also reports that the performance gap between “Let It Wag!” and ImageNet decreases for higher-capacity models, with a fitted slope of 1.58.Source: §5 · p. 7, Figure 6 · p. 8

Finding 2: Misalignment Between Concepts in Image-Text Pairs

  • Across pretraining datasets, 5–36% of image-text pairs are marked misaligned when independently identified image and caption concepts have no overlap.Table 3 reports the number of misaligned pairs and their fraction for each pretraining dataset.Source: §5 · p. 7, Table 3 · p. 8
  • The authors release the precise misaligned image-text samples to support better data curation.Across 40 tested CLIP models, Figure 6 shows large accuracy drops on “Let It Wag!” compared with ImageNet.Source: §5 · p. 8, Figure 6 · p. 8

Finding 3: Concept Frequencies Across Datasets are Correlated

  • Concept frequencies are strongly correlated across pretraining datasets despite differences in scale and curation, with the internet proposed as a shared source of their long-tailed patterns.The datasets span 3M–400M samples; the authors say this correlation inspired “Let It Wag!”.Source: §5 · p. 8, Table 4 · p. 8

6 Testing the Tail: Let It Wag!

Models underperform on “Let It Wag!”, a benchmark built from rare concepts, in both classification and image generation. The classification gap is large across model scales, while generation examples show failures to represent many concepts accurately.

Dataset Details

  • “Let It Wag!” contains 130K web-downloaded test images evenly distributed across 290 curated, long-tailed concept categories, with labels manually cleaned and verified.The concepts were selected as the least frequent across all pretraining datasets.Source: §6 · p. 8

Analysis Details

  • The evaluation uses 40 CLIP models with an 80-prompt ensemble for classification, and three text-to-image models for generation.Generation uses SD-XL [95], SD-v2 [104], and Dreamlike-Photoreal-v2.0.Source: §6 · p. 8

Text-Image Classification Results

  • Up to 20% lower absolute accuracy on “Let It Wag!” than ImageNet appears across model scales and 10 pretraining distributions, although higher-capacity models seem to narrow the gap.The fitted line for higher-capacity models has slope 1.58.Source: §6 · pp. 8–9

T2I Generation Results

  • Stable Diffusion models qualitatively fail on rare concepts, including by misrepresenting activities, generating the wrong concept, or not comprehending it.Prompts were generated with Gemini [121] and GPT-4 [12]; the figure marks generations as correct, incorrect, or ambiguous.Source: §6 · p. 9, Figure 7 · p. 9

Conclusion

  • Across classification and generation experiments, current multimodal models underperform on the long tail regardless of model scale or pretraining dataset.The authors identify better strategies for sample-efficient learning on the long tail as a need.Source: §6 · p. 9

7 Related Work

Prior work links pretraining data to downstream performance through data diversity, distribution differences, and dataset similarity. This work adds concept frequency and exponential sample inefficiency to that discussion, with broader modality and dataset coverage than a concurrent long-tail study.

Related work

  • Prior studies connect pretraining data to downstream performance through diversity, distribution differences, and similarity between pretraining and test datasets.The cited works are Fang et al. [43], Nguyen et al. [88], and Mayilvahanan et al..Source: §7 · p. 9
  • The authors distinguish their analysis from Parashar et al. by showing a log-linear trend that explicitly indicates exponential sample inefficiency in pretrained multimodal models.They also index both image and text modalities and cover diverse datasets at several scales.Source: §7 · p. 9
  • The study extends prior analyses with image-text misalignment quantification, cross-dataset concept-distribution comparisons, and control experiments testing the robustness of log-linear scaling results.These extensions build on prior work on pretraining data in CLIP [43] and language models [62] [82].Source: §7 · pp. 9–10

8 Conclusion

Across five pretraining datasets and 34 multimodal models, the study finds that linear improvements in zero-shot performance require exponentially more data, and that models underperform on the long-tail benchmark. The concept-frequency trend also persists across the two prompting strategies examined.

8 Conclusion

  • Across five pretraining datasets and 34 multimodal models, significant zero-shot performance improvements require exponentially more data, following a sample-inefficient log-linear trend.The study reports that this pattern persists when controlling for pretraining–downstream similarity and when testing on entirely synthetic data distributions.Source: §8 · p. 10
  • All tested models underperform on “Let It Wag!”, the long-tail test set constructed from the study’s findings.The authors say current generalization capabilities remain limited and that the key to “zero-shot” generalization under large-scale training remains to be found.Source: §8 · p. 10

A Concept Frequency is Predictive of Performance Across Prompting Strategies

B Concept Frequency is Predictive of Performance Across Retrieval Metrics

Across retrieval metrics, text-to-image generation, and independently searched image- and text-derived concepts, zero-shot performance follows a log-linear trend with concept frequency. The relationship also appears in a small human evaluation of generated images.

B Concept Frequency is Predictive of Performance Across Retrieval Metrics

C Concept Frequency is Predictive of Performance for T2I Models

  • For Stable Diffusion, a human evaluation of people concepts also finds a log-linear relationship between frequency and text-to-image performance.The study sampled 360 people for evaluation; the authors report inconsistent trends in HEIM human-rated scores and conducted their own small-scale evaluation.Source: C Concept Frequency is Predictive of… · p. 22, Figure 17 · p. 24

D Concept Frequency is Predictive of Performance across Concepts only from Image and Text Domains

E Generalization of findings to improved VLM training objectives

The log-linear scaling trend remains strong for SLIP and CyCLIP, and the appendix extends the analysis through dataset comparisons, a long-tail benchmark, frequency-estimation checks, and misalignment validation.

E Generalization of findings to improved VLM training objectives

F Extended Related Work; G.1 Setup of Mayilvahanan et al. [81]

  • The analysis compares its multi-dataset, image-and-text frequency estimates with Parashar et al. and describes a control set pruned by perceptual similarity.The authors attribute why Parashar et al. results roughly hold across datasets to correlated concept distributions; the control uses a 150M-sample set after pruning 50 million similar LAION samples.Source: F Extended Related Work · p. 27, §G.1 · p. 29

H.1 Final Set of Concepts in Let It Wag!; Further statistics of Let-It-Wag!

H.2 Let It Wag!: Classification Test Set Curation

  • The benchmark curation combines images from three online sources with temporal filtering, outlier removal, deduplication, manual verification, and class balancing.Images were retrieved after January 2023 to mitigate, but not ensure resolution of, possible overlap with pretraining data.Source: §H.2 · p. 31

I Why and How Do We Use RAM++; I.1 Why RAM++ and not CLIP or open-vocabulary detectors?; I.3 GPT-4 Descriptions for each extracted concept

  • RAM++ was selected for fine-grained concepts, and GPT-4 descriptions were added to cover synonyms and concept hierarchies.The authors report poor fine-grained performance from CLIP and OWL-v2 in their examples; the descriptions are intended to improve tagging precision.Source: §I.1 · p. 32, §I.3 · p. 32, Figure 22 · p. 32

I.2 How: Optimal RAM++ threshold for calculating concept frequencies

  • A RAM++ threshold of 0.7 gave the best frequency estimates; lower thresholds admitted noisier hits, although correlations stayed positive at all tested thresholds.The authors prioritize hit precision, even at the cost of sometimes missing correctly tagged images.Source: §I.2 · p. 32, Figure 24 · p. 33, Figure 23 · p. 33

J Clarification regarding 0-frequency points

K Misalignment Degree Results and Human Verification

  • Human annotations of 200 random image-text pairs per dataset agreed with estimated misalignment; YFCC-15M’s high rate likely reflects weak caption-quality filtering.The authors note that Flickr captions often provide high-level context rather than accurately describing image content.Source: K Misalignment Degree Results and… · p. 36, Table 7 · p. 37

L Analysis of dips in high frequency concepts

High-frequency classification points can dip when concepts are ambiguous or overly broad, and those points show greater performance spread. The appendix also includes further text-to-image evaluation on Let It Wag! concepts.

L Analysis of dips in high frequency concepts

  • High-frequency classification dips are attributed to ambiguous concepts and concepts that are too broad to match a consistent visual category.Examples of ambiguous concepts include homonyms and polysemous terms; broad categories can span diverse images.Source: L Analysis of dips in high frequency… · p. 38

M Variance in performance per point in the zero-shot classification plots

N T2I Models: Evaluation

N.1 Quantitative Results by Retrieval

The retrieval evaluation compares generated images for head and tail concepts against real-image galleries and reports a large performance gap. Qualitative examples show difficulty with fine-grained aircraft, activities, animal species, flowers, and objects.

N.1 Quantitative Results by Retrieval

  • The evaluation uses 25 least-frequent and 25 most-frequent concepts, four generated images per concept, and galleries of 100 real images per concept.DINOv2 ViT-S/14 supplies the retrieval features.Source: §N.1 · p. 40
  • A large retrieval performance gap separates head and tail concepts across the three tested text-to-image models.The comparison reports differences in CMC@k for k={1,2,5}.Source: §N.1 · p. 40, Table 8 · p. 40

N.2 Qualitative Results

O Classification Results: Let It Wag!

Let It Wag! accuracy is clearly lower than ImageNet accuracy across 40 tested models, spanning model sizes, architectures, and pretraining datasets. The appendix also documents compute and storage resources and asset licenses.

O Classification Results: Let It Wag!

P Compute and Storage Resources

  • RAM++ image indexing used NVIDIA GPU nodes, while text-index construction and search used a 48-core CPU server; exact storage and runtime costs are documented.Table 10 covers dataset disk space and shards, plus wall-clock runtimes for image tagging and text-index construction.Source: P Compute and Storage Resources · p. 47, Table 10 · p. 47

Q Licenses and Attributions

R Limitations, Open Questions and Future Directions

This section discusses limitations and open questions in measuring and explaining long-tail generalization, proposes research directions, and addresses potential risks associated with exposure to models trained on large-scale image-text datasets. It also notes concerns about web-data biases, informed consent, and the absence of IRB approval.

R Limitations, Open Questions and Future Directions

  • Image-text misalignments may stem from exact-match concept identification, RAM++ tagging inaccuracies, or noisy or irrelevant captions; recaptioning or image-grounded feedback could help.The authors propose explicitly recaptioning images [29] [89] or using grounded image concepts as an additional feedback signal.Source: R Limitations, Open Questions and… · p. 50
  • The study analyzes concepts in isolation, not their combinations, leaving compositional generalization from a data-centric perspective unresolved.The authors distinguish this challenge—generalizing to new combinations of concepts—from traditional zero-shot learning.Source: R Limitations, Open Questions and… · p. 50
  • The empirical log-linear scaling trend lacks a theoretical explanation, and how model scale changes its slope—and capacity-data-frequency tradeoffs—remains an open question.The authors say a theoretical framework could clarify data dependence and inform more efficient training strategies or algorithms.Source: R Limitations, Open Questions and… · p. 50
  • Potential long-tail remedies include LLM-derived descriptions, better data sequencing, and synthetic samples, while retrieval may compensate for pretrained models’ generalization shortcomings.The authors note that simply adding web data may not fill low-density regions of the pretraining distribution; they do not propose a specific solution.Source: R Limitations, Open Questions and… · p. 50

S Broader Impacts

  • Web-sourced datasets share long-tail biases, so models may underperform for digitally marginalized communities, while released artifacts may reflect hateful, harmful, or toxic content.The authors call for more inclusive training algorithms and hope that releasing artifacts will support research on fairer datasets.Source: S Broader Impacts · p. 51

4. Experimental Result Reproducibility

  • The authors report releasing code and more than 300GB of data artifacts, including concept descriptions, search counts, and per-concept evaluation results, to support reproducibility.They say the artifacts cover experiments across T2I and zero-shot settings.Source: §4 · p. 53

9. Code Of Ethics

  • LAION-Aesthetics is a subset of LAION-5B, which has been shown to contain CSAM; the authors plan to reproduce results on a cleaned set and say their 4,029 concepts contain no harmful content.They state that they are in contact with the LAION-5B authors and expect their main results to hold on cleaned data.Source: §9 · p. 55

10. Broader Impacts

  • The broader impact statement describes potential risks from exposure to models trained on large-scale image-text datasets, including Stable Diffusion [104] (v1.4).The risks are discussed in Appendix S.Source: §10 · p. 58
  • Volunteers provided informed consent for the research.Appendix C reports the consent.Source: §10 · p. 58
  • The authors report that IRB approval was not obtained.Source: §10 · p. 58
Loading 2404.04125v3…