PaperlayerView exact revision on arXiv

Source-linked AI summary

Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)

Alex Fang, Gabriel Ilharco, Mitchell Wortsman, Yuhao Wan, Vaishaal Shankar, Achal Dave, Ludwig Schmidt

arXiv:2205.01397v2cs.CVcs.CLcs.LG

TL;DR

The paper asks what causes the robustness gains of large language-image models under natural distribution shifts. Through controlled experiments and new paired-image datasets, it finds that diverse training distributions dominate robustness, while language supervision mainly helps assemble such data.

  • Problem

    The paper asks which of five training changes explains the unprecedented robustness of CLIP and related language-image models to natural distribution shifts.

  • Method

    It conducts controlled experiments isolating training-set size, training distribution, language supervision, test-time prompts, and contrastive loss, using ImageNet-Captions for same-image comparisons.

  • Results

    Robustness is dominated by the training distribution; with the same images, language supervision does not make models more robust than standard supervised learning.

  • Takeaways & Limitations

    More diverse training data, rather than language supervision itself, leads to more robust representations, while language supervision helps assemble diverse training sets.

  • Takeaways & Limitations

    The experiments are non-exhaustive, and whether language information improves ImageNet or broader vision-task performance remains open.

Abstract

from arXiv · show

Contrastively trained language-image models such as CLIP, ALIGN, and BASIC have demonstrated unprecedented robustness to multiple challenging natural distribution shifts. Since these language-image models differ from previous training approaches in several ways, an important question is what causes the large robustness gains. We answer this question via a systematic experimental investigation. Concretely, we study five different possible causes for the robustness gains: (i) the training set size, (ii) the training distribution, (iii) language supervision at training time, (iv) language supervision at test time, and (v) the contrastive loss function. Our experiments show that the more diverse training distribution is the main cause for the robustness gains, with the other factors contributing little to no robustness. Beyond our experimental results, we also introduce ImageNet-Captions, a version of ImageNet with original text annotations from Flickr, to enable further controlled experiments of language-image training.

1 Introduction

The paper investigates why CLIP and related language-image models are unusually robust to natural distribution shifts. Controlled experiments identify training-distribution diversity as the dominant factor, while introducing ImageNet-Captions and a simpler language-image baseline.

  • Motivation: CLIP, ALIGN, and BASIC achieve unprecedented robustness across challenging natural distribution shifts, motivating an investigation of the underlying cause.The paper focuses on shifts including ImageNetV2 and ObjectNet.
  • Research question: The study tests five possible causes: training-set size, training distribution, language supervision during training and testing, and the contrastive loss.
  • Main finding: Controlled experiments find that robustness is determined almost exclusively by the training distribution, not by language supervision when images are held constant.Language supervision has an indirect role by simplifying training on diverse image collections without consistent class labels.
  • Experimental framework: ImageNet-Captions augments 463,622 ImageNet 2012 images with original Flickr text, enabling language-image and class-based training comparisons on the same images.
  • Additional contribution: A simpler baseline uses SimCLR pre-training on YFCC-15M followed by fine-tuning through caption-to-class text matches, without a language model.The approach achieves accuracy similar to CLIP training.

2 Background

The paper measures robustness as performance beyond an ImageNet-based accuracy baseline under natural distribution shifts. It reviews evidence that training-set size alone is insufficient and introduces datasets supporting controlled language-image comparisons.

  • Robustness measurement: The framework compares accuracy on a reference distribution D1 and a shifted distribution D2 while controlling for the confounding effect of baseline accuracy.Figure 1 plots accD1(f) against accD2(f) for ImageNet models and measures vertical lift above their baseline.
  • Robustness measurement: Effective robustness measures accuracy under a shifted distribution beyond the baseline expected from ImageNet accuracy.For model f′, it is ρ(f′) = accD2(f′) − β(accD1(f′)).
  • Distribution shifts: Natural shifts arise from real variations such as lighting, geographic location, and crowdsourcing, unlike synthetic shifts created by computational perturbations.
  • Distribution shifts: The evaluation includes ImageNetV2, ImageNet-Sketch, ImageNet-R, ObjectNet, and ImageNet-A as examples of natural or naturally filtered shifts.
  • Prior evidence: Changing training-set size while holding its distribution constant changes accuracy but does not influence effective robustness, ruling out size as its cause.The training-set size remains important for ImageNet accuracy.
  • Datasets and related work: ImageNet-Captions preserves an ImageNet-like image distribution while pairing images with text, enabling comparisons between multiclass supervised and image-text training.

3 ImageNet-Captions

ImageNet-Captions augments ImageNet with original Flickr text while retaining class labels, enabling controlled comparisons between language-image and standard classification training on the same images.

  • Dataset desiderata: ImageNet-Captions was created to support controlled comparisons between contrastive image-text and standard classification training on identical images.The dataset was designed to isolate language supervision and loss-function effects while holding architecture and images constant.
  • Dataset desiderata: ImageNet-Captions uses original image-source text rather than templates or generated captions to better represent image-text data in the wild.The dataset was also designed to remain related to ImageNet and large enough for contemporary neural-network training.
  • Dataset construction: The dataset pairs 463,622 ILSVRC-2012 training images with original Flickr titles, descriptions, tags, and corresponding class labels.These annotations were recovered from Flickr metadata associated with ImageNet images.
  • Dataset construction: The construction filtered ImageNet’s Flickr-sourced URLs, restricted images to the 2012 competition classes, deduplicated them, and removed profanity-containing text.Filtering produced 642,147 images across 999 classes before deduplication and text cleaning reduced the dataset to 463,622 images.
  • Dataset properties: Captions span 127 languages, with 90% in English, and 94% of images contain the ImageNet class name in associated text.The class-label presence check indicates that most captions contain information relevant to the image class.
  • Caption ablations: The strongest caption variant combines title, tags, and description, while filtering for cleaner English captions reduces performance because it shrinks the dataset.Table 2 reports top-1 accuracy across ImageNet natural distribution shifts.

4 ImageNet-Captions experiments

ImageNet-Captions enables controlled comparisons between language-image and classification training on the same images. Across these experiments, language supervision and prompting do not improve robustness, while caption effects on ImageNet accuracy are mixed and the experiments are non-exhaustive.

  • ImageNet-Captions augments 463,622 ImageNet training images with original Flickr text, enabling matched language-image and classification comparisons.
  • Robustness: On most distribution shifts, ImageNet-Captions language-image models follow the same trends as classification models trained on the same images.Neither reaches the robustness of OpenAI’s CLIP models.
  • Robustness: Pre-trained OpenAI CLIP language-encoder weights improve accuracy but provide no additional effective robustness when transferred to ImageNet-Captions.
  • ImageNet performance: Template-only captions achieve 50.5% ImageNet top-1 accuracy, compared with 34.7% for templates prepended to captions and 48.7% for equivalent classification.The additional caption information hurts ImageNet performance in this comparison.
  • ImageNet performance: Using templates instead of captions can increase ImageNet performance but does not improve robustness; training on all ImageNet with templates behaves like equivalent classification.
  • ImageNet performance: The ImageNet accuracy experiments are non-exhaustive, leaving whether language improves ImageNet or broader vision-task performance for future work.

5 YFCC experiments

YFCC experiments test whether diverse image data, rather than language supervision, explains CLIP’s robustness. Image-only pre-training followed by simple classification and text matching achieves robustness close to CLIP, although reduced supervision limits from-scratch comparisons.

  • Image-only contrastive pre-training on YFCC followed by minimal text matching produces effective robustness close to CLIP.The experiment directly compares the contribution of YFCC image data with language supervision.
  • Dataset: YFCC-15M contains 14,829,396 images with natural-language captions, while YFCC-15M-Cls retains 1,694,125 images covering 953 ILSVRC classes.
  • Results: The YFCC-15M image-only baseline reaches 35.7% accuracy, close to the 37.9% achieved by CLIP trained on all YFCC-15M images and captions.
  • Results: From-scratch models trained on the roughly nine-times-smaller YFCC-15M-Cls dataset achieve much lower accuracy and are difficult to compare with CLIP.
  • Results: YFCC models consistently achieve higher effective robustness than ImageNet-trained models, supporting training-distribution diversity as the key factor.

6 Effect of test time prompts

Prompt experiments separate changes in test-time language inputs from robustness gains. Prompt variations can change performance, but apparent robustness improvements result from reduced accuracy rather than improved distribution-shift behavior.

  • The prompt search varies templates, class-name sources, template counts from 1 to 80, and synonym counts from 1 to 4.
  • Over a hundred prompt experiments show that specific prompt choices substantially affect performance.
  • Prompt-induced increases in effective robustness are entirely due to substantially reduced accuracy.
  • Random interpolation with a classifier that sees no distribution-shift performance change can reproduce the same effective-robustness change.

7 Effect of contrastive training losses

The paper examines whether contrastive pre-training itself explains CLIP’s robustness by comparing several contrastive methods and their learned representations. These methods use different mechanisms, but the supplied results do not establish contrastive loss as the main source of robustness.

  • Contrastive pre-training: Contrastive pre-training encourages similar pairs to be close and dissimilar pairs to be far apart in representation space.CLIP uses images and corresponding captions as similar pairs, while SimCLR uses the same images with different augmentations.
  • Experimental comparison: The experiments compare SimCLRv2, SimSiam, and SwAV after pre-training on ImageNet across ImageNet and five distribution shifts.The methods differ in augmentation strategies, memory banks, and feature-clustering techniques.

8 Conclusion

The investigation concludes that CLIP’s robustness is dominated by the training distribution rather than training set size, language supervision, or contrastive loss. The authors connect this finding to dataset design as an avenue for improving robustness to real-world distribution shifts.

  • 8 Conclusion: Changing the training distribution from ImageNet(-Captions) to YFCC substantially affects the robustness of the resulting models.The conclusion follows after ruling out training set size, language supervision, and the contrastive loss function as primary explanations.
  • 8 Conclusion: Training distribution plays a central role in mitigating real-world distribution shifts.The authors position this result within broader evidence connecting data choices to robustness.
  • 8 Conclusion: Figure 1 shows samples of the candle class from the distribution shifts evaluated in the experiments.The figure provides visual examples of the evaluated shifts rather than a quantitative result.

B ImageNet-Captions experiments training details

The ImageNet-Captions experiments use specified CLIP and classification training configurations, augmentations, model assumptions, and evaluation layouts. Template-based variants produce closely clustered ImageNet top-1 results around the base model.

  • Training configuration: CLIP experiments use AdamW, an initial learning rate of 0.001, cosine annealing with 500 warmup steps, batch size 1024, and different epoch counts for ImageNet-Captions and all ImageNet.The models train for 32 epochs on ImageNet-Captions and 90 epochs on all ImageNet.
  • Training configuration: ImageNet-Captions classification models use cross-entropy with SGD and Nesterov momentum for 90 epochs, decaying the learning rate at epochs 30, 50, and 70.Weight decay is 0.0001, momentum is 0.9, batch size is 256, and the initial learning rate is 0.1.
  • Augmentation: The default augmentation is a random resized crop to 224 followed by normalization, while additional augmentation adds random horizontal flips.The passage specifies the normalization means and standard deviations used by the experiments.
  • Model assumptions: All models use the ResNet-50-based CLIP visual encoder and train on a class-balanced subset of ImageNet-Captions.One configuration additionally uses a linear layer at the end of the visual encoder.
  • Evaluation: The evaluation layouts report top-1 percentages for ImageNet, ImageNetV2, ImageNet-R, ImageNet Sketch, ObjectNet, and ImageNet-A.The supplied table headers identify the experiments and these six evaluation columns.
  • Template and caption variants: Figure 8 reports that templates do not increase a model’s robustness.This figure concerns the template-based experiments described in the section.
  • Template and caption variants: Template experiments replace ImageNet class labels with templates, while caption experiments use captions for the ImageNet-Captions subset.The reported ImageNet top-1 results range from 76.0% to 76.9% across the listed variants.

H Self-supervised training variation experiments

The paper varies self-supervised training approaches and compares their robustness across natural distribution shifts. MAE is more robust than the other listed self-supervised approaches but remains much less robust than CLIP, while the dataset is filtered for offensive or profane captions.

  • Self-supervised training variation experiments: MAE provides more effective robustness than SimCLRv2 or MoCo but remains much less robust than CLIP.The source of MAE’s robustness is left as an open question.
  • Self-supervised training variation experiments: Figure 9 averages robustness over ImageNetV2, ImageNet-R, ImageNet Sketch, and ObjectNet, leaving out ImageNet-A because of its piecewise-function behavior.The figure concerns models pre-trained on ImageNet with various contrastive objectives.
  • Self-supervised training variation experiments: Figure 10 repeats the prior comparison while plotting each distribution shift separately and adding ImageNet-A.This view separates performance by shift instead of using only the aggregate comparison.
  • Dataset filtering: The dataset removes captions considered offensive or profane using bad-word matches, a profanity detector, and human annotations when applicable.These are the three independent filtering mechanisms described for the cleaning process.
  • Dataset filtering: 14,322 samples, approximately 3% of the data, are removed by the combined filtering steps.The total combines the three filtering procedures.

J NoCLIP ablations

The NoCLIP ablations vary label matching, initialization, augmentation, and sampling, while documenting training choices and ImageNet-Captions dataset characteristics. The experiments also note missing classes and a lower-accuracy from-scratch regime.

  • Training setup: NoCLIP uses RandAug augmentation, cosine-annealed learning rates, class-balanced sampling, one training epoch, and early stopping.The learning rate starts at 1e-3; training longer than one epoch hurts performance.
  • Ablation factors: Table 4 varies label matching, initialization, augmentation, and sampler choice in the NoCLIP ablations.Label matching searches for either ImageNet synsets or synsets plus synonyms; initialization uses scratch training or SimCLR pre-training.
  • Data coverage: 47 ImageNet classes do not appear in the YFCC captions.The passage provides the beginning of the omitted-class list, including tiger shark, boa constrictor, and partridge.
  • Model initialization: The YFCC-15M-Cls model trained from scratch operates in a much lower accuracy regime than the other evaluated models.This comparison concerns models trained on top of YFCC-15M versus a model trained from scratch on YFCC-15M-Cls.
  • Dataset characteristics: English appears among the top three languages in 91% of ImageNet-Captions captions.Table 5 reports the most frequent languages using PYCLD2 top-1 language identification.
  • Dataset characteristics: The median ImageNet-Captions caption length is 17 words, while only 3.6% of captions contain 100–4,924 words.Figure 11 limits the displayed x-axis to 100 words because the long-caption tail is small.
Loading 2205.01397v2…