PaperlayerView exact revision on arXiv

Source-linked AI summary

Is a Caption Worth a Thousand Images? A Controlled Study for Representation Learning

Shibani Santurkar, Yann Dubois, Rohan Taori, Percy Liang, Tatsunori Hashimoto

arXiv:2207.07635v1cs.CVcs.LGstat.ML

TL;DR

The paper asks whether language supervision improves visual transfer and compares matched CLIP and image-only models. It finds that dataset scale and caption properties shape which supervision works, and proposes caption interventions to help CLIP better use existing data.

  • Problem

    Prior side-by-side studies did not conclusively establish whether language supervision improves representation transfer because of algorithmic, architectural, and dataset confounders.

    Source: §2 · p. 3

  • Method

    The paper systematically compares matched SimCLR and CLIP models built on their shared contrastive-learning primitive to assess the effect of language supervision.

    Source: §1 · p. 2, §2.1 · p. 3

  • Results

    Neither supervision type is strictly better: at sufficient scale CLIP can outperform SimCLR despite more image data, while language supervision can hurt in low-data regimes.

    Source: §1 · p. 2, §6 · p. 12

  • Takeaways & Limitations

    Filtering low-quality captions and augmenting captions through sampling or paraphrasing are proposed ways to help CLIP better leverage existing datasets.

    Source: §1 · p. 3

    The analysis is limited to transfer learning via probing on certain object-recognition tasks, and the comparison retains some unaccounted inconsistencies.

    Source: §2.1 · p. 4, §6 · p. 12

Abstract

from arXiv · show

The development of CLIP [Radford et al., 2021] has sparked a debate on whether language supervision can result in vision models with more transferable representations than traditional image-only methods. Our work studies this question through a carefully controlled comparison of two approaches in terms of their ability to learn representations that generalize to downstream classification tasks. We find that when the pre-training dataset meets certain criteria -- it is sufficiently large and contains descriptive captions with low variability -- image-only methods do not match CLIP's transfer performance, even when they are trained with more image data. However, contrary to what one might expect, there are practical settings in which these criteria are not met, wherein added supervision through captions is actually detrimental. Motivated by our findings, we devise simple prescriptions to enable CLIP to better leverage the language information present in existing pre-training datasets.

1 Introduction

The paper asks whether language supervision, rather than other differences between CLIP and image-only models, improves transfer, and studies this through matched comparisons. Its findings make the answer conditional on dataset scale and caption properties.

1 Introduction

  • The study systematically compares matched SimCLR and CLIP versions to assess language supervision using their shared contrastive-learning primitive.Source: §1 · p. 2
  • At sufficiently large dataset scale, CLIP transfers better than SimCLR, and more SimCLR image data does not close the gap; in low-data regimes, language supervision hurts in- and out-of-distribution performance.Source: §1 · p. 2
  • Caption descriptiveness directly determines CLIP transfer, while caption variability can adversely affect performance.Source: §1 · pp. 2–3
  • A single descriptive image-caption pair is worth five less descriptive, uncurated YFCC captions.Source: §1 · p. 3
  • To better leverage dataset language, the paper proposes sampling captions from per-image pools, filtering low-quality captions, and paraphrasing captions.Source: §1 · p. 3
  • Dataset properties interact: YFCC's scale can partly compensate for its less-descriptive and variable captions.Source: §1 · p. 3

2 An apples-to-apples comparison

The authors compare CLIP with image-only SimCLR under controlled conditions to isolate language supervision’s value for downstream transfer. On COCO, CLIP leads on most transfer tasks, stochastic caption sampling improves transfer further, and image-augmentation choices substantially affect the comparison.

2 An apples-to-apples comparison

  • To address prior comparisons’ dataset and algorithm confounders, the authors evaluate frozen representations with linear probes on downstream tasks.Their setup follows the evaluation suite of Kornblith et al..Source: §2 · p. 3

2.1 Finding common ground

  • CLIP and SimCLR share contrastive learning, but differ in the transformation producing positive examples: captions for CLIP and image augmentations for SimCLR.The authors use COCO’s curated captions as a higher-quality setting and sample among its five captions per image to approximate CLIPS.Source: §2.1 · pp. 3–4, §2.2 · p. 5
  • Some mismatches remain: CLIP uses a text transformer, and its modality-specific encoders cannot share weights as SimCLR’s do.The authors attribute these residual inconsistencies partly to differences between modalities.Source: §2.1 · p. 4

2.2 A COCO case study

  • On most transfer tasks, CLIP averages 10% accuracy over the supervised baseline and 2% over SimCLR, despite SimCLR’s higher COCO classification accuracy than human-caption CLIP.Supervised models outperform self-supervised ones on COCO classification, but the ordering flips for out-of-distribution transfer.Source: §2.2 · p. 5
  • About 5% better on average in transfer, CLIPS also matches SimCLR in-distribution by sampling captions stochastically.The authors say it is unclear whether the gain reflects optimization or caption properties.Source: §2.2 · p. 5
  • A controlled comparison requires matching image augmentations: standard CLIP augmentations lower average transfer accuracy by 10%, while SimCLR’s performance drops by 50%.The authors use standard SimCLR augmentations for both methods unless otherwise specified.Source: §2.1 · p. 4, §2.2 · p. 5

3 The impact of pre-training data

Across COCO, CC, and YFCC, this section examines how pre-training scale and caption properties shape CLIP transfer. The results show that the value of language supervision depends on having enough data and captions that describe images consistently.

3 The impact of pre-training data

  • At 100K samples, CLIP on CC and YFCC performs worse than image-only training, while greater dataset scale partly compensates for less descriptive, more variable captions.Caption variability is greatest in YFCC and, to a lesser extent, CC.Source: §3 · p. 6, §3.3 · pp. 9–10

3.1 Scale matters

  • With sufficient data, CLIP outperforms SimCLR on all three datasets, and extra SimCLR data does not close the gap; below dataset-specific thresholds, language hurts transfer.The low-data thresholds are <50K, <200K, and <500K samples for COCO, CC, and YFCC, respectively.Source: §3.1 · p. 7, Figure 3 · p. 7

3.2 The importance of descriptive captions

  • One descriptive COCO image-caption pair is worth 2x CC samples and 5x YFCC samples; BLIP re-captioning removes CC/YFCC CLIP’s disadvantage relative to COCO.The study estimates caption descriptiveness using BLIP caption-image matching scores.Source: §3.2 · p. 8, Figure 4 · p. 8

3.3 The effect of intra-dataset variations in captions

  • On COCO, complete and consistent synthetic captions outperform human captions; greater inconsistency or incompleteness lowers transfer, while CLIPS is less affected.CLIPS uses stochastic caption sampling; with five synthetic captions per image, its improvement over CLIP is about 2%.Source: §3.3 · pp. 8–9
  • With BLIP-generated captions, CLIPS improves as captions per image increase, plateauing around 10; scaling image-caption pairs is more effective than adding captions per image.The comparison is made on CC and YFCC, with the pairs-versus-captions trade-off examined on YFCC.Source: Figure 5 · p. 9, §3.3 · p. 10

4 Making existing captions work

The section tests two ways to improve CLIP’s use of existing captions: filter for caption quality and generate paraphrases for stochastic training. Both interventions improve transfer in the reported settings.

Data pre-processing

  • The authors favor image-agnostic filtering over caption selection with image-language models, which they note can require costly image-text processing.The filtering approach selects captions by similarity to COCO captions rather than by matching captions to their images.Source: §4 · p. 10
  • Caption-only filtering of CC and YFCC yields moderate transfer gains of about 2% for a fixed pre-training data budget.A bag-of-n-grams sentence-embedding classifier retains captions predicted to be COCO-like.Source: §4 · p. 10, Table 2 · p. 11

Mitigating caption variability

  • For COCO, paraphrase-trained CLIPS bridges more than half the performance gap between single-caption CLIP and CLIPS trained with five human-written captions.The method uses GPT-J in-context examples to produce five paraphrases per caption.Source: §4 · pp. 10–11, Table 2 · p. 11

5 Related Work

Prior work studies representation transfer and language-based vision, while this paper positions its controlled comparison as complementary to studies of CLIP robustness and transfer. It frames captions as a possible alternative view in contrastive learning.

5 Related Work

  • Earlier representation-learning studies characterize how training choices such as architecture, datasets, and loss functions affect learned representations.Source: §5 · p. 11
  • Prior CLIP work examined robustness in zero-shot classification, while this study focuses on self-supervised representation learning and finds language’s impact depends on caption quality and quantity.Source: §5 · pp. 11–12
  • Devillers et al. argued language supervision did not improve transfer, few-shot learning, or adversarial robustness, but compared models trained on different datasets and objectives.Source: §5 · p. 12
  • The authors present their controlled comparison as enabling more direct claims about language supervision than Devillers et al..Source: §5 · p. 12
  • The study also frames captions as a possible better view of an image than standard augmentations in contrastive learning.Source: §5 · p. 12

6 Discussion

The discussion concludes that neither image-only nor language supervision is uniformly superior: the outcome depends on pre-training data, especially its scale and caption properties. The authors also identify limits in scope, remaining model inconsistencies, and unresolved dataset-bias concerns.

6 Discussion

  • Matched CLIP and SimCLR comparisons show neither supervision type is strictly better: some regimes favor CLIP beyond what more SimCLR image data can match, while others make language supervision harmful.Source: §6 · p. 12
  • The authors attribute these differing outcomes to CLIP’s sensitivity to pre-training scale, caption descriptiveness, and caption variability, and report CLIPS and caption filtering or augmentation as improvements.Source: §6 · p. 12
  • The analysis is limited to probing on certain object-recognition transfer tasks, leaves some CLIP–SimCLR inconsistencies unaccounted for, and does not assess whether learning dataset biases is desirable.Source: §6 · p. 12

A.1 Datasets

The appendix describes dataset examples and model-training choices. COCO examples include multiple human-written captions and object labels, while the model description specifies the encoders, projection heads, and downstream probe.

A.1 Datasets

  • The 2017 COCO data has five human-written captions and multi-object labels per image, with examples including differing descriptions of scenes.Source: §A.1 · p. 18
  • COCO, Conceptual Captions, and YFCC were scraped from online hosting services; ownership remains with uploaders, while research use is permitted under the services’ terms.Source: §A.1 · p. 18
  • COCO, Conceptual Captions, and YFCC were not extensively vetted and may contain identifying information or offensive content; the authors do not redistribute them.Source: §A.1 · p. 21

A.2 Models

  • Experiments use ResNet-50 image encoders, linear and MLP projection heads for CLIP and SimCLR respectively, and a transformer caption encoder for CLIP; downstream transfer uses a linear probe.Source: §A.2 · p. 21

A.3 Hyperparameters

The appendix points to default training hyperparameters and describes shared optimization and probe settings, with adjustments to training duration for selected dataset sizes.

A.3 Hyperparameters

  • The appendix identifies Table 3 as the source of default model-training hyperparameters, but the supplied table passage gives no values.Source: §A.3 · p. 21, Table 3 · p. 21
  • All models use Adam with a cosine learning-rate schedule, and otherwise retain defaults from standard SimCLR and CLIP implementations.Source: §A.3 · p. 21
  • CLIP and SimCLR training on CC/YFCC-2M lasts 100 epochs due to computational restrictions; datasets below 100K receive more epochs to keep iteration counts roughly comparable.Source: §A.3 · p. 21
  • Linear probes train for 250 epochs with SGD, three random seeds, and validation-set learning-rate search; COCO supervision predicts presence of each of 80 objects.Source: §A.3 · p. 22

A.4 Compute

Training each model on a corpus of about 100K takes roughly 8–10 hours on four NVIDIA A100 GPUs.

A.4 Compute

  • A corpus of about 100K takes 8–10 hours to train each model on four NVIDIA A100 GPUs.Source: §A.4 · p. 22

A.5 BLIP recaptioning

The appendix describes BLIP recaptioning and synthetic COCO caption construction, then outlines caption filtering and GPT-J paraphrase augmentation. These procedures vary caption generation, quality, and diversity in existing datasets.

A.5 BLIP recaptioning

  • BLIP captions for CC and YFCC use ViT-Base with nucleus sampling, varied seeds, and a 5–40 token length range.The sampling settings include top-p 0.9 and repetition penalty 1.1.Source: §A.5 · p. 23, Figure 7 · p. 23

A.6 Synthetic COCO captions

  • Synthetic COCO captions vary completeness, terminology consistency, and templates, with image objects randomly ordered.Completeness controls whether all or a random subset of objects are described; consistency controls whether objects use fixed terms or curated synonyms.Source: §A.6 · p. 24, Figure 8 · p. 24

A.7 Filtering captions

  • A fastText bag-of-n-grams classifier selects YFCC captions classified as COCO-like, but filtering remains limited by captions available in the source dataset.The classifier is trained to distinguish source captions from COCO validation captions.Source: §A.7 · pp. 24–25, Figure 9 · p. 25

A.8 Augmenting captions with GPT-J

  • GPT-J paraphrases dataset captions using four paired COCO caption–paraphrase examples as in-context prompts, with temperature sampling generating multiple variants.Source: §A.8 · pp. 25–26, Figure 10

B Additional experiments

Additional experiments report performance effects from changing CLIP’s default data augmentations and from using label information in SimCLR’s COCO training. The appendix also lists experiments on caption variation, filtering, and paraphrasing.

Additional experiments

Related papers

Loading 2207.07635v1…