PaperlayerView exact revision on arXiv

Source-linked AI summary

Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm

Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, Junjie Yan

arXiv:2110.05208v2cs.CV

TL;DR

CLIP achieves strong zero-shot and transfer performance but requires very large image-text datasets, limiting adoption. DeCLIP addresses this by combining multiple intrinsic supervision signals, achieving improved efficiency and transfer results across the supplied evaluations.

  • Problem

    CLIP requires 400M image-text pairs for pre-training, making its adoption data-hungry and resource-intensive.

  • Method

    DeCLIP combines self-supervision within each modality, cross-modal multi-view supervision, and nearest-neighbor supervision from similar pairs.

  • Results

    DeCLIP-ResNet50 outperforms CLIP-ResNet50 on 8 of 11 downstream visual datasets and achieves 60.4% ImageNet zero-shot top1 accuracy, 0.8% above CLIP using 7.1× fewer data.

  • Takeaways & Limitations

    The experiments support learning generic visual features more efficiently by exploiting broader supervision in multimodal data across architectures and data scales.

Abstract

from arXiv · show

Recently, large-scale Contrastive Language-Image Pre-training (CLIP) has attracted unprecedented attention for its impressive zero-shot recognition ability and excellent transferability to downstream tasks. However, CLIP is quite data-hungry and requires 400M image-text pairs for pre-training, thereby restricting its adoption. This work proposes a novel training paradigm, Data efficient CLIP (DeCLIP), to alleviate this limitation. We demonstrate that by carefully utilizing the widespread supervision among the image-text pairs, our De-CLIP can learn generic visual features more efficiently. Instead of using the single image-text contrastive supervision, we fully exploit data potential through the use of (1) self-supervision within each modality; (2) multi-view supervision across modalities; (3) nearest-neighbor supervision from other similar pairs. Benefiting from intrinsic supervision, our DeCLIP-ResNet50 can achieve 60.4% zero-shot top1 accuracy on ImageNet, which is 0.8% above the CLIP-ResNet50 while using 7.1 x fewer data. Our DeCLIP-ResNet50 outperforms its counterpart in 8 out of 11 visual datasets when transferred to downstream tasks. Moreover, Scaling up the model and computing also works well in our framework.Our code, dataset and models are released at: https://github.com/Sense-GVT/DeCLIP

1 INTRODUCTION

DeCLIP addresses CLIP’s data demands by exploiting supervision within modalities, across views, and from nearest-neighbor pairs. It reports stronger data efficiency and transfer performance than CLIP in the supplied results.

  • CLIP and ALIGN rely on enormous datasets containing 400M/1B image-text pairs, creating substantial storage and computing demands.
  • Prior methods use only single image-text contrastive supervision while overlooking widespread supervision within image-text pairs.
  • DeCLIP combines self-supervision within each modality, cross-modal multi-view supervision, and nearest-neighbor supervision from similar pairs.Its nearest-neighbor supervision samples similar text descriptions in embedding space as additional supervisory signals.
  • 60.4% zero-shot top1 accuracy on ImageNet is 0.8% above CLIP-ResNet50 while using 7.1× fewer data.The result uses a ResNet50 image encoder and Transformer text encoder.
  • DeCLIP-ResNet50 outperforms CLIP-ResNet50 on 8 out of 11 downstream visual datasets.
  • The work studies self-supervision and cross-modal multi-view supervision in million-scale image-text pre-training and advocates exploiting intrinsic multimodal supervision.

2 RELATED WORK

Related work spans supervised visual pre-training, language-image pre-training, self-supervised learning, nearest-neighbor supervision, and scalable two-tower vision-language models. DeCLIP extends these directions by combining intrinsic multimodal supervision with nearest-neighbor signals.

  • Supervised ImageNet pre-training transfers to downstream vision tasks but is difficult to scale because new visual concepts require arduous human labeling.
  • CLIP and ALIGN learn transferable visual features from 400M/1B image-text pairs, and DeCLIP targets improved data efficiency for this setting.
  • Prior multimodal work includes self-supervised CLIP, finer-grained patch-word alignment, and optimal-transport matching to mitigate image-text noise.
  • Nearest-neighbor supervision had mainly been studied for single-modality pre-training on relatively small datasets before DeCLIP applied it to multimodal learning.
  • Many vision-language models require object detectors or dedicated cross-modal transformers, whereas DeCLIP uses a two-tower framework with interaction only at the top.
  • Figure 4 contrasts standard paired image-text training with DeCLIP’s self-supervision, multi-view supervision, and nearest-neighbor supervision.

3 APPROACH

DeCLIP extends CLIP’s single image-text contrastive objective with self-supervision, cross-modal multi-view supervision, and nearest-neighbor supervision. Its framework combines these auxiliary signals with the original CLIP loss to exploit supervision throughout multimodal data.

  • Revisiting CLIP: DeCLIP uses a dual encoder with image and text features projected to a shared dimension, L2-normalized, and trained with symmetric InfoNCE losses.The image encoder can be a CNN or ViT, while the text encoder can be a Transformer or variant.
  • Self-Supervision: Self-supervision applies SimSiam to two augmented image views and masked language modeling to text within each modality.SimSiam maximizes similarity between augmented image representations; MLM predicts randomly masked original tokens.
  • Multi-View Supervision: Multi-view supervision generates two correlated image and text views, then computes contrastive losses across all 2 × 2 image-text combinations.The method reuses augmented image views and applies EDA text augmentation, producing three additional pairings beyond the original.
  • Nearest-Neighbor Supervision: Nearest-neighbor supervision retrieves semantically similar text features in embedding space and adds contrastive objectives between them and the image views.A FIFO feature queue approximates the million-scale data distribution for nearest-neighbor search.
  • Overall Objective: The overall DeCLIP objective weights the original CLIP, self-supervised, multi-view, and nearest-neighbor losses using α, β, and γ.The loss is LDeCLIP = (1 − α − β − γ)LCLIP + α(LISS + LT SS) + βLMV S + γLNNS.

4 EXPERIMENTS

DeCLIP is evaluated across pre-training datasets, architectures, ImageNet zero-shot recognition, downstream transfer, and ablations of its added supervision and training cost. It achieves stronger data efficiency and broad transfer gains while incurring higher per-iteration cost.

  • Zero-shot recognition: 60.4% ImageNet zero-shot top1 accuracy is achieved by DeCLIP-ResNet50 with 56M pairs, 0.8% above CLIP-ResNet50 using 7.1× fewer data.With full data, DeCLIP-ResNet50 and ViT-B/32 reach 62.5% and 66.2%, respectively.
  • Zero-shot recognition: 62.5% and 66.2% are the full-data ImageNet zero-shot accuracies of DeCLIP-ResNet50 and DeCLIP-ViT-B/32.These values are nearly 3.0% higher than the best reported numbers for the two architectures.
  • Downstream transfer: 8 out of 11 downstream datasets favor DeCLIP-ResNet50 over CLIP, with a 0.8% average improvement.DeCLIP performs worse than CLIP on some datasets, including SUN and Food101.
  • Additional-supervision ablation: MVS improves ImageNet zero-shot top1 accuracy by 4.2% over single image-text contrastive supervision in the CC3M ablation.The ablation uses a smaller 1024 batch size and follows the DeCLIP-ResNet50 protocol.
  • Training cost: 27.2% top1 accuracy remains 5.3% above time-equivalent CLIP despite DeCLIP iterations taking 1.5× as long.The comparison uses a smaller CC3M dataset and longer CLIP training.
  • Analysis: Nearest-neighbor pairs generally have similar textual meanings, providing high-quality supervision, although matching is weaker in noisier YFCC and web-crawled datasets.Matching performs particularly well on CC3M and CC12M.

5 CONCLUSION

DeCLIP learns visual representations from broader supervision than single image-text contrastive learning. It combines self-supervision, cross-modal multi-view supervision, and nearest-neighbor supervision to exploit intrinsic multimodal data supervision.

  • 5 CONCLUSION: DeCLIP combines self-supervision, cross-modal multi-view supervision, and nearest-neighbor supervision instead of relying only on image-text contrastive supervision.The paradigm is designed to exploit broader supervision within multimodal data.

A PSEUDO CODE OF DECLIP

The DeCLIP batch update encodes original and augmented image-text views, retrieves text nearest neighbors, and combines four loss components into the training objective.

  • B A PSEUDO CODE OF DECLIP: The batch update encodes I, ˜I, T, and ˜T, then retrieves TNN from a feature queue using current text features.These features feed the contrastive, self-supervised, multi-view, and nearest-neighbor losses.
  • B A PSEUDO CODE OF DECLIP: The DeCLIP loss combines CLIP, self-supervised, multi-view, and nearest-neighbor losses with weights determined by α, β, and γ.The pseudo-code assigns the remaining weight to the original CLIP loss.

B DATA AUGMENTATION

DeCLIP applies self-supervised image and text objectives using augmented views and masked language modeling, alongside specified image and text augmentation policies.

  • Self-supervision: Image self-supervision uses SimSiam, while text self-supervision uses masked language modeling.For text, 15% of tokens are selected and replaced by mask, random, or unchanged tokens with 80%, 10%, and 10% probabilities.
  • Image augmentations: Image augmentation uses random resized crops, color jitter, random grayscale, Gaussian blur, and horizontal flips.The passage specifies probabilities and parameter ranges for these transformations.
  • Text augmentations: Text augmentation uses synonym replacement, random swap, or random deletion, with one strategy selected randomly for each text.The policy is based on EDA.

C PRE-TRAINING DATASETS & IMPLEMENTATION DETAILS

DeCLIP uses open-source and web-crawled image-text data, with dataset sources and training infrastructure specified for reproducibility.

  • Pre-training datasets: The open-source pre-training data comprises CC3M, CC12M, and YFCC, though incomplete downloads reduce the obtained dataset sizes.CC3M contributes about 3M pairs and CC12M about 11M pairs after download failures.
  • Pre-training datasets: 59M web-crawled examples are collected using YFCC-derived tags and synonyms, followed by image, caption, and language filtering.WordNet supplies synonyms for query construction, while damaged, small, captionless, and Chinese-captioned data are filtered.
  • Implementation details: DeCLIP-ResNet50 and DeCLIP-ViT-B/32 are trained from scratch for 32 epochs using batches of 10,240 examples.The setup uses FP16-based optimization and large-scale distributed training.
  • Implementation details: Training requires 8 days for R50 and 10 days for V-B32 on 80 V100 GPUs, while the largest DeCLIP-RegNetY-64GF uses 160 GPUs for 21 days.The reported costs provide compute context for the model configurations.
  • Pre-training datasets: Figure 10 presents randomly sampled image-text pairs from Conceptual Captions, YFCC, Conceptual 12M, and web-crawled data.The figure compares examples across the four training-data sources.

D DOWNSTREAM DATASETS & IMPLEMENTATION DETAILS

The downstream evaluation uses 11 visual datasets and follows CLIP’s logistic-regression protocol with validation-based regularization selection.

  • Downstream datasets: The evaluation suite contains 11 datasets spanning food, objects, scenes, vehicles, textures, pets, general images, and flowers.Birdsnap is omitted because it cannot be downloaded, and PASCAL VOC 2007 is replaced by ImageNet-1K.
  • Implementation details: A logistic regression classifier is trained with L-BFGS for up to 1,000 iterations, and each dataset’s corresponding metric is reported.This follows the CLIP evaluation procedure.
  • Implementation details: The L2 regularization strength λ is selected by a validation-set sweep from 10^-6 to 10^6 using 96 logarithmically spaced steps.A parametric binary search is used to reduce the computation required by the sweep.

E PROMPT ENGINEERING

Zero-shot classification uses prompt templates that convert labels into natural-language descriptions and are matched to the CLIP evaluation setup.

  • Prompt engineering: The method uses prompts such as "a photo of a {label}" because single-word captions are relatively rare in the dataset.The prompt template supplies a natural-language form for zero-shot classification labels.
  • Prompt engineering: For ImageNet, DeCLIP uses the same prompts proposed by CLIP to ensure a fair comparison.Figure 11 lists the prompts used for zero-shot testing.
  • Prompt engineering: The prompts are described as reducing the domain gap between the training data and test set while considering different picture situations.This rationale is given for using prompt templates rather than single-word labels.

F ADDITIONAL STUDY

Additional studies examine performance across training datasets, differences from a YFCC re-implementation, and the memory cost of DeCLIP’s additional views.

  • Different pre-training datasets: Table 7 reports DeCLIP zero-shot ImageNet top1 performance across different training datasets.The table supports comparisons among dataset sources and scales.
  • Different pre-training datasets: 49.3% zero-shot top1 accuracy on ImageNet is achieved using DeCLIP with 29M open-source image-text pairs.The authors describe the open-source data as an affordable benchmark for exploration.
  • YFCC re-implementation: 41.9% zero-shot accuracy on ImageNet-1K is obtained by DeCLIP on YFCC data, 6.0% above the authors’ CLIP re-implementation.The re-implemented CLIP reaches 35.9%, compared with 31.1% reported in the original CLIP paper.
  • YFCC re-implementation: Table 8 reports DeCLIP zero-shot ImageNet top1 performance on YFCC datasets and notes differences from CLIP’s YFCC data-cleaning strategy.The table caption identifies data cleaning as a source of dataset difference.
  • Memory usage: DeCLIP uses more memory because of its additional views; doubling CLIP’s batch size and training it for 64 epochs still leaves it 4.9% below DeCLIP.The comparison is conducted on CC-3M with 16 V100 GPUs.
  • Memory usage: Table 9 presents the ablation study of memory usage.It summarizes the memory-usage comparison described in the additional study.
Loading 2110.05208v2…