PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust Cross-Modal Representation Learning with Progressive Self-Distillation

Alex Andonian, Shixing Chen, Raffay Hamid

arXiv:2204.04588v1cs.CVcs.LG

TL;DR

Web-harvested image-caption data contains noisy many-to-many correspondences, while CLIP pretraining is data- and compute-intensive. This paper uses progressive self-distillation to generate soft image-text alignments, and reports consistent gains over CLIP across benchmarks plus improved robustness to natural shifts.

  • Problem

    Web-harvested image-caption data can contain multiple caption alignments and incorrect pairings, while CLIP pretraining is data- and compute-intensive.

    Source: §1 · p. 1, §3.2 · p. 3

  • Method

    The framework uses progressive self-distillation to dynamically generate soft-alignment targets for a subset of image-caption samples in each minibatch.

    Source: §1 · p. 2, §5 · p. 8

  • Results

    Across 14 benchmark datasets, the method consistently outperforms CLIP in multiple settings and offers better effective robustness to natural shifts than ImageNet-trained models and CLIP.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    Its simplicity allows ready incorporation into existing and future methods.

    Source: §1 · p. 2

Abstract

from arXiv · show

The learning objective of vision-language approach of CLIP does not effectively account for the noisy many-to-many correspondences found in web-harvested image captioning datasets, which contributes to its compute and data inefficiency. To address this challenge, we introduce a novel training framework based on cross-modal contrastive learning that uses progressive self-distillation and soft image-text alignments to more efficiently learn robust representations from noisy data. Our model distills its own knowledge to dynamically generate soft-alignment targets for a subset of images and captions in every minibatch, which are then used to update its parameters. Extensive evaluation across 14 benchmark datasets shows that our method consistently outperforms its CLIP counterpart in multiple settings, including: (a) zero-shot classification, (b) linear probe transfer, and (c) image-text retrieval, without incurring added computational cost. Analysis using an ImageNet-based robustness test-bed reveals that our method offers better effective robustness to natural distribution shifts compared to both ImageNet-trained models and CLIP itself. Lastly, pretraining with datasets spanning two orders of magnitude in size shows that our improvements over CLIP tend to scale with number of training examples.

1. Introduction

CLIP’s reliance on large-scale image-text data is costly, while noisy captions and semantically similar negatives challenge its one-caption-per-image assumption. The paper proposes progressive self-distillation with soft alignments and reports consistent gains over CLIP across multiple settings.

1. Introduction

  • The Figure 1 illustration contrasts CLIP’s attraction of paired and repulsion of unpaired examples with the proposed soft alignments, including a stained-glass goldfish example.The caption says the proposed method correctly classifies the ImageNet-R goldfish, unlike CLIP.Source: Figure 1 · p. 1
  • CLIP’s training demands include 400M image-text pairs and an estimated 3,584 GPU days, making pretraining increasingly expensive as data scale grows.The paper frames these requirements as limiting sustainable, widespread adoption.Source: §1 · p. 1
  • Web-harvested captions can include words unrelated to image content, and large batches increase the chance of semantically similar negatives that can degrade representations.CLIP assumes each caption is accurately and exclusively related to its paired image.Source: §1 · pp. 1–2
  • Progressive self-distillation generates soft-alignment targets for random image-caption subsets in each minibatch, re-calibrating potentially mismatched pairs without identifying or pruning them.The network generates alignments for increasingly large minibatch subsets over training.Source: §1 · p. 2
  • The method consistently outperforms CLIP across multiple settings on 14 benchmark datasets, with better effective robustness to natural distribution shifts and gains that tend to scale with training examples.The paper also states that these improvements do not incur additional computational cost.Source: §1 · p. 2

2. Related Work

Related work spans self-supervised representation learning, vision-language pretraining, noisy-data methods, and knowledge distillation. The paper positions its approach as reusing noisy data through self-distillation rather than filtering it or adding computationally costly supervision sources.

Self-Supervised Representation Learning

  • Self-supervised learning derives supervision from data through pretext tasks, including instance discrimination that aligns augmented features while distinguishing other instances with a contrastive loss.Examples include spatial or temporal reasoning, visual-property prediction, reconstruction, and pseudo-labeling or pseudo-clustering.Source: §2 · p. 2

Vision-Language Pretraining

  • Unlike EfficientCLIP [75], this approach repurposes rather than removes noisy data, and unlike DeCLIP [39], it improves data-efficiency without added computational cost.The passage says DeCLIP uses intra-model contrastive learning and a nearest-neighbor feature bank to augment negatives, which can be computationally expensive.Source: §2 · p. 2

Learning from Noisy Data

  • Prior noise-robust methods address supervised detection and classification, but the passage says they do not directly transfer effectively to cross-modal pretraining, where robust VLP remains relatively unexplored.Approaches discussed include outlier-resistant losses, meta-learning, loss correction, regularization, and iterative noise filtering.Source: §2 · p. 2

Knowledge Distillation

  • Knowledge distillation transfers knowledge from teacher to student and can regularize models; this work applies progressive self-distillation to vision-language pretraining.The authors describe this context as among the first investigations of progressive self-distillation in vision-language pretraining.Source: §2 · p. 2

3. Methods

The method addresses noisy many-to-many image-text relations with teacher-generated soft alignments, progressively increasing self-distillation as the model’s representations become more reliable.

3. Methods

  • The method treats web-scale image-text pretraining as learning many-to-many relations and progressively distills its own knowledge to soften initially hard alignments.This is proposed as a way to learn more transferable representations from the same amount of training data.Source: §3 · p. 3

3.1. Preliminaries

  • Cross-modal contrastive pretraining learns image and text encoders whose normalized embeddings are close for related pairs and farther apart for unpaired instances.Source: §3.1 · p. 3

3.2. Contrastive Learning with InfoNCE Loss

  • InfoNCE treats each image as paired exclusively with its corresponding text, although images may match multiple captions and web-scale pairings may be incorrect or loosely correlated.The loss is cross-entropy between one-hot pair labels and estimated probabilities.Source: §3.2 · p. 3

3.3. Distillation through Soft-Alignments

  • A teacher uses opposite-modality predictions to create soft targets that can re-pair poorly captioned images and represent many-to-many batch relations.Swapped prediction derives each modality’s alignment scores from the other modality’s posterior probabilities.Source: §3.3 · p. 3

3.4.1 Teacher Network Selection

  • Using the current student as teacher avoids the secondary inference stage and extra model weights required by conventional teacher choices.The student’s current representations update its own contrastive targets as learning progresses.Source: §3.4.1 · p. 4

3.4.2 Progressing from Student to Teacher

  • As training advances, the model increasingly acts as its own teacher: random per-epoch partitions train the teacher on aligned pairs and supervise unaligned pairs, while α decreases.The decreasing α increases the teacher’s influence on learning.Source: §3.4.2 · p. 4

4. Experiments

Across matched CLIP comparisons, the method improves zero-shot classification, linear-probe transfer, and image-text retrieval. Analyses also report stronger effective robustness to natural shifts and advantages across CC12M data scales.

4.1. Pretraining Datasets

  • Experiments span COCO, CC3M, and CC12M, with classification measured by top-1 accuracy and retrieval by recall at K and mean rank.The evaluation includes ImageNet, Places365, ObjectNet, and natural-shift ImageNet variants.Source: §4.1 · p. 5, §4.3 · p. 5

4.4. Zero-Shot Image Classification

  • Up to 6.19% absolute zero-shot top-1 gain over CLIP on CC3M, with nearly 8.5% average gain on ImageNet-R using Conceptual Captions; the method also surpasses CLIP on ImageNet variants.The comparison covers zero-shot top-1 accuracy across ImageNet-based benchmarks and pretraining datasets of different scales.Source: §4.4 · pp. 5–6, Table 1 · p. 6
  • Across 600K–12M CC12M examples, the method maintains higher average zero-shot performance than CLIP, with gains appearing largest at the data-size boundaries.The authors suggest this pattern indicates improved data efficiency in data-limited regimes as well as larger gains with more data.Source: Figure 3 · p. 6, §4.4 · p. 6

4.5. Evaluation of Effective Robustness

  • The method’s effective-robustness trend has a steeper slope than CLIP’s as ImageNet accuracy increases, and it performs better on naturally shifted distributions at similar ImageNet performance.Mean transfer accuracy is computed over ImageNet-R/O/A/V2.Source: Figure 4 · p. 6, §4.5 · p. 6

4.6. Linear Probe Performance

  • The method outperforms CLIP on linear-probe performance across all four downstream datasets, indicating more transferable visual features.The table reports linear-probe top-1 accuracy and attributes the gains to improved within-modal feature alignment.Source: Table 2 · p. 7, §4.6 · p. 7

4.8. Ablation Study

  • Static posterior bootstrapping degrades representations, swapped prediction restores classification to slightly above baseline but leaves retrieval below it, and dynamic partitioning exceeds baseline performance.The authors suggest the static teacher may reinforce its own mistakes without loss penalties, causing partial collapse for unaligned examples.Source: Table 4 · p. 7, §4.8 · p. 7

4.7. Image-Text Retrieval

  • On COCO, the method consistently outperforms CLIP in both image-to-text and text-to-image zero-shot retrieval subtasks.Direct COCO pretraining yields higher downstream retrieval performance than CC12M pretraining despite CC12M’s 100× larger dataset; COCO fine-tuning further improves retrieval.Source: §4.7 · p. 7, Table 3 · p. 8

4.9. Qualitative Analysis

  • Similarity-score distributions show higher-mean, lower-variance positive-pair scores and higher negative-pair scores, while retrieval examples capture more query aspects than CLIP.For the final Figure 6 example, the method retrieves images reflecting the cat, chair, and living room, whereas CLIP focuses on the living room and misses the cat.Source: §4.9 · pp. 7–8, Figure 5 · p. 8

5. Conclusions

The framework uses progressive self-distillation to generate soft image-text targets for robust learning from noisy data, and reports consistent gains over CLIP across benchmark settings.

5. Conclusions

  • Progressive self-distillation dynamically generates soft-alignment targets for a subset of minibatch samples, helping the model learn robust representations from noisy data.The framework uses cross-modal contrastive learning and soft image-text alignments.Source: §5 · p. 8
  • Retrieved images from the method more holistically match text queries than CLIP’s, even after the ground-truth image appears in the ranking.Figure 6 shows the top ten retrieved images, ranked left to right.Source: Figure 6 · p. 8
  • Across 14 benchmark datasets, the method consistently outperforms its CLIP counterpart in multiple settings.The conclusion also reports better effective robustness to natural distribution shifts than existing state-of-the-art methods.Source: §5 · p. 8
  • The authors plan to investigate data redundancy, network architectures, and optimization algorithms to improve efficiency and generalize to larger scale with fewer resources.These are directions for future work, not reported results.Source: §5 · p. 8

Supplementary Materials

The supplied supplement passage identifies the paper’s arXiv version and date.

Supplementary Materials

  • The paper is identified as arXiv:2204.04588v1 [cs.CV], dated 10 Apr 2022.Source: MIT CSAIL · p. 13

1. Summary of Datasets

The dataset summary describes pretraining dataset sizes and evaluation-set composition, including the limits of available caption data and testing-only benchmarks.

1.1. Pretraining Datasets

  • COCO pretraining multiplies image-text pairs fivefold by randomly selecting among five captions per image; CC12M can fall 1M+ below its advertised size because some URLs are invalid.Table 1 lists exact pretraining dataset example counts.Source: §1.1 · p. 13, Table 1 · p. 13

1.2. Evaluation Datasets

  • Evaluation dataset details include class counts and train/test split sizes; ObjectNet and ImageNet-{A,O,R,V2} are testing-only, so only class counts and test sizes are listed.Source: §1.2 · p. 13

2. Detailed Experimental Settings

The experimental settings adapt CLIP’s pretraining implementation, report unchanged model and training costs, and specify the zero-shot and linear-probe evaluation procedures.

2.1. Implementation Details

  • The implementation reduces batch size, learning rate, and weight decay versus CLIP while increasing training epochs and warm-up iterations, and uses global all-gather for within-batch similarities.Table 3 summarizes hyperparameters shared by baseline CLIP and the proposed method.Source: §2.1 · p. 13, Table 3 · p. 13

2.2. Complexity Analysis

  • The ViT-B/32 and RN50 models retain CLIP’s parameter counts, with effectively identical iteration time and memory requirements; alternative distillation designs may require at least 2x compute and memory.The reported COCO iteration time is 0.253s using 8 Nvidia V100 GPUs.Source: §2.2 · p. 14

2.3. Prompt Engineering

  • For zero-shot classification, prompt templates replace {} with the category name and correct the preceding article from “a” to “an” when needed.Source: §2.3 · p. 14

2.4. Linear Probe Details

  • Linear-probe evaluation trains logistic regression on frozen image-encoder features for up to 1000 iterations using L-BFGS.The evaluation uses the train/test splits listed in Table 2.Source: §2.4 · p. 14

3. Comparison to Concurrent Works

Among methods using at most 15M pretraining examples, the paper reports the highest zero-shot ImageNet top-1 accuracy for its approach. Comparisons with concurrent works are not exact one-to-one analogs, and even matched settings can differ substantially.

3. Comparison to Concurrent Works

  • Among approaches using at most 15M pretraining examples, the method achieves the highest absolute zero-shot ImageNet top-1 accuracy.The concurrent-work comparisons vary in datasets, backbones, hyperparameters, and training details.Source: §3 · p. 14, Table 5 · p. 14
  • Accuracy differs by as much as 4.8% between architecture- and dataset-matched experiments, underscoring the difficulty of meaningful comparisons.The passage compares and.Source: §3 · p. 14

4. Additional Quantitative Results

Additional experiments report that the method’s trends extend to an RN50 visual backbone and that downstream performance is fairly robust to the choice of α schedule.

4.1. ResNet50 Visual Backbone Results

  • RN50 experiments show trends consistent with the ViT results, suggesting the method also works with CNN-based backbones.The comparison is reported in Table 6; the method received minimal hyperparameter tuning for the backbone change.Source: §4.1 · p. 14, Table 6 · p. 15

4.2. Ablation Studies for α-scheduling

  • For ViT, cosine versus linear α scheduling differs by 0.13% absolute and 0.67% relative on COCO, and −0.22% absolute and −0.63% relative on CC3M.The reported differences are in top-1 classification performance.Source: §4.2 · p. 14

5. Additional Qualitative Results

The qualitative examples illustrate more complete text-image retrievals, more reasonable predictions on ImageNet-R examples, and different treatment of semantically similar image-text pairs.

5.1. Additional Text-Image Retrievals

  • On COCO retrieval examples, the method more often captures the full caption meaning, while CLIP tends to focus narrowly on one aspect.For the cited cat query, the method retrieves cat images in 9 of the top 10 results; CLIP also retrieves dogs, a zebra, and boxes.Source: §5.1 · p. 15

5.2. Additional Robust Classification Examples

  • For ImageNet-R examples involving artistic styles, strong color patterns, or unusual contexts, the method more consistently produces reasonable top predictions than CLIP.The passage relates these examples to a reported average improvement of nearly 8.5% with Conceptual Captions pretraining.Source: §5.2 · p. 15

5.3. Similarity Matrix Visualization

  • Similarity matrices show elevated scores for semantically related unpaired pairs under the method, whereas CLIP emphasizes diagonal positives and low off-diagonal scores.The passage also reports larger scores for ground-truth positive pairs and more robust zero-shot classification performance.Source: §5.3 · p. 15

6. Ethical Considerations

The ethical discussion connects more efficient vision-language pretraining with accessibility and notes intended relevance to robust systems facing practical and environmental challenges.

6.1. Impact on ML and Related Scientific Fields

  • Improving VLP efficiency and reducing its computational and environmental costs are presented as aims that could make these methods more accessible to study, develop, and deploy.The paper frames VLP as a foundation-model training method with potential broad downstream impact.Source: §6.1 · pp. 15–16
  • The paper’s retrieval examples show more holistic matches to text descriptions than CLIP, including after the ground-truth image appears in the ranking.The figures display the top ten retrieved images for each text query.Source: Figure 1 · p. 16, Figure 2 · p. 17

6.2. Impact on Society

  • The framework is intended to address challenges in training robust and reliable systems amid difficult examples, sensitive applications, and diverse data sources.The discussion names autonomous vehicles and medical applications as sensitive contexts.Source: §6.2 · pp. 17–18

Related papers

Loading 2204.04588v1…