PaperlayerView exact revision on arXiv

Source-linked AI summary

SLIP: Self-supervision meets Language-Image Pre-training

Norman Mu, Alexander Kirillov, David Wagner, Saining Xie

arXiv:2112.12750v1cs.CV

TL;DR

The paper asks whether image self-supervision can improve language-supervised visual representations despite potential interference between their objectives. It introduces SLIP, which combines both objectives in a multi-task framework, and finds consistent gains across evaluation settings and datasets, with added training cost.

  • Problem

    Evidence is limited on whether self-supervised learning scales well to larger uncurated datasets and benefits language-supervised visual representation learning.

    Source: §1 · p. 2

  • Method

    SLIP combines CLIP language supervision and image self-supervision through a shared image encoder and evaluates representations using zero-shot transfer, linear classification, and end-to-end finetuning.

    Source: §1 · p. 2

  • Results

    SLIP improves zero-shot ImageNet performance across model sizes by +4.8% to +5.6% and shows consistent linear-classification gains, with improvements across most evaluations.

    Source: §1 · p. 2, §7 · p. 8

  • Takeaways & Limitations

    The findings support the complementarity of language supervision and self-supervision across ImageNet and additional classification benchmarks.

    Source: §7 · pp. 7–8

    SLIP increases activation count and memory footprint, taking 30.5 hours versus 22.3 hours for CLIP to train ViT-B/16 on 64 V100 GPUs.

    Source: §3.3 · p. 3, §7 · p. 8

Abstract

from arXiv · show

Recent work has shown that self-supervised pre-training leads to improvements over supervised learning on challenging visual recognition tasks. CLIP, an exciting new approach to learning with language supervision, demonstrates promising performance on a wide variety of benchmarks. In this work, we explore whether self-supervised learning can aid in the use of language supervision for visual representation learning. We introduce SLIP, a multi-task learning framework for combining self-supervised learning and CLIP pre-training. After pre-training with Vision Transformers, we thoroughly evaluate representation quality and compare performance to both CLIP and self-supervised learning under three distinct settings: zero-shot transfer, linear classification, and end-to-end finetuning. Across ImageNet and a battery of additional datasets, we find that SLIP improves accuracy by a large margin. We validate our results further with experiments on different model sizes, training schedules, and pre-training datasets. Our findings show that SLIP enjoys the best of both worlds: better performance than self-supervision (+8.1% linear accuracy) and language supervision (+5.2% zero-shot accuracy).

1. Introduction

SLIP asks whether image self-supervision can strengthen language-supervised visual representation learning despite possible interference between their objectives. It combines both forms of supervision and evaluates transfer across multiple settings and datasets.

  • Self-supervised learning has recently improved performance on larger datasets such as ImageNet, reducing reliance on labor-intensive annotation.Source: §1 · p. 1
  • ImageNet-centered evaluation can bias comparisons against models not pre-trained on ImageNet, while evidence for scaling self-supervision to larger uncurated datasets remains limited.Source: §1 · pp. 1–2
  • CLIP uses images and free-form captions, scales to large datasets and models, and renews language supervision as an alternative to label-based learning.Source: §1 · p. 2
  • SLIP combines language supervision and image self-supervision in a multi-task framework, although the objectives may encode conflicting information and cause interference.Source: §1 · p. 2
  • SLIP models are evaluated using zero-shot transfer, linear classification, and end-to-end finetuning across ImageNet and 25 additional classification benchmarks.Source: §1 · p. 2

2. Related Work

Related work establishes language supervision and self-supervision as complementary approaches to visual representation learning, while prior multi-modal methods add objectives or extend language coverage.

  • Language-supervised representation learning has included shared image-text embeddings, bag-of-words prediction, n-gram prediction, and caption modeling.Source: §2 · p. 2
  • Contrastive self-supervised methods improved scalability and effectiveness, alongside alternatives based on self-distillation and input reconstruction.Source: §2 · p. 2
  • Multi-modal multi-task approaches have extended image-text learning with crosslingual or additional training objectives.Source: §2 · p. 2

3. SLIP Framework

SLIP uses a shared image encoder to combine CLIP language supervision with view-based self-supervision, summing their objectives during training. The framework is designed to retain CLIP’s zero-shot capability while improving visual representations.

  • SLIP constructs separate image views for language and image self-supervision branches, which feed a shared image encoder during pre-training.Source: §3 · p. 3
  • CLIP embeds matching images and captions into a normalized shared space and trains with an InfoNCE loss over positive and negative pairs.Source: §3.1 · p. 3
  • The CLIP contrastive objective enables image classification without retraining dataset-specific classification layers through zero-shot transfer.Source: §3.1 · p. 3
  • SLIP primarily adapts SimCLR view-based self-supervision and uses Vision Transformers, while other self-supervised frameworks can be substituted.Source: §3.2 · p. 3
  • During each forward pass, SLIP computes CLIP and self-supervised losses on the shared encoder’s embeddings and sums them into one scalar loss.Source: §3.3 · p. 3
  • SLIP processes more images and produces approximately 3× more activations, increasing memory use and slowing training.Source: §3.3 · p. 3

4. Improved Training Procedure

The improved procedure adapts CLIP training for YFCC15M and specifies data, architecture, optimization, and evaluation choices. It uses caption-based language supervision alongside augmented image views for self-supervision.

  • The procedure achieves 34.6% zero-shot transfer to ImageNet with a modified ResNet-50 on YFCC15M, exceeding the original 31.3% result.Source: §4 · p. 3
  • Experiments primarily use YFCC15M, a filtered English-only subset of YFCC100M, and also evaluate CC3M and CC12M.Source: §4.1 · p. 3
  • Training samples a valid caption for each image, applies global crops to the CLIP branch, and generates two MoCo v3-augmented views for self-supervision.Source: §4.1 · p. 3
  • The image encoders use ViT-B/16, ViT-L/16, or ViT-S/16, while the text encoder is CLIP’s smallest 38M-parameter Transformer.Source: §4.1 · p. 3
  • The CLIP branch projects image and caption embeddings into a 512-dim space, whereas the self-supervised branch maps image embeddings into a 256-dim output space.Source: §4.1 · pp. 3–4
  • Training uses batch size 4096 with AdamW, β2 = 0.98, and distinct weight decay values of 0.5 for CLIP and 0.1 for SLIP.Source: §4.1 · p. 4
  • Zero-shot evaluation averages caption embeddings across prompts and compares them with image embeddings using cosine similarity.Source: §4.1 · p. 4
  • Linear evaluation freezes the representation and trains a final classifier, while end-to-end finetuning updates the model using task-specific training procedures.Source: §4.1 · p. 4

5. Empirical Evaluations

SLIP combines language supervision with image self-supervision and is evaluated against CLIP and self-supervised baselines across ImageNet settings and additional benchmarks. It generally improves representation quality, with benefits extending across model sizes, training schedules, and pre-training datasets.

Evaluation setup

  • SLIP evaluates representations using zero-shot transfer, linear classification, and end-to-end finetuning on ImageNet and additional classification benchmarks.Zero-shot transfer uses text prompts without updating model weights.Source: §5.1 · p. 4, Figure 3 · p. 5

ImageNet results

  • SLIP significantly improves ImageNet zero-shot transfer over CLIP across all three Vision Transformer model sizes.The improvement is approximately +5%, narrowing from +5.6% for ViT-Small to +4.8% for ViT-Large.Source: §5.1 · p. 5, Table 2 · p. 5
  • SLIP significantly outperforms SimCLR in ImageNet linear classification across all three model sizes.The gap is largest for ViT-Large, at almost +10%, while CLIP exceeds SimCLR by a smaller margin.Source: §5.1 · p. 5
  • On ImageNet end-to-end finetuning, SLIP slightly exceeds SimCLR for ViT-Large, while self-supervision addresses CLIP’s weak finetuning performance.CLIP ViT-L exceeds CLIP ViT-B by only +0.5%, whereas SimCLR ViT-L gains +3.0% over CLIP ViT-L.Source: §5.1 · pp. 5–6

Additional benchmarks

  • SLIP’s additional-benchmark results are strongest on datasets represented in YFCC15M, where larger models and longer training generally improve zero-shot accuracy.Some poorly matched datasets remain near chance, and low-resolution datasets show very poor zero-shot performance.Source: §5.3 · p. 6

Dataset and framework comparisons

  • SLIP maintains its improvement over CLIP across ImageNet settings on CC3M and CC12M, although dataset choice changes which evaluation setting benefits most.CC12M lowers zero-shot accuracy but raises linear and finetuning performance relative to YFCC15M.Source: §5.4 · p. 6

6. Further Analysis

Further analysis tests whether SLIP’s gains come from joint training, data augmentation, or coupling the self-supervised and language-supervised signals.

  • Jointly training CLIP and self-supervision outperforms initializing CLIP with self-supervised weights and then fine-tuning with CLIP.The initialization approach improves early progress but stalls during the 25-epoch training and underperforms SLIP across all three ImageNet evaluations.Source: §6 · p. 7
  • Additional color, blur, crop, and flip augmentations are evaluated to test whether augmentation alone explains SLIP’s improvements.Source: §6 · p. 7
  • SLIP-decoupled performs just as well as SLIP when self-supervision uses a disjoint image set sampled independently from the language-supervision images.Source: §6 · p. 7

7. Discussion

SLIP combines language supervision with image self-supervision and improves representation quality across evaluation settings, while introducing training costs and data-related limitations.

Results

  • SLIP’s gains over CLIP are consistent across zero-shot transfer, linear classification, and end-to-end finetuning evaluations.The comparison uses ImageNet and additional downstream benchmarks.Source: §7 · pp. 7–8

Comparison

  • Initializing CLIP with self-supervised weights performs noticeably worse than SLIP across all three ImageNet evaluation settings.Source: §7 · p. 7, Table 7 · p. 8
  • Color and blur augmentations improve CLIP slightly, but by much less than SLIP’s improvement.Source: §7 · p. 7, Table 8 · p. 8

Results

  • +4.8% to +5.6% zero-shot ImageNet performance gains are observed across model sizes.Source: §7 · p. 8

Practical limitations

  • SLIP’s forward pass increases activation memory and training time, taking 30.5 hours versus 22.3 hours for CLIP on ViT-B/16.Source: §7 · p. 8

Scope limitations

  • Uncurated pre-training alone appears inefficient for concepts unlikely to be widely shared on social media or the broader internet.Source: §7 · p. 8

A. Additional Implementation Details

The implementation details describe dataset preprocessing, dataset-specific filtering, training schedules, and finetuning settings used for the experiments.

Datasets

Pre-training

End-to-end Finetuning

B. Full Scaling Results

Scaling generally benefits SLIP across model sizes and training durations, but longer training can reduce ViT-L linear-classification performance while improving other evaluations.

Full Scaling Results

  • Larger models and longer training generally improve ImageNet performance across zero-shot transfer, linear classification, and end-to-end finetuning.Source: B. Full Scaling Results · p. 12
  • SLIP ViT-L/16 linear-classification performance degrades slightly with longer training and also worsens on average across other downstream benchmarks.Source: B. Full Scaling Results · p. 12
  • Zero-shot transfer and end-to-end finetuning improve with longer training, so the ViT-L behavior is not simple overfitting across evaluation settings.Source: B. Full Scaling Results · p. 12

C. Additional Linear Classification Benchmarks

Additional benchmark results show that SLIP generally outperforms CLIP, while transfer quality depends strongly on dataset overlap with pre-training data and task requirements.

Dataset dependence

Limitations

Scaling behavior

Additional Linear Classification Benchmarks

Related papers

Loading 2112.12750v1…