Source-linked AI summary
SLIP: Self-supervision meets Language-Image Pre-training
Norman Mu, Alexander Kirillov, David Wagner, Saining Xie
TL;DR
The paper asks whether image self-supervision can improve language-supervised visual representations despite potential interference between their objectives. It introduces SLIP, which combines both objectives in a multi-task framework, and finds consistent gains across evaluation settings and datasets, with added training cost.
Problem
Evidence is limited on whether self-supervised learning scales well to larger uncurated datasets and benefits language-supervised visual representation learning.
Method
SLIP combines CLIP language supervision and image self-supervision through a shared image encoder and evaluates representations using zero-shot transfer, linear classification, and end-to-end finetuning.
Results
SLIP improves zero-shot ImageNet performance across model sizes by +4.8% to +5.6% and shows consistent linear-classification gains, with improvements across most evaluations.
Takeaways & Limitations
The findings support the complementarity of language supervision and self-supervision across ImageNet and additional classification benchmarks.
Takeaways & Limitations
SLIP increases activation count and memory footprint, taking 30.5 hours versus 22.3 hours for CLIP to train ViT-B/16 on 64 V100 GPUs.
Abstract
from arXiv · showhide
Recent work has shown that self-supervised pre-training leads to improvements over supervised learning on challenging visual recognition tasks. CLIP, an exciting new approach to learning with language supervision, demonstrates promising performance on a wide variety of benchmarks. In this work, we explore whether self-supervised learning can aid in the use of language supervision for visual representation learning. We introduce SLIP, a multi-task learning framework for combining self-supervised learning and CLIP pre-training. After pre-training with Vision Transformers, we thoroughly evaluate representation quality and compare performance to both CLIP and self-supervised learning under three distinct settings: zero-shot transfer, linear classification, and end-to-end finetuning. Across ImageNet and a battery of additional datasets, we find that SLIP improves accuracy by a large margin. We validate our results further with experiments on different model sizes, training schedules, and pre-training datasets. Our findings show that SLIP enjoys the best of both worlds: better performance than self-supervision (+8.1% linear accuracy) and language supervision (+5.2% zero-shot accuracy).
1. Introduction
SLIP asks whether image self-supervision can strengthen language-supervised visual representation learning despite possible interference between their objectives. It combines both forms of supervision and evaluates transfer across multiple settings and datasets.
- Self-supervised learning has recently improved performance on larger datasets such as ImageNet, reducing reliance on labor-intensive annotation.
- ImageNet-centered evaluation can bias comparisons against models not pre-trained on ImageNet, while evidence for scaling self-supervision to larger uncurated datasets remains limited.
- CLIP uses images and free-form captions, scales to large datasets and models, and renews language supervision as an alternative to label-based learning.
- SLIP combines language supervision and image self-supervision in a multi-task framework, although the objectives may encode conflicting information and cause interference.
- SLIP models are evaluated using zero-shot transfer, linear classification, and end-to-end finetuning across ImageNet and 25 additional classification benchmarks.
2. Related Work
Related work establishes language supervision and self-supervision as complementary approaches to visual representation learning, while prior multi-modal methods add objectives or extend language coverage.
- Language-supervised representation learning has included shared image-text embeddings, bag-of-words prediction, n-gram prediction, and caption modeling.
- Contrastive self-supervised methods improved scalability and effectiveness, alongside alternatives based on self-distillation and input reconstruction.
- Multi-modal multi-task approaches have extended image-text learning with crosslingual or additional training objectives.
3. SLIP Framework
SLIP uses a shared image encoder to combine CLIP language supervision with view-based self-supervision, summing their objectives during training. The framework is designed to retain CLIP’s zero-shot capability while improving visual representations.
- SLIP constructs separate image views for language and image self-supervision branches, which feed a shared image encoder during pre-training.
- CLIP embeds matching images and captions into a normalized shared space and trains with an InfoNCE loss over positive and negative pairs.
- The CLIP contrastive objective enables image classification without retraining dataset-specific classification layers through zero-shot transfer.
- SLIP primarily adapts SimCLR view-based self-supervision and uses Vision Transformers, while other self-supervised frameworks can be substituted.
- During each forward pass, SLIP computes CLIP and self-supervised losses on the shared encoder’s embeddings and sums them into one scalar loss.
- SLIP processes more images and produces approximately 3× more activations, increasing memory use and slowing training.
4. Improved Training Procedure
The improved procedure adapts CLIP training for YFCC15M and specifies data, architecture, optimization, and evaluation choices. It uses caption-based language supervision alongside augmented image views for self-supervision.
- The procedure achieves 34.6% zero-shot transfer to ImageNet with a modified ResNet-50 on YFCC15M, exceeding the original 31.3% result.
- Experiments primarily use YFCC15M, a filtered English-only subset of YFCC100M, and also evaluate CC3M and CC12M.
- Training samples a valid caption for each image, applies global crops to the CLIP branch, and generates two MoCo v3-augmented views for self-supervision.
- The image encoders use ViT-B/16, ViT-L/16, or ViT-S/16, while the text encoder is CLIP’s smallest 38M-parameter Transformer.
- The CLIP branch projects image and caption embeddings into a 512-dim space, whereas the self-supervised branch maps image embeddings into a 256-dim output space.
- Training uses batch size 4096 with AdamW, β2 = 0.98, and distinct weight decay values of 0.5 for CLIP and 0.1 for SLIP.
- Zero-shot evaluation averages caption embeddings across prompts and compares them with image embeddings using cosine similarity.
- Linear evaluation freezes the representation and trains a final classifier, while end-to-end finetuning updates the model using task-specific training procedures.
5. Empirical Evaluations
SLIP combines language supervision with image self-supervision and is evaluated against CLIP and self-supervised baselines across ImageNet settings and additional benchmarks. It generally improves representation quality, with benefits extending across model sizes, training schedules, and pre-training datasets.
- Evaluation setup: SLIP evaluates representations using zero-shot transfer, linear classification, and end-to-end finetuning on ImageNet and additional classification benchmarks.Zero-shot transfer uses text prompts without updating model weights.
- ImageNet results: SLIP significantly improves ImageNet zero-shot transfer over CLIP across all three Vision Transformer model sizes.The improvement is approximately +5%, narrowing from +5.6% for ViT-Small to +4.8% for ViT-Large.
- ImageNet results: SLIP significantly outperforms SimCLR in ImageNet linear classification across all three model sizes.The gap is largest for ViT-Large, at almost +10%, while CLIP exceeds SimCLR by a smaller margin.
- ImageNet results: On ImageNet end-to-end finetuning, SLIP slightly exceeds SimCLR for ViT-Large, while self-supervision addresses CLIP’s weak finetuning performance.CLIP ViT-L exceeds CLIP ViT-B by only +0.5%, whereas SimCLR ViT-L gains +3.0% over CLIP ViT-L.
- Additional benchmarks: SLIP’s additional-benchmark results are strongest on datasets represented in YFCC15M, where larger models and longer training generally improve zero-shot accuracy.Some poorly matched datasets remain near chance, and low-resolution datasets show very poor zero-shot performance.
- Dataset and framework comparisons: SLIP maintains its improvement over CLIP across ImageNet settings on CC3M and CC12M, although dataset choice changes which evaluation setting benefits most.CC12M lowers zero-shot accuracy but raises linear and finetuning performance relative to YFCC15M.
6. Further Analysis
Further analysis tests whether SLIP’s gains come from joint training, data augmentation, or coupling the self-supervised and language-supervised signals.
- Jointly training CLIP and self-supervision outperforms initializing CLIP with self-supervised weights and then fine-tuning with CLIP.The initialization approach improves early progress but stalls during the 25-epoch training and underperforms SLIP across all three ImageNet evaluations.
- Additional color, blur, crop, and flip augmentations are evaluated to test whether augmentation alone explains SLIP’s improvements.
- SLIP-decoupled performs just as well as SLIP when self-supervision uses a disjoint image set sampled independently from the language-supervision images.
7. Discussion
SLIP combines language supervision with image self-supervision and improves representation quality across evaluation settings, while introducing training costs and data-related limitations.
- Results: SLIP’s gains over CLIP are consistent across zero-shot transfer, linear classification, and end-to-end finetuning evaluations.The comparison uses ImageNet and additional downstream benchmarks.
- Comparison: Initializing CLIP with self-supervised weights performs noticeably worse than SLIP across all three ImageNet evaluation settings.
- Comparison: Color and blur augmentations improve CLIP slightly, but by much less than SLIP’s improvement.
- Results: +4.8% to +5.6% zero-shot ImageNet performance gains are observed across model sizes.
- Practical limitations: SLIP’s forward pass increases activation memory and training time, taking 30.5 hours versus 22.3 hours for CLIP on ViT-B/16.
- Scope limitations: Uncurated pre-training alone appears inefficient for concepts unlikely to be widely shared on social media or the broader internet.
A. Additional Implementation Details
The implementation details describe dataset preprocessing, dataset-specific filtering, training schedules, and finetuning settings used for the experiments.
- Datasets: YFCC15M captions and titles are lightly preprocessed by unescaping HTML and removing tags and URLs.
- Datasets: CC3M filters 99.9% of five billion candidate images using image and text heuristics, while CC12M relaxes many filters to collect a larger, noisier dataset.
- Pre-training: YFCC15M is trained for 25 epochs, CC12M for 35 epochs, and CC3M for 40 epochs to reduce overfitting on its smaller dataset.
- End-to-end Finetuning: ImageNet finetuning uses AdamW, batch size 1024, learning rate 4e-3, weight decay 0.05, augmentation, and regularization.
B. Full Scaling Results
Scaling generally benefits SLIP across model sizes and training durations, but longer training can reduce ViT-L linear-classification performance while improving other evaluations.
- Full Scaling Results: Larger models and longer training generally improve ImageNet performance across zero-shot transfer, linear classification, and end-to-end finetuning.
- Full Scaling Results: SLIP ViT-L/16 linear-classification performance degrades slightly with longer training and also worsens on average across other downstream benchmarks.
- Full Scaling Results: Zero-shot transfer and end-to-end finetuning improve with longer training, so the ViT-L behavior is not simple overfitting across evaluation settings.
C. Additional Linear Classification Benchmarks
Additional benchmark results show that SLIP generally outperforms CLIP, while transfer quality depends strongly on dataset overlap with pre-training data and task requirements.
- Dataset dependence: Linear classification can massively improve accuracy on tasks poorly represented in YFCC100M, often from near-chance performance.
- Dataset dependence: Datasets sharing more overlap with YFCC100M, including Food-101, Caltech-101, and Caltech-UCSD Birds 2011, also show significant improvements.
- Limitations: HatefulMemes and Rendered SST2 remain around chance for all models because they require OCR capabilities.
- Scaling behavior: Relative rankings between 25- and 100-epoch models are unstable across datasets.
- Additional Linear Classification Benchmarks: SLIP outperforms CLIP on most of the 26 classification tasks, frequently by a significant margin.