PaperlayerView exact revision on arXiv

Source-linked AI summary

Improved baselines for vision-language pre-training

Enrico Fini, Pietro Astolfi, Adriana Romero-Soriano, Jakob Verbeek, Michal Drozdzal

arXiv:2305.08675v2cs.CV

TL;DR

The paper asks how much reported gains from non-contrastive vision-language losses reflect the losses themselves rather than training-recipe changes. It adapts visual self-supervised losses to image-text learning and evaluates them alongside an improved CLIP recipe. The improved recipe strengthens CLIP, which outperforms prior methods across four datasets, by up to 4% on the largest.

  • Problem

    Because prior vision-language work often combined modified objectives with advanced training techniques, their separate contributions are difficult to evaluate.

    Source: §1 · p. 1

  • Method

    The paper adapts four visual self-supervised non-contrastive losses for vision-language pre-training and evaluates them alongside a CLIP recipe with stronger augmentations, dropout, label smoothing, and projector changes.

    Source: §1 · p. 2, §3 · p. 5

  • Results

    CLIP 🚀outperforms methods with additional or modified losses across four datasets, by up to 4% over the best related method on the largest dataset.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Under the tested conditions, a well-tuned CLIP implementation performs better than the more complex loss extensions studied.

    Source: §1 · p. 2, §7 · p. 14

Abstract

from arXiv · show

Contrastive learning has emerged as an efficient framework to learn multimodal representations. CLIP, a seminal work in this area, achieved impressive results by training on paired image-text data using the contrastive loss. Recent work claims improvements over CLIP using additional non-contrastive losses inspired from self-supervised learning. However, it is sometimes hard to disentangle the contribution of these additional losses from other implementation details, e.g., data augmentation or regularization techniques, used to train the model. To shed light on this matter, in this paper, we first propose, implement and evaluate several baselines obtained by combining contrastive learning with recent advances in self-supervised learning. In particular, we use the loss functions that were proven successful for visual self-supervised learning to align image and text modalities. We find that these baselines outperform a basic implementation of CLIP. However, when a stronger training recipe is employed, the advantage disappears. Indeed, we find that a simple CLIP baseline can also be improved substantially, up to a 25% relative improvement on downstream zero-shot tasks, by using well-known training techniques that are popular in other subfields. Moreover, we discover that it is enough to apply image and text augmentations to make up for most of the improvement attained by prior works. With our improved training recipe for CLIP, we obtain state-of-the-art performance on four standard datasets, and consistently outperform prior work (up to +4% on the largest dataset), while being substantially simpler. The code is available at https://github.com/facebookresearch/clip-rocket

1 Introduction

The paper investigates whether non-contrastive losses improve vision-language pre-training beyond CLIP, while addressing the difficulty of separating objective changes from training-recipe changes. It finds that a stronger recipe substantially improves CLIP and outperforms more complex extensions.

1 Introduction

  • CLIP’s standard recipe omits common augmentations, label smoothing, and projector improvements, alongside limitations tied to contrastive learning.The paper identifies high computational cost, sensitivity to batch size and composition, and limited fine-grained alignment or cross-modality fusion.Source: §1 · p. 1
  • The study addresses a comparison problem: prior work often changed objectives and training techniques together, making their separate contributions difficult to evaluate.Existing non-contrastive VLP methods also had not faithfully translated or broadly explored successful visual SSL losses.Source: §1 · p. 2
  • The four non-contrastive baselines improve on standard CLIP by 1–2% on ImageNet.The paper tests loss functions adapted from visual self-supervised learning, both on their own and in composition with CLIP.Source: §1 · p. 2
  • Up to 11% on ImageNet (∼25% relative boost): the improved CLIP recipe raises zero-shot performance substantially.The recipe combines image and text augmentations, dropout, label smoothing, and projector design choices.Source: §1 · p. 2
  • On four datasets, CLIP 🚀outperforms methods with additional or modified losses, by up to 4% over the best related method on the largest dataset.The pre-training datasets range from 3M to 29M image-text pairs.Source: §1 · p. 2

2 Background

This section introduces paired image-text representation learning and contrasts CLIP’s batch-based contrastive objective with non-contrastive approaches. It describes consistency, redundancy-reduction, and clustering mechanisms used in visual self-supervised learning.

2 Background

  • Vision-language pre-training learns multimodal representations from paired data, with an image and its augmentation or textual description representing the same underlying concept.The objective is to learn robust encoders whose representations generalize to diverse downstream tasks.Source: §2 · pp. 2–3

2.1 Contrastive learning

  • Contrastive learning brings representations of paired samples together and pushes representations from different pairs apart; CLIP applies this objective across image and text encoders.The objective uses batch elements, temperature, and cosine similarity, and can be made symmetric by computing both directions.Source: §2.1 · p. 3
  • Contrastive learning can require large batches to diversify negatives, increasing memory use as the similarity matrix scales quadratically with batch size.Semantically similar samples may also be pushed apart; the text notes this issue may be mitigated on broad data but remains relevant for small datasets and can occur in large datasets with redundancy.Source: §2.1 · p. 3

2.2 Non-contrastive learning

  • Non-contrastive methods avoid negative samples by enforcing consistency between positive pairs, reducing batch-similarity computation and often performing better with small batches.Other families replace negatives with redundancy-reduction terms or cluster prototypes.Source: §2.2 · pp. 3–4
  • Consistency-based methods such as BYOL and SimSiam compare positive samples with an MSE or negative-cosine objective, using a predictor in an asymmetric architecture.BYOL uses a momentum encoder, whereas SimSiam uses the same encoder for both branches.Source: §2.2 · p. 4
  • Barlow Twins uses cross-correlation between two views to make feature components less redundant while keeping representations of the views similar.Its loss pushes the cross-correlation matrix toward the identity matrix.Source: §2.2 · p. 4
  • Clustering-based methods compare assignments for different views with cross-entropy; SwAV uses shared prototypes and Sinkhorn-Knopp assignments that encourage roughly equal cluster representation in a batch.In SwAV, the encoders for both modalities share the learned prototype set.Source: §2.2 · p. 4

3 Improved training recipe

CLIP 🚀 retains CLIP’s contrastive objective while adding stronger image and text augmentations, separate projectors for weak and strong views, and regularization. At inference, it averages similarity scores from the two projectors.

3 Improved training recipe

  • Strong image augmentations use wider-range random crops followed by randomly combined color changes, greyscaling, Gaussian blur, and horizontal flips.These create additional views of each input image.Source: §3 · p. 5
  • Text views use stop-word removal; strong views also sample a caption when multiple are available and apply synonym replacement, word insertion or deletion, or sentence shuffling.When only one caption is available, the same caption is used for each augmentation.Source: §3 · p. 5
  • Separate two-layer MLP projectors serve strong views, while linear projectors serve weak views.Each strong projector has two linear layers separated by BatchNorm and ReLU.Source: §3 · p. 5
  • Label smoothing softens the contrastive targets for strong views, but is not applied to weak-view loss.The recipe pairs strong augmentations and deep projectors with smoothing to discourage overconfident predictions.Source: §3 · p. 5
  • CLIP 🚀 trains on weak and strong image-text views, applies CLIP loss to each corresponding pair of views, and averages both projectors’ similarity scores at inference.The recipe also applies dropout to the text encoder and label smoothing to the strong-view loss.Source: §3 · p. 6

4 Non-contrastive baselines for vision-language pre-training

The paper combines CLIP’s contrastive loss with cross-modal non-contrastive objectives to create four vision-language baselines. The variants use consistency, redundancy reduction, or clustering criteria, with a modified SwALIP formulation used in most experiments.

4 Non-contrastive baselines for vision-language pre-training

  • The proposed baselines combine CLIP loss with a non-contrastive auxiliary loss, and the experiments found equal weights for the two losses worked best.Setting the contrastive weight to 1 and auxiliary weight to 0 gives CLIP alone; the reverse gives non-contrastive multimodal learning.Source: §4 · p. 6
  • All four baselines use multilayer projectors unless otherwise specified.Figure 2 summarizes SiamLIP, BYOLIP, BarLIP, and SwALIP, but omits the combined CLIP loss and projectors; BYOLIP’s momentum encoder is also omitted.Source: §4 · p. 6, Figure 2 · p. 7
  • SiamLIP and BYOLIP add bidirectional cross-modal consistency objectives; BYOLIP additionally uses momentum encoders for image and text.Both use two projectors and two predictors for the asymmetric image-to-text and text-to-image objectives.Source: §4 · p. 6
  • BarLIP adds a Barlow Twins-inspired loss that aligns image and text features while decorrelating feature components.The authors suggest this may help with noisy and duplicate-rich vision-language datasets.Source: §4 · p. 7
  • SwALIP compares cross-modal cluster assignments in both directions, but its original prototype-based formulation proved unstable in practice.Most experiments use a modified version with text representations as prototypes; it requires multiple views and cannot otherwise be used.Source: §4 · p. 7

5 Experiments

The experiments test whether non-contrastive losses or a stronger CLIP training recipe drive zero-shot performance. Across backbone, dataset, and benchmark comparisons, the recipe improves CLIP and often outperforms prior methods; ablations examine its components and cost.

5.1 Datasets and evaluation

  • Evaluation covers zero-shot ImageNet classification and 22 additional vision benchmarks, using pre-training datasets ranging from 3M to 29M image-text pairs, plus PMD46M for scaling experiments.Zero-shot classification selects the class whose text representation has the highest cosine similarity to the image representation.Source: §5.1 · pp. 7–8

5.2 Implementation details

  • The recipe trains on one weak and two strong image and text augmentations, with separate strong-view projectors; optimization uses AdamW, 4096-pair batches, and label smoothing of 0.1.The text encoder's dropout varies by dataset size, and the text augmentation includes stop-word removal and caption sampling.Source: §5.2 · p. 8

5.3.1 Comparison with baselines

  • With the base recipe, non-contrastive variants beat CLIP by 0.5%–2.4% with ViT-S/16, but not with ResNet-50; with the improved recipe, CLIP 🚀 scores 43.1% and 43.2%, ahead of those variants.The improved recipe raises original CLIP by 10.7% with ViT-S and 10.5% with ResNet-50; results vary somewhat across CC3M, CC12M, and YFCC15M.Source: §5.3.1 · p. 9

5.3.2 Comparison with the state-of-the-art

  • CLIP 🚀 consistently exceeds prior methods with the same architecture across datasets; it leads ViT-B/32 comparisons on YFCC15M by +0.6% but trails on Open29M by -0.9%.Table 2 compares zero-shot ImageNet accuracy, using ResNet-50 unless otherwise specified.Source: §5.3.2 · p. 10, Table 2 · p. 10
  • On Open29M, CLIP 🚀 leads DeCLIP by +4.0%; on PMD46M it beats the CLIP baseline by 5.8% and FLAVA by 3.3—5.3%, matching open-clip trained on roughly 10× more data and samples seen.The scaling results combine the Open29M ImageNet trend with PMD46M linear evaluation.Source: §5.3.2 · p. 10, Figure 3 · p. 10, Table 3 · p. 11
  • With ViT-{S,B,L}/16, CLIP 🚀 improves over original CLIP by 8.8%, 7.4%, and 9.5%, respectively, and outperforms SLIP across the reported training-duration comparisons.The reported comparisons use YFCC15M pre-training; the recipe includes multiple views with separate projectors.Source: §5.3.2 · p. 10, Figure 4 · p. 11
  • Across 23 vision benchmarks, CLIP 🚀 improves on baseline CLIP in 18/23, 20/23, and 18/23 benchmarks for ViT-{S,B,L}, respectively, after 25 epochs.The average-performance improvement holds across the reported settings; CLIP 🚀 also leads SLIP on 13 to 16 benchmarks.Source: §5.3.2 · p. 11, Table 4 · p. 11

5.3.3 Ablations

  • Image augmentations with multiple views add +10.7% over original CLIP, while text augmentations add +0.6%; stronger asymmetric BYOL-like image transformations do not help.Label smoothing from 0.0 to 0.1 adds +2.8%; dropout hurts on noisy YFCC15M but slightly helps on smaller, cleaner datasets.Source: §5.3.3 · p. 12

6 Related work

Prior VLP work has refined contrastive alignment and robustness, while other approaches add self-supervised losses or address noisy captions. CLIP 🚀 differs by using multiple augmented views of both images and text without unimodal losses.

6 Related work

  • CLIP and ALIGN use image-caption contrastive learning, while FILIP, PyramidCLIP, and SoftCLIP refine it with image-text part alignment.Other approaches target robustness through Hopfield networks, fused multimodal features, or image masking.Source: §6 · p. 13
  • SLIP, UniCLIP, and ViewCo use multiple image views, whereas CLIP 🚀 also uses multiple text views and omits unimodal losses.CLIP 🚀 removes caption stop-words, an approach inspired by BoW-CLIP.Source: §6 · p. 13
  • Prior non-contrastive VLP methods use unimodal or custom multimodal losses, but the paper reports that none outperform CLIP without retaining its contrastive loss.Masking approaches such as FLAVA and ViCHA are outside this work’s zero-shot-transfer focus because they usually require fine-tuning.Source: §6 · pp. 13–14

7 Conclusion

The paper finds that self-supervised-loss baselines improve on a basic CLIP implementation, but that advantage disappears with an improved recipe. The recipe yields state-of-the-art results across four datasets.

7 Conclusion

  • The SSL-inspired baselines outperform basic CLIP, but their advantage disappears when an improved training recipe is used.The recipe combines image and text augmentations, label smoothing, dropout, and deeper projector networks.Source: §7 · p. 14
  • The improved CLIP recipe achieves state-of-the-art performance on four datasets and outperforms related methods by significant margins.Source: §7 · p. 14
  • The study evaluates combining contrastive learning with advances in self-supervised and supervised learning through baselines using non-contrastive losses successful in visual SSL.Source: §7 · p. 14

A Modified SwALIP

The modified SwALIP variant replaces learned prototypes with representations from the other modality. It uses Sinkhorn-Knopp assignments and symmetric swapped-assignment losses, with additional averaging for implementation and stability.

A Modified SwALIP

  • Modified SwALIP uses representations from the other modality as prototypes instead of learned prototypes, requiring two strongly augmented views of each pair.Source: A Modified SwALIP · p. 18
  • It applies Sinkhorn-Knopp to multimodal similarities for cluster assignments, then computes swapped-assignment cross-entropy in both directions to make the loss symmetric.Assignments correspond to a correlated view, and prototypes are representations from the other modality.Source: A Modified SwALIP · p. 18
  • The implementation averages losses over both valid prototype-view choices and blends Sinkhorn-Knopp assignments with the identity matrix to stabilize training and avoid collapse.Source: A Modified SwALIP · p. 19

B Additional results

Across 22 vision datasets, results among improved non-contrastive baselines are spread, with BYOLIP only slightly ahead on average. The appendix also reports sensitivity to the contrastive and non-contrastive loss weights.

B Additional results

  • The best results are spread across the improved baselines on 22 datasets, with BYOLIP slightly better on average but no model consistently outperforming the others.Table 5 covers zero-shot transfer on ImageNet and other vision datasets, using YFCC15M pretraining for 32 epochs with ResNet-50.Source: Table 5 · p. 19, B Additional results · p. 19
  • With α fixed at 1, increasing β harms accuracy by up to 0.8%; with β fixed at 1, increasing α causes a drop of up to 3.7%.This ablation uses BarLIP 🚀, ResNet-50, YFCC15M, and 32 epochs.Source: B Additional results · p. 19
  • Using only the non-contrastive loss is problematic for zero-shot evaluation because the CLIP projector is not optimized.Source: B Additional results · p. 19

C Pseudo-code for multi-view

The pseudo-code describes CLIP training with weak and multiple strong image-text augmentations, using separate projections and contrastive losses for the views.

C Pseudo-code for multi-view

  • Each strong image augmentation is contrasted with all strong text augmentations, and vice versa.With two strong augmentations, each image view is compared against both text views.Source: C Pseudo-code for multi-view · p. 19
  • Weak views use linear projections, while strong views use normalized MLP projections for both modalities.The code defines separate vision and text projections for the weak and strong augmentations.Source: C Pseudo-code for multi-view · p. 20
  • The training step averages bidirectional cross-entropy losses across weak and strong views, then averages the image-to-text and text-to-image losses.Strong-view losses are averaged across augmentations before combination with the weak-view loss.Source: C Pseudo-code for multi-view · p. 20

Related papers

Loading 2305.08675v2…