PaperlayerView exact revision on arXiv

Source-linked AI summary

CyCLIP: Cyclic Contrastive Language-Image Pretraining

Shashank Goel, Hritik Bansal, Sumit Bhatia, Ryan A. Rossi, Vishwa Vinay, Aditya Grover

arXiv:2205.14459v2cs.CVcs.LG

TL;DR

Standard contrastive learning can produce image and text representations whose downstream predictions are inconsistent because their overall geometry is unconstrained. CyCLIP adds in-modal and cross-modal consistency constraints, and it improves over CLIP across zero-shot classification and natural distribution-shift benchmarks. The study's current scope does not establish whether these benefits persist at CLIP's 400M-pair training scale.

  • Problem

    Standard contrastive learning aligns matched image-text pairs but leaves the geometry of mismatched and within-modality pairs unconstrained, potentially producing inconsistent predictions across image and text spaces.

  • Method

    CyCLIP augments contrastive learning with in-modal and cross-modal cycle-consistency constraints that impose additional geometric structure on image-text representations.

  • Results

    CYCLIP improves over CLIP on zero-shot classification and natural distribution-shift robustness, including a 10.2% improvement on ImageNet1K zero-shot classification.

  • Takeaways & Limitations

    Improved representation consistency translates into stronger benchmark performance and better captures coarse- and fine-grained dataset concept hierarchies.

  • Takeaways & Limitations

    The study does not establish whether cycle-consistency benefits persist when scaling to CLIP's original 400M image-text-pair training scale.

Abstract

from arXiv · show

Recent advances in contrastive representation learning over paired image-text data have led to models such as CLIP that achieve state-of-the-art performance for zero-shot classification and distributional robustness. Such models typically require joint reasoning in the image and text representation spaces for downstream inference tasks. Contrary to prior beliefs, we demonstrate that the image and text representations learned via a standard contrastive objective are not interchangeable and can lead to inconsistent downstream predictions. To mitigate this issue, we formalize consistency and propose CyCLIP, a framework for contrastive representation learning that explicitly optimizes for the learned representations to be geometrically consistent in the image and text space. In particular, we show that consistent representations can be learned by explicitly symmetrizing (a) the similarity between the two mismatched image-text pairs (cross-modal consistency); and (b) the similarity between the image-image pair and the text-text pair (in-modal consistency). Empirically, we show that the improved consistency in CyCLIP translates to significant gains over CLIP, with gains ranging from 10%-24% for zero-shot classification accuracy on standard benchmarks (CIFAR-10, CIFAR-100, ImageNet1K) and 10%-27% for robustness to various natural distribution shifts. The code is available at https://github.com/goel-shashank/CyCLIP.

1 Introduction

Vision-language contrastive models learn useful image-text representations, but standard contrastive training can leave their geometry inconsistent across modalities. CyCLIP addresses this issue with cycle-consistency constraints and improves zero-shot classification and robustness over CLIP.

  • Vision-language pretraining learns joint image and text embeddings from paired web data, supporting zero-shot classification and robustness to natural distribution shifts.
  • Standard contrastive learning aligns matched image-text pairs but does not constrain mismatched pairs or relationships within each modality.
  • Irregular pairwise geometry can produce inconsistent downstream predictions between image-space and text-space reasoning.
  • CyCLIP adds in-modal and cross-modal symmetrization terms to impose geometric structure on representations.
  • 10.2% on ImageNet1K, 10.6% on CIFAR-10, and 23.9% on CIFAR-100 are CYCLIP's reported improvements over CLIP in zero-shot classification.
  • CYCLIP also achieves significant improvements over CLIP on robustness benchmarks and analyzes consistency through dataset concept hierarchies.

2 Cycle Consistent Representation Learning

CLIP learns aligned image-text embeddings and uses them for zero-shot inference, but its contrastive objective leaves broader geometry unconstrained, allowing inconsistent predictions across image and text spaces. CYCLIP addresses this by adding cross-modal and in-modal consistency regularizers that symmetrize these relationships.

  • 2.1 Preliminaries: CLIP combines image and text encoders with a contrastive objective that pulls matched pairs together and pushes unmatched pairs apart.The encoders map images and captions into a shared d-dimensional embedding space.
  • 2.1 Preliminaries: Zero-shot classification converts each category into a caption and predicts the category whose caption has the highest similarity to the test image.This inference procedure relies on the relationship between image and text representations.
  • 2.2 Inconsistent Representation Learning in CLIP: The standard contrastive objective constrains matched pairs but leaves mismatched pairs and within-modality geometry largely unrestricted.Consequently, image-space and text-space inference can produce different predicted labels.
  • 2.2 Inconsistent Representation Learning in CLIP: A consistency score measures synchrony between predicted labels in the image and text spaces.For k = 1, CLIP’s score was 44% on CIFAR-10, 16% on CIFAR-100, and 16% on ImageNet1K.
  • 2.3 Cycle Consistent Representation Learning via CYCLIP: CYCLIP augments CLIP with cross-modal and in-modal consistency regularizers that impose additional geometric structure on learned representations.The cross-modal term symmetrizes similarities between mismatched image-text pairs, while the in-modal term aligns image-image and corresponding text-text similarities.

3 Experiments

Experiments compare CLIP and CYCLIP on zero-shot classification, natural distribution shifts, and linear probing. CYCLIP improves substantially on zero-shot and shifted-distribution benchmarks, while showing smaller gains with added in-domain supervision.

  • 3.1 Zero-Shot Transfer: CYCLIP is pretrained on Conceptual Captions 3M and evaluated with prompt-based zero-shot classification on CIFAR-10, CIFAR-100, and ImageNet1K.Class-name prompts are encoded into normalized text embeddings and compared with image embeddings.
  • 3.1 Zero-Shot Transfer: 10% - 24% gains: CYCLIP outperforms CLIP across all datasets and TopK metrics for K=1.The reported comparison covers CIFAR-10, CIFAR-100, and ImageNet1K.
  • 3.2 Robustness to Natural Distribution Shifts: 10 - 27% improvement: CYCLIP outperforms CLIP on all evaluated natural distribution shift datasets.The evaluation includes ImageNetV2, ImageNetSketch, ImageNet-A, and ImageNet-R, where both models generally decline relative to ImageNet1K.
  • 3.3 Linear Probing: With linear probing, CYCLIP marginally outperforms CLIP on 9 out of 14 visual datasets.Both models recover most performance lost due to inconsistency when given extra in-domain and in-modality supervision.

4 Analysis

The analysis measures consistency between image- and text-space predictions, examines hierarchical classification, alignment and uniformity, retrieval, and robustness across training-data sizes. CYCLIP’s geometric constraints generally improve consistency and downstream performance, with especially large gains in coarse-grained classification.

  • 4.1 Consistency in Image and Text Spaces: CYCLIP is more consistent than CLIP across CIFAR-10, CIFAR-100, and ImageNet1K, supporting more interchangeable image and text representations.Consistency compares zero-shot text-space predictions with k-nearest-neighbor predictions in image space.
  • 4.2 Fine-grained and Coarse-grained Performance: CYCLIP’s advantage is larger for coarse-grained than fine-grained classification across datasets.The authors associate this pattern with better concept-level knowledge and less rigid separation of semantically similar negatives.
  • 4.3 Alignment and Uniformity on the Unit Hypersphere: CYCLIP balances alignment and uniformity, while I-CYCLIP has better alignment but poorer hypersphere coverage and C-CYCLIP is more uniform but poorly aligned.All three CYCLIP variants outperform CLIP on the three datasets, with CYCLIP best on ImageNet1K.
  • Cross-modal Retrieval: CYCLIP marginally outperforms CLIP on zero-shot image retrieval across Flickr30K and MSCOCO.The passage attributes the relatively low zero-shot retrieval performance of both models to the datasets’ multi-resolution image-text matching demands.
  • Effective Robustness: Across training sizes from 500K to 4M image-text pairs, CYCLIP significantly improves over CLIP and remains above the baseline robustness trend.The result holds across the evaluated data regimes for ImageNet1K accuracy and effective robustness on ImageNetV2.

5 Related Work

The work is situated within multimodal representation learning, especially contrastive and generative approaches. It extends contrastive learning with consistency regularizers applied to image-text embeddings rather than generative mappings.

  • Multimodal Pretraining: Multimodal pretraining learns representations across images, text, speech, and other modalities using contrastive or generative approaches.The paper places its contribution within broader unsupervised multimodal pretraining.
  • Contrastive Representation Learning: Contrastive representation learning brings augmented views closer and pushes representations of other samples apart, with later work adding redundancy-reduction constraints.This describes the unimodal origins and extensions of the contrastive framework.
  • Contrastive Language-Image Pretraining: CLIP, ALIGN, and BASIC extend contrastive learning to paired image-text data for zero-shot classification and robustness.The paper presents CYCLIP as complementary work that identifies consistency regularizers for this literature.
  • Generative Representation Learning: Prior cyclic-consistency work regularizes generative mappings between modalities, whereas CYCLIP applies cycle consistency to embeddings learned through contrastive training.The distinction separates CYCLIP from GAN- and normalizing-flow-based approaches.

6 Conclusion

CYCLIP uses cross-modal and in-modal consistency regularizers to improve alignment between image and text representation spaces. It outperforms CLIP on zero-shot classification and distributional-robustness benchmarks, while scaling to larger datasets remains an open limitation.

  • 6 Conclusion: CYCLIP prevents inconsistent inference by including cross-modal and in-modal consistency regularizers in multimodal representation learning.These regularizers encourage geometric consistency between image and text spaces.
  • 6 Conclusion: CYCLIP performs much better than CLIP on zero-shot classification and is more robust on distributional-robustness benchmarks.The representations also better capture concept-level knowledge in fine-grained and coarse-grained analyses.
  • 6 Conclusion: Further study is needed to determine whether cycle-consistency benefits remain when CYCLIP scales to the 400M image-text pairs used to train original CLIP.The authors identify scaling to larger datasets as a current limitation and future direction.

A Additional Results

The appendix evaluates CYCLIP variants that isolate cross-modal or in-modal consistency regularization, providing ablations of the full objective.

  • A Additional Results: C-CYCLIP retains only the cross-modal consistency regularizer, whereas I-CYCLIP retains only the in-modal consistency regularizer.C-CYCLIP sets λ1 = 0 and λ2 = 0.5; I-CYCLIP sets λ1 = 0.5 and λ2 = 0.

A.1 Zero-shot Transfer

Across zero-shot transfer, robustness, linear probing, and consistency analyses, CYCLIP variants generally outperform CLIP and improve agreement between image- and text-space predictions. The appendix also examines coarse- and fine-grained performance across ImageNet-family benchmarks.

  • A.1 Zero-shot Transfer: All CYCLIP variants outperform CLIP across the zero-shot datasets, while C-CYCLIP performs best on CIFAR-10 and CIFAR-100.CYCLIP outperforms its sub-variants and CLIP on ImageNet1K.
  • A.1 Zero-shot Transfer: All CYCLIP models outperform CLIP by a large margin across the natural distribution-shift datasets.C-CYCLIP performs best on three of the four distribution-shift datasets, while I-CYCLIP performs worst across all of them.
  • A.1 Zero-shot Transfer: CYCLIP models marginally outperform CLIP on all 14 linear-probing datasets except Flowers102 and OxfordIIITPet.The evaluation covers 14 visual classification datasets.
  • A.1 Zero-shot Transfer: All CYCLIP variants are more consistent than CLIP across the evaluated datasets, with C-CYCLIP identified as the most consistent model.The consistency score measures agreement between zero-shot text-space predictions and image-space k-nearest-neighbor predictions.
  • A.1 Zero-shot Transfer: CYCLIP variants outperform CLIP on coarse-grained and fine-grained classification, with larger improvement margins for coarse-grained analysis.The analysis spans ImageNet1K/V2/Sketch, ImageNet-A/R, and CIFAR-100 superclass structures.

A.6 Zero-shot Transfer: Additional Datasets

On additional visual datasets and cleaner test sets, CYCLIP improves zero-shot Top-1 classification over CLIP, while the accompanying figure highlights stronger gains in coarse-grained performance.

  • A.6 Zero-shot Transfer: Additional Datasets: 10.6% average gain in Top-1 classification accuracy is achieved by CYCLIP over CLIP across the additional datasets.The evaluation includes cleaner test sets for CIFAR-10, CIFAR-100, and ImageNet1K.
  • A.6 Zero-shot Transfer: Additional Datasets: The performance gap between CLIP and CYCLIP variants is larger in coarse-grained than fine-grained analysis.Figure 6 compares CLIP and CYCLIP across traditional datasets and natural-distribution-shift variants.

B Additional Discussion

The discussion highlights unresolved concerns about applying geometric consistency to noisy image-text data and emphasizes that its usefulness depends on the downstream task. It also notes that soft consistency regularization helps across settings, with gains varying by task and domain.

  • Noisy web captions may not fully describe their images, leaving unclear how strongly such distinctions should appear in the learned representations.
  • Task dependence creates a trade-off: bringing captions closer may harm distinctions needed for cat-versus-dog classification.
  • CLIP weights every image-text pair equally even though some captions describe their images more fully than others.
  • Soft consistency regularization generally helps downstream tasks, but relative gains differ across domains and are higher for coarse-grained than fine-grained classification.

C Pretraining and Implementation details

The implementation uses web-scraped image-caption data and CLIP’s original encoder architecture. Training uses specified regularization, optimization, batch-size, epoch, and tuning choices, with hyperparameters documented in a table.

  • Pretraining uses Conceptual Captions 3M, retaining 2,631,703 image-caption pairs after removing broken image URLs.
  • The models retain CLIP’s architecture: a ResNet-50 image encoder and transformer-based text encoder projecting to 1024-dimensional image-matched embeddings.
  • Training uses batch size 128 for 64 epochs, with cyclic-consistency weights selected through grid search and manual tuning on 480K pairs.
  • Table 13 lists the hyperparameters used for training the CLIP models.
Loading 2205.14459v2…