PaperlayerView exact revision on arXiv

Source-linked AI summary

SoftCLIP: Softer Cross-modal Alignment Makes CLIP Stronger

Yuting Gao, Jinfeng Liu, Zihan Xu, Tong Wu Enwei Zhang, Wei Liu, Jie Yang, Ke Li, Xing Sun

arXiv:2303.17561v2cs.CVcs.AI

TL;DR

Noisy web-crawled image-text data can contain local similarities beyond paired matches, challenging strict one-to-one contrastive alignment. SoftCLIP uses intra-modal self-similarity as softened alignment targets and disentangles negatives; on zero-shot ImageNet, it improves top-1 accuracy over CLIP with CC3M and CC12M pretraining.

  • Problem

    Noisy web-crawled image-text pairs may have local similarities across unpaired samples, conflicting with contrastive learning’s strict one-to-one alignment.

    Source: §1 · p. 1

  • Method

    SoftCLIP uses fine-grained intra-modal self-similarities as softened cross-modal targets and disentangles negative relations in the prediction and target distributions.

    Source: §1 · p. 2

  • Results

    6.8%/7.2% higher top-1 accuracy than CLIP on zero-shot ImageNet with CC3M/CC12M pretraining, respectively.

    Source: §1 · p. 2

  • Takeaways & Limitations

    SoftCLIP models commonly occurring many-to-many relationships in noisy web-crawled image-text datasets.

    Source: §5 · p. 9

Abstract

from arXiv · show

During the preceding biennium, vision-language pre-training has achieved noteworthy success on several downstream tasks. Nevertheless, acquiring high-quality image-text pairs, where the pairs are entirely exclusive of each other, remains a challenging task, and noise exists in the commonly used datasets. To address this issue, we propose SoftCLIP, a novel approach that relaxes the strict one-to-one constraint and achieves a soft cross-modal alignment by introducing a softened target, which is generated from the fine-grained intra-modal self-similarity. The intra-modal guidance is indicative to enable two pairs have some local similarities and model many-to-many relationships between the two modalities. Besides, since the positive still dominates in the softened target distribution, we disentangle the negatives in the distribution to further boost the relation alignment with the negatives in the cross-modal learning. Extensive experiments demonstrate the effectiveness of SoftCLIP. In particular, on ImageNet zero-shot classification task, using CC3M/CC12M as pre-training dataset, SoftCLIP brings a top-1 accuracy improvement of 6.8%/7.2% over the CLIP baseline.

1. Introduction

Image-text contrastive learning typically brings paired samples closer and pushes unpaired ones away, although noisy web-crawled data can contain local similarities across unpaired pairs. SoftCLIP uses intra-modal self-similarity to soften alignment, disentangles negative relations, and reports higher zero-shot ImageNet accuracy than CLIP.

1. Introduction

  • Noisy web-crawled pairs can have local similarities, making it unreasonable to push every unpaired image-text pair completely apart.Equal weighting of all negatives ignores their differing relationships.Source: §1 · p. 1
  • SoftCLIP uses fine-grained intra-modal self-similarities as softened targets to relax one-to-one alignment and model many-to-many relationships.This allows some similarity between unpaired images and texts.Source: §1 · p. 2
  • SoftCLIP disentangles negative logits from positive logits in both prediction and target distributions, then renormalizes and aligns the resulting distributions.This is intended to strengthen relation alignment with negatives when the positive dominates the softened target.Source: §1 · p. 2
  • 6.8%/7.2% higher top-1 accuracy than CLIP on zero-shot ImageNet with CC3M/CC12M pretraining, respectively.SoftCLIP reached 24.2%/43.2% top-1 accuracy in those settings.Source: §1 · p. 2
  • SoftCLIP also replaces conventional cross-entropy with symmetric KL-Divergence and reports steady improvements across pretraining scales and model architectures.The introduction reports experiments across several downstream tasks.Source: §1 · p. 2

2. Related Work

Related work places SoftCLIP in the dual-stream vision-language pretraining and softened-target literature. It contrasts its intra-modal guidance with prior approaches to alignment and label smoothing.

2.1. Vision Language Pre-training

  • Dual-stream models align modalities, single-stream models fuse them, and hybrid models combine these structures; this work adopts the dual-stream architecture.The section situates CLIP and related methods within this landscape and distinguishes SoftCLIP’s use of intra-modal self-similarities from one-hot labels.Source: §2.1 · p. 2

2.2. Softened Target

  • Prior softened-target approaches include label smoothing and teacher-logit distillation, while PyramidCLIP uses label smoothing for image-text alignment.The section argues that treating all negative samples indiscriminately is unreasonable.Source: §2.2 · p. 2
  • SoftCLIP is described as working under self-distillation, with softened targets derived from pre-extracted object ROI features and corresponding tags rather than images and texts.Source: §2.2 · p. 3

3. Methodology

SoftCLIP replaces strict pair-only supervision with intra-modal guidance from ROI and tag similarities, then adds negative-focused alignment and a symmetric KL training objective. This methodology is built on CLIP’s dual-stream contrastive setup.

3.1. CLIP Preliminaries and Label Smoothing

  • CLIP aligns dual-stream image and text embeddings with InfoNCE, while label smoothing softens its one-hot targets.The smoothing hyper-parameter α is set to 0.2.Source: §3.1 · p. 3

3.2. Soft Alignment under Intra-modal Guidance

  • ROI and tag self-similarities guide cross-modal targets, blended with hard labels and trained with KL-Divergence instead of cross-entropy.The mixing coefficient β is set to 0.3.Source: §3.2 · pp. 3–4

3.3. Boosting Relation Alignment with Negatives

  • SoftCLIP discards positive logits and renormalizes negatives to strengthen negative-relation alignment, addressing positive dominance and faulty pairs.The text identifies faulty positives as a case in which the issue becomes more serious.Source: §3.3 · pp. 4–5

3.4. Training Objective

  • The training objective adds a reversed KL term to symmetrize KL, then uses soft and relation-enhanced losses to regulate the original CLIP loss.The loss weights λ and µ are set to 1.0 and 0.5, respectively.Source: §3.4 · p. 5

4. Experiments

SoftCLIP is evaluated across zero-shot classification, image retrieval, image-text retrieval, and ablations. It improves on CLIP in the reported comparisons and distributes retrieval probability beyond the top match.

4.1. Pre-training and Evaluation Details

  • Experiments use ResNet50, ViT-B/32, or ViT-B/16 encoders and CC3M, CC12M, or YFCC15M-V2, evaluating classification and retrieval tasks.Source: §4.1 · p. 5

4.2. Zero-shot Image Classification

4.3. Image Retrieval

4.4. Zero-shot Image-text Retrieval

  • 7.2% and 5.9% top-1 hit accuracy improvement on Flickr30K ResNet50 image-to-text and text-to-image retrieval, respectively; SoftCLIP also exceeds DeCLIP on the same dataset.Source: §4.4 · p. 6, Table 5 · p. 7

4.5. Ablation Study

  • Adding naive soft loss to CLIP significantly improves performance; relation-enhanced soft loss and symmetric KL-Divergence further improve it.Source: Table 6 · p. 7, §4.5 · p. 7
  • ROI/tag guidance performs better than image/text guidance; performance peaks with β between 0.1 and 0.5, declines above 0.8, and drops sharply below 0.2 without relation-enhanced loss.Source: §4.5 · pp. 7–8, Figure 4 · p. 7

4.6. Visualization

  • Retrieval examples show SoftCLIP matching captions beyond narrow terms and localizing nouns such as “cars” and “boat” more precisely than CLIP.Source: Figure 5 · p. 8, §4.6 · p. 8

4.7. Analysis

  • On Flickr30K, CLIP's top-1 logit dominates, whereas SoftCLIP spreads probability across other ranked matches, allowing a soft cross-modal relation.Source: Figure 6 · p. 8, §4.7 · p. 8

5. Conclusions

SoftCLIP relaxes strict one-to-one alignment by using intra-modal self-similarity as a softened target and disentangling negatives. The authors report effectiveness across several tasks and its ability to model many-to-many relationships in noisy web-crawled image-text datasets.

  • SoftCLIP relaxes the one-to-one constraint through intra-modal self-similarity targets and disentangled negatives, modeling many-to-many relationships in noisy web-crawled datasets.Source: §5 · p. 9

A.1. Pre-training Datasets

The appendix describes the three pre-training datasets and distinguishes their filtering and collection processes, including differences between YFCC15M versions.

A.1. Pre-training Datasets

  • CC3M and CC12M use the same web-source filtering pipeline, but CC12M applies a more relaxed filter and has fewer valid URLs than advertised.Source: §A.1 · p. 9
  • YFCC15M-V2 uses DeCLIP's different filtering strategy, adds internet-crawled data, and is described as higher quality than YFCC15M-V1.Source: §A.1 · p. 9
  • SoftCLIP uses CC3M, CC12M, and YFCC15M-V2 for pre-training.Source: Table 7 · p. 9, §A.1 · p. 9

A.2. Downstream Zero-shot Classification Datasets

The appendix specifies the zero-shot classification transfer benchmarks and provides additional retrieval examples comparing SoftCLIP with CLIP.

A.2. Downstream Zero-shot Classification Datasets

  • Transfer evaluation covers ImageNet and six other datasets, using CLIP's data split and evaluation metric for comparison.Source: Table 8 · p. 9, §A.2 · p. 9

B. More Visualizations

  • Additional MS-COCO retrieval examples indicate that SoftCLIP retrieves images matching query text better than CLIP.Source: B. More Visualizations · p. 9

C. More Ablation

SoftCLIP’s performance is largely insensitive to the tested supervision form: swapping the intra-modal similarities causes no degradation, while cross-modal similarities cause only a slight drop.

C.1. Different Supervision Forms

  • Swapping R2R and A2A does not degrade performance.SoftCLIP uses R2R for the vision-to-language loss and A2A for the language-to-vision loss.Source: §C.1 · p. 9
  • Replacing R2R and A2A with R2A and A2R causes only a slight performance drop.Source: §C.1 · p. 9
  • The authors conclude that introducing ROI features and corresponding tags matters more than the particular supervision form.Source: §C.1 · p. 9

C.2. Varying the Training Data Size

On YFCC15M-V2, SoftCLIP maintains an improvement over CLIP across the tested training-data sizes, which the authors take as evidence of effectiveness and robustness.

C.2. Varying the Training Data Size

  • SoftCLIP maintains a significant improvement over CLIP across all tested training-data sizes.Source: §C.2 · p. 9
  • The ResNet50-image-encoder sweep randomly sampled subsets of 2.5, 5, 7.5, 10, 12.5, and 15M examples from YFCC15M-V2.Source: §C.2 · p. 9
  • The authors interpret the improvement across sizes as indicating SoftCLIP’s effectiveness and robustness.Source: §C.2 · p. 9

C.3. Aggregation of ROI Features

SoftCLIP aggregates ROI features through MHSA layers to obtain the image representation; the reported comparison finds some gains over raw-ROI aggregation.

C.3. Aggregation of ROI Features

  • In SoftCLIP, the ROI feature sequence enters the rear image-encoder portion, where MHSA layers aggregate it into a final representation vector through an additional cls token.Source: §C.3 · p. 9
  • MHSA-processed ROI features bring some gains compared with raw ROI features.Source: §C.3 · p. 10
  • The raw-ROI alternatives aggregate each representation-vector element using the mean, maximum, or minimum across the ROI feature sequence.Source: §C.3 · p. 10

Related papers

Loading 2303.17561v2…