PaperlayerView exact revision on arXiv

Source-linked AI summary

PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretraining

Yuting Gao, Jinfeng Liu, Zihan Xu, Jun Zhang, Ke Li, Rongrong Ji, Chunhua Shen

arXiv:2204.14095v2cs.CVcs.AI

TL;DR

Web-crawled image-text pairs often violate the one-to-one correspondence assumed by existing methods through semantic mismatch and compatibility among nominal negatives. PyramidCLIP addresses this with hierarchical multi-level alignment and softened negative-sample loss, achieving stronger downstream performance and improved data efficiency.

  • Problem

    Existing vision-language pre-training methods rely on web-crawled image-text pairs having perfect one-to-one correspondence, although semantic mismatch and compatibility among unpaired samples occur in practice.

  • Method

    PyramidCLIP builds visual and linguistic input pyramids and uses peer-level semantics alignment, cross-level relation alignment, and softened negative-sample loss.

  • Results

    PyramidCLIP achieves state-of-the-art results on five downstream tasks, including 47.8%/46.0%/50.7% ImageNet zero-shot top-1 accuracy with ResNet50/ViT-B32/ViT-B16 on YFCC15M-V2.

  • Takeaways & Limitations

    PyramidCLIP’s results support improved image-text alignment and higher data efficiency than CLIP across the reported downstream evaluations.

Abstract

from arXiv · show

Large-scale vision-language pre-training has achieved promising results on downstream tasks. Existing methods highly rely on the assumption that the image-text pairs crawled from the Internet are in perfect one-to-one correspondence. However, in real scenarios, this assumption can be difficult to hold: the text description, obtained by crawling the affiliated metadata of the image, often suffers from the semantic mismatch and the mutual compatibility. To address these issues, we introduce PyramidCLIP, which constructs an input pyramid with different semantic levels for each modality, and aligns visual elements and linguistic elements in the form of hierarchy via peer-level semantics alignment and cross-level relation alignment. Furthermore, we soften the loss of negative samples (unpaired samples) so as to weaken the strict constraint during the pre-training stage, thus mitigating the risk of forcing the model to distinguish compatible negative pairs. Experiments on five downstream tasks demonstrate the effectiveness of the proposed PyramidCLIP. In particular, with the same amount of 15 million pre-training image-text pairs, PyramidCLIP exceeds CLIP on ImageNet zero-shot classification top-1 accuracy by 10.6%/13.2%/10.0% with ResNet50/ViT-B32/ViT-B16 based image encoder respectively. When scaling to larger datasets, PyramidCLIP achieves the state-of-the-art results on several downstream tasks. In particular, the results of PyramidCLIP-ResNet50 trained on 143M image-text pairs surpass that of CLIP using 400M data on ImageNet zero-shot classification task, significantly improving the data efficiency of CLIP.

1 Introduction

PyramidCLIP addresses imperfect web-crawled image-text correspondence by hierarchically aligning visual and linguistic elements and softening negative-sample constraints. It reports stronger ImageNet zero-shot classification and state-of-the-art results across downstream tasks.

  • Existing CLIP-like methods assume web-crawled image-text pairs have perfect one-to-one correspondence and no correlation with unpaired samples.
  • Web-crawled pairs exhibit caption redundancy, image redundancy, and cast deficiency, creating semantic mismatch between visual and linguistic modalities.Captions may contain irrelevant information, images may extend beyond the relevant region, and text may omit salient objects.
  • Existing methods treat other pairs as negative samples despite potential compatibility, which may confuse the model.
  • PyramidCLIP constructs visual and linguistic input pyramids with global, local, and object-level semantic representations, then applies peer-level and cross-level alignment.The visual pyramid uses global images, local regions, and salient-object ROI features; the linguistic pyramid uses summaries, captions, and object descriptions.
  • PyramidCLIP achieves 47.8%/46.0%/50.7% ImageNet zero-shot top-1 accuracy with ResNet50/ViT-B32/ViT-B16 on YFCC15M-V2, versus CLIP’s 37.2%/32.8%/40.7%.
  • The method achieves state-of-the-art results on several downstream tasks, including with 143M pairs surpassing CLIP trained on 400M pairs for ImageNet zero-shot classification using ResNet50.

2 Related Work

Related work spans single-stream and dual-stream vision-language pre-training and increasingly fine-grained cross-modal alignment. PyramidCLIP follows a dual-stream design while organizing each modality into multiple independently encoded semantic levels.

  • Single-stream methods jointly model image and text, whereas dual-stream methods use separate image and text encoders.
  • Fine-grained alignment addresses irrelevant text and missing object descriptions that complicate direct image-text alignment.
  • PyramidCLIP constructs three visual and three linguistic semantic levels, encoding each level separately and computing six contrastive loss terms.

3 Method

PyramidCLIP is a dual-stream architecture that builds multi-level visual and linguistic inputs, aligning them through peer-level semantics and cross-level relations. It combines global, local, and object-level representations with corresponding textual semantics for hierarchical contrastive learning.

  • Overall Architecture: PyramidCLIP uses dual-stream image and text encoders whose outputs are projected and normalized into a shared embedding space.The image encoder is composed of front and rear parts, while both encoders end with a linear projector and normalization operator.
  • Peer-level Semantics Alignment: Global image views pair with text summaries, while local image views pair with original captions through peer-level contrastive alignment.Global views retain broad context, whereas local views focus on image sub-regions and preserve finer-grained information.
  • Overall Architecture: The method constructs visual and linguistic input pyramids with global, local, and object-level semantic representations.Visual inputs include global images, local regions, and salient-object ROI features; linguistic inputs include summaries, captions, and object-attribute descriptions.
  • Cross-level Relation Alignment: Object ROI features and category-attribute descriptions provide fine-grained supervision for modeling relations among salient image objects.ROI features are projected into the image encoder’s latent dimension and processed through its rear attention or transformer layers.
  • Cross-level Relation Alignment: Cross-level alignment minimizes distances for additional pairings between global, local, and object-level visual and linguistic representations.The additional positive pairs include (vg, la), (vr, ls), (vl, la), and (vr, lt), complementing peer-level alignment.
  • Image Encoder Design: The image encoder supports CNN and ViT backbones, with ROI sequences processed by attention pooling or the rear transformer layers.For ViT, LeFF is applied in the front part to improve patch-level local perception and interaction, but not to embedded ROI sequences.

4 Experiments

PyramidCLIP is evaluated across five downstream tasks and multiple dataset scales, with comparisons covering zero-shot classification, retrieval, transfer learning, detection, segmentation, ablations, and feature visualizations. The experiments report stronger results than CLIP and other baselines across these settings.

  • Downstream Tasks: PyramidCLIP is evaluated on five downstream tasks, including zero-shot classification, image-text retrieval, linear probing, object detection, and instance segmentation.Classification uses 11 datasets; retrieval uses Flickr30K and MS-COCO, while detection and segmentation use PASCAL VOC and MS-COCO.
  • Large-scale Datasets: 6.1%/3.8%/3.5% improvements over CLIP are reported on ImageNet classification with 143M pre-training pairs and ResNet50/ViT-B32/ViT-B16.With ResNet50, PyramidCLIP trained on 143M pairs surpasses CLIP trained on 400M pairs.
  • Small-scale Classification Datasets: Average accuracy on 10 small classification datasets exceeds CLIP trained on 400M data for both zero-shot and linear-probe tasks.The pre-training data used is less than 36% of CLIP's data.
  • Detection and Segmentation: PyramidCLIP significantly outperforms random initialization and surpasses CLIP and DECLIP on object detection and instance segmentation.The experiments initialize Faster R-CNN and Mask R-CNN backbones with the PyramidCLIP-ResNet50 visual model and fine-tune all parameters.
  • Ablation Study: Cross-level global-relation and local-relation alignment, LeFF in ViT, and softened objectives each provide significant gains beyond peer-level alignment.These component effects are measured on ImageNet zero-shot classification after eight pre-training epochs on YFCC15M-V1.

5 Conclusion

The paper concludes that PyramidCLIP improves visual-linguistic alignment by constructing pyramidal semantic inputs on both sides of a dual-stream network. It reports benefits from softened peer-level alignment and cross-level relation alignment, with state-of-the-art results across five downstream tasks.

  • Conclusion: PyramidCLIP addresses imperfect webly crawled image-text correspondence by constructing pyramidal semantic inputs for both modalities.The method uses a hierarchical pre-training design to improve alignment between visual and linguistic modalities.
  • Conclusion: Softened peer-level semantics alignment and cross-level relation alignment interact across modalities and are beneficial.The conclusion identifies both alignment mechanisms and softened objectives as contributing components.
  • Conclusion: PyramidCLIP achieves state-of-the-art results on five downstream tasks.The paper presents this outcome as evidence of the method's superiority.

A.1 Datasets

The study draws on multiple web-scale image-text datasets and evaluates transfer across 10 downstream classification datasets. It also specifies model architecture and training settings for reproducibility.

  • Datasets: The 143M pre-training corpus combines SBU, YFCC15M variants, CC3M, CC12M, and LAION400M.YFCC15M-V1 follows CLIP’s filtering rule, while YFCC15M-V2 uses DeCLIP’s strategy and includes additional Internet-crawled data.
  • Datasets: YFCC15M-V2 is described as higher quality than YFCC15M-V1 because it uses a different filtering strategy and additional crawled data.
  • Model Architectures: The architecture follows CLIP for PyramidCLIP-ResNet50, ViT-B32, and ViT-B16, with LeFF incorporated into the ViT feed-forward module.The image input resolution is 224×224, and the text encoder supports a maximum context length of 77.
  • Model Architectures: The object-attribute detector uses VinVL-pretrained Faster R-CNN and selects 10 highest-confidence objects with ROI features and attribute descriptions.Images are resized to 640×640 before detection.
  • Pre-training Setup: Training uses AdamW with cosine learning-rate decay, linear warmup over 10% of total steps, weight decay 0.2, and automatic mixed precision.Models are trained from scratch for either 8 or 32 epochs in the reported experiments.
  • Downstream Evaluation: Transferability is evaluated on 10 classification datasets beyond ImageNet, using CLIP’s data splits and evaluation metrics for fair comparison.

B.2 Implementation Details

Downstream evaluations use task-specific protocols spanning zero-shot classification, retrieval, linear probing, detection, segmentation, and end-to-end fine-tuning.

  • Downstream Zero-shot Image Classification: Zero-shot classification represents each class with 80 prompt templates and ensembles the resulting text representations for image-label similarity.The prompts include templates such as “a photo of label”.
  • Downstream Evaluation: The downstream classification datasets and their details are listed in Table 7.
  • Downstream Zero-shot Image-text Retrieval: Zero-shot image-text retrieval ranks image-text pairs by similarity on Flickr30K and MS-COCO for image and text retrieval.
  • Downstream Image Linear Probe: Linear probing trains an L-BFGS logistic regression classifier on frozen visual features, selecting regularization strength C through validation-set sweeps.The classifier uses a maximum of 1,000 iterations.
  • Downstream Object Detection and Instance Segmentation: Object detection and instance segmentation fine-tune all parameters under task-specific training schedules and image-scale settings.For PASCAL VOC detection, training lasts 24k steps and inference uses image scale 800.
  • Downstream Image End-to-end Fine-tuning: End-to-end classification fine-tuning first trains the classifier layer alone for 8 epochs while freezing the remaining parameters.

C Fair Comparison on YFCC15M-V1 and LAION15M

Under matched pre-training-data conditions on YFCC15M-V1 and LAION15M, PyramidCLIP is compared with CLIP on retrieval and ImageNet zero-shot classification.

  • Fair Comparison: PyramidCLIP surpasses CLIP on zero-shot image-text retrieval and ImageNet classification when both are trained on the same pre-training data.The comparisons use YFCC15M-V1 and LAION15M settings.
  • YFCC15M-V1: Table 8 reports MS-COCO zero-shot image-text retrieval and ImageNet top-1 accuracy after YFCC15M-V1 pre-training for 32 epochs.SLIP is the exception, using 100 epochs.
  • Fair Comparison: The comparison includes an implementation entry and a released-model evaluation marked separately in the reported results.
  • LAION15M: Table 9 reports MS-COCO zero-shot image-text retrieval and ImageNet top-1 accuracy after LAION15M pre-training for 32 epochs.
  • Downstream Dense Prediction: Table 10 provides a fair comparison of object detection and instance segmentation using ResNet50 as the backbone.

D Downstream Task: Linear Probe

Linear-probe evaluations test transferability using frozen visual features. PyramidCLIP’s 143M-data model exceeds CLIP’s 400M-data model for ViT-B/32 and outperforms CLIP on more than half of the small datasets across encoders.

  • Downstream Task: Linear Probe: PyramidCLIP pre-trained on 143M data exceeds CLIP using 400M data in average linear-probe accuracy with a ViT-B/32 image encoder.
  • Downstream Task: Linear Probe: Across image encoders, PyramidCLIP outperforms CLIP on more than half of the small datasets despite using about 36% as much pre-training data.
  • Downstream Task: Linear Probe: The evaluation measures transferability through linear probing on downstream classification tasks.

E Downstream Task: End-to-end Fine-tuning

PyramidCLIP transfers effectively through end-to-end fine-tuning, outperforming both CLIP and a supervised ResNet50 while using substantially less pre-training data than CLIP.

  • PyramidCLIP pre-trained on 143M image-text pairs exceeds both CLIP and supervised ResNet50 in end-to-end fine-tuning across 11 downstream classification datasets.The evaluation uses a ResNet50 backbone and reports accuracy on 11 datasets.
  • PyramidCLIP uses only 36% of CLIP’s pre-training data in this comparison.The comparison is against CLIP, while the supervised ResNet50 uses manually labeled ImageNet-1K data.
  • The fine-tuning evaluation compares PyramidCLIP with CLIP and a supervised ImageNet-1K ResNet50 baseline.Table 12 summarizes end-to-end fine-tuning accuracy across 11 downstream classification datasets.

F.1 Supplementary Ablation of PyramidCLIP Components

Supplementary ablations show that PyramidCLIP’s components improve multiple downstream tasks, while alternative semantically mismatched alignments are unreliable and the selected transformer-layer setting matters.

  • Ablation of Each Component on Other Downstream Tasks: All additional PyramidCLIP components individually improve accuracy on MS-COCO zero-shot retrieval and PASCAL VOC object detection when added to peer-level alignment.The ablation evaluates components on two downstream tasks beyond ImageNet zero-shot classification.
  • The Influence of Different Ls Settings: Ls = 9 achieves the best ImageNet zero-shot classification result among the tested transformer-layer settings.Ls = 0 omits LeFF from all 12 layers, whereas Ls = 12 applies it to all layers and performs poorly.
  • Granular Ablation of Peer-level Alignment: PyramidCLIP includes six losses for peer-level semantics and global- or local-relation cross-level alignment; LRA, LGT, and LLS are alternative candidates.LRA corresponds to peer-level alignment, while LGT and LLS represent semantically mismatched alignments.
  • Granular Ablation of Peer-level Alignment: Adding global contrast LGS improves performance, while adding LRA yields no further gain and is therefore excluded from PyramidCLIP.The authors attribute the lack of additional gain from LRA to precise pairing between ROI features and object-attribute descriptions.
  • Ablation of Semantically Mismatched Alignment: Adding semantically mismatched alignments LGT and LLS provides no stable benefit and degrades performance in most cases.Consequently, LGT and LLS are not included in the PyramidCLIP paradigm.
  • Grad-CAM Visualizations: Grad-CAM visualizations show CLIP attending more to scene or background regions, whereas PyramidCLIP attends more to salient objects during text-to-image retrieval.The authors associate PyramidCLIP’s object-focused attention with cross-level relation alignment.
Loading 2204.14095v2…