PaperlayerView exact revision on arXiv

Source-linked AI summary

CALIP: Zero-Shot Enhancement of CLIP with Parameter-free Attention

Ziyu Guo, Renrui Zhang, Longtian Qiu, Xianzheng Ma, Xupeng Miao, Xuming He, Bin Cui

arXiv:2209.14169v2cs.CVcs.AIcs.MM

TL;DR

Existing CLIP adaptations rely on learnable modules, few-shot data, and additional training, limiting efficient downstream transfer. CALIP instead uses parameter-free bidirectional cross-modal attention to update visual and textual representations. Across 14 2D and 3D datasets, it improves zero-shot performance over CLIP, while CALIP-FS achieves leading few-shot results after adding linear layers.

  • Problem

    Existing CLIP adaptations require task-specific learnable parameters, few-shot data, and additional training, hindering efficient zero-shot knowledge transfer and deployment.

  • Method

    CALIP adds parameter-free bidirectional cross-modal attention that interacts spatial visual and textual features to enhance CLIP without extra downstream data or training.

  • Results

    Across 14 datasets spanning zero-shot 2D image and 3D point-cloud classification, CALIP improves zero-shot performance over CLIP, while CALIP-FS achieves leading few-shot performance.

  • Takeaways & Limitations

    Parameter-free attention provides an efficient CLIP enhancement framework for zero-shot transfer, with a parametric extension supporting competitive few-shot adaptation.

Abstract

from arXiv · show

Contrastive Language-Image Pre-training (CLIP) has been shown to learn visual representations with great transferability, which achieves promising accuracy for zero-shot classification. To further improve its downstream performance, existing works propose additional learnable modules upon CLIP and fine-tune them by few-shot training sets. However, the resulting extra training cost and data requirement severely hinder the efficiency for model deployment and knowledge transfer. In this paper, we introduce a free-lunch enhancement method, CALIP, to boost CLIP's zero-shot performance via a parameter-free Attention module. Specifically, we guide visual and textual representations to interact with each other and explore cross-modal informative features via attention. As the pre-training has largely reduced the embedding distances between two modalities, we discard all learnable parameters in the attention and bidirectionally update the multi-modal features, enabling the whole process to be parameter-free and training-free. In this way, the images are blended with textual-aware signals and the text representations become visual-guided for better adaptive zero-shot alignment. We evaluate CALIP on various benchmarks of 14 datasets for both 2D image and 3D point cloud few-shot classification, showing consistent zero-shot performance improvement over CLIP. Based on that, we further insert a small number of linear layers in CALIP's attention module and verify our robustness under the few-shot settings, which also achieves leading performance compared to existing methods. Those extensive experiments demonstrate the superiority of our approach for efficient enhancement of CLIP.

Introduction

CLIP enables transferable zero-shot recognition, but existing downstream adaptations require task-specific learnable parameters, few-shot data, and additional training. CALIP addresses this efficiency gap with parameter-free cross-modal attention that enhances CLIP across zero-shot and few-shot settings.

  • CALIP: CALIP updates both modalities through attention, strengthening visual regions and corresponding category features while preserving cross-modal interactions.The attention map assigns weights based on feature similarity between categories and image pixels or sites.
  • Motivation: CLIP aligns image and text features in a shared embedding space, enabling zero-shot recognition of unseen classes from category names.Its downstream classification becomes an image-text matching problem in open-vocabulary settings.
  • Motivation: Existing adaptations improve CLIP with learnable prompts or adapters but require few-shot labeled data, extra training, and separate task-specific model copies.These requirements hinder efficient zero-shot knowledge transfer and deployment.
  • CALIP: CALIP uses intermediate spatial visual features and textual features in a non-parametric, bidirectional cross-modal attention module.The design exchanges image-text information without learnable parameters before producing the final zero-shot prediction.
  • Evaluation: CALIP is evaluated on 14 datasets for zero-shot 2D image and 3D point-cloud classification, while CALIP-FS adds a small number of linear layers for few-shot fine-tuning.The authors report favorable zero-shot performance for CALIP and leading or competitive performance for CALIP-FS among existing methods.

Method

CALIP enhances CLIP through parameter-free cross-modal attention that exchanges information between spatial visual and textual features. A parametric extension, CALIP-FS, adds lightweight projections for few-shot adaptation while freezing CLIP’s encoders.

  • CALIP with Parameter-free Attention: CALIP applies parameter-free cross-modal attention between CLIP’s intermediate spatial visual feature and textual feature without projection layers.The attention map is obtained directly by matrix multiplication.
  • CALIP with Parameter-free Attention: The attention map represents category-to-image-site similarities and supports bidirectional updates of visual and textual features.Textual features become adaptive to each input image, while spatial features provide fine-grained information.
  • CALIP with Parameter-free Attention: CALIP combines original CLIP, visual-guided, and textual-blended logits through weighted aggregation for zero-shot prediction.The resulting process requires neither few-shot fine-tuning nor additional data.
  • CALIP-FS with Parametric Attention: CALIP-FS inserts learnable pre- and post-projection layers into the attention module while freezing CLIP’s pre-trained encoders.The added projections enable parametric attention for few-shot settings.
  • CALIP-FS with Parametric Attention: CALIP-FS uses weighted summation of the same three logit terms and achieves competitive performance after few-shot fine-tuning.The method is described as significantly improving over zero-shot CALIP.

Experiments

Experiments evaluate CALIP and CALIP-FS across diverse 2D and 3D benchmarks, distribution shifts, and ablations. The results show consistent gains from parameter-free enhancement and strong few-shot performance from CALIP-FS.

  • Evaluation Setup: CALIP is evaluated on 11 image datasets and 3 point-cloud datasets without using downstream training sets.The setup uses full test sets for direct evaluation, with ResNet-50 and a 12-layer transformer as encoders.
  • Zero-shot Performance: +0.92% average accuracy: CALIP outperforms CLIP across all 2D downstream benchmarks.It also improves Caltech101 by +3.77% and EuroSAT by +1.36%.
  • Zero-shot Performance: +1.70% average accuracy: CALIP enhances PointCLIP across three 3D point-cloud classification datasets without parameters.
  • Few-shot Performance: CALIP-FS achieves the highest average accuracy across 11 datasets and improves substantially over zero-shot CLIP, especially on DTD and EuroSAT.Its gains over CLIP on DTD and EuroSAT range from +20% to +50%.
  • Few-shot Performance: CALIP-FS lags behind Tip-Adapter-F only on OxfordPets while outperforming other few-shot methods on DTD, EuroSAT, and SUN397.Unlike Tip-Adapter-F’s two-step procedure with stored training samples, CALIP-FS uses one-step training.
  • Distribution Shift: CALIP-FS surpasses CoOp on four out-of-distribution datasets under 16-shot fine-tuning.
  • Ablation Studies: Ablations show that combining logits with attention applied to only one modality performs better than including the fourth term updated in both modalities.The authors attribute the degradation to excessive cross-modal interaction harming CLIP’s pre-trained alignment.
  • Ablation Studies: Using both pre- and post-projection layers for both modalities gives CALIP-FS its best performance.The design separates the attention calculation space from the preceding embedding space.

Visualization

Visualization shows that both CALIP variants focus attention around object pixels and make spatial visual features more distinctive using category text guidance.

  • Attention maps in CALIP and CALIP-FS concentrate around object pixels, while category-guided visual features become more distinctive.After few-shot fine-tuning, both attention-map and feature distributions become more intensive.

Conclusion

The paper concludes that CALIP provides parameter-free, training-free enhancement of CLIP across 2D and 3D benchmarks, while CALIP-FS extends the approach to competitive few-shot performance.

  • CALIP performs zero-shot visual-textual interaction through a parameter-free attention module without additional training.
  • CALIP achieves favorable performance across a wide range of 2D and 3D benchmarks.
  • CALIP-FS adds a parametric attention version for few-shot fine-tuning and obtains competitive results among existing state-of-the-art methods.
  • The authors identify extending parameter-free methods to CLIP-based object detection and semantic segmentation as future work.

Few-shot Training Settings

Few-shot experiments compare CALIP-FS with established baselines across 11 2D datasets and multiple shot counts, while reporting selected hyperparameter settings and attention magnitudes.

  • Few-shot Protocol: CALIP-FS is compared with CoOp, CLIP-Adapter, and Tip-Adapter-F on 11 2D datasets using 1, 2, 4, 8, and 16 shots per category.

Additional Ablation Study

The study adds ablations for zero-shot CALIP and 16-shot CALIP-FS to further explain the method's behavior. Hyperparameters are tuned on validation data, with test-set results reported for reference.

  • The ablation study examines both zero-shot CALIP and 16-shot CALIP-FS to further explain the method's theory.
  • All hyperparameters are tuned on the official validation set.
  • Corresponding test-set results are reported for reference after validation-based tuning.

Logits Weights β1, β2, β3

The experiments analyze how the three logits contribute to prediction, finding that zero-shot CALIP benefits from stronger visual-guided logits while CALIP-FS relies more on pretrained logits.

  • For zero-shot CALIP, a visual-guided logit weight of 5 achieves the best Caltech101 performance.The textual features use cross-modal interactions to explore informative visual features from the image.
  • The 5-weight visual-guided logit contributes more than CLIP's original 1-weight logit in zero-shot CALIP.
  • For CALIP-FS, the pretrained logit is most important, while β2 and β3 show limited sensitivity across datasets within a certain range.

Parameter-free Magnitudes αt, αs

The parameter-free attention uses equal magnitude modulators for textual and visual attention, while CALIP and CALIP-FS remain stronger than their baselines across visual encoders.

  • Setting both textual and visual attention modulators αt and αs to 2 produces the highest zero-shot Caltech101 accuracy.
  • Equal magnitude modulation indicates comparative roles for the two modalities in bidirectional feature updates.
  • Across different visual networks on ImageNet, CALIP and CALIP-FS outperform their respective zero-shot CLIP and CoOp baselines.

Best Dataset Hyperparameters

Dataset-specific experiments show that the three logits require different proportions, with zero-shot CALIP favoring larger β2 and CALIP-FS generally favoring smaller, balanced β2 and β3.

  • Different domains require different proportions of the three logits.The best β1, β2, and β3 values are listed for all datasets in Table 9.
  • For zero-shot CALIP, larger β2 improves recognition accuracy and surpasses β1 on 7 datasets.
  • CALIP-FS generally prefers equal β2 and β3 values, both normally smaller than β1.

Additional Visualization

CALIP’s visualizations show that cross-modal interactions make features more semantically aligned for final matching. Medical-dataset experiments further assess transfer under larger domain gaps, with CALIP and CALIP-FS leading existing methods.

  • Ground-truth textual attention concentrates on corresponding objects in 16-shot CALIP-FS visualizations.After inter-modal interactions, visual and textual features become more semantically aligned for final matching.
  • Medical experiments use ISIC and BCCD to test transfer across pathological skin-injury and blood-cell recognition.These datasets have larger domain gaps than the default eleven 2D datasets.
  • CALIP and CALIP-FS show leading performance among existing methods on ISIC and BCCD in zero-shot and 16-shot evaluations.Table 12 reports zero-shot classification, while Table 13 reports 16-shot classification.
Loading 2209.14169v2…