PaperlayerView exact revision on arXiv

Source-linked AI summary

CALIP: Zero-Shot Enhancement of CLIP with Parameter-free Attention

Ziyu Guo, Renrui Zhang, Longtian Qiu, Xianzheng Ma, Xupeng Miao, Xuming He, Bin Cui

arXiv:2209.14169v2cs.CVcs.AIcs.MM

TL;DR

Existing CLIP adaptations rely on learnable modules, few-shot data, and additional training, limiting efficient downstream transfer. CALIP instead uses parameter-free bidirectional cross-modal attention to update visual and textual representations. Across 14 2D and 3D datasets, it improves zero-shot performance over CLIP, while CALIP-FS achieves leading few-shot results after adding linear layers.

  • Problem

    Existing CLIP adaptations require task-specific learnable parameters, few-shot data, and additional training, hindering efficient zero-shot knowledge transfer and deployment.

    Source: Introduction · p. 2

  • Method

    CALIP adds parameter-free bidirectional cross-modal attention that interacts spatial visual and textual features to enhance CLIP without extra downstream data or training.

    Source: Introduction · p. 2, CALIP with Parameter-free Attention · p. 3

  • Results

    Across 14 datasets spanning zero-shot 2D image and 3D point-cloud classification, CALIP improves zero-shot performance over CLIP, while CALIP-FS achieves leading few-shot performance.

    Source: Introduction · p. 2, Conclusion · p. 7

  • Takeaways & Limitations

    Parameter-free attention provides an efficient CLIP enhancement framework for zero-shot transfer, with a parametric extension supporting competitive few-shot adaptation.

    Source: Conclusion · p. 7

Abstract

from arXiv · show

Contrastive Language-Image Pre-training (CLIP) has been shown to learn visual representations with great transferability, which achieves promising accuracy for zero-shot classification. To further improve its downstream performance, existing works propose additional learnable modules upon CLIP and fine-tune them by few-shot training sets. However, the resulting extra training cost and data requirement severely hinder the efficiency for model deployment and knowledge transfer. In this paper, we introduce a free-lunch enhancement method, CALIP, to boost CLIP's zero-shot performance via a parameter-free Attention module. Specifically, we guide visual and textual representations to interact with each other and explore cross-modal informative features via attention. As the pre-training has largely reduced the embedding distances between two modalities, we discard all learnable parameters in the attention and bidirectionally update the multi-modal features, enabling the whole process to be parameter-free and training-free. In this way, the images are blended with textual-aware signals and the text representations become visual-guided for better adaptive zero-shot alignment. We evaluate CALIP on various benchmarks of 14 datasets for both 2D image and 3D point cloud few-shot classification, showing consistent zero-shot performance improvement over CLIP. Based on that, we further insert a small number of linear layers in CALIP's attention module and verify our robustness under the few-shot settings, which also achieves leading performance compared to existing methods. Those extensive experiments demonstrate the superiority of our approach for efficient enhancement of CLIP.

Introduction

CLIP enables transferable zero-shot recognition, but existing downstream adaptations require task-specific learnable parameters, few-shot data, and additional training. CALIP addresses this efficiency gap with parameter-free cross-modal attention that enhances CLIP across zero-shot and few-shot settings.

CALIP

  • CALIP updates both modalities through attention, strengthening visual regions and corresponding category features while preserving cross-modal interactions.The attention map assigns weights based on feature similarity between categories and image pixels or sites.Source: Figure 1 · p. 1

Motivation

  • CLIP aligns image and text features in a shared embedding space, enabling zero-shot recognition of unseen classes from category names.Its downstream classification becomes an image-text matching problem in open-vocabulary settings.Source: Introduction · p. 1
  • Existing adaptations improve CLIP with learnable prompts or adapters but require few-shot labeled data, extra training, and separate task-specific model copies.These requirements hinder efficient zero-shot knowledge transfer and deployment.Source: Introduction · pp. 1–2

CALIP

  • CALIP uses intermediate spatial visual features and textual features in a non-parametric, bidirectional cross-modal attention module.The design exchanges image-text information without learnable parameters before producing the final zero-shot prediction.Source: Introduction · p. 2, RELATED WORK · p. 3

Evaluation

  • CALIP is evaluated on 14 datasets for zero-shot 2D image and 3D point-cloud classification, while CALIP-FS adds a small number of linear layers for few-shot fine-tuning.The authors report favorable zero-shot performance for CALIP and leading or competitive performance for CALIP-FS among existing methods.Source: Introduction · p. 2, Figure 2 · p. 3

Method

CALIP enhances CLIP through parameter-free cross-modal attention that exchanges information between spatial visual and textual features. A parametric extension, CALIP-FS, adds lightweight projections for few-shot adaptation while freezing CLIP’s encoders.

CALIP with Parameter-free Attention

  • CALIP applies parameter-free cross-modal attention between CLIP’s intermediate spatial visual feature and textual feature without projection layers.The attention map is obtained directly by matrix multiplication.Source: CALIP with Parameter-free Attention · pp. 3–4
  • The attention map represents category-to-image-site similarities and supports bidirectional updates of visual and textual features.Textual features become adaptive to each input image, while spatial features provide fine-grained information.Source: CALIP with Parameter-free Attention · pp. 3–4
  • CALIP combines original CLIP, visual-guided, and textual-blended logits through weighted aggregation for zero-shot prediction.The resulting process requires neither few-shot fine-tuning nor additional data.Source: CALIP with Parameter-free Attention · p. 4

CALIP-FS with Parametric Attention

  • CALIP-FS inserts learnable pre- and post-projection layers into the attention module while freezing CLIP’s pre-trained encoders.The added projections enable parametric attention for few-shot settings.Source: CALIP-FS with Parametric Attention · p. 4
  • CALIP-FS uses weighted summation of the same three logit terms and achieves competitive performance after few-shot fine-tuning.The method is described as significantly improving over zero-shot CALIP.Source: CALIP-FS with Parametric Attention · p. 5

Experiments

Experiments evaluate CALIP and CALIP-FS across diverse 2D and 3D benchmarks, distribution shifts, and ablations. The results show consistent gains from parameter-free enhancement and strong few-shot performance from CALIP-FS.

Evaluation Setup

  • CALIP is evaluated on 11 image datasets and 3 point-cloud datasets without using downstream training sets.The setup uses full test sets for direct evaluation, with ResNet-50 and a 12-layer transformer as encoders.Source: Zero-shot CALIP · p. 5

Zero-shot Performance

  • +0.92% average accuracy: CALIP outperforms CLIP across all 2D downstream benchmarks.It also improves Caltech101 by +3.77% and EuroSAT by +1.36%.Source: Zero-shot CALIP · p. 6
  • +1.70% average accuracy: CALIP enhances PointCLIP across three 3D point-cloud classification datasets without parameters.Source: Zero-shot CALIP · p. 6

Few-shot Performance

  • CALIP-FS achieves the highest average accuracy across 11 datasets and improves substantially over zero-shot CLIP, especially on DTD and EuroSAT.Its gains over CLIP on DTD and EuroSAT range from +20% to +50%.Source: Few-shot CALIP-FS · p. 6, Figure 4 · p. 6
  • CALIP-FS lags behind Tip-Adapter-F only on OxfordPets while outperforming other few-shot methods on DTD, EuroSAT, and SUN397.Unlike Tip-Adapter-F’s two-step procedure with stored training samples, CALIP-FS uses one-step training.Source: Few-shot CALIP-FS · p. 6

Distribution Shift

Ablation Studies

  • Ablations show that combining logits with attention applied to only one modality performs better than including the fourth term updated in both modalities.The authors attribute the degradation to excessive cross-modal interaction harming CLIP’s pre-trained alignment.Source: Ablation Study · p. 7
  • Using both pre- and post-projection layers for both modalities gives CALIP-FS its best performance.The design separates the attention calculation space from the preceding embedding space.Source: Ablation Study · p. 7

Visualization

Visualization shows that both CALIP variants focus attention around object pixels and make spatial visual features more distinctive using category text guidance.

  • Attention maps in CALIP and CALIP-FS concentrate around object pixels, while category-guided visual features become more distinctive.After few-shot fine-tuning, both attention-map and feature distributions become more intensive.Source: Visualization · p. 7

Conclusion

The paper concludes that CALIP provides parameter-free, training-free enhancement of CLIP across 2D and 3D benchmarks, while CALIP-FS extends the approach to competitive few-shot performance.

  • CALIP performs zero-shot visual-textual interaction through a parameter-free attention module without additional training.Source: Conclusion · p. 7
  • CALIP achieves favorable performance across a wide range of 2D and 3D benchmarks.Source: Conclusion · p. 7
  • CALIP-FS adds a parametric attention version for few-shot fine-tuning and obtains competitive results among existing state-of-the-art methods.Source: Conclusion · p. 7
  • The authors identify extending parameter-free methods to CLIP-based object detection and semantic segmentation as future work.Source: Conclusion · p. 7

Few-shot Training Settings

Few-shot experiments compare CALIP-FS with established baselines across 11 2D datasets and multiple shot counts, while reporting selected hyperparameter settings and attention magnitudes.

Few-shot Protocol

Additional Ablation Study

The study adds ablations for zero-shot CALIP and 16-shot CALIP-FS to further explain the method's behavior. Hyperparameters are tuned on validation data, with test-set results reported for reference.

Logits Weights β1, β2, β3

The experiments analyze how the three logits contribute to prediction, finding that zero-shot CALIP benefits from stronger visual-guided logits while CALIP-FS relies more on pretrained logits.

Parameter-free Magnitudes αt, αs

The parameter-free attention uses equal magnitude modulators for textual and visual attention, while CALIP and CALIP-FS remain stronger than their baselines across visual encoders.

Best Dataset Hyperparameters

Dataset-specific experiments show that the three logits require different proportions, with zero-shot CALIP favoring larger β2 and CALIP-FS generally favoring smaller, balanced β2 and β3.

Additional Visualization

CALIP’s visualizations show that cross-modal interactions make features more semantically aligned for final matching. Medical-dataset experiments further assess transfer under larger domain gaps, with CALIP and CALIP-FS leading existing methods.

Related papers

Loading 2209.14169v2…