PaperlayerView exact revision on arXiv

Source-linked AI summary

SuS-X: Training-Free Name-Only Transfer of Vision-Language Models

Vishaal Udandarao, Ankush Gupta, Samuel Albanie

arXiv:2211.16198v4cs.CVcs.CLcs.MM

TL;DR

Vision-language models often need target-distribution data or fine-tuning for strong downstream transfer, but both can be costly, unstable, or unavailable. SuS-X enables training-free name-only transfer by curating proxy support sets and applying TIP-X, improving zero-shot classification across 19 datasets and extending to training-free few-shot adaptation.

  • Problem

    Downstream adaptation of vision-language models commonly requires fine-tuning or target-distribution samples, despite the cost, instability, and limited availability of those resources.

  • Method

    SuS-X curates support sets from target category names through generated or retrieved images, then uses TIP-X to adapt vision-language models without training or target-task samples.

  • Results

    SuS-X outperforms zero-shot methods by 4.60% for CLIP, 5.97% for BLIP, and 11.37% for TCL in average accuracy across 19 benchmark datasets.

  • Takeaways & Limitations

    Curated support sets provide task-specific knowledge that improves zero-shot predictions, while TIP-X also supports training-free few-shot adaptation.

  • Takeaways & Limitations

    Name-only transfer depends on target concepts having appeared during pre-training, and its LAION-5B and Stable Diffusion sources may contain harmful content.

Abstract

from arXiv · show

Contrastive Language-Image Pre-training (CLIP) has emerged as a simple yet effective way to train large-scale vision-language models. CLIP demonstrates impressive zero-shot classification and retrieval on diverse downstream tasks. However, to leverage its full potential, fine-tuning still appears to be necessary. Fine-tuning the entire CLIP model can be resource-intensive and unstable. Moreover, recent methods that aim to circumvent this need for fine-tuning still require access to images from the target distribution. In this paper, we pursue a different approach and explore the regime of training-free "name-only transfer" in which the only knowledge we possess about the downstream task comprises the names of downstream target categories. We propose a novel method, SuS-X, consisting of two key building blocks -- SuS and TIP-X, that requires neither intensive fine-tuning nor costly labelled data. SuS-X achieves state-of-the-art zero-shot classification results on 19 benchmark datasets. We further show the utility of TIP-X in the training-free few-shot setting, where we again achieve state-of-the-art results over strong training-free baselines. Code is available at https://github.com/vishaal27/SuS-X.

1. Introduction

Vision-language models enable strong zero-shot transfer, but adapting them can require unstable fine-tuning or inaccessible target-distribution samples. SuS-X addresses this with training-free name-only transfer using curated support sets and TIP-X.

  • Motivation: Vision-language models support transfer across classification, retrieval, and segmentation, with CLIP pioneering zero-shot classification on unseen datasets.CLIP converts class labels into textual prompts and predicts the class whose prompt most closely matches the query image.
  • Motivation: CLIP’s zero-shot performance drops when downstream images diverge strongly from its pre-training distribution, motivating adaptation methods.Existing approaches commonly use labelled or unlabelled target-task data, while fine-tuning can be unstable and lead to overfitting.
  • SuS-X: SuS-X performs training-free, name-only downstream adaptation using only target category names and no target-task samples.The framework contains SuS, which curates support sets, and TIP-X, which performs zero-shot classification with those sets.
  • SuS-X: SuS creates proxy few-shot support sets either by generating images with text-to-image models or retrieving real images from large vision-language data banks.These support sets provide task-specific information for downstream predictions without access to the target distribution.
  • Results: SuS-X outperforms zero-shot methods across 19 benchmark datasets, improving average accuracy by 4.60% for CLIP, 5.97% for BLIP, and 11.37% for TCL.The reported improvements cover three different vision-language models.

2. Related Work

Related work has improved vision-language transfer through zero-shot inference and adaptation methods, but the paper focuses on training-free name-only transfer without target-distribution samples. This setting uses only target category names to adapt vision-language models.

  • Vision-language models: Vision-language foundation models learn joint image-text representations from web-scale datasets for downstream visual tasks.CLIP introduced this model family using 400M web-acquired image-text pairs.
  • Zero-shot transfer: CLIP’s training objective enables zero-shot classification by assigning similarities to image-text pairs.This capability supports adaptation at inference time even when target dataset classes are not known in advance.
  • Adaptation methods: Prior CLIP adaptation methods are distinguished by whether they require training, labelled target-task samples, or any target-task distribution samples.This taxonomy frames the resource requirements that separate existing approaches.
  • Name-only transfer: The paper studies training-free name-only transfer, requiring neither explicit training nor target-distribution samples and assuming only target category names.The terminology follows earlier use of name-only transfer for semantic segmentation.

3. SuS-X: Training-Free Name-Only Transfer

SuS-X enables training-free name-only transfer by constructing visual support sets from category names and using TIP-X to adapt VLM predictions through image-text affinities. Its inference pipeline replaces target-domain few-shot features with these support-set features and blends support-derived logits with zero-shot logits.

  • Framework: SuS-X combines SuS support-set construction with TIP-X training-free inference for downstream VLM classification.The framework is designed for name-only transfer without direct access to target-task samples.
  • SuS Construction: SuS constructs a support set by generating N images per class with Stable Diffusion, producing NC labelled support images.The class labels are used to prompt image generation; customized prompts can further diversify generated images.
  • SuS Construction: SuS can alternatively retrieve task-specific images from LAION-5B using textual class prompts.This provides a non-parametric support-set construction route alongside Stable Diffusion generation.
  • TIP-X Inference: TIP-X uses support-set affinities as attention weights over support labels and blends the resulting logits with the original zero-shot logits.In SuS-X, support-set features replace the few-shot features used by TIP-X, yielding the SuS-X-LC and SuS-X-SD variants.
  • TIP-X Inference: TIP-X addresses potentially uncalibrated image-image similarities by comparing test and support features through image-text signatures and KL-divergence affinities.The resulting affinity matrix avoids direct reliance on image-image similarities and is rescaled and negated before use as attention weights.

4. Experiments

The experiments evaluate SuS-X across 19 datasets and multiple vision-language models, then test TIP-X in training-free few-shot adaptation and analyze its components and limitations. SuS-X improves over zero-shot and competing training-free methods, while support-set quality and domain matching remain important.

  • Experimental Settings: The evaluation covers 19 datasets spanning object, scene, and fine-grained categories, using six zero-shot or name-only baselines.The study reports results for CLIP and transfers the evaluation to TCL and BLIP.
  • Transfer to Different VLMs: 11.37% and 5.97% average gains over zero-shot models are reported for SuS-X across TCL and BLIP, respectively.These averages are computed across 19 datasets and support transfer beyond CLIP.
  • Few-shot Adaptation: 0.91% average accuracy separates TIP-X from TIP-Adapter across shots in training-free few-shot classification.TIP-X also reports a 2.1% average gain on OxfordPets.
  • Component Analysis: SuS-X uses SuS construction and TIP-X inference as complementary building blocks, and the component analysis finds both important for the best results.The analysis compares replacing TIP-X with TIP-Adapter and examines prompt choices.
  • Support-Set Analysis: SuS samples can mimic the true data distribution, but domain characteristics such as lighting, backgrounds, and confounding objects may remain mismatched.The paper identifies unsupervised domain matching as needed to close the gap to true few-shot support sets.
  • Limitations: Name-only transfer requires related concepts to have appeared during CLIP pre-training, and the approach relies on potentially harmful, relatively uncurated LAION-5B and Stable Diffusion sources.The authors state that real-world deployment requires careful mitigation strategies.

5. Conclusion

The paper studies training-free name-only classification by curating support sets without target-distribution samples and using TIP-X for inference. It reports superior performance to prior state-of-the-art methods while extending TIP-X to related adaptation settings.

  • Conclusion: The study targets training-free name-only classification without access to samples from the target distribution.Its support sets provide task-specific knowledge for improving CLIP’s zero-shot predictions.
  • Conclusion: SuS-X combines systematically curated support sets with TIP-X, motivated partly by limitations of intra-modal similarity spaces.The conclusion frames these components as the basis for the proposed transfer approach.
  • Conclusion: The authors report superior performance to prior state-of-the-art methods using the two building blocks.The conclusion summarizes the method-level outcome without specifying a dataset or metric.

A. Dataset Details

The appendix documents dataset modifications, support-set sizes, implementation details, baselines, and additional visualizations for the experiments. It also extends the few-shot results across 11 datasets and describes the support samples used across datasets.

  • Dataset Details: The dataset appendix reports two modifications: removing specified Caltech101 classes and using each UCF101 video’s middle frame as its image sample.These changes follow the standard-dataset modifications described in CoOp.
  • Implementation: Stable Diffusion generation uses a fixed checkpoint, guidance scale, diffusion-step count, and 512×512 output resolution before downscaling to CLIP’s input size.The appendix also describes LAION-5B retrieval as the second support-set curation strategy.
  • Few-shot Results: The extended few-shot evaluation reports results across 11 datasets and an average 0.91% improvement over TIP-Adapter across all shots.This extends the few-shot comparison described in the main paper.
  • Support Sets: Support-set sizes are enumerated per dataset, and the appendix notes that support-set size can affect performance in dataset-specific ways.The main experiments use a fixed number of support samples per dataset.
  • VLM Transfer: The appendix transfers baselines to TCL and BLIP by swapping CLIP’s frozen image and text encoders while preserving other experimental settings.The transfer uses standard ViT-B/16 retrieval-fine-tuned models for both VLMs.
  • Visualizations: Additional figures show support samples generated or retrieved by SuS across datasets and compare them with true training-distribution samples.The figures identify the source ordering or labels for the displayed samples.

G. Hyperparameter Settings

The appendix specifies the searched hyperparameter ranges, reports fixed-parameter results, and analyzes the roles of intra-modal and inter-modal distance terms. It also notes prompt-specific visual diversity and fine-grained-task improvements.

  • Hyperparameter Settings: The hyperparameter search covers α from 0.1 to 50, β from 1 to 50, and γ from 0.1 to 30.The main paper’s sensitivity analysis assesses whether accuracy varies substantially across settings.
  • Results Without Tuning: Large gains over Zero-shot CLIP remain when α=0.1, β=1.0, and γ=0.1 are fixed without tuning.These results are reported across all 19 datasets.
  • Distance-Term Analysis: Small inter-modal distance contributions generally outperform removing them, supporting the importance of both intra-modal and inter-modal terms.The comparison uses optimal, fixed, and γ=0 settings for datasets where α≫γ.
  • Prompting Strategies: CuPL prompting produces more diverse support sets than Photo prompting, while the best results use LC-Photo and SD-CuPL strategies.Diversity is defined through within-class pairwise image cosine similarities.
  • Fine-grained Tasks: Figure 9 reports improvements for fine-grained tasks using SuS.The supplied figure label identifies the comparison but does not provide numerical values.

I. Compute Cost Comparison

SuS-X adds support-set curation and TIP-X inference while remaining training-free, and its design uses diversity and complementary distance components to improve adaptation.

  • Compute cost: SuS-X support sets can be cached, amortising their construction costs across queries.Compute requirements are measured for one ImageNet class and 50 test images.
  • Support-set diversity: Support-set diversity is computed from mean pairwise cosine similarity within classes and then averaged across classes.Higher mean pairwise similarity implies lower diversity, so diversity is defined as 1 − MPCS.
  • Component analysis: The ablations identify both SuS construction and TIP-X as crucial components of SuS-X.The component analysis compares SuS with TIP-Adapter and TIP-X across 19 datasets.
  • TIP-X design: SuS-X combines zero-shot, intra-modal, and inter-modal distance terms, whereas TIP-Adapter uses only the first two.The three terms are zero-shot logits, intra-modal distances, and inter-modal distances.
  • TIP-X design: Both intra-modal and inter-modal distances provide large, complementary gains over the zero-shot term alone.Combining both distance types produces the best results in the reported 19-dataset ablation.

K.2. Comparing name-only SuS-X to few-shot methods

SuS-X remains competitive with labelled few-shot adaptation despite using no target-task images, but its performance can fall when synthetic or retrieved supports differ from the true distribution.

  • Name-only versus few-shot: SuS-X-LC is competitive with 16-shot TIP-Adapter and 16-shot TIP-X despite using no target-task images.The comparison uses 16 labelled images per class for each few-shot method.
  • Name-only versus few-shot: SuS-X-LC can severely underperform few-shot methods when a domain gap separates SuS images from true labelled images.The reported explanation attributes these cases to mismatch between support and target distributions.
  • LAION-5B retrieval: LAION-5B supports retrieved with CuPL prompts diverge from natural target concepts, producing a larger domain gap than Photo prompting.The passage links this divergence to poorer performance for CuPL retrieval than Photo prompting.
  • Stable Diffusion generation: Stable Diffusion supports face an inherent distribution shift because target datasets mostly contain natural images.This places generated supports at a disadvantage relative to LAION-5B supports.
  • Stable Diffusion generation: Within Stable Diffusion supports, CuPL mildly outperforms Photo by generating more diverse lighting conditions and background scenes.The added diversity is reported to reduce the domain gap to the target dataset to a small extent.

L. Extended Results on all Datasets

Extended evaluation reports training-free zero-shot/name-only results across the full 19-dataset suite and across CLIP, TCL, and BLIP model configurations.

  • Dataset and model coverage: The extended results cover all 19 datasets and also report averages on the 11-dataset subset used by earlier CLIP adaptation work.The individual datasets include object, scene, and fine-grained recognition tasks.
  • CLIP: The tables distinguish ZS-CLIP and CuPL+ensemble baselines and include original and re-implemented CALIP results.The extended CLIP table reports averages for both the 11-dataset subset and the entire 19-dataset suite.
  • TCL: TCL results use SuS-X-SD with CuPL and SuS-X-LC with Photo, the two best configurations from the CLIP experiment.Table 18 contains the full values used for the main-paper TCL results.
  • BLIP: BLIP results likewise use SuS-X-SD with CuPL and SuS-X-LC with Photo.Table 19 reports the full values for the BLIP architecture.

M. Results with different Visual Backbones

SuS-X is evaluated with ResNets and Vision Transformers, and accuracy increases monotonically as visual-backbone model capacity increases.

  • Backbone comparison: Accuracy values monotonically improve as visual-backbone capacity increases across ResNet-50, ResNet-101, ViT-B/32, and ViT-B/16.The comparison spans two ResNet and two Vision Transformer backbone classes.
  • Backbone comparison: The main results use the ResNet-50 visual backbone for CLIP’s image encoder.The extended comparison adds ResNet-101, ViT-B/32, and ViT-B/16.

N. Results with different Text-to-Image Generation Models

SuS-X-SD remains robust across text-to-image generation models, while comparisons with fine-tuning show a trade-off between peak performance and training-free flexibility.

  • Text-to-Image Generation Models: SuS-X-SD uses Stable-Diffusion-v1.4 in the main-paper results and shows similar gains with three other text-to-image generation models.The additional models are evaluated in Table 21.
  • Fine-tuning Comparison: Fine-tuned methods outperform training-free SuS-X on some datasets, but their gains are limited by the domain gap between generated and real test data.The comparison covers full and parameter-efficient fine-tuning methods.
  • Fine-tuning Comparison: SuS-X and full fine-tuning are complementary, producing a large performance boost for the fine-tuned SuS-X-F variant.
  • Fine-tuning Comparison: SuS-X prioritizes flexibility and scalability by supporting arbitrary numbers of rare categories without training, especially when categories change frequently.The paper contrasts this setting with repetitive fine-tuning and notes that fine-tuning can cause forgetting of pretrained performance.
Loading 2211.16198v4…