PaperlayerView exact revision on arXiv

Source-linked AI summary

Task Residual for Tuning Vision-Language Models

Tao Yu, Zhihe Lu, Xin Jin, Zhibo Chen, Xinchao Wang

arXiv:2211.10277v2cs.CV

TL;DR

Efficient transfer learning must preserve useful VLM knowledge while adapting to downstream tasks with limited data, but prompt and adapter tuning can respectively damage or over-rely on that prior. TaskRes freezes the pre-trained text-based classifier and adds a prior-independent residual, achieving state-of-the-art or comparable performance across 11 datasets with minimal implementation effort, while negative transfer remains on some datasets and transfer difficulty is assessed heuristically.

  • Problem

    Existing ETL methods can damage VLM prior knowledge or insufficiently explore task-specific knowledge when adapting with limited data.

  • Method

    TaskRes freezes the pre-trained text-based classifier and adds prior-independent tunable parameters as a residual for target-task adaptation.

  • Results

    TaskRes achieves state-of-the-art or state-of-the-art-comparable performance across 11 benchmark datasets and few-shot settings while significantly surpassing zero-shot CLIP.

  • Takeaways & Limitations

    Decoupling frozen prior knowledge from an independent task residual supports flexible task-specific exploration with minimal implementation effort.

  • Takeaways & Limitations

    TaskRes encounters negative transfer on OxfordPets and Food101 in specified few-shot settings, while transfer-difficulty assessment remains heuristic.

Abstract

from arXiv · show

Large-scale vision-language models (VLMs) pre-trained on billion-level data have learned general visual representations and broad visual concepts. In principle, the well-learned knowledge structure of the VLMs should be inherited appropriately when being transferred to downstream tasks with limited data. However, most existing efficient transfer learning (ETL) approaches for VLMs either damage or are excessively biased towards the prior knowledge, e.g., prompt tuning (PT) discards the pre-trained text-based classifier and builds a new one while adapter-style tuning (AT) fully relies on the pre-trained features. To address this, we propose a new efficient tuning approach for VLMs named Task Residual Tuning (TaskRes), which performs directly on the text-based classifier and explicitly decouples the prior knowledge of the pre-trained models and new knowledge regarding a target task. Specifically, TaskRes keeps the original classifier weights from the VLMs frozen and obtains a new classifier for the target task by tuning a set of prior-independent parameters as a residual to the original one, which enables reliable prior knowledge preservation and flexible task-specific knowledge exploration. The proposed TaskRes is simple yet effective, which significantly outperforms previous ETL methods (e.g., PT and AT) on 11 benchmark datasets while requiring minimal effort for the implementation. Our code is available at https://github.com/geekyutao/TaskRes.

1. Introduction

Efficient transfer learning for vision-language models must preserve transferable prior knowledge while exploring task-specific knowledge with limited data. TaskRes addresses this tension by adding a prior-independent residual to a frozen text-based classifier and achieves strong performance across 11 datasets with minimal implementation effort.

  • ETL on pre-trained VLMs aims to inherit their learned knowledge structure while exploring task-specific knowledge using limited data.
  • Prompt tuning can damage prior knowledge, whereas adapter-style tuning may insufficiently explore task-specific knowledge because it relies on pre-trained features.
  • TaskRes freezes the original text-based classifier and adds prior-independent tunable parameters as a task residual for target-task adaptation.
  • Experiments across 11 benchmark datasets show that task residuals significantly enhance transfer performance and yield state-of-the-art performance across diverse datasets.
  • TaskRes requires minimal implementation effort, technically requiring only one additional line of code.

2. Related Work

Related work establishes large-scale VLMs as transferable models that align language and images in a common space, while efficient transfer methods adapt them with few parameters and examples. Existing approaches mainly use prompt tuning or adapters, motivating TaskRes as a prior-independent alternative.

  • Language-driven visual representation learning maps text and images into a common space so text representations can support visual classification.
  • Large-scale VLMs use billion-level image-text data and powerful neural networks to achieve strong zero-shot transferability.
  • ETL tunes a small number of parameters with limited data and is commonly divided into prompt tuning and adapter-style tuning.
  • Adapter methods improve performance while relying heavily on pre-trained features, which limits exploration of new task-specific knowledge.
  • TaskRes addresses this limitation through a prior-independent task residual for more flexible task-specific knowledge learning.

3. Preliminaries

CLIP aligns image and text representations through contrastive pre-training and classifies images using text-derived classifiers. Existing efficient tuning methods either optimize prompt contexts or transform pre-trained features with adapter modules.

  • CLIP aligns image and text features in a unified embedding space using contrastive learning on 0.4 billion image-text pairs.
  • CLIP classifies an image by comparing its image embedding with text embeddings generated from natural-language class prompts.
  • The predicted class probabilities use cosine similarity between the image embedding and each class text embedding, scaled by CLIP’s learned temperature.
  • CoOp replaces fixed prompt contexts with learnable context vectors while keeping the pre-trained VLM parameters frozen.
  • Adapter-style tuning adds trainable modules that transform pre-trained features, with CLIP-Adapter using two linear layers and an intervening ReLU activation.

4. Approach

TaskRes addresses limitations in existing efficient transfer learning by separating preservation of pre-trained knowledge from exploration of task-specific knowledge. It freezes the base classifier and image branch while tuning a prior-independent residual added to the text-based classifier.

  • Motivation: Existing efficient transfer methods can damage pre-trained knowledge or insufficiently explore task-specific knowledge under downstream distribution shifts.Prompt tuning may replace the original classification boundary, while adapter-style tuning can be constrained by pre-trained features.
  • Task Residual Tuning: TaskRes explicitly decouples maintenance of pre-trained knowledge from learning task-specific knowledge.This design is intended to avoid the pitfalls identified in prompt tuning and adapter-style tuning.
  • Fixed base classifier: The method freezes the text-based base classifier to prevent damage to the pre-trained classification boundary.The base classifier consists of text embeddings from a pre-trained vision-language model.
  • Prior-independent task residual: TaskRes adds a scaled, element-wise prior-independent task residual to the frozen base classifier to form a target-task classifier.The residual is a tunable parameter set independent of the base classifier.
  • Tuning for downstream tasks: During tuning, only the task residual is updated while the base classifier and image branch remain fixed.The downstream loss updates the residual through standard backpropagation, while fixed image embeddings are used for prediction.
  • Tuning for downstream tasks: A text-embedding tuning strategy avoids image-embedding overfitting because testing image embeddings differ from training embeddings.The paper also notes that image-embedding diversity is not conducive to learning stable task-level information.

5. Experiment

Experiments evaluate TaskRes across few-shot recognition, domain generalization, classifier-component ablations, scaling choices, visual backbones, and transfer-difficulty analysis. TaskRes consistently improves the pre-trained base and performs strongly across the tested settings.

  • Experimental setup: The evaluation uses 1/2/4/8/16 shots per class across 11 diverse benchmark datasets, with results averaged over three random seeds.The datasets cover generic and fine-grained objects, actions, scenes, textures, and satellite images.
  • Few-shot performance: Ours and Ours* achieve SOTA or SOTA-comparable performance across all few-shot settings and significantly surpass Zero-shot CLIP.The comparison includes CoOp, CLIP-Adapter, and Tip-Adapter-F.
  • Domain generalization: TaskRes models consistently outperform Zero-shot CLIP, Linear Probe CLIP, and CoOp on ImageNet-to-variant domain generalization across multiple visual backbones.The target variants are ImageNet-V2, ImageNet-Sketch, ImageNet-A, and ImageNet-R.
  • Ablation study: The ablation supports explicit prior preservation and prior-independent tuning: directly adapting the classifier performs worse than the original, adapter-style, and TaskRes classifiers.The comparison constructs target classifiers from the pre-trained classifier, adapter module, and task residual on ImageNet.
  • Task residual effectiveness: 16.14% accuracy gain is obtained in the 16-shot setting when TaskRes is added to the regular base classifier averaged over 11 datasets.TaskRes also improves the enhanced base classifier, indicating benefits for both base-classifier versions.

6. Conclusion, Limitation and Future Work

The paper concludes that TaskRes separates an undamaged prior-rich base classifier from a prior-independent task residual, whose magnitude relates to transfer difficulty. It also reports negative transfer in specific settings and identifies heuristic transfer-difficulty assessment as a limitation.

  • Conclusion: TaskRes decouples the classifier into an undamaged base classifier containing prior knowledge and a task residual for exploring task-specific knowledge.The paper presents this decoupling as the basis of its tuning approach.
  • Conclusion: Learned task-residual magnitude is highly related to the difficulty of transferring pre-trained VLMs to a target task.The paper suggests this relation may motivate modeling task-to-task transfer difficulty for efficient transfer learning.
  • Limitations: Negative transfer occurs on OxfordPets at 1 shot and Food101 at 1/2/4/8 shots.The authors conjecture that these cases combine high relative transfer difficulty with fair zero-shot CLIP precision.
  • Future work: The transfer-difficulty assessment used in this work is heuristic.The paper calls for more precise and reliable transfer-difficulty metrics and broader distribution analyses.

A. Summary of Datasets

The experiments use diverse few-shot and domain-generalization benchmarks spanning multiple visual-recognition settings. Supplementary results provide full numerical comparisons and report the best averaged performance across shot settings and datasets.

  • Generalization datasets: Domain-generalization evaluation trains on ImageNet and tests on ImageNet-V2, ImageNet-Sketch, ImageNet-A, and ImageNet-R, which have substantially different data distributions.ImageNet-V2 and ImageNet-Sketch share ImageNet’s label space; ImageNet-A and ImageNet-R use sub-spaces.
  • Supplementary results: The supplementary full numerical results report the best averaged performance across all shot settings and datasets, with averaged gains from 5.08% to 16.14%.Tip-Adapter-F was re-run and evaluated at the end of training for a fair comparison.

B.2. Training Efficiency

TaskRes is designed for efficient training by avoiding additional network modules and repeated text-encoder execution. The reported timing is substantially lower than for prompt tuning and slightly lower than for adapter-style tuning.

  • Training efficiency: 11 minutes is the reported TaskRes training time, compared with 121 minutes for prompt tuning and 16 minutes for adapter-style tuning.The efficiency is attributed to having no additional adapter modules and not running the text encoder every time.

B.3. Ablation Study

TaskRes consistently improves both regular and enhanced base classifiers across few-shot settings, while scaling-factor sensitivity is generally limited and dataset-dependent.

  • TaskRes effectiveness: 16.14% accuracy gain occurs when TaskRes equips the regular base classifier in the 16-shot setting.Across 1-, 2-, 4-, 8-, and 16-shot settings, gains are 5.08%, 8.06%, 10.65%, 13.80%, and 16.14%, respectively.
  • TaskRes effectiveness: 3.17% to 5.84% accuracy gains remain for the enhanced base classifier across the five few-shot settings.The gains are 3.17%, 4.73%, 5.84%, 3.44%, and 2.69% from 1- through 16-shot settings.
  • Scaling factor: 2.90% accuracy gain is achieved even with scaling factor α = 0.1, while α ∈[0.3, 1] generally yields insensitive performance.On average, α = 0.5 performs well, but the best value differs across datasets.
  • Scaling factor: Learned α attains the most favorable average result by adaptively selecting the scaling factor.Larger α performs better on Flower102 and EuroSAT, whereas smaller α is better for OxfordPets and Food101.

B.4. Learned Task Residual

Learned task-residual magnitudes track transfer difficulty and increase with more shots, while their variance decreases; TaskRes also changes fewer correct predictions into errors than competing methods.

  • Residual magnitude and difficulty: Task-residual magnitudes positively correlate with CLIP’s relative transfer difficulty across all shot settings.This indicates that the residual supplements old knowledge according to task difficulty.
  • Residual magnitude and shots: Mean and median residual magnitudes across 11 datasets tend to increase as the number of shots increases.The trend indicates that more downstream samples allow TaskRes to explore more task-specific knowledge.
  • Residual magnitude and shots: Residual-magnitude variance decreases with more shots, as shown by shrinking line shadows.The paper interprets this as task-specific knowledge being captured with less variance.
  • Decision-boundary preservation: TaskRes records 3037/1702 Wrong2Right/Right2Wrong images, compared with 4161/4599 for CoOp and 3542/2925 for CLIP-Adapter.These results are presented as evidence that TaskRes more effectively preserves pre-trained decision boundaries.

C. Discussion

TaskRes differs from adapter and prompt-tuning approaches by adding prior-independent residual parameters directly to the classifier, supporting flexible task-specific learning and strong performance on specialized tasks.

  • TaskRes versus adapters: TaskRes adds a learnable bias independent of old features, unlike CLIP-Adapter’s feature-dependent transformation.This design decouples task-specific learning from pre-trained features.
  • TaskRes versus adapters: 62.93% accuracy for TaskRes exceeds 61.27% for the nonlinear adapter and 60.93% and 60.90% for linear adapters with and without bias.The comparison uses 4-shot ImageNet and supports prior-independent learnable parameters as the key distinction.
  • TaskRes versus Tip-Adapter: TaskRes tunes classifier weights with prior-independent residuals, whereas Tip-Adapter(-F) adjusts classification results through training-set feature retrieval or matching.The methods therefore operate from different perspectives: classifier weights versus produced logits.
  • Task specialization: TaskRes improves specialized tasks by 7.85% ∼ 23.71% with 1-shot data and 25.25% ∼45.01% with 16-shot data.Examples include EuroSAT, DTD, and Flowers102, where more specialized knowledge is needed.
  • Scope: TaskRes’s residual-weight concept is presented as potentially applicable beyond CLIP to other vision-language, vision, and language models.The experiments themselves are conducted using CLIP.
Loading 2211.10277v2…