PaperlayerView exact revision on arXiv

Source-linked AI summary

VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks

Yi-Lin Sung, Jaemin Cho, Mohit Bansal

arXiv:2112.06825v2cs.CVcs.AIcs.CLcs.LG

TL;DR

Rapidly growing V&L models make full fine-tuning increasingly impractical, motivating more parameter-efficient transfer methods. The paper benchmarks adapter-based techniques in a unified multitask setup and finds that shared-weight adapters can match full fine-tuning while updating only a small fraction of parameters.

  • Problem

    Rapid model growth makes storing and fine-tuning entire V&L models costly, while adapter methods had not been evaluated on challenging downstream V&L tasks such as question answering and captioning.

  • Method

    The paper applies Adapter, Hyperformer, and Compacter methods to CLIP-BART and CLIP-T5 in unified multitask experiments across image-text and video-text benchmarks, including shared adapter weights.

  • Results

    Shared-weight adapter training matches full fine-tuning while updating 4.18% of parameters for image-text tasks and 3.39% for video-text tasks.

  • Takeaways & Limitations

    A single shared-weight vanilla adapter provides the best reported combination of accuracy, efficiency, and simplicity across the studied tasks.

  • Takeaways & Limitations

    Results may not generalize to other tasks because architectures and task data distributions require different hyper-parameters, and only three adapter variants were tested.

Abstract

from arXiv · show

Recently, fine-tuning language models pre-trained on large text corpora have provided huge improvements on vision-and-language (V&L) tasks as well as on pure language tasks. However, fine-tuning the entire parameter set of pre-trained models becomes impractical since the model size is growing rapidly. Hence, in this paper, we introduce adapter-based parameter-efficient transfer learning techniques to V&L models such as VL-BART and VLT5. We evaluate our methods in a unified multi-task setup on both image-text and video-text benchmarks. For the image-text tasks, we use four diverse V&L datasets: VQAv2, GQA, NLVR2 , and MSCOCO image captioning. For video-text tasks, we use TVQA, How2QA, TVC, and YC2C. With careful training and thorough experiments, we benchmark three popular adapter-based methods (Adapter, Hyperformer, Compacter) against the standard full fine-tuning and the recently proposed prompt-tuning approach. We also enhance the efficiency and performance of adapters by sharing their weights to attain knowledge across tasks. Our results demonstrate that training the adapter with the weight-sharing technique (4.18% of total parameters for image-text tasks and 3.39% for video-text tasks) can match the performance of fine-tuning the entire model. Lastly, we present a comprehensive analysis including the combination of adapter and task-specific prompts and the impact of V&L pre-training on adapters. Our code is available at: https://github.com/ylsung/VL_adapter.

1. Introduction

The paper targets the memory and storage burden of full V&L-model fine-tuning by applying parameter-efficient adapters in a unified multitask setting. Across diverse image-text and video-text benchmarks, shared-weight adapters approach full fine-tuning while updating a small parameter fraction.

  • Motivation: 700 GB of memory would be required to store all parameters of a 175B-parameter GPT-3 backbone.The paper presents this as an example of the storage cost created by rapidly growing V&L models.
  • Approach: Adapters add small modules to intermediate model layers, enabling downstream learning by updating only a small parameter subset.The paper describes adapters as modules inserted into intermediate layers and contrasts them with updating all model parameters.
  • Evaluation: The study benchmarks Adapter, Hyperformer, Compacter, and prompt-based methods in a unified multitask setup using encoder-decoder V&L models.BART and T5 generate outputs for diverse tasks without task-specific architectures.
  • Evaluation: The experiments cover VQAv2, GQA, NLVR2, MSCOCO captioning, TVQA, How2QA, TVC, and YC2C.These benchmarks span both image-text and video-text V&L tasks.
  • Results: 4.18% of parameters for image-text tasks and 3.39% for video-text tasks can match full fine-tuning with weight-shared adapter training.The paper also reports that a single shared adapter offers the best accuracy-efficiency trade-off among its adapter configurations.

2. Related Work

Related work establishes pretraining and encoder-decoder modeling as foundations for V&L systems, while parameter-efficient training reduces the cost of adapting increasingly large models. The paper positions its approach among adapters, prompts, sparse updates, and low-rank methods, while freezing CLIP and tuning the language model for efficient V&L transfer.

  • Language Model Pre-training: BART and T5 are selected as text-generative encoder-decoder models because prior studies reported their effectiveness.The paper uses them to represent multiple V&L tasks as text generation.
  • V&L Pre-training: V&L pretraining commonly combines vision encoders with language models and trains them on large-scale multimodal data before downstream fine-tuning.Prior systems use architectures such as ViLBERT, UNITER, LXMERT, and encoder-decoder V&L models.
  • V&L Pre-training: The paper freezes CLIP and focuses on efficient language-model tuning because competitive downstream V&L performance requires tuning the language model.It evaluates training and freezing combinations for CLIP and BART before focusing on adapter methods.
  • Parameter-Efficient Training: Parameter-efficient training includes adding new parameters, sparsely updating existing parameters, and using low-rank factorization.The related-work taxonomy distinguishes these three broad method types.

3. Methods

The paper applies parameter-efficient adapter methods to a unified encoder-decoder framework for diverse vision-and-language tasks. It compares adapter variants, introduces cross-task weight sharing, and freezes CLIP to improve the performance–efficiency trade-off.

  • Unified Framework for V&L Tasks: The unified model combines CLIP with BART or T5 and formulates image- and video-text tasks as text generation.Visual and sentence representations are concatenated for the encoder-decoder model, whose decoder generates the text label.
  • Adapters for V&L Models: Adapters insert small downsampling–upsampling subnetworks with residual connections after attention and feed-forward layers, updating only a small parameter subset.Their added parameter complexity is O(d_id), typically 2–3% of the model.
  • Adapter Variants: Hyperformer generates adapter weights from shared task and layer embeddings, while Compacter reduces parameters through hypercomplex multiplication, sharing, and low-rank decomposition.Hyperformer must remain smaller than independently trained adapters to provide a memory benefit.
  • Weight Sharing: Sharing adapter weights across tasks enables cross-task information transfer while retaining task-specific parameters, including half-shared and single-adapter configurations.The paper evaluates shared upsampling or downsampling layers and the extreme case of one adapter set for all tasks.
  • Where to Add Adapters?: The final design freezes CLIP and adds adapters to the language model because freezing CLIP provides a better performance–parameter-efficiency trade-off.The setup always updates the visual projection layer and language-model layer normalizations, while freezing the output layer.

4. Experimental Setup

The experiments evaluate the unified framework across four image-text datasets and four video-text tasks. The setup uses CLIP-based visual inputs, subtitles for videos, and an encoder-decoder language model.

  • Datasets: Image-text experiments cover VQAv2 and GQA for visual question answering, NLVR2 for visual reasoning, and MSCOCO for image captioning.These are four diverse vision-and-language datasets.
  • Datasets: Video-text experiments use TVQA and How2QA for video question answering, plus TVC and YC2C for video captioning.All four video tasks come from the VALUE benchmark.
  • Architecture Details: Video inputs use CLIP ViT-B/32 features from one uniformly sampled frame per second, concatenate sampled-frame outputs, and cap visual input length at 64.Subtitles are included as additional video information.

5. Results and Analysis

The experiments compare parameter-efficient methods with full fine-tuning across image-text and video-text tasks, while analyzing adapter architectures, visual representations, freezing choices, and trainable components. Weight sharing and the Single Adapter provide strong accuracy-efficiency trade-offs, including parity with full fine-tuning in several settings.

  • Multi-Task Parameter-Efficient Fine-Tuning: 77.6 vs. 76.7 shows a small improvement from CLIP features over R-CNN features despite CLIP using smaller 224 × 224 images.The comparison is between CLIP-BART and VL-BART without V&L pre-training, supporting the effectiveness of pre-trained cross-modality features.
  • Multi-Task Parameter-Efficient Fine-Tuning: 77.4 vs. 77.6 shows Single Adapter’s performance is as competitive as full fine-tuning on the reported image-text comparison.Half-shared Adapters perform on par with Multiple Adapters using fewer parameters, while the main boost comes from the smaller NLVR2 dataset.
  • Multi-Task Parameter-Efficient Fine-Tuning: 4.18% vs. 5.79% shows Single Adapter is more parameter-efficient than Hyperformer, while 77.4 vs. 76.4 shows higher reported effectiveness.The authors suggest hyper-network optimization difficulty may contribute to Hyperformer’s outcome; Compacter does not stand out in these experiments.
  • Training or Freezing Visual Encoder: Freezing CLIP improves the performance-parameter-efficiency trade-off because adding CLIP training yields only a 1% improvement.Freezing CLIP is therefore used consistently to save memory, while freezing BART and tuning only CLIP reaches 39.4 accuracy, far below updating BART.
  • Ablations and Analysis: Updating only the visual projection layer is insufficient, while adding layer normalization improves accuracy but remains well behind full fine-tuning.The finding differs from prior image-classification results where updating layer normalization was comparable to or better than training adapters inside CLIP.

6. Discussion and Conclusion

The paper evaluates adapter-based parameter-efficient training for challenging image-text and video-text tasks, finding that shared-weight adapters can match full fine-tuning with far fewer updated parameters.

  • Discussion and Conclusion: The study benchmarks Adapter, Hyperformer, and Compacter in a unified multitask setup across challenging image-text and video-text tasks.It also analyzes trainable modules, task-specific prompts, and the effect of V&L pre-training.
  • Discussion and Conclusion: The experiments do not guarantee generalization to other tasks because architectures have different best hyper-parameters and task data distributions vary.The study evaluates four V&L tasks using CLIP-BART and CLIP-T5.
  • Discussion and Conclusion: The three evaluated adapter variants do not represent all adapter-based approaches.This limits how broadly the reported findings characterize adapter methods.

A. Details for Prompt-tuning

Prompt-tuning adapts a frozen language model by learning input prompt parameters transformed through a three-layer network, with prompt length constrained by attention memory costs.

  • Details for Prompt-tuning: Prompt-tuning adds trainable parameters to encoder inputs without changing the model.The trainable parameters include prompt embeddings and a three-layer neural network.
  • Details for Prompt-tuning: The prompt network uses an embedding layer followed by downsampling and upsampling layers to transform prompt embeddings to the language model’s dimension.Its parameters are θE ∈ R^Np×di, θD ∈ R^di×d, and θU ∈ R^d×di.
  • Details for Prompt-tuning: Prompt embeddings are combined with vision and sentence embeddings before being fed into the model and trained by backpropagation.Tanh is used as the activation function.
  • Details for Prompt-tuning: Longer prompts increase memory usage significantly because the attention layer has quadratic cost in input length.The prompt length is maximized to use approximately the same 40 GB memory as adapter-based approaches.

B. Details for LoRA

LoRA learns low-rank weight updates instead of modifying inputs or network modules, and unlike adapters or prompt-tuning it adds no inference overhead after merging updates.

  • Details for LoRA: LoRA learns two low-rank matrices, A and B, to approximate updates to an initial layer weight θ.The low-rank dimension d is much smaller than the input and output dimensions.
  • Details for LoRA: The adapted layer output is written as fθ+AB(h).This expresses the learned update as a low-rank additive modification to the original weight.
  • Details for LoRA: LoRA applies the low-rank update to attention layers and also updates model bias terms.The cited description excludes feed-forward layers.
  • Details for LoRA: LoRA has no extra inference overhead because its updates can be added back to the model after training.Adapters and prompt-tuning instead modify the network or inputs and cause extra inference computation.

C. Hyper-parameter Search

The hyper-parameter search combines a small learning-rate sweep with configuration-specific searches for adapter, Hyperformer, Compacter, and prompt-tuning parameters.

  • Hyper-parameter Search: Learning rates are searched over {1 × 10^-4, 3 × 10^-4, 1 × 10^-3} for each hyper-parameter configuration.A randomly selected configuration is searched first, and its learning rate is reused when results remain stable.
  • Hyper-parameter Search: Additional learning-rate searches are performed when results diverge for some tasks.NLVR2 can become very low at certain learning rates.
  • Hyper-parameter Search: Adapter searches vary hidden dimension d and whether half-shared variants share upsampling or downsampling layers.Hyperformer searches the projected hidden dimension de.
  • Hyper-parameter Search: Compacter searches the number of Kronecker products k, hidden dimension d, weight sharing, and low-rank parameterization, while prompt-tuning tunes dm.

C.1. CLIP-BART Hyper-parameter Search

Table 11 reports the hyper-parameter search for CLIP-BART, while Table 13 lists the final configurations used in the main paper. The Single and Multiple approaches share hyper-parameters for consistency, even when another setting performs better for Multiple Prompts.

  • Table 11 reports CLIP-BART hyper-parameter search results, with final main-paper configurations shown in bold and listed in Table 13.The selected configurations are presented across both tables.
  • The Single and Multiple approaches use the same hyper-parameters despite their potentially different best settings.This shared configuration is an explicit consistency choice.
  • Multiple Prompts perform better with d_m = 100, but both Multiple Prompts and Single Prompt use d_m = 800.The paper identifies these as the J and K rows in Table 11.

C.2. CLIP-T5 Hyper-parameter Search

Table 12 reports CLIP-T5 hyper-parameter search results, and Table 13 gives the final configurations for each parameter-efficient method. Compacter is distinctive in T5 because it can perform similarly to Single Adapter with fewer parameters.

  • Table 12 reports CLIP-T5 hyper-parameter search results, while Table 13 lists final configurations for each method.The tables cover the parameter-efficient training approaches evaluated in this search.
  • Compacter can perform similarly to Single Adapter in T5 while using fewer parameters.The paper attributes this contrast to Compacter’s validation on T5 in prior work.
  • The Compacter result is identified as F.1 in Table 12, while the comparable Single Adapter result is identified as C.2.These labels specify the configurations being compared.

D. Learderboard Results of the Test-dev split for VQA

The appendix reports test-dev VQA results and states that their trend remains similar to the Karpathy-split evaluation. Single Adapter remains the best parameter-efficient training method, while the tables document the evaluation and configuration contexts.

  • Test-dev VQA results are reported for all approaches in Table 14 alongside the main Karpathy-split evaluation.The appendix uses the test-dev split as an additional VQA evaluation setting.
  • The test-dev trend remains similar to the Karpathy-split trend.This comparison is stated directly in the accompanying discussion.
  • Single Adapter performs best among the parameter-efficient training methods on the reported test-dev VQA results.The passage identifies this method as the strongest parameter-efficient approach.
  • Table 12 describes multi-task CLIP-T5 evaluation on VQA, GQA, NLVR2, and COCO Caption across adapter-based approaches and hyper-parameters.The table notes that V&L pre-training is not used for every model and that NLVR2 results may depend on learning-rate choices.
  • Table 13 lists the best hyperparameter configurations for the evaluated parameter-efficient training approaches.It provides configuration information complementary to the reported results.
Loading 2112.06825v2…