VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks

Yi-Lin Sung; Jaemin Cho; Mohit Bansal

Source-linked AI summary

VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks

Yi-Lin Sung, Jaemin Cho, Mohit Bansal

arXiv:2112.06825v2Dec 13, 2021cs.CVcs.AIcs.CLcs.LG

TL;DR

Rapidly growing V&L models make full fine-tuning increasingly impractical, motivating more parameter-efficient transfer methods. The paper benchmarks adapter-based techniques in a unified multitask setup and finds that shared-weight adapters can match full fine-tuning while updating only a small fraction of parameters.

Problem
Rapid model growth makes storing and fine-tuning entire V&L models costly, while adapter methods had not been evaluated on challenging downstream V&L tasks such as question answering and captioning.
Source: §1 · pp. 1–2
Method
The paper applies Adapter, Hyperformer, and Compacter methods to CLIP-BART and CLIP-T5 in unified multitask experiments across image-text and video-text benchmarks, including shared adapter weights.
Source: §1 · p. 2, §3.2 · pp. 4, 6
Results
Shared-weight adapter training matches full fine-tuning while updating 4.18% of parameters for image-text tasks and 3.39% for video-text tasks.
Source: §1 · p. 2
Takeaways & Limitations
A single shared-weight vanilla adapter provides the best reported combination of accuracy, efficiency, and simplicity across the studied tasks.
Source: §1 · p. 2, §6 · p. 10
Results may not generalize to other tasks because architectures and task data distributions require different hyper-parameters, and only three adapter variants were tested.
Source: §6 · p. 10

Abstract

from arXiv · show

Recently, fine-tuning language models pre-trained on large text corpora have provided huge improvements on vision-and-language (V&L) tasks as well as on pure language tasks. However, fine-tuning the entire parameter set of pre-trained models becomes impractical since the model size is growing rapidly. Hence, in this paper, we introduce adapter-based parameter-efficient transfer learning techniques to V&L models such as VL-BART and VLT5. We evaluate our methods in a unified multi-task setup on both image-text and video-text benchmarks. For the image-text tasks, we use four diverse V&L datasets: VQAv2, GQA, NLVR2 , and MSCOCO image captioning. For video-text tasks, we use TVQA, How2QA, TVC, and YC2C. With careful training and thorough experiments, we benchmark three popular adapter-based methods (Adapter, Hyperformer, Compacter) against the standard full fine-tuning and the recently proposed prompt-tuning approach. We also enhance the efficiency and performance of adapters by sharing their weights to attain knowledge across tasks. Our results demonstrate that training the adapter with the weight-sharing technique (4.18% of total parameters for image-text tasks and 3.39% for video-text tasks) can match the performance of fine-tuning the entire model. Lastly, we present a comprehensive analysis including the combination of adapter and task-specific prompts and the impact of V&L pre-training on adapters. Our code is available at: https://github.com/ylsung/VL_adapter.

1. Introduction

The paper targets the memory and storage burden of full V&L-model fine-tuning by applying parameter-efficient adapters in a unified multitask setting. Across diverse image-text and video-text benchmarks, shared-weight adapters approach full fine-tuning while updating a small parameter fraction.

Motivation

700 GB of memory would be required to store all parameters of a 175B-parameter GPT-3 backbone.The paper presents this as an example of the storage cost created by rapidly growing V&L models.Source: §1 · p. 1

Approach

Adapters add small modules to intermediate model layers, enabling downstream learning by updating only a small parameter subset.The paper describes adapters as modules inserted into intermediate layers and contrasts them with updating all model parameters.Source: §1 · p. 1

Evaluation

The study benchmarks Adapter, Hyperformer, Compacter, and prompt-based methods in a unified multitask setup using encoder-decoder V&L models.BART and T5 generate outputs for diverse tasks without task-specific architectures.Source: §1 · p. 2
The experiments cover VQAv2, GQA, NLVR2, MSCOCO captioning, TVQA, How2QA, TVC, and YC2C.These benchmarks span both image-text and video-text V&L tasks.Source: §1 · p. 2

Results

4.18% of parameters for image-text tasks and 3.39% for video-text tasks can match full fine-tuning with weight-shared adapter training.The paper also reports that a single shared adapter offers the best accuracy-efficiency trade-off among its adapter configurations.Source: §1 · p. 2

2. Related Work

Related work establishes pretraining and encoder-decoder modeling as foundations for V&L systems, while parameter-efficient training reduces the cost of adapting increasingly large models. The paper positions its approach among adapters, prompts, sparse updates, and low-rank methods, while freezing CLIP and tuning the language model for efficient V&L transfer.

Language Model Pre-training

BART and T5 are selected as text-generative encoder-decoder models because prior studies reported their effectiveness.The paper uses them to represent multiple V&L tasks as text generation.Source: §2 · p. 3

V&L Pre-training

V&L pretraining commonly combines vision encoders with language models and trains them on large-scale multimodal data before downstream fine-tuning.Prior systems use architectures such as ViLBERT, UNITER, LXMERT, and encoder-decoder V&L models.Source: §2 · pp. 2–3
The paper freezes CLIP and focuses on efficient language-model tuning because competitive downstream V&L performance requires tuning the language model.It evaluates training and freezing combinations for CLIP and BART before focusing on adapter methods.Source: §2 · p. 3

Parameter-Efficient Training

Parameter-efficient training includes adding new parameters, sparsely updating existing parameters, and using low-rank factorization.The related-work taxonomy distinguishes these three broad method types.Source: §2 · p. 3

3. Methods

The paper applies parameter-efficient adapter methods to a unified encoder-decoder framework for diverse vision-and-language tasks. It compares adapter variants, introduces cross-task weight sharing, and freezes CLIP to improve the performance–efficiency trade-off.

Unified Framework for V&L Tasks

The unified model combines CLIP with BART or T5 and formulates image- and video-text tasks as text generation.Visual and sentence representations are concatenated for the encoder-decoder model, whose decoder generates the text label.Source: §3.1 · pp. 3–4

Adapters for V&L Models

Adapters insert small downsampling–upsampling subnetworks with residual connections after attention and feed-forward layers, updating only a small parameter subset.Their added parameter complexity is O(d_id), typically 2–3% of the model.Source: §3.2 · pp. 4–5

Adapter Variants

Hyperformer generates adapter weights from shared task and layer embeddings, while Compacter reduces parameters through hypercomplex multiplication, sharing, and low-rank decomposition.Hyperformer must remain smaller than independently trained adapters to provide a memory benefit.Source: §3.2 · p. 5

Weight Sharing

Sharing adapter weights across tasks enables cross-task information transfer while retaining task-specific parameters, including half-shared and single-adapter configurations.The paper evaluates shared upsampling or downsampling layers and the extreme case of one adapter set for all tasks.Source: §3.2 · pp. 5–6

Where to Add Adapters?

The final design freezes CLIP and adds adapters to the language model because freezing CLIP provides a better performance–parameter-efficiency trade-off.The setup always updates the visual projection layer and language-model layer normalizations, while freezing the output layer.Source: §3.2 · p. 6

4. Experimental Setup

The experiments evaluate the unified framework across four image-text datasets and four video-text tasks. The setup uses CLIP-based visual inputs, subtitles for videos, and an encoder-decoder language model.

Datasets

Image-text experiments cover VQAv2 and GQA for visual question answering, NLVR2 for visual reasoning, and MSCOCO for image captioning.These are four diverse vision-and-language datasets.Source: §4 · p. 6
Video-text experiments use TVQA and How2QA for video question answering, plus TVC and YC2C for video captioning.All four video tasks come from the VALUE benchmark.Source: §4 · p. 6

Architecture Details

Video inputs use CLIP ViT-B/32 features from one uniformly sampled frame per second, concatenate sampled-frame outputs, and cap visual input length at 64.Subtitles are included as additional video information.Source: §4 · p. 6

5. Results and Analysis

The experiments compare parameter-efficient methods with full fine-tuning across image-text and video-text tasks, while analyzing adapter architectures, visual representations, freezing choices, and trainable components. Weight sharing and the Single Adapter provide strong accuracy-efficiency trade-offs, including parity with full fine-tuning in several settings.

Multi-Task Parameter-Efficient Fine-Tuning

77.6 vs. 76.7 shows a small improvement from CLIP features over R-CNN features despite CLIP using smaller 224 × 224 images.The comparison is between CLIP-BART and VL-BART without V&L pre-training, supporting the effectiveness of pre-trained cross-modality features.Source: §5.1 · p. 7
77.4 vs. 77.6 shows Single Adapter’s performance is as competitive as full fine-tuning on the reported image-text comparison.Half-shared Adapters perform on par with Multiple Adapters using fewer parameters, while the main boost comes from the smaller NLVR2 dataset.Source: §5.1 · p. 7
4.18% vs. 5.79% shows Single Adapter is more parameter-efficient than Hyperformer, while 77.4 vs. 76.4 shows higher reported effectiveness.The authors suggest hyper-network optimization difficulty may contribute to Hyperformer’s outcome; Compacter does not stand out in these experiments.Source: §5.1 · p. 7

Training or Freezing Visual Encoder

Freezing CLIP improves the performance-parameter-efficiency trade-off because adding CLIP training yields only a 1% improvement.Freezing CLIP is therefore used consistently to save memory, while freezing BART and tuning only CLIP reaches 39.4 accuracy, far below updating BART.Source: §5.2 · p. 8

Ablations and Analysis

Updating only the visual projection layer is insufficient, while adding layer normalization improves accuracy but remains well behind full fine-tuning.The finding differs from prior image-classification results where updating layer normalization was comparable to or better than training adapters inside CLIP.Source: §5.3 · pp. 8–9

6. Discussion and Conclusion

The paper evaluates adapter-based parameter-efficient training for challenging image-text and video-text tasks, finding that shared-weight adapters can match full fine-tuning with far fewer updated parameters.

Discussion and Conclusion

The study benchmarks Adapter, Hyperformer, and Compacter in a unified multitask setup across challenging image-text and video-text tasks.It also analyzes trainable modules, task-specific prompts, and the effect of V&L pre-training.Source: §6 · p. 10
The experiments do not guarantee generalization to other tasks because architectures have different best hyper-parameters and task data distributions vary.The study evaluates four V&L tasks using CLIP-BART and CLIP-T5.Source: §6 · p. 10
The three evaluated adapter variants do not represent all adapter-based approaches.This limits how broadly the reported findings characterize adapter methods.Source: §6 · p. 10

A. Details for Prompt-tuning

Prompt-tuning adapts a frozen language model by learning input prompt parameters transformed through a three-layer network, with prompt length constrained by attention memory costs.

Details for Prompt-tuning

Prompt-tuning adds trainable parameters to encoder inputs without changing the model.The trainable parameters include prompt embeddings and a three-layer neural network.Source: A. Details for Prompt-tuning · p. 12
The prompt network uses an embedding layer followed by downsampling and upsampling layers to transform prompt embeddings to the language model’s dimension.Its parameters are θE ∈ R^Np×di, θD ∈ R^di×d, and θU ∈ R^d×di.Source: A. Details for Prompt-tuning · p. 12
Prompt embeddings are combined with vision and sentence embeddings before being fed into the model and trained by backpropagation.Tanh is used as the activation function.Source: A. Details for Prompt-tuning · p. 12
Longer prompts increase memory usage significantly because the attention layer has quadratic cost in input length.The prompt length is maximized to use approximately the same 40 GB memory as adapter-based approaches.Source: A. Details for Prompt-tuning · p. 13

B. Details for LoRA

LoRA learns low-rank weight updates instead of modifying inputs or network modules, and unlike adapters or prompt-tuning it adds no inference overhead after merging updates.

Details for LoRA

LoRA learns two low-rank matrices, A and B, to approximate updates to an initial layer weight θ.The low-rank dimension d is much smaller than the input and output dimensions.Source: B. Details for LoRA · p. 13
The adapted layer output is written as fθ+AB(h).This expresses the learned update as a low-rank additive modification to the original weight.Source: B. Details for LoRA · p. 13
LoRA applies the low-rank update to attention layers and also updates model bias terms.The cited description excludes feed-forward layers.Source: B. Details for LoRA · p. 13
LoRA has no extra inference overhead because its updates can be added back to the model after training.Adapters and prompt-tuning instead modify the network or inputs and cause extra inference computation.Source: B. Details for LoRA · p. 13

C. Hyper-parameter Search

The hyper-parameter search combines a small learning-rate sweep with configuration-specific searches for adapter, Hyperformer, Compacter, and prompt-tuning parameters.

Hyper-parameter Search

Learning rates are searched over {1 × 10^-4, 3 × 10^-4, 1 × 10^-3} for each hyper-parameter configuration.A randomly selected configuration is searched first, and its learning rate is reused when results remain stable.Source: C. Hyper-parameter Search · p. 13
Additional learning-rate searches are performed when results diverge for some tasks.NLVR2 can become very low at certain learning rates.Source: C. Hyper-parameter Search · pp. 13–14
Adapter searches vary hidden dimension d and whether half-shared variants share upsampling or downsampling layers.Hyperformer searches the projected hidden dimension de.Source: C. Hyper-parameter Search · p. 14
Compacter searches the number of Kronecker products k, hidden dimension d, weight sharing, and low-rank parameterization, while prompt-tuning tunes dm.Source: C. Hyper-parameter Search · p. 14

C.1. CLIP-BART Hyper-parameter Search

Table 11 reports the hyper-parameter search for CLIP-BART, while Table 13 lists the final configurations used in the main paper. The Single and Multiple approaches share hyper-parameters for consistency, even when another setting performs better for Multiple Prompts.

Table 11 reports CLIP-BART hyper-parameter search results, with final main-paper configurations shown in bold and listed in Table 13.The selected configurations are presented across both tables.Source: §C.1 · p. 14
The Single and Multiple approaches use the same hyper-parameters despite their potentially different best settings.This shared configuration is an explicit consistency choice.Source: §C.1 · p. 14
Multiple Prompts perform better with d_m = 100, but both Multiple Prompts and Single Prompt use d_m = 800.The paper identifies these as the J and K rows in Table 11.Source: §C.1 · p. 14

C.2. CLIP-T5 Hyper-parameter Search

Table 12 reports CLIP-T5 hyper-parameter search results, and Table 13 gives the final configurations for each parameter-efficient method. Compacter is distinctive in T5 because it can perform similarly to Single Adapter with fewer parameters.

Table 12 reports CLIP-T5 hyper-parameter search results, while Table 13 lists final configurations for each method.The tables cover the parameter-efficient training approaches evaluated in this search.Source: §C.2 · p. 14
Compacter can perform similarly to Single Adapter in T5 while using fewer parameters.The paper attributes this contrast to Compacter’s validation on T5 in prior work.Source: §C.2 · p. 14
The Compacter result is identified as F.1 in Table 12, while the comparable Single Adapter result is identified as C.2.These labels specify the configurations being compared.Source: §C.2 · p. 14

D. Learderboard Results of the Test-dev split for VQA

The appendix reports test-dev VQA results and states that their trend remains similar to the Karpathy-split evaluation. Single Adapter remains the best parameter-efficient training method, while the tables document the evaluation and configuration contexts.

Test-dev VQA results are reported for all approaches in Table 14 alongside the main Karpathy-split evaluation.The appendix uses the test-dev split as an additional VQA evaluation setting.Source: D. Learderboard Results of the… · p. 14, Table 14 · p. 15
The test-dev trend remains similar to the Karpathy-split trend.This comparison is stated directly in the accompanying discussion.Source: D. Learderboard Results of the… · p. 14
Single Adapter performs best among the parameter-efficient training methods on the reported test-dev VQA results.The passage identifies this method as the strongest parameter-efficient approach.Source: D. Learderboard Results of the… · p. 14
Table 12 describes multi-task CLIP-T5 evaluation on VQA, GQA, NLVR2, and COCO Caption across adapter-based approaches and hyper-parameters.The table notes that V&L pre-training is not used for every model and that NLVR2 results may depend on learning-rate choices.Source: Table 12 · p. 15
Table 13 lists the best hyperparameter configurations for the evaluated parameter-efficient training approaches.It provides configuration information complementary to the reported results.Source: Table 13 · p. 15