Source-linked AI summary
Prompt Tuning for Generative Multimodal Pretrained Models
Hao Yang, Junyang Lin, An Yang, Peng Wang, Chang Zhou, Hongxia Yang
TL;DR
Finetuning large pretrained models is computationally costly, while prompt tuning has shown promise in language and vision but remained unexplored for generative multimodal models. This work evaluates prompt tuning on a unified sequence-to-sequence multimodal model, finding comparable performance with finetuning, stronger adversarial robustness, and slow convergence as a limitation.
Problem
Finetuning large pretrained models is computationally costly, and prompt tuning’s effectiveness for generative multimodal pretrained models had not been explored.
Source: §1 · pp. 1–2Method
The study implements prefix prompt tuning for generative multimodal pretrained models and evaluates it across understanding and generation tasks.
Source: §1 · p. 2Results
Takeaways & Limitations
Prompt length, prompt depth, and reparameterization influence performance, and prompt tuning may be an alternative to finetuning.
Source: §6 · p. 8Prompt tuning can require at least 40 epochs to approach its best performance on some datasets, including RefCOCO.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Prompt tuning has become a new paradigm for model tuning and it has demonstrated success in natural language pretraining and even vision pretraining. In this work, we explore the transfer of prompt tuning to multimodal pretraining, with a focus on generative multimodal pretrained models, instead of contrastive ones. Specifically, we implement prompt tuning on the unified sequence-to-sequence pretrained model adaptive to both understanding and generation tasks. Experimental results demonstrate that the light-weight prompt tuning can achieve comparable performance with finetuning and surpass other light-weight tuning methods. Besides, in comparison with finetuned models, the prompt-tuned models demonstrate improved robustness against adversarial attacks. We further figure out that experimental factors, including the prompt length, prompt depth, and reparameteratization, have great impacts on the model performance, and thus we empirically provide a recommendation for the setups of prompt tuning. Despite the observed advantages, we still find some limitations in prompt tuning, and we correspondingly point out the directions for future studies. Codes are available at \url{https://github.com/OFA-Sys/OFA}
1 Introduction
Prompt tuning offers a lower-cost alternative to finetuning, but its effectiveness for generative multimodal pretrained models remains unknown. This work applies prefix tuning to that setting and evaluates performance, robustness, and design choices.
1 Introduction
- Prompt tuning is unexplored for generative multimodal pretrained models, despite prior work on CLIP-like models and visual prompts for frozen language models.The study asks whether lightweight prompt tuning transfers effectively to generative multimodal models.Source: §1 · p. 2
- The study implements prefix tuning on generative multimodal pretrained models, updating roughly 1% of parameters and reducing computation costs such as memory relative to finetuning.Prefix tuning adds trainable prompt embeddings while leaving most pretrained parameters frozen.Source: §1 · p. 2
- Lightweight prompt tuning achieves performance comparable with finetuning across four multimodal tasks spanning understanding and generation.The paper also reports that prompt tuning surpasses other lightweight methods and shows improved robustness against adversarial attacks.Source: §1 · p. 2
- Prompt length, prompt depth, and reparameterization affect performance; the authors favor lengths above 20 tokens, often 64, and report no significant gains from reparameterization.Longer prompts can increase computation costs and degrade performance.Source: §1 · p. 2
- The authors identify slow convergence and training instability as limitations and point to faster, more stable tuning as a future direction.Some datasets require at least 40 epochs to reach nearly best performance, which can add computation costs.Source: §1 · p. 2
2 Related Work
Related work covers generative and contrastive multimodal pretraining alongside prompt-based learning. The paper positions prompt tuning for generative multimodal pretrained models as an open topic.
2 Related Work
- The review groups multimodal pretraining into generative and contrastive approaches and surveys prompt-based learning in NLP and cross-modal representation learning.It notes that encoder-decoder multimodal models support both understanding and generation tasks and can unify tasks as sequence-to-sequence learning.Source: §2 · pp. 2–3, §2.1 · p. 3
2.1 Multimodal Pretraining
- Encoder-decoder multimodal pretraining supports understanding and generation and can unify tasks in a sequence-to-sequence format for multitask pretraining.The authors connect this framework to their motivation for prompt tuning.Source: §2.1 · p. 3
- Contrastive multimodal pretraining, exemplified by CLIP, learns from image-text pairs and supports cross-modal retrieval and zero-shot classification with prompt ensembling.Prompt ensembling combines model outputs from multiple handcrafted prompts.Source: §2.1 · p. 3
2.2 Prompt-based Learning
- Prompt-based learning includes soft prompts and adapters for CLIP-like models, visual prompts for frozen language models, and Flamingo's in-context learning approach.The paper focuses on prompt tuning for generative multimodal pretrained models.Source: §2.2 · p. 3
3 Method
The method applies prompt tuning to OFA, a unified sequence-to-sequence multimodal pretrained model. It adds tunable prompts across Transformer layers while freezing pretrained parameters, then examines prompt design choices.
3.1 Preliminaries
- The method uses a unified sequence-to-sequence framework that handles understanding and generation tasks, implementing prompt tuning on OFA's Transformer encoder-decoder model.The overview places tunable prompt embeddings in each encoder and decoder layer.Source: §3.1 · pp. 3–4, Figure 1 · p. 4
- The study addresses downstream transfer of a pretrained multimodal model and leaves prompt learning during pretraining for future work..Source: §3.1 · p. 4
3.2 Prompt Tuning for Generative Multimodal Pretrained Models
- Prefix tuning adds trainable soft prompt embeddings at each encoder and decoder layer, prefixing them to that layer's input hidden states.Prompt length is a predefined hyperparameter, and prompts are added at Transformer layers.Source: §3.2 · p. 4
- During downstream tuning, only the added prompt embeddings are updated while the pretrained model parameters remain frozen, reducing the number of parameters that need tuning.The paper gives approximately 1% as an example of the tunable parameter share.Source: §3.2 · p. 4
- Table 1 covers RefCOCO, RefCOCO+, RefCOCOg, SNLI-VE, COCO image captioning, and VQA; prompt tuning trails finetuning at base size but is comparable at large size.The caption states this overall model-size pattern without providing table-cell values.Source: Table 1 · p. 5
- The method study varies prompt depth across the encoder, decoder, or both, and examines prompt length and reparameterization with an added encoder such as an MLP.Prompt depth is simplified to the choice of which module or modules receive prompt embeddings.Source: §3.2 · pp. 4–5
4 Experiments
Experiments test prompt tuning on multimodal understanding and generation tasks, comparing it with finetuning and other parameter-efficient methods. Analyses examine adversarial robustness and how prompt length, insertion depth, and reparameterization affect performance.
4.1 Datasets & Metrics
- Evaluation spans approximately 180M and 470M models across referring-expression comprehension, visual entailment, image captioning, and VQA, using task-specific metrics.Source: §4 · p. 5, §4.1 · p. 5
4.2 Experimental Results
- For large-size models, prompt tuning achieves comparable performance with finetuning, while it underperforms by larger margins on base-size models.For referring expression comprehension, the average gaps are 5.64 for base-size and 0.59 for large-size models; for VQA, the gaps are 3.63 for base-size and 2.17 on the large-size test-std set.Source: §4.2 · pp. 5–6
- On all downstream tasks, prompt tuning surpasses Adapter and BitFit in the large-size-model comparison.Source: Table 2 · p. 6, §4.2 · p. 6
4.3 Analyses
- Longer prompts generally improve downstream performance, but gains can diminish or reverse; 64 tokens is recommended for better average performance.Source: Figure 2 · p. 6, §4.3 · p. 7
- Under FGSM adversarial attack, prompt tuning consistently shows better robustness than finetuning across all tasks.The experiments use a one-step gradient computation on text and image embeddings.Source: Figure 3 · p. 7, §4.3 · p. 7
- Inserting prompts at every Transformer layer performs best but is less computation-efficient; encoder-only insertion outperforms decoder-only insertion across multiple tasks.Source: Table 3 · p. 7, §4.3 · p. 8
- MLP reparameterization improves some datasets but harms others, so neither it nor direct embedding updates is uniformly preferable.MLP brings consistent improvements on RefCOCO and RefCOCOg, but relatively negative impacts on SNLI-VE and VQA; direct updates are associated with unstable optimization and a slight performance drop.Source: Table 4 · p. 8, §4.3 · p. 8
5 Discussion
The discussion argues that prompt tuning is not yet a replacement for finetuning, despite its advantages. It identifies slow convergence and hyperparameter search as limitations while highlighting stronger robustness against adversarial attacks.
5 Discussion
- Prompt tuning still cannot replace finetuning, and the authors identify slow convergence and training instability as salient limitations.On some datasets such as RefCOCO, it takes at least 40 epochs to reach nearly best performance, potentially increasing computation costs.Source: §5 · p. 8
- Finding a suitable hyperparameter setup is difficult, and finetuning’s hyperparameter experience does not transfer to prompt tuning.The authors report that prompt tuning for generative multimodal pretrained models is less sensitive to hyperparameters than prompt tuning for pretrained language models.Source: §5 · p. 8
- Prompt tuning demonstrates significantly better robustness against adversarial attack.The authors recommend future work explore how to leverage this merit.Source: §5 · p. 8
6 Conclusion
The paper studies prompt tuning for generative multimodal pretrained models and reports comparable performance to finetuning with far fewer tunable parameters. It also reports advantages over other lightweight methods and improved robustness, while noting limitations that remain.
6 Conclusion
- Lightweight prompt tuning achieves comparable performance with finetuning using much fewer tunable parameters, e.g., 1%, and surpasses Adapter and BitFit.The paper evaluates prompt tuning across four multimodal downstream tasks spanning understanding and generation.Source: §6 · p. 8
- Prompt tuning demonstrates improved robustness against adversarial attack.Source: §6 · p. 8
- The analysis examines how prompt length, prompt depth, and reparameterization influence performance and provides empirical setup recommendations.Source: §6 · p. 8
- The authors identify slow convergence and training instabilities as limitations that future studies should address.Source: §6 · p. 8
A.1 Experimental Setups
The experimental setups cover multimodal understanding and generation tasks, including referring expression comprehension, visual entailment, image captioning, and visual question answering.
A.1 Experimental Setups
- Referring expression comprehension experiments use RefCOCO, RefCOCO+, and RefCOCOg, reporting Acc@0.5 on validation and test sets.Source: §A.1 · p. 12
- Visual entailment uses SNLI-VE and reports accuracy on dev and test sets; image captioning reports BLEU@4, METEOR, CIDEr, and SPICE on the Karpathy test split.Image captioning uses cross-entropy loss without further CIDEr optimization.Source: §A.1 · p. 12
- The resource-consumption comparison reports GPU-hours for finetuning and prompt tuning on large-size models.Source: Table 5 · p. 12
- Visual question answering experiments use VQA 2.0 and report the test-std score, with EMA at decay 0.9999 during finetuning.Source: §A.1 · p. 12
A.2 Additional Experimental Results
Additional experiments report an average-performance recommendation for prompt length and compare tuning efficiency and resource use. Prompt tuning is faster per sample, but its total resource advantage depends on the task.
A.2 Additional Experimental Results
- A prompt length of 64 achieves the best average performance across downstream tasks.Source: §A.2 · p. 12, Table 6 · p. 13
- Prompt tuning consistently processes 100 samples faster than finetuning and is around 2 times faster on the huge-size model.The comparison spans OFA base models from 93M to 930M parameters.Source: §A.2 · pp. 12–13
- Faster per-sample processing does not necessarily reduce computation resources: prompt tuning uses fewer GPU-hours for image captioning and VQA but more for other tasks.The authors report that prompt tuning can incur more carbon footprints on tasks less similar to pretraining tasks, especially those with less pretraining data.Source: §A.2 · p. 13