VPGTrans: Transfer Visual Prompt Generator across LLMs

1 Introduction

• The study examines transfer between different LLM sizes and between different LLM types, including models with different architectures.It asks whether small-to-large transfer can reduce cost while maintaining comparable performance, and whether transfer can cross model types efficiently.Source: §1 · p. 2
• VPGTrans combines projector warm-up with vanilla fine-tuning to reduce adaptation steps and avoid performance drops from directly inheriting a VPG.The warm-up uses a well-initialized projector trained at 5 × lr.Source: §1 · p. 2
• VPGTrans transfers BLIP-2 ViT-G OPT2.7B→6.7B with less than 10% of original-training GPU hours and 10.7% of its training data, improving over BLIP-2 OPT6.7B by 2.9 on VQAv2 and 3.4 on OKVQA.For same-type LLM transfers, VPGTrans achieves at most 10 times acceleration.Source: §1 · p. 2
• For same-type small-to-large transfers, VPGTrans can avoid the performance drop of direct inheritance and achieve up to 10 times acceleration; different-type transfers can reach up to 5 times acceleration with comparable or better performance.Efficient transfer was not achieved between two small LLMs in the different-type setting, attributed to a large embedding-space gap.Source: §1 · pp. 2–3
• The paper identifies transfer-efficiency factors, proposes VPGTrans, and releases the VL-LLaMA and VL-Vicuna multimodal models.It also presents findings and possible explanations intended to inform further research.Source: §1 · p. 3

2 Preliminary

• The preliminary section introduces existing multimodal LLMs and describes the exploratory-analysis settings.The supplied passage states the section’s scope but gives no further substantive detail.Source: §2 · p. 3

2.1 MLLM

• MLLMs commonly combine a visual prompt generator, a projector, and a backbone LLM, which uses projected visual prompts to generate sentences.The VPG encodes images or videos into fixed-length soft prompts, and the linear projector aligns their dimensions to the LLM’s word embeddings.Source: §2.1 · p. 3
• Training commonly updates the VPG and projector while freezing the LLM or tuning only a small portion; this study focuses on transfer across frozen LLMs.PaLM-E updates the full VPG, while BLIP-2 and Flamingo tune their Q-Former and Resampler, respectively.Source: §2.1 · pp. 3–4

2.2 Experiment Settings

• Exploratory experiments use BLIP-2 components, 1.4 million COCO and SBU image-text pairs, OPT and FlanT5 transfer settings, and five captioning and VQA datasets.Performance is evaluated after pre-training without task-specific fine-tuning, using CIDEr for captioning and accuracy for VQA; training uses 10 epochs with model-family-specific batch sizes and numeric formats.Source: §2.2 · p. 4

3.1 Exploratory Analysis: Identifying Key Factors for VPG Transfer

• Projector warm-up avoids the observed performance drop and reduces the joint-training steps, but adds a costly training stage.The exploratory experiment freezes the VPG and LLM during warm-up; for a 6.7B LLM, freezing the VPG saves only 5.4% of A100 hours.Source: §3.1 · p. 4, Figure 4 · p. 5
• Inheriting an OPT125M VPG accelerates OPT350M training, especially on caption tasks, but directly continuing training can harm fine-grained VQA performance.The authors hypothesize that gradients through a randomly initialized projector may mislead the pretrained VPG.Source: §3.1 · p. 4, Figure 3 · p. 5
• Initializing the target projector by merging the source projector with a word-embedding converter reduces warm-up from 3 epochs to 2 in an OPT125M-to-OPT1.3B experiment.The converter maps source-model word embeddings to target-model embeddings; training it uses text-only caption data.Source: §3.1 · p. 5, §3.2 · pp. 5–6
• At 5 times the original learning rate, projector warm-up reaches a COCO caption CIDEr score of 133.1 after 1 epoch, exceeding the 3-epoch result without initialization.The comparison concerns linear-projector warm-up with and without word-embedding initialization.Source: §3.1 · p. 5, Table 2 · p. 5

3.2 A Two-stage VPG Transfer Framework

• The word converter is trained on text-only caption data to map source and target word embeddings, using cosine similarity as the loss.When tokenization differs, optimization uses overlapping tokens; the converter's linear mapping is merged with the source projector to initialize the target projector.Source: §3.2 · pp. 5–6
• VPGTrans first inherits the source VPG, initializes and warms up the target projector, then jointly fine-tunes the VPG and projector for n epochs.The framework's warm-up trains only the projector with the VPG and LLM frozen, for 1 epoch at 5 times the normal learning rate.Source: §3.2 · pp. 5–6, Figure 5 · p. 6

4.1 Experimental Settings

• The experiments cover four OPT sizes and three FlanT5 sizes, but instability limits the main reported FlanT5 transfers to base and XL.The section studies transfers across model sizes and evaluates five tasks without further task-specific fine-tuning.Source: §4.1 · p. 6, §4 · p. 6

4.2 VPGTrans Enabling Faster Convergence without Performance Drop under TaS

• VPGTrans consistently accelerates convergence, particularly on COCO caption and NoCaps, with evaluation immediately after pre-training and no task-specific fine-tuning.The figure compares methods across three TaS variants and five tasks.Source: §4.2 · p. 6, Figure 6 · p. 7
• At least 4 times speed-up occurs on 40% of transfer-task variants; OPT125M→2.7B gives 10 times speed-up on both caption datasets.Speed-up is measured by comparing epochs needed to exceed the best training-from-scratch performance.Source: §4.2 · p. 7, Table 3 · p. 7
• Compared with VPG inheritance in Fig. 6, VPGTrans achieves a higher speed-up rate on all caption-task variants and better performance on most variants, except OPT1.3B→2.7B.Source: §4.2 · p. 7
• For a fixed target OPT model, smaller source models generally yield faster convergence and better optimal performance; the authors hypothesize this reflects less damage to fine-grained VPG perception.The reported comparison finds OPT125M→2.7B better than OPT1.3B→2.7B on three VQA tasks; the perception analysis uses SPICE on COCO caption.Source: §4.2 · pp. 7–8, Figure 7 · p. 8

4.3 Scale-up Experiments

• 10.7 times speed-up with 10.7% training data for OPT2.7B→6.7B; VQAv2 and OKVQA improve by over 2 points.For FlanT5XL→XXL, the reported speed-up is 21.1 times with less than 5% training data; performance is the same on VQAv2, higher on GQA, and slightly lower on OKVQA.Source: §4.3 · p. 8, Table 4 · p. 8

5.1 Experimental Settings

• The experiments compare VPGTrans with training from scratch on five tasks, using OPT350M/FlanT5base and OPT2.7B/FlanT5XL transfers.Other experimental details follow the TaS setup.Source: §5.1 · p. 9

5.2 VPGTrans Enabling Faster Convergence only on Large LLMs under TaT

• For the OPT350M-to-FlanT5base transfer, VPGTrans converges more slowly than training from scratch in the initial epochs.The section reports that speed-up occurs for larger LLMs, including OPT2.7B and FlanT5XL.Source: §5.2 · p. 9
• At least 2 times speed-up is reported when transferring from FlanT5XL to OPT2.7B with VPGTrans.For the OPT350M-to-FlanT5base transfer, convergence is slower than training from scratch in the initial epochs.Source: §5.2 · p. 9

• VL-LLaMA outperforms the original BLIP-2 OPT6.7B on all datasets after receiving its VPG through VPGTrans.The transferred VPG was trained on BLIP-2 OPT6.7B.Source: §6 · p. 9
• VL-Vicuna shows better visual perception than MiniGPT-4 after transfer from BLIP-2 OPT6.7B and instruction tuning on MiniGPT-4 self-instruct data [62].The paper also reports ranking and ELO comparisons with seven other multimodal chatbots on the Multimodality Chatbot Arena platform.Source: §6 · pp. 9–10, Figure 9 · p. 10

7 Conclusion

• VPGTrans investigates VPG transfer across LLM sizes and types and uses a two-stage framework to reduce training costs while achieving comparable or better performance.The authors also demonstrate customizing new MLLMs through transfer from existing ones.Source: §7 · p. 10

A.1 Vision and Language Models

• In MLLMs, visual encoders convert vision signals into LLM-understandable soft prompts, called VPGs, while connecting and tuning a VPG remains costly.The paper cites BLIP-2's largest model as requiring over 600 A100 GPU hours on over 100M image-text pairs.Source: §A.1 · p. 15

A.2 Prompt Transfer

• VPG transfer uses soft prompts representing specific input content, unlike task-prompt transfer, which tunes prompts for particular tasks.The authors state that their VPG can be shared across LLMs with different embedding dimensions, leaving dimension changes to a simple projector.Source: §A.2 · p. 15

B Extended Findings in Exploratory Analysis

• Tuning only the projector falls short of training from scratch on COCO caption and NoCaps, and word-embedding-based projector initialization cannot replace warm-up training.The paper reports soft-prompt norms typically around 10 times the average word-embedding norm; using only linear projector initialization yields random performance.Source: B Extended Findings in Exploratory… · pp. 15–16, Figure 10 · p. 16, Figure 11 · p. 16
• A 5 times normal learning rate crashes VPG training, while a 10 times rate lets projector warm-up converge without crashing but adds no acceleration.Source: B Extended Findings in Exploratory… · p. 16

C.1 VPGTrans Enabling Stable Training under TaS

• Lowering the stage-2 learning rate enables stable VPGTrans training on FlanT5large, whose training is otherwise unstable even at one-tenth its original learning rate.The paper plots the COCO caption performance curve for this setting.Source: §C.1 · p. 16

C.2 VPGTrans Enabling Training with Less Data under TaS

• Reducing TaS training data to COCO causes no obvious performance drop, but retaining the same number of SBU examples instead can hurt performance, especially captioning.The authors emphasize that retained data should be high quality.Source: §C.2 · pp. 16–17

C.3 Comparison between VPGTrans and VPG Inherit

• Over 74% of Transfer-Task variants favor VPGTrans over VPG inherit, and it also speeds up 69.2% of the conditions where it performs better.The comparison spans 7 TaS variants on 5 tasks; Table 6 reports speed-up rates.Source: §C.3 · p. 17, Table 6 · p. 17, Figure 13 · p. 18

C.4 Scale-up Experiment Implementation Details

• Stage 1 uses COCO and SBU for projector warm-up; stage 2 uses COCO, VG, and Laion-COCO, while VL-Vicuna also receives 3,439 MiniGPT-4 self-instruct pairs.Including VG and Laion-COCO in stage 1 showed no obvious difference, so the authors use COCO and SBU for simplicity.Source: §C.4 · p. 17

D Extended TaT Experiments

• Figure 14 presents a more complete version of Figure 8 for the extended TaT experiments.The section points back to the finding that transfer between two small LLMs is harder because their visual prompts have weaker linear transferability.Source: D Extended TaT Experiments · p. 19, §D.1 · p. 19

D.1 Linear Transfer Gap between Different LLM’s Visual Prompts

• 25.7 points separates linear transfer from the best result for FlanT5base → OPT350M on COCO captioning, versus 7.2 points for FlanT5XL → OPT2.7B.The authors leave systematic exploration of small-to-large TaT transfers, which must consider both VPG visual perception and transfer gap, for future work.Source: §D.1 · p. 19, Table 7 · p. 19

E Extended Results for MLLM Customization

• VL-Vicuna identifies three people and their roles where MiniGPT-4 mistakes them for two, and links Leonardo in an image to films including Titanic.Source: E Extended Results for MLLM… · p. 19

F Potential Impact and Limitations

• VPGTrans is designed to build new MLLMs with shorter training time and less training data, supporting transfers to larger or different-type LLMs.Source: F Potential Impact and Limitations · p. 19
• The approach relies on an already-aligned VPG, and its resulting MLLMs can still generate sentences based on falsely recognized visual facts.The authors cite VL-Vicuna as an example of this common content-generation problem.Source: F Potential Impact and Limitations · p. 19