Source-linked AI summary
Reformulating Vision-Language Foundation Models and Datasets Towards Universal Multimodal Assistants
Tianyu Yu, Jinyi Hu, Yuan Yao, Haoye Zhang, Yue Zhao, Chongyi Wang, Shan Wang, Yinxv Pan, Jiao Xue, Dahai Li, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun
TL;DR
MLLM development faces challenges in aligning visual modules with LLMs and building instruction-tuning datasets that capture complementary image information. The paper proposes Muffin's pre-trained VLM bridge and UniMM-Chat's merged-annotation conversations; Muffin achieves state-of-the-art performance across a wide range of tasks.
Problem
MLLM architecture alignment and multimodal instruction-dataset construction remain under-explored, while existing architectures have capacity, computation, or extra-training drawbacks and prior data approaches can miss complementary annotations.
Source: §1 · p. 1Method
Muffin uses pre-trained VLMs as bridges to LLMs, while UniMM-Chat merges same-image annotations from different datasets to generate knowledge-intensive conversations.
Source: §1 · p. 2Results
Muffin achieves state-of-the-art performance on a wide range of tasks and on UniMM-Bench and the LLaVA test set, significantly surpassing the evaluated baselines.
Source: §1 · p. 2, §5.2 · p. 5Takeaways & Limitations
The authors conclude that Muffin with UniMM-Chat exhibits strong reasoning capabilities and abundant knowledge.
Source: §5.2 · p. 5
Abstract
from arXiv · showhide
Recent Multimodal Large Language Models (MLLMs) exhibit impressive abilities to perceive images and follow open-ended instructions. The capabilities of MLLMs depend on two crucial factors: the model architecture to facilitate the feature alignment of visual modules and large language models; the multimodal instruction tuning datasets for human instruction following. (i) For the model architecture, most existing models introduce an external bridge module to connect vision encoders with language models, which needs an additional feature-alignment pre-training. In this work, we discover that compact pre-trained vision language models can inherently serve as ``out-of-the-box'' bridges between vision and language. Based on this, we propose Muffin framework, which directly employs pre-trained vision-language models to act as providers of visual signals. (ii) For the multimodal instruction tuning datasets, existing methods omit the complementary relationship between different datasets and simply mix datasets from different tasks. Instead, we propose UniMM-Chat dataset which explores the complementarities of datasets to generate 1.1M high-quality and diverse multimodal instructions. We merge information describing the same image from diverse datasets and transforms it into more knowledge-intensive conversation data. Experimental results demonstrate the effectiveness of the Muffin framework and UniMM-Chat dataset. Muffin achieves state-of-the-art performance on a wide range of vision-language tasks, significantly surpassing state-of-the-art models like LLaVA and InstructBLIP. Our model and dataset are all accessible at https://github.com/thunlp/muffin.
1 Introduction
The paper frames MLLM development around aligning visual modules with language models and constructing instruction-tuning data. It proposes Muffin, which uses pre-trained VLMs as bridges, and UniMM-Chat, which merges complementary image annotations into richer conversations.
1 Introduction
- Existing MLLM architectures use linear projectors or visual feature re-samplers, with capacity, image-resolution computation, or additional-training drawbacks.The paper notes that projector feature-sequence length grows quadratically with input image resolution, while re-samplers require additional modality-alignment training.Source: §1 · pp. 1–2
- Muffin uses pre-trained VLMs as bridges between vision modules and LLMs, aiming to avoid capacity loss and additional alignment training.The authors report state-of-the-art performance among existing MLLMs.Source: §1 · p. 2
- Prior instruction-data approaches either retain limited response formats or convert isolated datasets into conversations without merging their complementary information.The authors associate this omission with knowledge scarcity in generated data.Source: §1 · p. 2
- UniMM-Chat merges annotations for the same image and uses ChatGPT to generate high-quality, knowledge-intensive conversations, yielding over 1.1M instructions.The paper also introduces UniMM-Bench for evaluating reasoning and world knowledge.Source: §1 · p. 2
2 Related Work
Related work covers pre-trained vision-language models, multimodal LLM architectures, and multimodal instruction-tuning datasets. The section contrasts their alignment approaches and instruction-following data strategies.
Vision Language Models
- Pre-trained VLMs align visual and text modalities through large-scale image-text training but generally lack human-instruction-following capabilities.The section mentions contrastive, masked-data-modeling, and image-text-matching objectives, alongside architectural work such as UNITER, VinVL, and VLMo.Source: §2 · p. 2
Multimodal Large Language Models
- MLLMs connect visual modules to pre-trained LLMs using bridges such as BLIP-2's Q-Former or LLaVA's linear projection.Related approaches include modified or instruction-aware Q-Former variants, and the section says Table 1 summarizes model structures.Source: §2 · p. 2
Multimodal Instruction Tuning Datasets
- Existing multimodal instruction datasets use rewritten descriptions, templated public datasets, or GPT-generated instructions, but single-dataset annotations may leave image knowledge incomplete.The section describes LLaVA's instructions as more diverse than MiniGPT-4's, while noting their knowledge scarcity.Source: §2 · p. 3
3 Muffin Framework
Muffin uses a pre-trained VLM to summarize visual inputs for an LLM, then trains the system in pre-training and multimodal instruction-tuning stages. Its architecture uses deeply fused query and image representations.
3.1 Architecture
- Muffin inserts trainable text-space query vectors and image inputs into a pre-trained VLM's text and visual channels, whose blocks deeply fuse the modalities.The final hidden states at the query positions represent the image and are projected into the LLM embedding space as prefix context.Source: §3.1 · p. 3, Figure 2 · p. 3
3.2 Pre-training
- During pre-training, Muffin aligns the VLM and LLM on image-text pairs while freezing the LLM and prefixing captions with randomly selected instructions.The stated objective is to maximize target-response probability given the image and instruction.Source: §3.2 · p. 3
3.3 Multimodal Instruction Tuning
- During multimodal instruction tuning, the LLM becomes trainable and the model learns to decode answer spans from conversational snippets.The paper uses the same objective as in pre-training.Source: §3.3 · p. 3
4 UniMM-Chat
UniMM-Chat combines annotations for shared images and uses ChatGPT to generate diverse, knowledge-intensive dialogues. The section also describes dataset statistics and UniMM-Bench, a benchmark for reasoning and world knowledge.
4 UniMM-Chat
- UniMM-Chat contains 1.1M diverse instructions built by combining complementary annotations and using ChatGPT to generate multi-turn dialogues for each image.The combined annotations provide richer image context for generating knowledge-intensive conversations.Source: §4 · pp. 3–4, Figure 3 · p. 4
4.1 Dataset Construction
- The construction process aggregates annotations from five COCO-sourced datasets, including question-answer pairs, AOKVQA rationales, and five COCO captions per image.The annotations are structured into prompts with human-written few-shot examples for ChatGPT to generate image-centered multi-turn dialogues.Source: §4.1 · p. 4
4.2 Dataset Statistics
- UniMM-Chat contains 117,238 dialogues averaging 9.89 turns each, and its seven most common question types and associated nouns or verbs cover 44% of instructions.The paper uses this distribution to illustrate the breadth of intents and formats.Source: §4.2 · p. 4, Figure 4 · p. 4
4.3 UniMM-Bench
- UniMM-Bench evaluates reasoning and world knowledge using questions drawn from existing vision-language benchmarks and GPT-4 scoring of model outputs.The described questions come from OKVQA, AOKVQA, GQA, and VQAv2; one hundred samples are taken from each test set.Source: §4.3 · pp. 4–5
5 Experiments
Experiments find that Muffin outperforms the baselines across both benchmark suites and human-evaluation criteria. Ablations and qualitative examples support the roles of UniMM-Chat and the chosen training settings.
5.1 Experimental Settings
- Evaluation uses GPT-4 scoring on UniMM-Bench and the LLaVA test set, while training combines LLaVA-Instruct-150K and UniMM-Chat.Pre-training uses 180M image-text pairs for 100K steps; instruction tuning lasts 3,200 steps.Source: §5.1 · p. 5
5.2 Main Results
- Muffin gains 5.1 points on average over LLaVA and 1.2 points when both use LLaVA-Instruct-150K for instruction tuning.Excluding UniMM-Chat leads to substantial drops across visual question-answering tasks.Source: §5.2 · p. 5
- Muffin achieves state-of-the-art performance on both UniMM-Bench and the LLaVA test set, significantly surpassing all baselines.Table 3 reports average scores over three trials.Source: §5.2 · p. 5, Table 3 · p. 6
5.3 Human Evaluation
- Muffin outperforms LLaVA, InstructBLIP, and MiniGPT-4 on helpfulness, correctness, and question-answer consistency in pairwise human evaluation.The evaluation used 80 UniMM-Bench samples, 20 LLaVA test samples, and six annotators; model names were hidden.Source: §5.3 · pp. 5–6, Figure 5 · p. 6
5.4 Ablation Results
- UniMM-Chat avoids the in-domain performance drop seen with UniMM-Chat-sep, while original datasets alone fail to establish a chat model on the LLaVA test set.UniMM-Chat-sep is constructed without merging annotations across datasets.Source: Table 4 · p. 6, §5.4 · p. 6
- Lowering image resolution to 224 for pre-training and 448 for instruction tuning drops UniMM-Bench performance; freezing the LLM limits performance across evaluated datasets.The authors attribute the resolution result to difficulty retaining detailed image information needed for complex multimodal tasks.Source: Table 5 · p. 6, §5.4 · pp. 6, 8
5.5 Qualitative Results
- Muffin answers examples requiring visual detail and broad knowledge more accurately than LLaVA and InstructBLIP, and gives a more helpful, detailed response in another example.Examples include identifying an airplane's country of origin and naming traditional Indian attire.Source: Figure 6 · p. 7, §5.5 · p. 8
6 Conclusion
The paper concludes that Muffin's pre-trained-VLM bridge and UniMM-Chat dataset are effective, with state-of-the-art performance across a wide range of tasks.
6 Conclusion
- Muffin directly uses pre-trained VLMs to bridge visual signals and LLMs, while UniMM-Chat merges same-image annotations into diverse multimodal instructions; experiments show state-of-the-art performance across a wide range of tasks.The authors plan to apply the framework and dataset to more VLM and LLM combinations.Source: §6 · p. 8
A Prompts
The appendix documents prompts for constructing UniMM-Chat, evaluating UniMM-Bench, and pre-training Muffin.
A Prompts
- The appendix lists prompts for UniMM-Chat construction, UniMM-Bench evaluation, and Muffin pre-training.It includes raw prompts and demonstrations for reproducibility.Source: A Prompts · p. 9
A.1 UniMM-Chat Construction Prompts
- The UniMM-Chat construction prompt appendix describes organizing human demonstrations with the raw generation prompt and reports question and answer length distributions.It also identifies the annotation-amassing prompt in Table 6 and dialogue-generation prompt in Table 7.Source: §A.1 · p. 9
- Table 6 provides the template used to amass annotations from different vision-language datasets.Source: Table 6 · p. 9
A.2 UniMM-Bench Evaluation Prompt
- The UniMM-Bench evaluation prompt includes ground-truth answers and related annotations to support GPT-4 scoring.The prompt includes human answers and confidences from VQAv2, plus rationales from AOKVQA.Source: §A.2 · pp. 9–10
A.1 UniMM-Chat Construction Prompts
- Table 7 contains the detailed prompt guiding ChatGPT to generate conversations.Source: Table 7 · p. 10
A.3 Pre-training Prompts
- Muffin pre-training uses prompts adopted from LLaVA, with the full caption-generation prompt list in Table 9.Source: §A.3 · p. 10
B UniMM-Chat-sep
UniMM-Chat-sep is a version of UniMM-Chat built without merging annotations across vision-language datasets.
B UniMM-Chat-sep
- UniMM-Chat-sep generates dialogues for each dataset separately; results indicate that combining datasets improves visual question-answering and visual chat performance, while simple mixing may miss their complementary information.COCO Caption uses the pre-training instructions because its caption text is already flexible in format.Source: B UniMM-Chat-sep · p. 10
C UniMM-Bench
UniMM-Bench is assembled from 400 validation samples across four VQA datasets, with images used in UniMM-Chat excluded to prevent data leakage. Its examples illustrate evaluation of reasoning and world-knowledge abilities.
C UniMM-Bench
- UniMM-Bench combines 100 samples each from OKVQA, GQA, and AOKVQA with 100 VQAv2 samples selected to retain question-type diversity.For VQAv2, two samples per question type were first selected, yielding 130 candidates before the final random selection.Source: C UniMM-Bench · p. 10
- Images in UniMM-Bench were excluded from the image set used to construct UniMM-Chat to prevent evaluation data leakage.Source: C UniMM-Bench · p. 10
- Examples from UniMM-Bench show that it evaluates model abilities in reasoning and world knowledge.Source: C UniMM-Bench · p. 10
D Human Evaluation
The appendix provides human-evaluation statistics comparing Muffin with baseline models and includes materials for UniMM-Bench evaluation and multimodal pre-training.
D Human Evaluation
- Figure 8 presents UniMM-Bench question examples alongside results generated by Muffin and other models.Source: Figure 8 · p. 11
- Table 8 contains the GPT-4 evaluation prompt used for UniMM-Bench.Source: Table 8 · p. 12
- Table 9 contains instructions for generating captions during multimodal pre-training.Source: Table 9 · p. 12
- Table 10 reports human-evaluation win/loss sample counts comparing Muffin with baseline models across different metrics.The text identifies the table as a detailed reference for human-evaluation results.Source: D Human Evaluation · p. 10, Table 10 · p. 12