Source-linked AI summary
Efficient Multimodal Learning from Data-centric Perspective
Muyang He, Yexin Liu, Boya Wu, Jianhao Yuan, Yueze Wang, Tiejun Huang, Bo Zhao
TL;DR
Large MLLMs provide strong visual reasoning but are expensive to train and deploy, while smaller backbones commonly lose performance. Bunny combines flexible lightweight backbones with curated training data and reports that Bunny-4B/8B outperforms state-of-the-art large MLLMs across multiple benchmarks.
Problem
MLLM deployment is hindered by high training and inference costs, while using smaller vision and language models often causes significant performance drops.
Method
Bunny combines plug-and-play lightweight vision and language backbones with curated data selection and a two-stage multimodal training procedure.
Results
Bunny-4B/8B outperforms state-of-the-art large MLLMs on multiple benchmarks.
Takeaways & Limitations
Bunny demonstrates that smaller MLLMs can achieve strong benchmark performance through flexible architecture and more informative training data.
Takeaways & Limitations
The current framework offers a limited set of backbone choices, with additional LLMs, vision encoders, and projectors left for future work.
Abstract
from arXiv · showhide
Multimodal Large Language Models (MLLMs) have demonstrated notable capabilities in general visual understanding and reasoning tasks. However, their deployment is hindered by substantial computational costs in both training and inference, limiting accessibility to the broader research and user communities. A straightforward solution is to leverage smaller pre-trained vision and language models, which inevitably cause significant performance drops. In this paper, we demonstrate the possibility of training a smaller but better MLLM with high-quality training data. Specifically, we introduce Bunny, a family of lightweight MLLMs with flexible vision and language backbones for efficient multimodal learning from selected training data. Experiments show that our Bunny-4B/8B outperforms the state-of-the-art large MLLMs on multiple benchmarks. We expect that this work can provide the community with a clean and flexible open-source tool for further research and development. The code, models, and data can be found in https://github.com/BAAI-DCAI/Bunny.
1 Introduction
MLLMs offer strong visual understanding but are costly to deploy, while shrinking their backbones often reduces performance. Bunny addresses this trade-off by pairing lightweight models with more informative, curated training data.
- MLLM deployment is limited by substantial training and inference costs, despite strong visual understanding and reasoning capabilities.
- Lightweight MLLMs reduce model size but often sacrifice performance because smaller models have less capacity.
- Bunny uses curated data selection to compensate for reduced model size and train smaller but better MLLMs.
- Bunny-4B/8B outperforms state-of-the-art small MLLMs and even larger models on popular benchmarks.
2 Related Work
Related work progresses from large multimodal fusion systems toward smaller MLLMs designed for lower-cost and faster deployment. Existing lightweight approaches combine compact backbones with architectural or training optimizations.
- MLLM research builds unified cross-modality understanding by fusing visual tokens with pre-trained language models.
- Miniaturized MLLMs target lower-cost and faster deployment by using small language models and vision components.
- Lightweight MLLM studies also optimize model architecture and training recipes to offset reduced model size.
3 Bunny: A Family of Lightweight Multimodal Models
Bunny is a modular lightweight MLLM framework combining selectable vision and language backbones through a cross-modality projector. It uses condensed, curated data and a two-stage training procedure to support efficient multimodal learning.
- Architecture and Backbone: Bunny combines an LLM backbone, vision encoder, and cross-modality projector in a flexible modular architecture.
- Architecture and Backbone: The framework offers lightweight language backbones ranging from Phi-1.5 to Llama-3-8B and vision encoders SigLIP and EVA-CLIP.
- Architecture and Backbone: A two-layer GELU MLP serves as the cross-modality projector that aligns the vision encoder with the LLM.
- Architecture and Backbone: The framework currently provides a defined set of backbone choices, with more LLMs, vision encoders, and projectors planned for future release.
- Training Data Construction: Bunny constructs informative pre-training and instruction-tuning data, including a 2M coreset condensed from LAION-2B and the Bunny-695K dataset.
- Training Procedure: Training has two stages: projector-only pre-training aligns visual and text embeddings, followed by visual instruction tuning with LoRA and fully tuned vision components.
4 Experiment
Bunny is evaluated across 11 multimodal benchmarks and extensive ablations of tuning, data, resolution, merging, and backbone choices. Bunny-4B/8B generally outperform similarly sized MLLMs, with performance sensitive to training configuration.
- Benchmark comparison: Bunny is evaluated on 11 benchmarks spanning perception, cognition, visual question answering, reasoning, science, and hallucination-related assessment.The evaluation includes MME, MMBench, SEED-Bench-1, MMMU, VQA-v2, GQA, ScienceQA-IMG, and POPE.
- Benchmark comparison: 1644.1 MMEP is achieved by Bunny-8B, exceeding LLaVA-NeXT-Llama-3-8B’s 1603.7.Bunny-8B uses SigLIP-SO with Llama-3-8B.
- Benchmark comparison: 7.5 and 4.7 improvements are reported for Bunny-4B over MM1-3B-Chat on MMMU validation and test splits, respectively.The comparison concerns the challenging MMMU benchmark.
- Ablation study: LoRA empirically outperforms fully tuning the LLM during fine-tuning.The authors attribute this likely advantage to mitigating catastrophic forgetting in smaller models.
- Ablation study: Jointly training on Bunny-695K and LLaVA-665K for one epoch generally improves multiple benchmarks over alternatives with less data diversity.The comparison indicates that greater data diversity improves performance under the same training cost; ALLaVA-Instruct-4V adds further improvements.
- Ablation study: Unfreezing the vision encoder improves results when using Bunny-695K + LLaVA-665K, while learning-rate effects are not consistently monotonic.Learning rates of 5 × 10−4 and 8 × 10−4 are described as relatively good choices, whereas 1 × 10−3 is sub-optimal.
- Ablation study: S2-Wrapper yields a large performance improvement with Bunny-695K + LLaVA-665K, motivating its selection for the final high-resolution solution.The higher-resolution experiments compare positional embedding interpolation with sub-image decomposition.
- Ablation study: Simple weighted model merging greatly enhances performance, although some benchmarks such as MME are sensitive to the weight factor.The best merged models form Bunny-4B/8B and avoid hyper-parameter searching for this improvement.
5 Conclusion
Bunny is a lightweight, flexible multimodal model family that combines plug-and-play vision and language backbones with more informative selected training data. Bunny-4B/8B reportedly outperforms state-of-the-art large MLLMs on multiple benchmarks and demonstrates broad qualitative capabilities.
- Conclusion: Bunny provides plug-and-play vision encoders and language backbones, including EVA-CLIP, SigLIP, Phi variants, Qwen1.5-1.8B, StableLM-2, MiniCPM-2B, and Llama-3-8B.The framework is designed to support multiple backbone combinations rather than a single fixed architecture.
- Conclusion: Bunny-4B/8B outperforms state-of-the-art large MLLMs on multiple benchmarks.The conclusion presents this as the paper’s principal performance result.
- Backbone combinations: SigLIP-SO with Llama-3-8B achieves the best performance among the evaluated backbone combinations.The comparison covers various vision-encoder and language-model pairings within Bunny.
- Qualitative demonstrations: Bunny-8B demonstrates visual understanding, common-sense reasoning, external-knowledge referring, optical character recognition, and mathematical problem-solving.The qualitative demonstrations characterize Bunny-8B as a versatile visual assistant.
- Qualitative demonstrations: Bunny-8B also exhibits Chinese instruction-following ability despite the demonstrations’ stated focus on qualitative evaluation.The paper reports this capability in its Chinese test cases.