Source-linked AI summary
LLaVA-Phi: Efficient Multi-Modal Assistant with Small Language Model
Yichen Zhu, Minjie Zhu, Ning Liu, Zhicai Ou, Xiaofeng Mou, Jian Tang
TL;DR
Existing vision-language models often require substantial computation, limiting real-time and edge deployment. This paper combines Phi-2 with the LLaVA methodology and high-quality visual instruction data to build LLaVA-Phi, which performs competitively across multimodal benchmarks despite its compact size. The model also demonstrates qualitative abilities in empathetic reasoning, code generation, and mathematical problem solving.
Problem
Many open-source vision-language models use at least 7B parameters, creating test-time and inference-speed challenges for time-sensitive or real-time applications.
Method
LLaVA-Phi combines LLaVA-1.5 with Phi-2, using a LLaVA-style two-stage pipeline and high-quality visual instruction data.
Results
With 3 billion parameters, LLaVA-Phi performs comparably to or better than some multimodal models three times larger and shows strong benchmark and qualitative results.
Takeaways & Limitations
Small vision-language models can perform effectively on standard benchmarks when combined with the LLaVA training methodology and selected high-quality data.
Takeaways & Limitations
Because Phi-2 uses the codegenmono tokenizer and LLaVA-Phi was not fine-tuned for multilingual instructions, it cannot process instructions in multiple languages, including Chinese.
Abstract
from arXiv · showhide
In this paper, we introduce LLaVA-$φ$ (LLaVA-Phi), an efficient multi-modal assistant that harnesses the power of the recently advanced small language model, Phi-2, to facilitate multi-modal dialogues. LLaVA-Phi marks a notable advancement in the realm of compact multi-modal models. It demonstrates that even smaller language models, with as few as 2.7B parameters, can effectively engage in intricate dialogues that integrate both textual and visual elements, provided they are trained with high-quality corpora. Our model delivers commendable performance on publicly available benchmarks that encompass visual comprehension, reasoning, and knowledge-based perception. Beyond its remarkable performance in multi-modal dialogue tasks, our model opens new avenues for applications in time-sensitive environments and systems that require real-time interaction, such as embodied agents. It highlights the potential of smaller language models to achieve sophisticated levels of understanding and interaction, while maintaining greater resource efficiency.The project is available at {https://github.com/zhuyiche/llava-phi}.
1. Introduction
Existing vision-language models achieve strong multimodal interaction but commonly rely on at least 7B parameters, increasing inference cost for real-time deployment. LLaVA-Phi addresses this gap with a compact Phi-2-based assistant evaluated across diverse benchmarks.
- Motivation: Many vision-language models use language models with at least 7B parameters, increasing test time and reducing inference speed for time-sensitive applications.The cited examples include autonomous driving and robotics.
- Motivation: Small language models such as TinyGSM and Phi-2 have demonstrated capabilities in mathematics, language reasoning, commonsense reasoning, and code generation.TinyGSM has 2.6 billion parameters and exceeds 80% accuracy on GSM8k, while Phi-2 supports language and coding abilities.
- Approach: LLaVA-Phi combines LLaVA-1.5 with Phi-2 and uses a two-stage training pipeline with high-quality visual instruction data.The model was evaluated across eight diverse benchmarks.
- Reported outcome: With 3 billion parameters, LLaVA-Phi achieves performance comparable to or exceeding some multimodal models three times larger.The paper specifically reports exceptional ScienceQA performance and qualitative abilities in challenging questions, code generation, and mathematics.
2. Related Work
Related work establishes strong progress in vision-language dialogue while highlighting a recurring deployment challenge: open-source models often require substantial computation and are difficult to use on edge devices. Compact alternatives exist, but some are proprietary or remain limited in accessibility.
- Vision-language models: LLM-based vision-language models, including LLaVA, BLIP, and MiniGPT-4, substantially improve visual-centric dialogue and question answering.These systems use large language models as language encoding modules.
- Efficiency constraints: Open-source vision-language models commonly require 7B to 65B parameters, creating deployment challenges for edge and mobile devices.The limitation is especially relevant to real-time applications.
3. LLaVA-Phi
LLaVA-Phi uses a LLaVA-1.5-style architecture and two-stage training to connect visual inputs with Phi-2. Qualitative examples show capabilities in empathetic reasoning, code generation, and mathematical problem solving.
- Architecture: LLaVA-Phi uses CLIP ViT-L/14 at 336x336 resolution as its visual encoder and a two-layer MLP to connect the encoder with the language model.The overall architecture is similar to LLaVA-1.5.
- Training: Supervised fine-tuning uses selected high-quality ShareGPT data, and the authors report modest improvements across most benchmarks.The training ran for two epochs with an initial learning rate of 3e-5.
- Training: The training pipeline first trains an efficient projector with the vision encoder and Phi-2 frozen, then fine-tunes the projector and language model together.This two-stage procedure targets visual comprehension and language processing.
- Qualitative results: LLaVA-Phi gives a more empathetic interpretation of a meme than LLaVA-1.5-13B in one qualitative comparison.Its response connects the dog’s laid-back demeanor with stress or fatigue associated with a new workweek.
- Qualitative results: For an Excel-to-bar-chart task, LLaVA-Phi follows the plotting instructions with labeled matplotlib code, unlike the simpler comparison output.The authors associate this capability with Phi-2’s code-focused pretraining.
- Qualitative results: For image-based mathematics, LLaVA-Phi produces the correct answer without chain-of-thought reasoning after recognizing the equation, while LLaVA-1.5-13B misrecognizes numbers and symbols.The paper connects this qualitative result with stronger ScienceQA performance.
4. Experiments
LLaVA-Phi is evaluated on broad multimodal benchmarks against larger vision-language models. It surpasses several 7B-based systems and consistently outperforms MobileVLM, although the reported margins are generally modest except on ScienceQA.
- Evaluation setup: The evaluation covers visual question answering, object hallucination, visual understanding, and visual commonsense reasoning across multiple academic benchmarks.The listed benchmarks include VQA-v2, VizWizQA, ScienceQA, TextQA, POPE, MME, MMBench, and MMVet.
- Evaluation setup: LLaVA-Phi is benchmarked against state-of-the-art large vision-language models using the same publicly available pretraining and instruction-tuning datasets as LLaVA1.5.The shared datasets provide the stated comparison context.
- Results: LLaVA-Phi outperforms models backed by 7B-or-larger language models, including IDEFICS and InstructBLIP, and achieves its best performance on ScienceQA.The authors attribute the ScienceQA result to Phi-2’s code-generation and mathematical training.
- Results: 11.5% and 23.8% are LLaVA-Phi’s reported MMBench advantages over Otter and InstructBLIP, respectively.The paper presents these comparisons as evidence of stronger performance than several 7B-LLM-based models.
- Results: Across all five benchmarks, LLaVA-Phi outperforms MobileVLM, but the margins are modest except on ScienceQA.The authors primarily attribute the performance disparity to differences in language-model pretraining.
5. Conclusion, Limitation, and Future Works
LLaVA-Phi shows that a compact Phi-2-based vision-language assistant can perform effectively on standard benchmarks when trained with LLaVA methodology and high-quality data. The paper targets lightweight multimodal reasoning for edge devices, while acknowledging multilingual limitations and proposing future training and encoder improvements.
- Conclusion: LLaVA-Phi combines the compact Phi-2 language model with high-quality data and LLaVA training methodology to support effective vision-language reasoning.The project is designed to help create lightweight multimodal models for edge-device operation.
- Limitation: Its inability to process multilingual instructions, including Chinese, results from Phi-2's codegenmono tokenizer and the absence of multilingual instruction fine-tuning.
- Future Works: Future work will examine visual-encoder size and refine small-language-model training with methods including direct preference optimization and RLHF.