Source-linked AI summary
TinyLLaVA: A Framework of Small-scale Large Multimodal Models
Baichuan Zhou, Ying Hu, Xi Weng, Junlong Jia, Jie Luo, Xien Liu, Ji Wu, Lei Huang
TL;DR
Small-scale LMM research lacks a unified understanding of how architecture choices, data, and training recipes affect performance, despite the cost barriers of larger models. TinyLLaVA provides a framework for systematically analyzing these factors and training compact LMMs. Its experiments show that better data and recipes enable smaller models to achieve on-par performance with larger ones, with TinyLLaVA-3.1B outperforming existing 7B models overall.
Problem
Existing small-scale LMM attempts examine only parts of the design space, while larger models impose expensive computational requirements that restrict research access.
Method
TinyLLaVA unifies a vision encoder, small-scale LLM, connector, training data, and training pipelines to empirically study model and training choices.
Results
TinyLLaVA-3.1B achieves better overall performance against existing 7B models such as LLaVA-1.5 and Qwen-VL.
Takeaways & Limitations
Better-quality data combined with better training recipes can enable smaller LMMs to achieve on-par performance with bigger ones.
Takeaways & Limitations
The framework allows partial adjustment of LLM and vision-encoder parameters because connector-only training may not sufficiently align vision and text for small-scale LLMs.
Abstract
from arXiv · showhide
We present the TinyLLaVA framework that provides a unified perspective in designing and analyzing the small-scale Large Multimodal Models (LMMs). We empirically study the effects of different vision encoders, connection modules, language models, training data and training recipes. Our extensive experiments showed that better quality of data combined with better training recipes, smaller LMMs can consistently achieve on-par performances compared to bigger LMMs. Under our framework, we train a family of small-scale LMMs. Our best model, TinyLLaVA-3.1B, achieves better overall performance against existing 7B models such as LLaVA-1.5 and Qwen-VL. We hope our findings can serve as baselines for future research in terms of data scaling, training setups and model selections. Our model weights and codes will be made public.
1. Introduction
TinyLLaVA addresses the cost and access barriers of large multimodal models by systematically studying small-scale LMM design choices. Its experiments find that suitable data and training recipes enable smaller models to match or exceed larger counterparts.
- Large multimodal models require expensive computation, making training and inference less affordable and restricting research access.
- Existing small-scale LMM efforts examine only parts of the broad design space, complicating model design and comparison.
- TinyLLaVA unifies a vision encoder, small-scale LLM decoder, intermediate connector, and training pipelines for systematic analysis.
- The framework evaluates vision encoders, connection modules, language models, training data, and training recipes.
- With better training recipes and higher-quality data, smaller LMMs can achieve on-par performance with larger counterparts.
- TinyLLaVA-3.1B achieves better overall performance than existing 7B models such as LLaVA-1.5 and Qwen-VL.
2. Related Work
Related work develops large multimodal models by combining language and vision models, while recent efforts reduce their computational burden through smaller language models. TinyLLaVA differs by offering unified analysis of small-scale LMM design choices, data, and training.
- Large Multimodal Models: Large multimodal models extend autoregressive language models with visual signals for vision-language learning and multimodal understanding.
- Large Multimodal Models: Visual instruction tuning improves the zero-shot capabilities of LMMs by aligning them with human preferences.
- Small-scale LMMs: Small-scale LMMs reduce computation overhead by using compact language models such as Phi-2, TinyLlama, and StableLM-2.
- Small-scale LMMs: Prior small-scale LMM studies mainly build and train specific models, whereas TinyLLaVA analyzes how model selection, recipes, and data contribute to performance.
- Small-scale LMMs: TinyLLaVA focuses on small-scale LMMs, whose experimentally observed behaviors differ from those of standard LMMs.
3. TinyLLaVA Framework
TinyLLaVA generalizes the LLaVA design into a unified framework with a small-scale language model, vision encoder, connector, and two-stage training pipeline. The framework supports partially trainable components to align visual and textual information.
- Model Architecture: TinyLLaVA consists of a small-scale LLM Fθ, vision encoder Vφ, and connector Pϕ that produces text from image-text inputs.
- Model Architecture: The small-scale LLM receives embedded vectors and generates the corresponding next-token predictions.
- Model Architecture: The vision encoder maps an image to visual patch features, while the connector maps those features into the LLM text-embedding space.
- Training Pipeline: Training data consists of image-text pairs whose text is organized as multi-turn conversations with human instructions and assistant responses.
- Training Pipeline: Pre-training uses image-caption data to align vision and text embeddings by autoregressively maximizing the likelihood of the target response.
- Training Pipeline: Partial parameter training is allowed because connector-only training may not sufficiently align vision and text when using small-scale LLMs.
- Training Pipeline: Supervised fine-tuning maximizes likelihood only over assistant-response tokens and permits partial adjustment of language-model and vision-encoder parameters.
- Training Recipes: The base recipe freezes the vision encoder and small-scale LLM while updating only the connector; the share recipe updates the remaining model after freezing the first 12 vision layers.
4. Experiments
The experiments comprehensively investigate how model architectures, datasets, and training recipes affect the performance of small-scale LMMs within the TinyLLaVA framework.
- Experiments evaluate the effects of model architectures, datasets, and training recipes on small-scale LMM performance.
4.1. Experimental Settings
TinyLLaVA instantiates a unified framework by varying small-scale language models, vision encoders, connectors, datasets, and training recipes. Experiments compare these choices across image-question-answering and comprehensive multimodal benchmarks.
- Model Components: The framework instantiates models with three small-scale LLMs, two vision encoders, and selected connectors.The LLMs are TinyLlama, StableLM-2, and Phi-2; the vision encoders are CLIP and SigLIP.
- Training Data: Training uses LLaVA-1.5 and ShareGPT4V datasets to study how data quality affects LMM performance.The datasets contain separate pre-training and supervised fine-tuning stages, with ShareGPT4V providing more extensive and diverse caption data.
- Training Recipes: The study compares base and share training recipes that differ primarily in which model components are updated during pre-training.The base recipe updates only the connector during pre-training, whereas the share recipe updates most vision-encoder layers and other model components.
- Ablation Design: Under the base recipe, six variants combine three small-scale LLMs with two vision encoders on the LLaVA-1.5 dataset.Figure 4 organizes subplots by vision encoder.
- Evaluation: Connector exploration compares MLP and Resampler using CLIP, TinyLlama, and the LLaVA-1.5 dataset.The evaluation includes four image question-answering benchmarks and five comprehensive benchmarks.
4.2. Experimental Results
The experiments show that architecture, data, and training choices substantially affect small-scale LMM performance. With suitable data and recipes, smaller variants can match or exceed larger models, including a 3.1B model outperforming existing 7B systems.
- Model architectures: Phi-2 variants perform exceptionally well across configurations and benchmarks, while TinyLlama variants are smallest and achieve better POPE accuracy than StableLM-2 variants.The authors attribute Phi-2’s strength partly to its larger parameter count and textbook training, while TinyLlama shows slightly lower overall performance.
- Model architectures: SigLIP variants substantially improve performance over CLIP, especially on TextVQA and LLaVA-W.SigLIP uses higher input resolution and more visual tokens, which may provide more information for fine-grained image understanding.
- Model architectures: Using a resampler connector degrades performance compared with MLP under similar parameter settings.The authors describe this as a preliminary connector exploration and identify broader connector comparisons as future work.
- Data mixtures: ShareGPT4V generally improves evaluation performance over the LLaVA-1.5 dataset, but TinyLlama variants show notable degradation on POPE.StableLM-2 and Phi-2 variants improve comprehensively, whereas the authors speculate TinyLlama cannot adequately fit the larger dataset.
- Training recipes: The share recipe significantly improves all variants when using ShareGPT4V, and partially fine-tuning the vision encoder may benefit small-scale LMMs.The authors conjecture that this effect depends on both accompanied LLM size and training-data size.
- Training recipes: More trainable parameters can improve larger-data training for smaller LLMs but increase hallucinations for StableLM-2 and Phi-2 variants.Under the share recipe, StableLM-2 and Phi-2 improve on most benchmarks while declining on POPE.
- Overall comparison: Smaller TinyLLaVA variants can match larger counterparts with appropriate data and training recipes, while TinyLLaVA-share-Sig-Phi with 3.1B parameters comprehensively surpasses LLaVA-1.5 with 7B parameters.It also matches MoE-LLaVA on VQAv2 with fewer parameters and exceeds it on POPE accuracy.
5. Conclusion
TinyLLaVA offers a unified framework for analyzing small-scale LMM design choices. The authors conclude that high-quality data and improved training recipes can let smaller models achieve on-par performance with larger models, including their 3.1B model against existing 7B models.
- Conclusion: TinyLLaVA provides a unified perspective for designing and analyzing small-scale LMMs.The framework is intended to support analysis of data scaling, training setups, and model selection.
- Conclusion: With better-quality data and training recipes, smaller LMMs can consistently achieve on-par performance with bigger models.The conclusion contrasts this with the same-settings result that larger models perform better than smaller ones.
- Conclusion: TinyLLaVA-3.1B achieves better overall performance than existing 7B models.The authors present the finding as a result of using insights from their ablation studies.
- Conclusion: The authors suggest that the design space of LMMs remains vastly under-explored and propose their findings as baselines for future research.They identify data scaling, training setups, and model selections as relevant directions.
A. Brief Overviews of Evaluation Benchmark.
The evaluation suite covers visual recognition, reasoning, OCR, hallucination, generalizability, and broad multimodal capability. Its benchmarks combine image-question answering tasks with comprehensive evaluations using automated or GPT-4-based assessment.
- Image question answering: VQAv2 evaluates visual recognition, visual grounding, spatial reasoning, and language understanding using image-question-answer tuples from COCO.
- Image question answering: GQA evaluates visual and compositional reasoning using data organized around Visual Genome scene graphs.
- Image question answering: TextVQA tests recognition and reasoning over textual information embedded in images.
- Image question answering: ScienceQA-IMG evaluates reasoning about scientific knowledge through image-based questions, choices, and relevant contexts from lectures and textbooks.
- Comprehensive benchmarks: POPE evaluates hallucination handling by requiring accurate recognition of positive objects and correct rejection of non-existent objects.
- Comprehensive benchmarks: MM-Vet evaluates six multimodal dimensions, including visual recognition, spatial reasoning, common-knowledge deduction, language generation, visual math reasoning, and OCR recognition.GPT-4 evaluates the generated outputs.
- Comprehensive benchmarks: LLaVA-W uses 24 images and 60 questions to assess challenging-task performance and generalizability in novel domains.
- Comprehensive benchmarks: MME measures perception and cognition across 14 subtasks, while MMBench assesses capabilities across 20 dimensions.MME is automatically evaluated by GPT-4, and MMBench by ChatGPT.
B. TinyLLaVA Variants.
Table A1 compares TinyLLaVA variants trained with a base LLaVA-1.5 setup and a ShareGPT4V recipe. The examples highlight capabilities in OCR, mathematics, memes, and explaining unusual scenes.
- Larger models, diverse datasets, and carefully designed training recipes can improve overall TinyLLaVA performance.
- Table A1 comprehensively compares TinyLLaVA variants using the LLaVA-1.5 dataset for the base setup and ShareGPT4V for the shared recipe.Best and second-best results are marked with boldface and underlining, respectively.
- The examples include a city map with two green dots, an order table listing fares and payment details, and an unusual ironing scene on a yellow taxi.
- TinyLLaVA-3.1B can perceive detailed OCR content and visually perceive and solve math problems.
- TinyLLaVA-3.1B can understand and explain memes and explain unrealistic scenes.