Source-linked AI summary
LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training
Xiang An, Yin Xie, Kaicheng Yang, Wenkang Zhang, Xiuwei Zhao, Zheng Cheng, Yirui Wang, Songcen Xu, Changrui Chen, Didi Zhu, Chunsheng Wu, Huajie Tan, Chunyuan Li, Jing Yang, Jie Yu, Xiyao Wang, Bin Qin, Yumeng Wang, Zizhen Yan, Ziyong Feng, Ziwei Liu, Bo Li, Jiankang Deng
TL;DR
High-performing multimodal models are often proprietary, limiting insight into how to build them from scratch. LLaVA-OneVision-1.5 addresses this gap with an open framework combining curated datasets, efficient staged training, and lightweight RL post-training, achieving competitive performance across diverse benchmarks under resource constraints.
Problem
Proprietary high-performing multimodal models expose neither their training data nor source code, limiting insight into building such systems from scratch.
Method
LLaVA-OneVision-1.5 combines curated multimodal datasets, a three-stage efficient training pipeline, public assets, and lightweight RL post-training for eliciting reasoning capabilities.
Results
LLaVA-OneVision-1.5 achieves superior performance across multiple multimodal benchmarks, while scaling high-quality knowledge-learning data consistently improves performance across ten benchmarks.
Takeaways & Limitations
The results support constructing competitive multimodal models from scratch through open, efficient, and reproducible frameworks under limited computational budgets.
Abstract
from arXiv · showhide
We present LLaVA-OneVision-1.5, a novel family of Large Multimodal Models (LMMs) that achieve state-of-the-art performance with significantly reduced computational and financial costs. Different from the existing works, LLaVA-OneVision-1.5 provides an open, efficient, and reproducible framework for building high-quality vision-language models entirely from scratch. The LLaVA-OneVision-1.5 release comprises three primary components: (1) Large-Scale Curated Datasets: We construct an 85M concept-balanced pretraining dataset LLaVA-OneVision-1.5-Mid-Traning and a meticulously curated 22M instruction dataset LLaVA-OneVision-1.5-Instruct. (2) Efficient Training Framework: We develop a complete end-to-end efficient training framework leveraging an offline parallel data packing strategy to facilitate the training of LLaVA-OneVision-1.5 within a $16,000 budget. (3) State-of-the-art Performance: Experimental results demonstrate that LLaVA-OneVision-1.5 yields exceptionally competitive performance across a broad range of downstream tasks. Specifically, LLaVA-OneVision-1.5-8B outperforms Qwen2.5-VL-7B on 18 of 27 benchmarks, and LLaVA-OneVision-1.5-4B surpasses Qwen2.5-VL-3B on all 27 benchmarks. (4) RL-based Post-training: We unlock the model's latent potential through a lightweight RL stage, effectively eliciting robust chain-of-thought reasoning to significantly boost performance on complex multimodal reasoning tasks.
1 Introduction
LLaVA-OneVision-1.5 addresses the lack of open, reproducible recipes for high-performing multimodal models by combining curated data, efficient training, and lightweight RL post-training. The resulting family targets competitive performance with limited computational cost.
- Proprietary LMMs lead performance, but their training data and source code are unavailable, limiting insight into how to build them from scratch.
- LLaVA-OneVision-1.5 is a fully open-source LMM family designed to achieve superior performance with limited computational cost.
- 85M concept-balanced pretraining samples and 22M curated instruction samples form the release’s large-scale data resources.
- An offline parallel data-packing strategy supports end-to-end training within a $16,000 compute budget.
- The framework releases datasets, training code, and Base and Instruct checkpoints to the community.
- A lightweight RL post-training stage uses discrepancy-driven selection and outcome verification to elicit latent reasoning capabilities on complex tasks.
2 Architecture
The architecture retains the ViT–MLP–LLM design while replacing the vision encoder with region-aware RICE-ViT. Its components target variable-resolution processing, local visual semantics, OCR, and multimodal reasoning.
- Overall architecture: The system uses three modules: a vision encoder, a projector, and a large language model.
- Vision encoder: RICE-ViT provides region-aware visual and OCR representations through local semantic modeling and 2D rotary positional encoding.
- Projector: A projector groups adjacent patch features, concatenates them, and maps the result into the LLM text-embedding space with a two-layer MLP.
- Language model: The LLM integrates projected visual embeddings with linguistic context for reasoning, instruction following, and generation.
- Vision encoder: 2D rotary positional encoding supports variable input resolutions without resolution-specific fine-tuning.
- Training objective: A single cluster-discrimination loss is reported to match SigLIP2 performance while reducing architectural complexity and training overhead.
3 Data
The data pipeline combines broad image-text sources with concept balancing and curated multimodal instructions. It produces an 85M mid-training corpus and a 22M instruction corpus spanning diverse capabilities.
- LLaVA-1.5 558K is used to align visual features with the LLM word-embedding space before large-scale multimodal training.
- Mid-training data: The mid-training dataset contains 85M high-quality image-text pairs, including 20M Chinese and 65M English pairs.
- Concept balancing: Top-k concept assignment uses cosine similarity between normalized image and concept embeddings to assign concepts to images.
- Concept balancing: Inverse-frequency concept weighting produces an 85M dataset with a more balanced concept distribution without relying on original captions.
- Instruction data: The instruction corpus comprises 22M curated samples balanced across captioning, charts and tables, code and math, VQA, grounding, OCR, science, and domain-specific tasks.
4 Training Strategy
Training proceeds through alignment, knowledge learning, and visual instruction tuning, with data packing and hybrid parallelism improving efficiency. The pipeline also includes benchmark evaluation and supports RL-related extensions.
- Training stages: The three-stage pipeline consists of language-image alignment, high-quality knowledge learning, and visual instruction tuning.
- Stage 1: Language-Image Alignment: Stage 1 trains the projection layer on LLaVA-1.5 558K to align visual features with the LLM embedding space.
- Stage 1.5: High-Quality Knowledge Learning: Stage 1.5 performs full-parameter training on the 85M mid-training dataset to inject knowledge while balancing computational efficiency.
- Stage 2: Visual Instruction Tuning: Stage 2 performs full-parameter visual instruction tuning with LLaVA-OneVision-1.5-Instruct and FineVision.
- Evaluation: Table 1 compares vision-language models across benchmark groups, reporting accuracy percentages unless otherwise specified.
- Efficient training: Offline parallel data packing consolidates shorter samples into packed sequences to reduce padding overhead and improve GPU utilization.
- Efficient training: Native-resolution mid-training uses 128 × A800 GPUs for 3.7 days on 85 million captions.
5 Post-training
LLaVA-OneVision-1.5 applies lightweight, outcome-based RL post-training through discrepancy-driven data selection, reward filtering, and a two-stage curriculum. The design elicits chain-of-thought reasoning while preserving concise perceptual skills.
- RL data construction: Discrepancy-driven selection targets tasks where Pass@N exceeds Pass@1, indicating latent capability that the policy does not reliably elicit.The authors frame RL as capability elicitation rather than knowledge injection.
- RL data construction: Reward-based sampling retains medium-difficulty examples by discarding instances with trivial or unsolvable candidate responses.Multiple responses are generated and filtered using their average automatic reward.
- RL data construction: The RL corpus contains about 67K instances spanning STEM, grounding, spatial, counting, coding, OCR, and diagram tasks.Each instance is tagged by answer-only or chain-of-thought prompting for curriculum design.
- Optimization: The framework uses rule-based, answer-type-specific rewards and GRPO implemented in asynchronous AReaL for parallel generation and training.The setup omits the KL penalty and explicit format reward, relying on outcome-based correctness with PPO-style clipping.
- Training curriculum: The two-stage curriculum first trains answer-only RL on normal data, then applies chain-of-thought RL to long-reasoning data.Stage 1 reinforces basic perceptual precision, while Stage 2 rewards only final-answer correctness and uses reasoning traces as auxiliary guidance.
- Training curriculum: A small proportion of normal-set examples is interleaved during Stage 2 to reduce forgetting of short, perception-heavy skills.These answer-only samples act as anchors while RL emphasizes deeper reasoning.
6 Experiments
The experiments evaluate LLaVA-OneVision-1.5 across broad visual question answering, reasoning, OCR, chart, and other multimodal benchmarks. The models achieve strong cross-benchmark results against Qwen2.5-VL baselines under the reported evaluation setup.
- Evaluation setup: Evaluation uses LMMs-Eval with default prompts across four downstream categories and 27 benchmarks.Categories include general VQA, multimodal reasoning, OCR and chart understanding, and other capabilities.
- Overall results: LLaVA-OneVision-1.5-8B surpasses Qwen2.5-VL-7B on 18 of 27 benchmarks, while the 4B model surpasses Qwen2.5-VL-3B on all 27.This is the paper’s headline cross-task comparison.
- Multimodal reasoning: The 4B model leads on MathVistamini 67.9, WeMath 24.9, MathVision 24.2, MMMUval 52.7, MMMU-Prostandard 35.3, and MMMU-Provision 25.4.The 8B model gains 1.0%, 0.3%, 3.2%, 4.1%, and 1.1% over Qwen2.5-VL-7B on five listed reasoning benchmarks.
- Other capabilities: The 8B model scores 62.2 on PixmoCount, 88.2 on CountBench, 46.7 on VL-RewardBench, and 78.0 on V*.These results are reported as on par with Qwen2.5-VL-7B for counting, perception, and grounding.
6.6 Performance of RL Post-training
RL post-training consistently improves multimodal reasoning while maintaining or slightly improving general VQA and OCR performance. Additional analyses attribute gains to both chain-of-thought reasoning and direct outcome feedback, alongside benefits from RICE-ViT and curated data scaling.
- RL performance: In thinking mode, RL gains +7.9 on WeMath, +8.8 on MathVision, and +10.5 on MMMU-Provision.These are the largest reported improvements over the supervised baseline.
- RL performance: RL maintains or slightly improves SFT performance on MMBench and DocVQA, avoiding regression in general VQA and OCR capabilities.The reported specialization is in hard reasoning while core capabilities remain stable.
- Extended capability analysis: RL improves spatial tasks such as SAT and Ref-L4 in fast mode, while thinking mode can lower strictly perceptual Ref-L4 IoU.The authors suggest verbose generation may interfere with precise coordinate regression.
- Extended capability analysis: Thinking mode achieves the highest scores on Design2Code and UniSVG, indicating benefits for structural code generation.The coding gains are reported for both RL models relative to the SFT baseline.
- Vision encoders: RICE-ViT outperforms SigLIPv2 on 9 of 14 benchmarks at 378 pixels, including InfoVQA +4.4%, DocVQA +3.5%, and ChartQA +4.9%.At 336 pixels, it also gains +6.3% on InfoVQA and +5.0% on OCRBench over CLIP.
- Data curation: Concept-balanced 2M mid-training data outperforms random-sampled 2M data on 25 of 27 downstream benchmarks.The balanced dataset is described as producing a smoother concept distribution.
- Instruction data: Merged46M SFT data outperforms LLaVA-OneVision-1.5-Inst-Data and FineVision on most of 16 evaluated benchmarks.The base instruction dataset performs comparably to FineVision.
7 Conclusions
LLaVA-OneVision-1.5 presents an open and efficient framework for training competitive multimodal models from scratch. Its curated data, budget-conscious training system, and benchmark results support reproducible development under resource constraints.
- Contributions: The paper combines a curated multimodal dataset, an efficient end-to-end training framework, and extensive benchmark evaluation.The conclusion identifies these as the central contributions.
- Implications: The framework demonstrates the feasibility of training competitive multimodal models from scratch under strict computational constraints.The conclusion emphasizes improved efficiency and reproducibility.
- Implications: LLaVA-OneVision-1.5 surpasses strong baselines such as Qwen2.5-VL-7B in resource-constrained settings.The authors position open, efficient frameworks as enabling broader access to state-of-the-art multimodal performance.
A LLaVA-OV-1.5 vs. Qwen2.5-VL with Same LLM
Using the same LLM for both evaluations, LLaVA-OneVision-1.5-3B is compared fairly with Qwen2.5-VL-3B and performs better across most downstream benchmarks.
- The evaluation compares LLaVA-OneVision-1.5-3B and Qwen2.5-VL-3B using the same LLM for both evaluations.LLaVA-OneVision-1.5-3B is based on Qwen2.5-3B-Instruct.
- 17 out of 27 downstream benchmarks favor LLaVA-OneVision-1.5-3B over Qwen2.5-VL-3B.
- Figure 9 reports the comparison on public datasets.
B Mid-Training Data: Concept Distribution and Top 50 Topics
The Mid-Traning mixture is analyzed through concept distributions and 50-topic modeling. Its source datasets show strong long-tail bias, while the extracted topics span diverse visual and semantic domains.
- Concept Distribution: The original distributions across eight common vision datasets exhibit pronounced long-tail bias, indicating incomplete concept coverage.
- Concept Distribution: Obelics has the broadest and most uniform concept distribution, whereas ImageNet-21K and SA-1B show fewer concepts with greater frequency concentration.
- Top 50 Topics: Table 3 reports topic-modeling results organized into 50 topics.
- Top 50 Topics: Other topic groups include engineering and mechanics, technology and innovation, biological diagrams, transportation and fiction, and vast landscapes.
- Top 50 Topics: The topics cover domains including healthcare and fitness, software and development, typography and design, cuisine and cooking, and home interiors.