PaperlayerView exact revision on arXiv

Source-linked AI summary

LLaVA-OneVision: Easy Visual Task Transfer

Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, Chunyuan Li

arXiv:2408.03326v3cs.CVcs.AIcs.CL

TL;DR

Open multimodal models have rarely combined strong performance across single-image, multi-image, and video scenarios in one system. LLaVA-OneVision consolidates prior data, model, and visual-representation insights into a scaled open recipe, achieving broad cross-scenario performance and image-to-video task transfer. Its scope remains shaped by fixed-compute representation choices and empirically selected component assumptions.

  • Problem

    Open multimodal models have rarely reported strong performance across single-image, multi-image, and video scenarios in one model.

  • Method

    LLaVA-OneVision consolidates LLaVA-NeXT insights and scales a recipe combining stronger language models, curated data, and scenario-specific visual representations.

  • Results

    LLaVA-OneVision improves open LMM performance across single-image, multi-image, and video benchmarks and demonstrates image-to-video task transfer.

  • Takeaways & Limitations

    The open model and released resources provide a starting point for building visual assistants and stronger multimodal systems across diverse vision scenarios.

  • Takeaways & Limitations

    The design is developed without extensively de-risking individual components and uses representation configurations under a fixed compute budget.

Abstract

from arXiv · show

We present LLaVA-OneVision, a family of open large multimodal models (LMMs) developed by consolidating our insights into data, models, and visual representations in the LLaVA-NeXT blog series. Our experimental results demonstrate that LLaVA-OneVision is the first single model that can simultaneously push the performance boundaries of open LMMs in three important computer vision scenarios: single-image, multi-image, and video scenarios. Importantly, the design of LLaVA-OneVision allows strong transfer learning across different modalities/scenarios, yielding new emerging capabilities. In particular, strong video understanding and cross-scenario capabilities are demonstrated through task transfer from images to videos.

1 Introduction

LLaVA-OneVision consolidates prior LLaVA-NeXT insights into an open multimodal model family designed for single-image, multi-image, and video scenarios. Its central contribution is cross-scenario task transfer, particularly from images to videos.

  • The project scales a cost-efficient vision-encoder–LLM recipe using newly accumulated larger datasets and stronger language models.
  • The model consolidates prior explorations of visual representations, architectures, training strategies, and multimodal instruction data.
  • LLaVA-OneVision is an open family of large multimodal models targeting single-image, multi-image, and video vision settings.
  • Its design supports task transfer across scenarios, with strong video understanding emerging from image-trained capabilities.
  • The authors release multimodal instruction data, code, model checkpoints, and a visual chat demo as open-source assets.

2 Related Work

Existing open multimodal models commonly specialize in individual visual scenarios, leaving a gap in one model performing well across images, multiple images, and video. LLaVA-OneVision addresses this gap while building on earlier versatile-model efforts.

  • Proprietary multimodal models perform across single-image, multi-image, and video settings, but open research has often developed separate scenario-specific systems.
  • Most open-model work emphasizes single-image performance, while multi-image research is newer and video models may trade image performance for video understanding.
  • LLaVA-OneVision aims to provide one open model with strong performance across all three visual scenarios.
  • LLaVA-OneVision inherits LLaVA-NeXT-Interleave’s training recipe and data while seeking improved performance through broader training and curated instruction data.

3 Modeling

LLaVA-OneVision uses a minimal LLaVA-style architecture with a language model, vision encoder, and projector, alongside Higher AnyRes visual representations. Its scenario-specific token allocations are designed to support capability transfer under compute constraints.

  • Network Architecture: The architecture retains LLaVA’s minimal design, combining a pretrained language model, visual model, and connection components for scalable multimodal learning.
  • Network Architecture: Qwen-2 serves as the language model, SigLIP encodes visual inputs, and a 2-layer MLP functions as the projector.
  • Visual Representations: Visual encoding quality depends on raw-image resolution and feature-space token count, with resolution scaling observed to be more effective than token scaling.
  • Visual Representations: Higher AnyRes divides images into crops and can reduce tokens per crop through bilinear interpolation when a token threshold is exceeded.
  • Visual Representations: Single images receive many tokens to facilitate image-to-video transfer, while multi-image inputs use base resolution and videos reduce per-frame tokens to process more frames.
  • Visual Representations: The representation configurations target capability transfer under a fixed compute budget, with more tokens available when computational resources increase.

4 Data

LLaVA-OneVision prioritizes balanced, high-quality visual instruction data across single-image, multi-image, and video scenarios. Its data pipeline separates single-image training from mixed-scenario training to support transfer across modalities.

  • Data Quality: High-quality knowledge learning is prioritized over raw data quantity to improve compute efficiency through carefully curated data.The authors note that pre-trained LLMs and ViTs already contain substantial knowledge, motivating refinement with curated data.
  • Data Organization: Visual instruction tuning data is organized by vision input, language instruction, and response, with categories covering general QA, OCR, documents, charts, screens, math reasoning, and language.The hierarchy connects visual scenarios with the skills and response formats expected from the model.
  • Cross-Scenario Transfer: Instruction data is split into single-image and all-scenario groups because stronger image models transfer better to multi-image and video tasks, while image datasets are more abundant.The split reflects both observed transfer relationships and differences in available training-data quantity and quality.
  • Single-Image Data: 3.2 million single-image samples form a balanced collection used for the initial visual instruction tuning stage.The collection is designed around single-image capabilities and is summarized in Figure 4.
  • OneVision Data: The OneVision stage mixes 560K multi-image, 350K video, and 800K single-image samples, totaling 1.6 million examples.The single-image portion is sampled from the earlier collection rather than newly introduced at this stage.

5 Training Strategies

LLaVA-OneVision uses curriculum learning that progressively expands visual complexity and trains first on single-image instructions, then on mixed scenarios. This staged design supports capability transfer while reusing intermediate checkpoints.

  • Curriculum Learning: Curriculum learning presents increasingly difficult objectives and examples stage by stage, producing reusable checkpoints under a fixed compute budget.The strategy decomposes training into stages for experimentation and reuse.
  • Learning Stages: Training proceeds through language-image alignment, high-quality knowledge learning, and visual instruction tuning.The first stage aligns visual features with the LLM embedding space, while later stages add knowledge and task-following abilities.
  • Scenario Expansion: The model first trains on 3.2 million single-image instructions, then on mixed video, single-image, and multi-image data to learn cross-scenario transfer.The authors describe this post-training sequence as a simple and cost-efficient route to multi-image and video understanding.
  • Visual Scaling: Visual capacity increases during training, from 729 base image tokens in Stage-1 to up to 5 times more in Stage 1.5 and 10 times more in Stage 2.The curriculum progressively increases image resolution and visual-token counts, while later stages update the full model.

6 Experimental Results

LLaVA-OneVision is evaluated reproducibly across single-image, multi-image, and video-related settings using standardized zero-shot, greedy-decoding benchmarks. The 72B model reaches performance between GPT-4V and GPT-4o on most benchmarks, while remaining gaps persist on complex visual chat.

  • Evaluation Setup: LLaVA-OneVision is evaluated across single-image, multi-image, and video benchmarks using standardized LMMs-Eval settings with greedy decoding and zero-shot evaluation.Results from original papers are used when available; otherwise, models are evaluated under consistent settings.
  • Overall Results: The 72B model achieves performance between GPT-4V and GPT-4o on most benchmarks across three model sizes.The 0.5B, 7B, and 72B variants target different performance-throughput trade-offs.
  • Single-Image Benchmarks: LLaVA-OneVision surpasses GPT-4V and approaches GPT-4o on chart, diagram, and document understanding benchmarks.The evaluated benchmarks include AI2D, ChartQA, DocVQA, and InfoVQA.
  • Single-Image Benchmarks: LLaVA-OneVision significantly outperforms GPT-4V on several perception and reasoning benchmarks and is comparable to GPT-4o on MathVista.The evaluation covers MME, MMBench, MMVet, MathVerse, MathVista, and MMMU.
  • Single-Image Benchmarks: LLaVA-OneVision remains behind GPT-4V and GPT-4o on real-world understanding and visual chat but is competitive with similarly sized open-source models.Its MM-LiveBench performance is highlighted for updated internet content, broad world knowledge, and generalization.
  • Multi-Image Benchmarks: LLaVA-OneVision (SI) consistently outperforms existing multi-image LMMs, while OneVision tuning improves performance further, especially on multi-view benchmarks.The reported gains include complex multi-image reasoning, difference identification, and 3D-environment understanding.

GPT-4o

LLaVA-OneVision is evaluated across single-image, multi-image, and video benchmarks against open and commercial models. Its video performance is competitive with GPT-4V on several benchmarks, while scaling and image-only training affect tasks differently.

  • Cross-modal evaluation: Table 2 compares LLaVA-OneVision models from 0.5B to 72B parameters with commercial models across multiple modalities.The comparison spans diverse evaluation benchmarks, with † marking training-set overlap.
  • Video benchmarks: LLaVA-OneVision achieves comparable or better results than larger open-source models on video benchmarks.The advantage is particularly evident on EgoSchema and VideoMME.
  • Video benchmarks: LLaVA-OneVision performs competitively with GPT-4V on ActivityNet-QA, MLVU, and VideoMME.These comparisons are reported in the video benchmark evaluation.
  • Video benchmarks: Scaling the LLaVA-OV language model from 0.5B to 7B improves PerceptionTest by 0.5 points, versus at least 5 points on other datasets.The smaller gain suggests PerceptionTest depends more on the vision module than on language-model scaling.
  • Video benchmarks: The smallest improvement from LLaVA-OV-7B (SI) to LLaVA-OV-7B occurs on ActivityNet-QA.Many ActivityNet-QA questions can be answered from a single frame, reducing the need for sequence understanding.

7 Emerging Capabilities with Task Transfer

LLaVA-OneVision exhibits capabilities that transfer across single-image, multi-image, and video settings. These behaviors combine previously learned skills such as OCR, visual referring, relational reasoning, and task composition.

  • Transfer to multi-image: LLaVA-OneVision transfers diagram-and-chart understanding from single images to coherent multi-image reasoning.The joint task is not present in the multi-image training data.
  • Transfer to multi-image: The model recognizes iPhone GUI screenshots and provides instructions for searching for and opening TikTok.This combines OCR learned from single-image scenarios with relational reasoning from multi-image scenarios.
  • Task composition: LLaVA-OneVision performs set-of-mark reasoning despite the task not being explicitly included in training data.The paper hypothesizes that this ability is composed from visual referring and OCR.
  • Transfer to video: The model generates detailed video-creation prompts from static images by combining image analysis with video comprehension.The paper hypothesizes generalization from single-image editing instructions and video description tasks.
  • Transfer to video: LLaVA-OneVision analyzes differences between video sequences, extending multi-image spot-the-difference capability to videos.It compares sequences with different endings or different foreground objects.
  • Transfer to video: The model interprets multi-camera self-driving footage, composed sub-videos, and visual prompts in video without corresponding visual-prompt video training.These examples transfer multi-view, multi-image, and single-image abilities into video understanding.
  • Transfer to video: The model can refer to an image query while answering questions about a video, a capability not seen in LLaVA-NeXT or LLaVA-Interleave.The paper suggests strong base single-image training is required for this capability.

8 Conclusions

LLaVA-OneVision is presented as an open multimodal model spanning single-image, multi-image, and video scenarios. Its examples emphasize transferred capabilities such as GUI interaction, set-of-mark reasoning, video comparison, and cross-image/video reference.

  • Conclusion: LLaVA-OneVision is trained across single-image, multi-image, and video scenarios to support broad task transfer.The conclusion identifies image-to-video transfer as a source of strong visual understanding.
  • Multi-image capabilities: The model transfers diagram and table understanding to multi-image inputs and interprets them coherently.Table 6 presents this cross-scenario capability.
  • Multi-image capabilities: LLaVA-OneVision recognizes multiple iPhone screenshots and provides operational instructions for automating tasks.Table 7 frames this as multimodal GUI agent behavior.
  • Task composition: The model exhibits set-of-mark prompting by referring to numerical marks when answering questions about image content.This demonstrates fine-grained visual-content comprehension.
  • Video capabilities: The model generates detailed video creation prompts from static images and analyzes differences between video sequences.The examples cover image-to-video editing and video-to-video comparison.
  • Video capabilities: LLaVA-OneVision analyzes multi-camera self-driving footage, composed sub-videos, and highlighted subjects in videos.Tables 12–14 illustrate multi-view, composed-video, and visual-prompting behaviors.
  • Cross-modal reference: The model relates visual queries across images and videos, including identifying the same individual or determining that the individual is absent.Table 15 reports these image-and-video reference cases.

A Development Roadmap from LLaVA-NeXT to LLaVA-OneVision

LLaVA-OneVision consolidates LLaVA-NeXT explorations in scaling, visual representation, training, and cross-scenario data. Its architecture balances token budgets across images, multiple images, and videos to support transfer between modalities.

  • Development roadmap: LLaVA-OneVision consolidates LLaVA-NeXT blog insights and scales them with larger datasets and stronger language models.The development process used accumulated high-quality data and implemented the model within available compute.
  • LLaVA-NeXT foundations: AnyRes enabled an image-only-trained LLaVA-NeXT model to perform strongly on video tasks through zero-shot modality transfer.The design digests vision signals as a sequence of images.
  • LLaVA-NeXT foundations: Scaling the LLM within the cost-efficient recipe produced performance comparable to GPT-4V on selected benchmarks.The explored models included LLaMA3 8B and Qwen models up to 110B.
  • LLaVA-NeXT foundations: LLaVA-NeXT established an extensible prototype through explorations of video transfer, LLM scaling, ablations, and multi-image or multi-view scenarios.These explorations informed the OneVision design.
  • Visual representations: The model uses AnyResMax-9 for single images, padding for up to 12 multi-image inputs, and 196 tokens per frame for up to 32 video frames.Single-image inputs use 729 visual tokens per crop or grid, while video frames are reduced through 2 × 2 bilinear interpolation.
  • Visual representations: Approximately equal maximum token counts across modalities are intended to balance training data and facilitate transfer.The design treats high-resolution images as compositions of images and multiple images as shorter videos.
  • Training and evaluation: The training and evaluation setup interleaves modality-specific visual inputs with language tokens and uses selected benchmarks to guide development.The evaluation process includes image, multi-image, and video-oriented benchmarks and post-evaluations with LMMs-Eval.

D.3 Evaluation Task Information

The evaluation uses standardized LMMs-Eval prompts and tasks spanning single-image, video, and multi-image scenarios. Task names provide the information needed to retrieve prompts and generation settings for reproduction.

  • The evaluation adopts the default post_prompt and pre_prompt from LMMs-Eval, consistent with earlier LLaVA-NeXT evaluations.
  • Single-image: Single-image evaluation covers tasks including AI2D, ChartQA, DocVQA, MME, RealWorldQA, MathVista, OCRBench, MMMU, and other benchmarks.
  • Videos: Video evaluation includes ActivityNetQA, VideoChatGPT, NExT-QA, EgoSchema, Video-MME, PerceptionTest, MLVU, and MVBench.
  • Multi-image: Multi-image evaluation includes LLaVA-Interleave-Bench and MuirBench.
  • Researchers can retrieve task-specific generation arguments and prompt information from the listed task names, supporting result reproduction.

E.1 Single-Image Data Curation

The data-curation process prioritizes quality, validation, balanced task coverage, and modality diversity. It combines curated single-image data with multi-image and video sources to support LLaVA-OneVision across visual scenarios.

  • Single-Image Data Curation: Single-image curation follows a quality-over-quantity principle, using high-quality data to maximize performance under limited resources.
  • Dataset Evolution: The initial LLaVA-NeXT datasets grew from 760K to 790K samples by adding COCO Caption data, supporting subsequent model releases.
  • Curation Process: Each dataset was manually inspected for quality and question-answer format, with duplicate or specially formatted sources handled selectively.
  • Language Data: The pipeline replaced ShareGPT text data with Magpie-Pro to maintain or improve language capabilities for large Qwen2-based models.
  • Re-annotation: Approximately 4 million raw SFT samples were obtained, with GPT-4V and GPT-4o re-annotation targeting underrepresented scenarios such as charts, diagrams, and Chinese descriptions.
  • Validation: Data validation adds approximately 100K samples from each new source to the 790K dataset and investigates additions that reduce performance in a specified experimental setting.
  • Single-Image Distribution: Manual tags produced a final distribution of 3.2 million single-image samples, while free-form data was adjusted against multiple academic and visual-chat benchmarks.
  • Multi-Image and Video Data: Multi-image and video datasets extend coverage through interleaved tasks, multi-view data, multi-frame sources, and re-annotated temporal datasets.
Loading 2408.03326v3…