PaperlayerView exact revision on arXiv

Source-linked AI summary

LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligence

Xiang An, Yin Xie, Feilong Tang, Yunyao Yan, Huajie Tan, Didi Zhu, Changrui Chen, Xiuwei Zhao, Bin Qin, Kaicheng Yang, Yifei Shen, Yuanhan Zhang, Kaichen Zhang, Wenkang Zhang, Zheng Cheng, Nansen Zhang, Chunsheng Wu, Chunjiang Ge, Zimin Ran, Dehua Song, Chunyuan Li, Shikun Feng, Ming Hu, Zhangquan Chen, Junbo Niu, Bo Li, Ziyong Feng, Ziwei Liu, Zongyuan Ge, Jiankang Deng

arXiv:2605.25979v1cs.CV

TL;DR

Existing video models often allocate evidence uniformly, while fine-grained grounding in densely repeated motion remains underrepresented. LLaVA-OneVision-2 uses codec-stream tokenization to allocate tokens adaptively and achieves broad gains, including 74.9 JumpScore mAP versus Qwen3-VL-8B’s 30.1.

  • Problem

    Existing approaches lack selective evidence allocation for sparse semantic, spatial, and temporal structure, while densely repeated motion remains underrepresented in temporal evaluations.

  • Method

    LLaVA-OneVision-2 adaptively groups compressed video by bit-cost dynamics and condenses motion-residual evidence into visual canvases within a unified attention interface.

  • Results

    LLaVA-OV-2-8B improves over Qwen3-VL-8B by +4.3 video-task points, +5.3 spatial-task points, and +15.6 average J&F on tracking, reaching 74.9 JumpScore mAP versus 30.1.

  • Takeaways & Limitations

    Codec-stream inputs favor long-video tasks with coarse temporal structure, whereas frame sampling remains stronger for detail-sensitive motion reasoning requiring dense trajectory continuity.

  • Takeaways & Limitations

    Codec-stream tokenization trades frame-level evidential resolution for event-level coverage and is weaker than frame sampling on dense trajectory and detail-sensitive motion tasks.

Abstract

from arXiv · show

We introduce LLaVA-OneVision-2 (LLaVA-OV-2), the most capable vision-language model in the LLaVA-OneVision series to date, achieving superior performance across a broad range of multimodal benchmarks. The model builds on a native OneVision-Encoder and incorporates Windowed Attention for efficient local computation while maintaining native resolution. Its key advance is codec-stream tokenization: it treats compressed video as a continuous bit-cost stream, where bit-cost dynamics determine adaptive temporal groups, and motion-residual cues select salient spatial evidence into compact visual canvases. This allocation concentrates a limited token budget on event-bearing content, enabling more stable long-video token compression than fixed groups of pictures. A shared 3D RoPE further places codec canvases, sampled frames, and images in a unified spatiotemporal coordinate system. Furthermore, we build the LLaVA-OV-2 data and training stack around large-scale open supervision: approximately 8M re-captioned video samples for pretraining, a 4M-sample spatial corpus for fine-tuning. We also introduce JumpScore, a temporal-localization benchmark targeting fine-grained grounding in high-frequency, densely repeated motion, a regime underrepresented by existing video evaluations. A standout capability of LLaVA-OV-2 is its unified perception across video understanding, temporal grounding, spatial grounding, and manipulation-trace reasoning. On JumpScore, LLaVA-OneVision-2-8B reaches 74.9 JumpScore mAP, surpassing Qwen3-VL-8B (30.1) by +44.8 points; under matched visual-token budgets on the same benchmark, codec-stream inputs improve temporal grounding over frame sampling by +9.7 points. Across standard benchmarks, LLaVA-OneVision-2-8B further outperforms Qwen3-VL-8B by +4.3 average points on video tasks, +5.3 on spatial tasks, and +15.6 average J&F on tracking tasks.

1 Introduction

LLaVA-OneVision-2 advances beyond frame-centric observation by allocating visual evidence within predictive codec streams, using bit-cost and motion-residual cues for stable long-video compression. It combines this codec-aligned design with scaled supervision and JumpScore, achieving strong gains across video, spatial, tracking, and temporal-grounding evaluations.

  • Motivation and approach: LLaVA-OneVision-2 moves beyond uniform frame observation by selectively allocating evidence because sparse deviations carry discriminative semantic, spatial, and temporal structure.The motivation contrasts contextual pixel continuity with sparse, information-bearing deviations in predictive visual streams.
  • Results: 74.9 JumpScore mAP for LLaVA-OneVision-2-8B versus Qwen3-VL-8B’s 30.1, a +44.8-point advantage; codec-stream inputs add +9.7 points over frame sampling for temporal grounding.Codec-stream inputs favor long-video tasks with coarse temporal structure, whereas frame sampling remains preferable for detail-sensitive queries.
  • Motivation and approach: Codec-stream tokenization treats compressed video as a continuous bit-cost stream, aligning token allocation with bit-cost dynamics and motion-residual evidence.This design aims to enable stable long-video token compression while preserving native resolution through codec-adaptive attention.
  • Data and evaluation: Approximately 8M re-captioned video samples and a 4M-sample 2D/3D spatial corpus scale training, alongside JumpScore for fine-grained grounding in high-frequency dense motion.JumpScore targets localization of the correct action instance among many visually similar cycles.
  • Results: +4.3 points across 18 video tasks, +5.3 points across 11 spatial-reasoning tasks, and +15.6 average J&F across 4 tracking tasks over Qwen3-VL-8B.These are reported average improvements for LLaVA-OV-2-8B over the named baseline.

2 Architecture

LLaVA-OneVision-2 unifies sampled-frame videos, codec-stream videos, and native-resolution images through a shared OneVision-Encoder, lightweight connector, and Qwen3-8B decoder. Codec-stream tokenization adaptively assigns temporal groups by prediction bit-cost and selects spatial evidence using motion-residual saliency for compact visual canvases.

  • Architecture overview: The shared OneVision-Encoder maps sampled frames, codec-stream videos, and static images into visual tokens with patch embeddings, 3D coordinates, and group assignments.Shared 3D RoPE provides a common spatiotemporal coordinate system, while group-visible masks control token visibility.
  • Architecture overview: A two-layer MLP connector and shared Qwen3-8B autoregressive decoder process all input forms without codec-specific adapters or language-side branches.Codec processing changes the selected evidence and attention-group assignment while preserving the encoder–connector–decoder pathway.
  • Temporal grouping: Adaptive GOP partitioning uses P/B-frame packet bit-cost rather than elapsed time, producing shorter groups for high-change intervals and longer groups for predictable intervals.I-frame packets are excluded because P/B packets expose inter-frame prediction difficulty, motion, and residual change.
  • Spatial evidence selection: Motion vectors and luma residuals form a dense saliency map, while codec patch-GOP selection aggregates evidence into 2×2 patch blocks aligned with encoder merging.The block-level primitive preserves four spatially coherent patch tokens and incorporates a normalized patch-level bit-cost prior.
  • Canvas packing: Stratified temporal allocation constructs P-canvases from ranked 2×2 candidate blocks to avoid concentrating the token budget on one high-response frame.Candidates are grouped by source frame and ranked by block saliency before packing.

3 Training Data

The training-data recipe combines inherited image–text and video-instruction resources with an approximately 8M-clip, length-stratified video-caption corpus spanning 30 seconds to 15 minutes. Length stratification supplies long-context capability, while inherited video corpora provide instruction-following diversity without additional long-video instruction synthesis.

  • Image–text foundation: ~85M image–text pairs and ~22M instruction samples provide the image-text foundation, covering OCR, GUI, documents, grounding, counting, charts, and diagrams.The model initializes from LLaVA-OneVision-1.5 and reuses its mid-training and instruction corpora, supplemented by FineVision.
  • Inherited video instruction: 1.6M video-instruction samples from LLaVA-Video-178K, alongside VideoChat-Flash-Training-Data, Molmo2, and TimeLens, supply diverse video instruction.These inherited corpora cover captioning, open-ended QA, and multiple-choice QA, and are general video-instruction data rather than long-form sources.
  • Inherited video instruction: No additional long-video instruction data is synthesized: long-context capability comes from the length-stratified caption corpus, while inherited corpora provide instruction-following diversity.This explicitly separates the sources of long-context learning and instruction-following variety.
  • Length stratification as a design choice: ~8M captioned clips spanning 30 seconds to 15 minutes are stratified by length to improve temporal continuity beyond short-clip-dominated captioning.The corpus uses four length buckets and targets maintaining state across extended videos rather than only describing individual scenes.
  • Codec-aware re-encoding at Stage 4: 384 frames and 768 frames are both used for the 10–15-minute bucket in Stage 4 under codec-aware re-encoding, re-aligning the same caption to a denser visible-patch index.The densified pass produces no new captions.

4 Training Recipe

The LLaVA-OneVision-2 training recipe progresses through four stages, starting from an image-pretrained checkpoint and expanding from frame-sampled captions to long-video and spatial supervision. Stage 4 adds codec-stream training at enlarged visual budgets alongside structured spatial and tracking data.

  • The recipe runs in four progressive stages, spanning Sections 4.1–4.4.
  • Stage 1: Training initializes from the image-pretrained LLaVA-OneVision-1.5 8B checkpoint and bootstraps video awareness with short video-caption data.This stage combines LLaVA-OneVision-1.5-Mid-Training-85M with 30s-Video-Caption-4.2M, using standard frame sampling rather than codec-stream tokenization.
  • Stage 2: Stage 2 expands instruction training to medium-length clips, increasing the maximum frame budget from 30 to 90.Its mixture includes approximately 22M instruction samples, approximately 24M FineVision samples, and newly released 30s–60s and 60s–180s caption corpora.
  • Stage 3: Stage 3 extends training to long-form video understanding by combining established video instruction corpora with 10min–15min-Video-Caption-350K.
  • Stage 4: Stage 4 introduces codec-stream tokenization for long-video captions, using up to 384 frames and a densified variant with up to 768 frames.It also adds the 4M-pair LLaVA-OneVision-2-Spatial-4M corpus plus Molmo2-VideoTrack and Molmo2-VideoPoint for spatial, tracking, and spatio-temporal pointing supervision.

5 Implementation Details

LLaVA-OneVision-2’s training recipe uses cross-stage design choices spanning mixed-batch composition, progressively increasing frame budgets, and late-stage codec-stream scheduling. These choices combine codec-aware and frame-faithful evidence while extending observation from short videos to dense long-form inputs.

  • Cross-stage Design: Three cross-stage design choices govern the training recipe beyond the data mixture of any individual stage.These shared choices cannot be ablated by reading any single stage’s data list.
  • Mixed-batch Composition: Each training step mixes approximately 50% codec-patchified video, 37.5% uniform chunk-wise video, and 12.5% image inputs.The mixture exposes the model to stream-aware event allocation and frame-faithful visual evidence while preserving image understanding.
  • Frame-budget Schedule: The visible per-clip frame budget rises from 30 frames in Stage 1 to 60/90 in Stage 2, 384 in Stage 3, and 384/768 in Stage 4.The schedule matches supervision temporal span, progressing from short-video captions to long-video captions and codec re-encoding.
  • Cross-stage Codec Scheduling: Stages 1–3 use standard frame sampling, while codec-stream training begins in Stage 4 with variable-length-GOP, bit-cost-scored processing.The 10–15 minute video-caption corpus is re-encoded under the Stage-4 configuration at 384 and 768 frames.

6 JumpScore

JumpScore evaluates fine-grained temporal grounding in high-frequency, densely repeating jump-rope motion, where adjacent cycles are nearly identical and cycle boundaries carry the discriminative signal. It contains 189 in-the-wild videos with decimal-second annotations for every cycle start, defined when the rope passes behind the legs.

  • Motivation: JumpScore targets fine-grained grounding in high-frequency, densely repeating motion with visually near-identical adjacent cycles and boundary-based evidence.This contrasts with standard benchmarks focused on one-shot event localization where adjacent visual evidence is distinguishable.
  • Benchmark: 189 in-the-wild jump-rope videos provide decimal-second annotations for every cycle start, defined as the moment the rope passes behind the legs.The dataset is publicly released on Hugging Face.
  • Dataset Construction: The source videos cover multiple indoor scenes, camera angles, and capture devices, with durations concentrated in the 30–90 s range and typically tens of sub-second cycles.Every clip is at least 1280×720, while more than 84% are 1920×1080 or higher; timestamps are verified against the source frame rate.
  • Task and Metric: Each video uses the same natural-language prompt requesting decimal-precision start timestamps for every jump rope performed by the main character.The start is explicitly defined as the moment the rope is behind the legs, with no auxiliary modality provided.

7 Evaluation

LLaVA-OneVision-2-8B leads the 8B class across video and spatial benchmarks, while preserving strong but not specialized image and document understanding. Its largest advantages occur in temporal grounding, JumpScore, and spatial tasks where existing baselines are weakest.

  • Evaluation Setup: Comparisons use four 8B-class baselines, with matched token budgets, default prompts, and identical decoding strategies unless otherwise specified.The baselines are Qwen3-VL-8B, Keye-VL-1.5-8B, InternVL-3.5-8B, and LLaVA-OV-1.5-8B; PLM-8B appears where results are available.
  • Video Understanding: LLaVA-OneVision-2-8B achieves the highest average score across 18 video tasks, improving over Qwen3-VL-8B by +4.3 points (62.5 vs. 58.2).Its largest gains are on temporal grounding, visual–spatial video reasoning, and JumpScore temporal localization.
  • Spatial Reasoning: LLaVA-OneVision-2-8B achieves the best spatial average, improving over Qwen3-VL-8B by +5.3 points (63.5 vs. 58.2) across 11 tasks.CrossPoint improves by +35.0 points over Qwen3-VL-8B (61.9 vs. 26.9), while TraceSpatial-3D reaches nearly 4× the next-best 8B score (31.0 vs. 8.0).
  • Image and Document Understanding: LLaVA-OneVision-2-8B remains competitive on DocVQA (95.2), ChartQA (85.9), CountBench (89.0), Pixmo-Count (64.0), and RealWorldQA (69.7).It leads the 8B class on V∗-Bench (85.9).
  • Image and Document Understanding: On text-dense or diagram-heavy tasks, LLaVA-OneVision-2-8B trails specialized baselines, including Keye-VL-1.5 on MMStar and OCRBench and Qwen3-VL-8B on InfoVQA.PLM-8B leads AI2D, indicating that the model is not OCR- or document-specialized.

8 Ablation of Codec-Stream Tokenization

Codec-stream tokenization improves event-level temporal grounding by allocating visual evidence according to compressed-stream dynamics and motion residuals, while preserving parity or small gains on long-form video QA. Its advantage is clearest for densely repeated motion and discrete transitions, whereas frame sampling remains stronger for dense trajectory continuity.

  • Ablation Overview: 17.3-point average gain on JumpScore and 9.7-point average gain on benchmarks over uniform frame sampling demonstrate codec-stream tokenization’s event-level advantage.Comparisons keep the backbone, language model, decoder, prompts, and evaluation protocol fixed.
  • Long-form Video Question Answering: Codec-stream inputs preserve parity or small gains on LVBench, VideoMME-Long with subtitles, and VideoEval-Pro.The result indicates that reallocating capacity toward event-bearing moments does not sacrifice broad semantic coverage in long-form video QA.
  • Temporal Grounding: Bit-cost dynamics identify informative temporal segments, while motion-residual cues retain spatial evidence around perceptual transitions for temporal grounding.This allocation avoids missing short event intervals that fall between uniformly sampled frames.
  • Fine-grained Grounding in Densely Repeated Motion: Codec-stream tokenization localizes transitions between visually similar cycles, where uniform frame sampling and similarity-based deduplication are least reliable.JumpScore targets high-frequency, densely repeated motion whose cycle boundaries distinguish the correct action instance.
  • Competence Boundary: Frame sampling remains stronger for dense trajectory continuity, including future event prediction, dyadic interaction, motion properties, and motion trajectory estimation.Codec-stream inputs instead gain on salient spatial snapshots or discrete state transitions, including recognition, ordering, and counting.

9 Related Work

LLaVA-OneVision-2 is positioned among open video MLLMs, video-token efficiency methods, temporal grounding, spatial cognition, and referring video object segmentation. Its codec-stream tokenization is distinguished from prior approaches by patch-level adaptive selection and an explicit, unified training-and-inference pipeline.

  • Open video MLLMs: 8B-class open multimodal models commonly combine a vision transformer, a connector, and an instruction-tuned language model, with Qwen, InternVL, Keye-VL, NVILA, and LLaVA-OneVision among representative releases.Other cited representatives include VideoLLaMA 3, MiniCPM-V, Aria, Eagle 2, Apollo, Oryx-MLLM, Tarsier 2, LongVU, LongVILA, InternVideo2.5, and Penguin-VL.
  • Video-token efficiency: Prior efficiency methods include token dropout, token merging, hierarchical clip-to-video compression, one-token-per-frame summarisation, compressed-domain tokenization, and adaptive frame sampling.These approaches address redundancy because most patches across most frames are not informative, but they differ in where and how compression is applied.
  • Video-token efficiency: LLaVA-OneVision-2 selects evidence at patch level under a global clip budget, whereas adaptive frame samplers select whole frames and leave the kept frames’ quadratic self-attention cost unmitigated.Its codec-stream pipeline makes variable-length GOP, bit-cost versus MV+residual scoring, and stratified block selection explicit and tunable.
  • Temporal grounding: Temporal-grounding work extends frame-sampled video LLMs with timestamp tokens, boundary-aware training, per-event captioning supervision, or specialised time-token vocabularies.Representative systems include VTimeLLM, TimeChat, LITA, VTG-LLM, and Grounded-VideoLLM.
  • Spatial cognition and R-VOS: Related evaluation lines address long-horizon spatial state, spatiotemporal evidence, spatial-functional intelligence, and cross-frame reference coherence through Cambrian-S, ReVSI, VAEX-Bench, SFI-Bench, MeViS, and ReVOS.R-VOS systems commonly pair an LLM with a promptable segmentation backbone such as SAM 2; VideoLISA uses a single <TRK> token with sparse-dense sampling.

10 Conclusion · 11 Contributors

The conclusion presents LLaVA-OneVision-2 as a codec-aligned long-video MLLM that allocates perceptual evidence from compressed video streams rather than relying only on frame-centric sampling. It also summarizes the model’s native-resolution architecture and open training stack, while the following section identifies contributors and project leaders.

  • 10 Conclusion: LLaVA-OneVision-2 moves long-video observation beyond frame-centric sampling toward stream-aware perceptual evidence allocation.The conclusion characterizes the model as a codec-aligned long-video multimodal language model.
  • 10 Conclusion: Bit-cost dynamics determine adaptive temporal groups, while motion-residual evidence is condensed into compact visual canvases.These mechanisms operate on compressed video treated as a continuous bit-cost stream.
  • 10 Conclusion: The model uses a native-resolution OneVision-Encoder and a unified group-visible attention interface.The interface processes the compact visual canvases produced from motion-residual evidence.
  • 10 Conclusion: The progressive open training stack includes approximately 8M re-captioned video samples and a 4M-sample 2D&3D spatial corpus.These datasets are presented as part of the conclusion’s summary of the training stack.
  • 11 Contributors: The contributors section lists core contributors including Xiang An, Yin Xie, Feilong Tang, Yunyao Yan, and Huajie Tan.The passage continues with additional core-contributor names.
  • 11 Contributors: The project leaders are Bo Li, Ziyong Feng, Ziwei Liu, Zongyuan Ge, and Jiankang Deng.The passage labels these individuals as project leaders.

12 Details

Codec-stream inputs generally outperform uniform frame sampling, with the largest gains on motion-sensitive temporal localization under limited frame budgets. LLaVA-OneVision-2-8B also achieves stronger tracking results than Qwen3-VL-8B across the reported evaluation splits.

  • Comparison setup: Matched nominal frame budgets compare codec-stream inputs against uniformly sampled RGB frames.“Uniform” denotes standard uniform frame sampling, while “Codec” denotes the codec-stream representation.
  • Long-video QA: 1.2 points, 1.7 points, and 2.6 points are the average gains on VideoMME-L-sub, LVBench, and VideoEval-Pro, respectively.Gains are mostly positive but not monotonic at every budget; VideoMME-L-sub drops slightly at 32 and 128 frames.
  • JumpScore: 17.3 points is codec-stream’s average JumpScore gain, increasing the score from 37.9 to 55.2 across six budgets.At 4 frames, the score rises from 32.5 to 39.4, a 6.9-point or 21.2% relative improvement.
  • Temporal grounding: 15.4, 17.1, and 26.6 points are QVHighlights gains at 4, 8, and 16 frames, while Charades-STA rises from 17.4 to 42.4 at 4 frames.ActivityNet Captions improves at every evaluated budget, with gains ranging from 2.3 to 13.3 points.
  • Tracking: +10.2 absolute points is LLaVA-OneVision-2-8B’s overall tracking advantage over Qwen3-VL-8B, with J&F gains across all four splits.The split gains are +17.4 on DAVIS, +17.3 on MeViS-U, +20.4 on REVOS-Referring, and +7.3 on REVOS-Reasoning.

13 Case Study

The case studies demonstrate unified temporal grounding, referring-video tracking, spatial grounding, and online robot manipulation. Codec-stream sampling sharply improves cycle-start localization over uniform sampling at matched visual-token budgets.

  • Referring video object segmentation: The ReasonVOS case tracks a cat moving diagonally across a 36-frame clip, with per-frame tracking points converted into dense masks by SAM2.The case reports J&F 0.939 and HOTA 0.954.
  • Referring video object segmentation: The model emits per-frame tracking points that remain aligned through tire smoke, motion blur, and viewpoint change in a 52-frame Ref-DAVIS17 sport-car case.Dense masks are recovered downstream with SAM2.
  • Robot manipulation: On two tabletop manipulation tasks, the model re-queries online three times and returns image-space (x, y, z) waypoints for downstream inverse kinematics.Waypoint counts contract near the target, and trajectories update when the scene state changes.
  • JumpScore validation: 82 of 85 cycle starts are correctly attributed by codec-stream sampling, versus 14 of 85 for uniform 128-frame sampling at matched visual-token budget.Mean IoU is 0.894 for codec-stream sampling versus 0.116 for uniform sampling, because tokens concentrate on motion-residual regions where cycle boundaries occur.
  • Spatial grounding: The case studies include eight 2D pointing examples and five 3D pick-and-place examples, using pixel coordinates and continuous image-space trajectories with relative depth.The 2D cases cover object references, relational queries, and free-space queries.
Loading 2605.25979v1…