PaperlayerView exact revision on arXiv

Source-linked AI summary

Open-o3-Video-v2: Seek, Ground, and Reason with Verifiable Spatio-Temporal Evidence

Jiahao Meng, Zhongyuan Bao, Yue Tan, Haiquan Wen, Renrui Zhang, Anna Wang, Jason Li, Yanwei Fu, Yunhai Tong, Ming-Hsuan Yang

arXiv:2610.17656v1cs.CV

TL;DR

Reliable long-video reasoning needs visual evidence that supports both intermediate reasoning and final answers, but fixed sparse observations can miss decisive clues. Open-o3-Video-v2 actively acquires and grounds observations within its reasoning trajectory, and its 4B model improves V-STaR results over its Qwen3-VL-4B backbone.

  • Problem

    Sparse observations may miss brief actions or small objects, while dense processing is costly; reasoning also needs visual evidence linked to the claims it supports.

    Source: §1 · pp. 1, 3

  • Method

    Open-o3-Video-v2 identifies missing information, acquires targeted observations, grounds supporting moments and objects, and reuses that evidence in subsequent reasoning.

    Source: §1 · p. 3

  • Results

    Open-o3-Video-v2-4B achieves 40.4% mAM and 56.3% mLGM on V-STaR, improving over Qwen3-VL-4B by 8.5% and 12.6%, respectively, with gains across other benchmarks.

    Source: §1 · p. 3, §6 · p. 14

  • Takeaways & Limitations

    Experiments show consistent improvements in grounded reasoning, general video understanding, and temporal localization from jointly seeking, grounding, and reasoning with explicit visual evidence.

    Source: §6 · p. 14

    The predecessor Open-o3-Video remains restricted to fixed visual context and has grounded supervision limited in scale and diversity.

    Source: §3.3 · p. 5

Abstract

from arXiv · show

Reliable video reasoning requires models to produce correct answers and expose supporting visual evidence. Open-o3-Video enables verifiable reasoning by embedding timestamps and object-level bounding boxes into reasoning traces, but is restricted to frames from an initial sparse sampler. We introduce Open-o3-Video-v2, which formulates video reasoning as a unified agentic evidencegrounded process. The model identifies missing information, acquires additional observations, grounds the moments and objects supporting its reasoning, and writes this evidence into the trajectory before continuing or answering. To support this behavior, we construct STGR-v2-CoT135K, including 52K multi-turn agentic grounded reasoning trajectories, and STGR-v2-RL-13K for reinforcement learning. We further propose Clue-Seek GRPO (CS-GRPO), which tracks how tool calls change accumulated evidence, rewards clue coverage and step-wise gains, and penalizes excessive, repetitive, or uninformative retrieval. On V-STaR, Open-o3-Video-v2-4B achieves 40.4% mAM and 56.3% mLGM, outperforming its Qwen3-VL-4B backbone by 8.5% and 12.6%, respectively. It also delivers consistent gains across general video understanding, long-video reasoning, and temporal grounding benchmarks. These results show that active evidence acquisition and verifiable object-level grounding are complementary for reliable video reasoning. The code and models are released at https://msalab-pku.github.io/projects/Open-o3-Video-v2.

1 Introduction

Reliable long-video reasoning must connect its inferences to visual evidence, yet sparse fixed observations can miss decisive clues. Open-o3-Video-v2 seeks additional observations, grounds them to moments and objects, and reuses them in its reasoning trajectory.

1 Introduction

  • Sparse sampling can miss brief actions or small objects, while dense processing is costly and may introduce irrelevant context.Reliable reasoning therefore requires locating visual evidence and linking it to defensible inferences.Source: §1 · pp. 1–2
  • Open-o3-Video embeds timestamps and object boxes in its reasoning traces, but remains restricted to initially sampled frames.Its datasets jointly supervise temporal and spatial grounding, general video QA, and spatio-temporal grounded reasoning.Source: §1 · pp. 2–3
  • Open-o3-Video-v2 identifies unresolved information, acquires targeted observations, grounds supporting moments and objects, and reuses that evidence in later reasoning.The model can revise its interpretation, continue seeking, or answer from the accumulated evidence.Source: Figure 1 · p. 2, §1 · p. 3
  • STGR-v2-CoT-135K provides 52K multi-turn grounded trajectories, while STGR-v2-RL-13K supplies informative rollouts for policy optimization.The trajectories connect sparse observations, targeted crops, returned frames, object-level grounding, and answers; CS-GRPO rewards clue coverage and useful evidence gains while regularizing uninformative retrieval.Source: §1 · p. 3
  • 40.4% mAM and 56.3% mLGM on V-STaR: Open-o3-Video-v2-4B improves over Qwen3-VL-4B by 8.5% and 12.6%, respectively.The paper also reports gains over the baseline on general video understanding and temporal-grounding benchmarks.Source: §1 · p. 3

2 Related Work

Prior video-reasoning work includes fixed-context text-based approaches and systems that actively acquire video observations. Open-o3-Video-v2 combines iterative evidence acquisition with explicit spatio-temporal grounding in the reasoning process.

Video Spatial and Temporal Grounding

  • Video grounding methods localize event times or objects, while joint spatio-temporal methods combine temporal and spatial evidence in different architectures.Open-o3-Video places timestamps, object references, and boxes in a reasoning trace, but operates in one round on the initial observation.Source: §2 · pp. 3–4

Video Reasoning

  • Text-only video reasoning produces textual intermediate states but does not expose precise visual support for them.These approaches use chain-of-thought supervision, reinforcement learning, preference optimization, or staged captions, memories, and retrieval.Source: §2 · p. 4

Thinking with Video

  • Agentic video methods vary from requesting clips or frames to temporal cropping, multi-hop clue seeking, and instance-level spatio-temporal perception.Most retrieve additional observations without linking them to the spatio-temporal evidence supporting reasoning steps.Source: §2 · p. 4
  • Open-o3-Video-v2 extends timestamped object-box reasoning with iterative acquisition, grounding retrieved observations to supporting moments and objects for human-verifiable evidence.Temporal calls recover observations missing from the initial context.Source: §2 · p. 4

3 Open-o3-Video

Open-o3-Video represents reasoning with timestamps and object boxes, supported by grounded data and training rewards. Its fixed visual context and limited supervision motivate the move to active evidence acquisition.

3 Open-o3-Video

  • Open-o3-Video links reasoning claims to timestamped frames, object names, and bounding boxes in parseable grounded traces.Its STGR-CoT-30K and STGR-RL-36K corpora mix temporal grounding, spatial grounding, spatio-temporal reasoning, and general video QA.Source: §3.1 · p. 4, §3 · p. 4

3.1 STGR Data Construction

  • STGR data construction combines annotation, bounding-box filtering, and self-consistency checks; the v2 pipeline adds multi-turn evidence annotation, CoT rewriting with tool observations, and quality checks.STGR-v2-CoT-135K contains 135K SFT samples, including 59K spatio-temporal grounded reasoning samples; STGR-v2-RL-13K contains 13K filtered RL samples.Source: Figure 2 · p. 5, §3.1 · p. 5

3.2 Training with Grounded Thinking Rewards

  • SFT teaches responses with <think> and <answer> blocks, while RL scores answer correctness, grounded thinking, and output format.The grounded-thinking reward evaluates cited evidence: temporal proximity is gradually tightened for point supervision, and spatial box IoU is rewarded only when the timestamp is sufficiently accurate.Source: §3.2 · p. 5

3.3 Remaining Challenges in Open-o3-Video

  • Open-o3-Video cannot request denser observations when sparse sampling misses a critical clue, and its grounded supervision remains limited in scale and diversity.Temporal cropping can retrieve intervals, but does not by itself identify which moment and visual entity support a reasoning step.Source: §3.3 · p. 5

4 Open-o3-Video-v2

Open-o3-Video-v2 shifts grounded reasoning from fixed observations to active acquisition, grounding, and integration of missing visual evidence. It combines multi-turn supervision with optimization of evidence acquisition over interaction trajectories.

4 Open-o3-Video-v2

  • Open-o3-Video-v2 extends grounded reasoning to an active process in which the model identifies missing information, inspects relevant intervals, and grounds observations for later inference.The model can continue seeking, revise its interpretation, or answer from accumulated evidence.Source: §4 · p. 6

4.1 Task Formulation

  • The task formulation interleaves reasoning, clip requests, returned observations, timestamp-object-box evidence, and a final answer, conditioned on the interaction history.A sparse sampler supplies the initial observation; tool calls specify an interval and sampling density.Source: §4.1 · pp. 6–7

4.2 STGR-v2: Data Optimization

  • The 52K agentic trajectories are filtered for answerability, annotated with objects and boxes, rewritten into grounded tool-use reasoning, and checked for localization, call validity, and evidence support.Samples failing the checks are removed unless their inconsistency can be resolved unambiguously.Source: §4.2 · p. 7
  • STGR-v2-CoT-135K combines 52K multi-turn and 7K single-turn grounded QA samples with tool-use, localization, general QA, and caption data; audits found over 90% met criteria at each audited stage.The RL corpus combines v1, agentic tool-use, and temporal-grounding data.Source: §4.2 · pp. 7–8

4.3 Clue-Seek GRPO

  • CS-GRPO tracks evolving clue coverage, rewards sufficient step-wise evidence gains, and regularizes excessive or uninformative acquisition patterns.Its trajectory-level score makes search location, evidence improvement, and stopping decisions optimizable.Source: §4.3 · pp. 8–9

4.4 Reward Design

  • The reward objective combines answer, format, grounded-thinking, and tool-use components, with thinking rewards providing primary supervision for object-level grounding correctness.The tool reward follows the temporal evidence-acquisition score defined for CS-GRPO.Source: §4.4 · pp. 9–10

4.5 Training Recipes

  • Training applies assistant-token-only supervised fine-tuning to conversational samples, then uses CS-GRPO to optimize complete agent-loop trajectories.Evidence-state transitions supervise successive observation decisions during the RL stage.Source: §4.5 · p. 10

4.6 Training Infrastructure

  • The training infrastructure uses ms-swift for scalable mixed-corpus SFT and a verl AgentLoop to execute crop_video between model turns during RL.Returned frames are context, while model-generated tokens remain policy actions.Source: §4.6 · p. 10

5 Experiments

The experiments evaluate grounded question answering, general video understanding, and temporal grounding. Open-o3-Video-v2 improves V-STaR and broader benchmarks, while ablations examine its training stages, reasoning components, and evidence-seeking rewards.

5 Experiments

  • Evaluation covers grounded question answering, general video understanding, and temporal grounding, with V-STaR assessing two reasoning orders and answer, temporal, and visual alignment.The general-understanding and temporal-grounding suites include benchmarks for long-video comprehension, multi-hop reasoning, and interval localization.Source: §5 · p. 10, Table 2 · p. 11

5.1 Main Results

  • On V-STaR, relative to Qwen3-VL-4B, Open-o3-Video-v2-4B gains 8.5% mAM and 12.6% mLGM; the 8B model reaches 40.7% mAM and 56.5% mLGM.Source: §5.1 · p. 11, Table 2 · p. 11
  • Across broader benchmarks, v2-4B improves Video-Holmes by 10.7%, LVBench by 7.2%, and Charades-TimeLens mIoU by 4.2%.The reported gains span multi-hop clue integration, long-video understanding, and temporal grounding.Source: Table 3 · p. 12, §5.1 · p. 12
  • In two qualitative comparisons, v2 inspects the relevant intervals and grounds supporting objects, while Video-o3 searches incorrectly and gives a wrong prediction.The examples show pigs grounded at 1310 s and three girls grounded at 469 s for a group handshake.Source: Figure 4 · p. 13

5.2 Ablation Studies and Analysis

  • SFT raises the Qwen3-VL-4B baseline by 6.3% mAM and 9.7% mLGM but lowers VideoMME accuracy from 65.3% to 63.7%; CS-GRPO then improves all three measures.The final model exceeds the base by 8.5% mAM, 12.6% mLGM, and 1.3% VideoMME accuracy.Source: Table 4 · p. 13, §5.2 · p. 13
  • With the same backbone, data, and evaluation setting, multi-turn clue seeking raises Video-Holmes, LVBench, and V-STaR mLGM; explicit grounding further raises them to 48.0%, 53.2%, and 56.3%.The comparisons isolate seeking beyond the initial context and the additional value of associating observations with localized evidence.Source: §5.2 · pp. 13–14, Table 5 · p. 14
  • CS-GRPO outperforms GRPO by 1.3, 1.2, and 1.5 points on Video-Holmes, LVBench, and VideoMME; removing either clue-gain rewards or retrieval penalties degrades performance.The reward tracks accumulated evidence changes and regularizes unhelpful retrieval.Source: Table 6 · p. 14, §5.2 · p. 14

6 Conclusion

Open-o3-Video-v2 makes video reasoning an agentic, evidence-grounded process and introduces data and training methods to support it. Experiments show consistent improvements across grounded reasoning, general video understanding, and temporal localization.

6 Conclusion

  • Open-o3-Video-v2 jointly seeks, grounds, and reasons with explicit visual evidence, with experiments showing consistent improvements across grounded reasoning, general video understanding, and temporal localization.The model identifies missing information, acquires targeted observations, and integrates decisive moments and grounded objects into subsequent inference.Source: §6 · p. 14
  • The paper introduces STGR-v2-CoT-135K, STGR-v2-RL-13K, and CS-GRPO to make agentic evidence-grounded reasoning effective and efficient.Source: §6 · p. 14

Appendix

  • The appendices cover data and annotation quality, training and inference settings, reward and scaling analyses, and successful and failed evidence-acquisition cases.Source: Appendix · p. 20

A.1 Data Sources and Composition

The data mixture combines temporal, spatial, spatio-temporal grounding, and general video QA. New spatio-temporal annotations link answer reasoning to key frames and objects, with filtering and reward mechanisms supporting grounded supervision.

A.1 Data Sources and Composition

  • STGR-CoT-30K and STGR-RL-36K combine temporal, spatial, spatio-temporal, and general video QA data to teach grounding while preserving broad video understanding.The SFT corpus interleaves reasoning with timestamps and boxes; the RL corpus uses task-specific rewards for answer correctness and grounded thinking.Source: §A.1 · p. 20

A.2 Spatio-Temporal Annotation

  • Each new spatio-temporal sample links a question–answer pair and reasoning trace to one to five key frames and one to three object boxes per frame.The reasoning trace must cite the annotations so textual rationale aligns with machine-verifiable temporal and spatial evidence.Source: §A.2 · p. 20

A.3 Filtering and Consistency Checking

  • Three filtering levels remove oversized or mismatched boxes, missing structural references, and reasoning sentences unsupported by their visual evidence.Samples with invalid boxes or inconsistent visual-text evidence are discarded.Source: §A.3 · pp. 20–21

A.4 Grounded Thinking Reward Details

  • Adaptive Temporal Proximity anneals the temporal-reward neighborhood from broad to narrow, while Temporal Gating computes spatial reward only when a predicted timestamp is sufficiently close.The spatial reward uses matched box IoU, with the gating threshold denoted by δ.Source: §A.4 · p. 21

B Additional Details on STGR-v2

Human audits assess annotation and trajectory quality, while agreement statistics indicate consistency across evidence annotation and trajectory-quality review. The audit also identifies recurring grounding and reasoning error patterns used in filtering.

B Additional Details on STGR-v2

  • 94.0% and 91.6% of audited examples satisfy the quality criteria after stages (2) and (3), respectively.The 500-example audit checks object references and boxes after stage (2), then tool-call validity, reasoning quality, and evidence consistency after stage (3).Source: B Additional Details on STGR-v2 · p. 21
  • Audited error patterns include object-referent mismatches, inaccurate boxes, invalid tool-call intervals, hallucinated descriptions, and reasoning-evidence mismatches.Representative patterns are incorporated into consistency checking and filtering.Source: B Additional Details on STGR-v2 · p. 21
  • Annotator agreement is 94.2% with κ = 0.88 after stage (2), and 92.6% with κ = 0.84 after stage (3).Disagreements are resolved by jointly re-inspecting the video, annotations, and rewritten reasoning trajectory.Source: B Additional Details on STGR-v2 · pp. 21–22

C Additional Implementation Details

Open-o3-Video-v2 uses a two-stage SFT and CS-GRPO training recipe, with video observations configured differently for training and inference. Inference permits additional video crops within a bounded interaction budget.

C Additional Implementation Details

  • For the 4B model, SFT trains for one epoch on STGR-v2-CoT-135K, and CS-GRPO starts from the SFT checkpoint with eight sampled trajectories per prompt.The SFT global batch size is 16; CS-GRPO batches contain 32 prompts, and its actor objective uses a low-variance KL loss with coefficient 0.001.Source: C Additional Implementation Details · p. 22
  • SFT samples up to 96 frames per observation, while CS-GRPO uses an initial cap of 32 frames and crop caps of 8, 16, and 32 frames.SFT targets 1 FPS initially and 1, 2, and 4 FPS for coarse, medium, and fine crops; CS-GRPO uniformly samples within requested intervals.Source: C Additional Implementation Details · p. 22
  • At inference, the model starts with up to 96 frames sampled at a target 1 FPS and may use crop_video within a maximum of five assistant turns.Five turns is an upper bound, allowing the model to stop when it has accumulated sufficient evidence.Source: C Additional Implementation Details · p. 22

D Open-o3-Video Ablation Studies

The ablations find complementary gains from supervised grounded traces, reward-based optimization, and spatio-temporal supervision. Evidence-removal and confidence-aware voting tests further examine whether grounded evidence supports predictions and can guide inference-time selection.

Contribution of the two training stages

  • SFT improves the Qwen2.5-VL-7B backbone by 9.2% mAM and 14.7% mLGM; RL adds 5.2% mAM, 9.5% mLGM, and 2.9% VideoMME accuracy over SFT.Together, the stages exceed the backbone by 14.4% mAM, 24.2% mLGM, and 1.2% VideoMME accuracy.Source: D Open-o3-Video Ablation Studies · p. 23

Grounded-thinking reward

  • Removing adaptive temporal proximity reduces V-STaR mAM/mLGM by 0.7%/1.4%, while removing temporal gating reduces them by 1.4%/1.7%.Temporal gating reserves spatial reward for temporally valid moments; adaptive proximity broadens early reward before tightening the criterion.Source: D Open-o3-Video Ablation Studies · p. 23, Table 10 · p. 23

Spatio-temporal supervision

  • Adding filtered and rewritten VideoEspresso samples improves mAM/mLGM by 2.8%/7.4%, and adding the authors’ annotations yields a further 2.6%/3.0%.The combined gains are 5.4% mAM and 10.4% mLGM.Source: D Open-o3-Video Ablation Studies · p. 23

Faithfulness of explicit evidence

Confidence-aware test-time scaling

  • Confidence-aware voting improves the single-response result by 1.0% on WorldSense and 1.8% on VideoMMMU, and exceeds majority voting by 1.2% and 1.0%, respectively.It weights each of eight sampled responses by the average evidence score of its cited object crops.Source: D Open-o3-Video Ablation Studies · p. 24, Table 13 · p. 24

E More Analysis

The analysis finds that moderate initial observation and bounded interaction can support adaptive evidence seeking, including on long-horizon questions. It also reports improved answer–grounding consistency and illustrates how targeted observations and object grounding help with fine-grained reasoning and counting.

E More Analysis

  • CS-GRPO remains relatively stable across tested reward-weight ranges, while increasing some weights from very small values to defaults benefits optimization.The varied terms include evidence-gain rewards and penalties on inefficient tool use.Source: E More Analysis · p. 24, Table 15 · p. 25
  • Increasing initial frames from 32 to 96 yields the largest gains; with five turns, LVBench rises from 46.2% to 52.7% and Video-Holmes from 44.5% to 47.0%.Beyond 96 frames, LVBench largely saturates around 53–54%, while Video-Holmes changes little through 192 frames and slightly declines at 384.Source: E More Analysis · pp. 24–25, Figure 5 · p. 25
  • More interaction turns help, but gains diminish as budgets grow; LVBench largely saturates by five turns.The analysis supports a moderate initial frame budget and bounded interaction horizon rather than dense input or excessive tool use.Source: E More Analysis · pp. 24–25, Figure 5 · p. 25
  • With a comparable frame budget, adaptive seeking improves VideoMME, Video-Holmes, and LVBench by 1.2, 0.6, and 7.7 points over uniform sampling.The full model averages 2.2 assistant turns and approximately 120 frames per question; Table 16's uniform baseline uses 120 frames.Source: E More Analysis · p. 25, Table 16 · p. 26
  • Replacing retrieved clips with length-matched random intervals reduces accuracy by 2.0% on VideoMME, 0.9% on Video-Holmes, and 7.0% on LVBench.The test keeps call count and returned-frame count unchanged.Source: E More Analysis · pp. 25–26, Table 16 · p. 26
  • On a 149-question LVBench subset of videos longer than 45 minutes, Open-o3-Video-v2 reaches 48.3% accuracy, beating Qwen3-VL-4B and Video-o3 by 10.7% and 10.0%.The subset covers event chains, ordinal localization, and repeated-instance tracking, with median video length of 79 minutes.Source: E More Analysis · p. 26, Table 17 · p. 26
  • Open-o3-Video-v2 raises joint QA–spatial–temporal success on V-STaR Chain1 from 11.7% to 21.7%, while the grounded fraction of correctly answered questions rises from 19.6% to 34.5%.QA–spatial correctness rises from 40.1% to 50.1%, and QA–temporal correctness from 15.6% to 26.4%.Source: E More Analysis · p. 26

F.1 Successful Cases

  • Qualitative cases show targeted interval inspection and object grounding revealing decisive details for fine-grained action, event, and counting questions.Examples include distinguishing food images and car occupants, and identifying repeated removal of a pineapple’s fibrous core.Source: §F.1 · pp. 26–27

F.2 Failure Analysis

The failure analysis identifies temporal evidence retrieval and event selection as the largest error category on LVBench. Examples also expose remaining challenges in identity knowledge, instance association across changing views, and choosing the relevant event among nearby alternatives.

F.2 Failure Analysis

  • On LVBench, temporal evidence retrieval and event selection account for approximately 43% of errors, while object grounding and fine-grained perception account for 27%.Knowledge-related errors account for approximately 6%; remaining errors mainly involve reasoning inconsistencies or cannot be assigned to one factor.Source: §F.2 · p. 27
  • Representative failures include misidentifying a retrieved face, confusing instances during a camera pan, and grounding a scoring event other than the game-ending one.The examples point to limits in identity knowledge, cross-view instance association, and temporal event selection.Source: §F.2 · p. 27, Figure 8 · p. 30
  • Future work targets longer-horizon agents that integrate visual evidence with world knowledge and analyze extremely long videos.The authors also identify fine-grained perception and temporal event discrimination as areas needing improvement.Source: §F.2 · p. 27

Related papers

Loading 2610.17656v1…