Open-o3-Video: Grounded Video Reasoning with Explicit Spatio-Temporal Evidence

Jiahao Meng; Xiangtai Li; Haochen Wang; Yue Tan; Tao Zhang; Lingdong Kong; Yunhai Tong; Anran Wang; Zhiyang Teng; Yujing Wang; Zhuochen Wang

Source-linked AI summary

Open-o3-Video: Grounded Video Reasoning with Explicit Spatio-Temporal Evidence

Jiahao Meng, Xiangtai Li, Haochen Wang, Yue Tan, Tao Zhang, Lingdong Kong, Yunhai Tong, Anran Wang, Zhiyang Teng, Yujing Wang, Zhuochen Wang

arXiv:2510.20579v2Oct 23, 2025cs.CVcs.AIcs.MM

TL;DR

Video reasoning models typically lack explicit evidence for when and where relevant events occur, especially in dynamic scenes. Open-o3-Video addresses this gap with unified spatio-temporal data and staged grounded training, achieving state-of-the-art V-STAR results and consistent gains across video benchmarks.

Problem
Reliable fine-grained video reasoning requires joint temporal and spatial evidence, but existing resources and models often provide only text-only or unsynchronized grounding supervision.
Method
Open-o3-Video combines STGR datasets with cold-start initialization and reinforcement learning using adaptive temporal proximity and temporal gating.
Results
+14.4% mAM and +24.2% mLGM over Qwen2.5-VL on V-STAR, with consistent gains across diverse video understanding benchmarks.
Takeaways & Limitations
Grounded reasoning traces make predictions more traceable and support confidence-aware test-time scaling for improved answer reliability.
Takeaways & Limitations
Long videos with complex scenes and small objects, multi-step reasoning beyond direct grounding, and audio or speech cues remain challenging.

Abstract

from arXiv · show

Most video reasoning models only generate textual reasoning traces without indicating when and where key evidence appears. Recent models such as OpenAI-o3 have sparked wide interest in evidence-centered reasoning for images, yet extending this ability to videos is more challenging due to the need for joint temporal tracking and spatial localization across dynamic scenes. We introduce Open-o3-Video, a non-agent framework that integrates explicit spatio-temporal evidence into video reasoning by highlighting key timestamps, objects, and bounding boxes, making the reasoning process traceable and verifiable. To enable this capability, we first construct high-quality datasets STGR that provide unified spatio-temporal supervision, which is absent in existing resources. We further adopt a cold-start reinforcement learning strategy with specially designed rewards that jointly encourage answer accuracy, temporal alignment, and spatial precision. On the V-STAR benchmark, Open-o3-Video achieves state-of-the-art performance, improving mAM by 14.4% and mLGM by 24.2% over the Qwen2.5-VL baseline, and shows consistent gains across a range of video understanding benchmarks. Beyond accuracy, the grounded reasoning traces produced by Open-o3-Video support confidence-aware test-time scaling, improving answer reliability.

1 Introduction

Open-o3-Video addresses the difficulty of reliable video reasoning by grounding textual reasoning in explicit timestamps and object regions. It combines STGR data with staged training and reports state-of-the-art V-STAR performance plus gains across other video tasks.

Motivation: Explicit visual operations in image reasoning motivate extending evidence-centered reasoning to videos.Video extension is harder because evidence must remain coherent across changing timestamps and regions.
Motivation: Video reasoning remains difficult because dynamic scenes require precise localization across time and space amid motion, occlusions, and camera changes.Prior approaches often provide only textual rationales or coarse temporal grounding.
Contribution: Open-o3-Video embeds timestamped key frames, localized object regions, and reasoning links into a unified spatio-temporal evidence process.Its STGR datasets include 5.9k newly annotated spatiotemporal samples with structured grounding supervision.
Method: The method uses cold-start training followed by reinforcement learning with adaptive temporal proximity and temporal gating.These mechanisms provide progressively stricter temporal supervision and reliable spatial evaluation.
Results: +14.4% mAM and +24.2% mLGM over Qwen2.5-VL on V-STAR, with consistent gains across VideoMME, WorldSense, VideoMMMU, LongVideo-Reason-eval, and TVGBench.Evidence-aligned frames also support confidence-aware test-time scaling, including +1.2% on WorldSense and 1.0% on VideoMMMU.

2 Related works

Related work spans reinforcement-learning approaches to video reasoning, temporal and spatial grounding, and explicit visual operations for multimodal reasoning. Open-o3-Video builds on these directions while targeting joint spatio-temporal evidence.

Video Reasoning: Video-R1 and related methods use reinforcement learning and temporal-aware reasoning to improve video understanding, while VideoChat-R1 extends toward grounding and tracking.These approaches motivate post-training methods that move beyond direct question answering.
Temporal and Spatial Grounding: Temporal grounding methods improve localization of relevant time spans, whereas spatial grounding methods focus on object-centric localization and geometric reasoning.The two lines address complementary dimensions of video evidence.
Thinking with Images: “Thinking with Images” methods use cropping, zoom-in, and region selection to provide intermediate visual evidence within reasoning chains.This image-based paradigm motivates explicit evidence operations in video reasoning.

3 STGR Data Construction

STGR constructs unified supervision linking timestamps, object boxes, and reasoning chains, addressing the fragmentation of existing temporal-only, spatial-only, and text-only resources. Its pipeline annotates, filters, and consistency-checks samples for grounded training.

Motivation: Existing datasets lack synchronized temporal and spatial supervision plus reasoning chains that explicitly connect visual evidence to answers.This prevents coherent localization and verifiable reinforcement-learning rewards in dynamic scenes.
Dataset Composition: STGR-CoT-30k and STGR-RL-36k combine existing resources with 5.9k newly annotated spatio-temporal samples containing timestamped frames, localized boxes, and structured reasoning.The SFT and RL corpora provide complementary supervision for the two training stages.
Data Annotation Pipeline: The annotation pipeline uses Gemini 2.5 Pro to generate question-answer pairs, key frames, object boxes, and evidence-linked reasoning.Each sample targets one to three salient objects across one to five key frames.
Data Annotation Pipeline: Bounding-box filtering removes uninformative boxes and retains crops verified to match their named objects.Boxes covering over 80% of a frame are removed, and Qwen2.5-VL verifies object identity.
Data Annotation Pipeline: Self-consistency checks align timestamps, boxes, and reasoning references, discarding samples with missing or semantically inconsistent evidence.These checks improve annotation quality for cold-start grounded training.

4 Open-o3-Video

Open-o3-Video trains grounded video reasoning through cold-start supervised fine-tuning followed by sequence-level reinforcement learning with task-specific rewards. Its reward design jointly evaluates answer correctness, temporal alignment, spatial overlap, and output format while adaptive proximity and temporal gating stabilize spatio-temporal supervision.

The two-stage recipe starts with cold-start fine-tuning and then applies reinforcement learning using adaptive temporal proximity and temporal gating.Cold-start initialization provides basic grounding and structured reasoning, reducing reward sparsity before reinforcement learning.
GSPO optimizes complete reasoning traces with sequence-level importance ratios and clipping, matching rewards defined over timestamps and bounding boxes.This avoids high-variance token-wise corrections and targets global grounding quality.
The total reward combines task-specific accuracy, thinking, and format terms before group normalization for GSPO optimization.Accuracy rewards cover multiple-choice, question answering, spatial grounding, and temporal grounding tasks.
Adaptive temporal proximity supplies progressively precise timestamp supervision, while temporal gating computes spatial rewards only when predictions are sufficiently close to ground truth.The design addresses the trade-off between sparse feedback from overly strict constraints and weak temporal learning from overly loose constraints.
The spatial reward matches each predicted timestamp to the nearest annotated time and uses maximal box overlap at the matched frame.This prevents rewarding salient but irrelevant objects at incorrect timestamps.

5 Experiments

Open-o3-Video is evaluated on spatio-temporal grounding and broader video-understanding benchmarks, with ablations examining training, reward design, data, evidence faithfulness, and test-time scaling. It achieves strong benchmark gains while producing grounded evidence that supports interpretable reasoning.

Main Results: 69.4% accuracy on LongVideoReason-eval and +4.5% mIoU on TVGBench demonstrate gains in long-range reasoning and temporal localization.The model also reports improvements on long videos and perception-related tasks.
Ablation and Analysis: 33.7% mAM and 46.6% mLGM are reached by combining SFT and RL, while GSPO exceeds GRPO on grounding metrics and Chain1 temporal IoU.RL provides larger gains than SFT alone, and their combination is reported as synergistic.
Ablation and Analysis: Removing temporal gating lowers performance by 1.4% mAM and 1.7% mLGM, while removing adaptive proximity lowers it by 0.7% and 1.4%.The ablation supports roles for temporal filtering and timestamp alignment in the reward design.
Ablation and Analysis: +5.4% mAM and +10.4% mLGM result from adding the paper’s spatio-temporal annotations beyond filtered VideoEspresso data.The experiments identify high-quality spatio-temporal supervision as important for grounding performance.
Ablation and Analysis: The model produces averages of 1.15 bounding boxes and 1.37 timestamps per VideoMME instance, and confidence-aware voting outperforms naive majority voting.Removing frames nearest predicted timestamps causes a larger performance drop than removing random frames.

6 Conclusion

Open-o3-Video generates explicit timestamps and bounding boxes within its reasoning without external models or tools. Its curated data, two-stage training, and reward design improve accuracy and grounding across video tasks, although limitations remain for future discussion.

Conclusion: Open-o3-Video generates explicit spatio-temporal evidence as part of reasoning without relying on external models or tools.The evidence consists of timestamps and bounding boxes.
Conclusion: Curated training data, SFT plus GSPO reinforcement learning, and adaptive temporal proximity and gating rewards jointly target accuracy, temporal alignment, and spatial grounding.The conclusion presents these components as the framework’s central design.
Conclusion: Open-o3-Video achieves state-of-the-art V-STAR performance and remains broadly competitive across diverse video-understanding tasks.The conclusion also states that the approach surpasses strong baselines including GPT-4o on V-STAR.

A Appendix

The appendix provides supplementary implementation details, related work, dataset and prompt analyses, ablations, additional model and benchmark results, and further discussion of training and evaluation.

Appendix: The appendix covers implementation details, video extensions of thinking with images, dataset preparation, annotation prompts, and training ablations.It also includes analyses of general VideoQA-data ratios and o3-style objectives.
Appendix: Additional sections report results for Qwen3-VL, STAR, CameraBench, and other experiments, alongside further analyses and discussion.The supplied appendix overview identifies these as supplementary materials.

A.1 More Implementation Details

Open-o3-Video uses a two-stage cold-start and GSPO training process with matched one-epoch learning rates, annealed thinking-reward variance, and temporal gating at 3 seconds.

More Implementation Details: The cold-start stage trains on STGR-CoT-30k for one epoch, followed by one GSPO epoch on STGR-RL-36k, both at 1 × 10^-6 learning rate.The two stages use separate supervised and reinforcement-learning datasets.
More Implementation Details: The thinking-reward standard deviation σ is annealed from 4 to 1, while the gating mechanism uses a temporal threshold τ of 3s.These are specified as implementation settings for training.
More Implementation Details: At test time, the model first generates spatio-temporally grounded reasoning before producing the final response.The passage identifies vLLM as the test-time framework.

A.2 More Related Works

Recent video reasoning work increasingly seeks explicit evidence retrieval and temporal exploration rather than relying solely on textual rationales. These approaches use visual tools, evidence-frame selection, or progress-bar interaction to support reasoning over videos.

Temporal Evidence Seeking: Video-oriented extensions of “thinking with images” primarily improve temporal evidence seeking through clips, zoom-in operations, or evidence-frame selection.VITAL crops relevant clips, LongVT and VideoZoomer iteratively retrieve them, and Conan supports cross-frame deduction with adaptive stopping.
Interactive Video Reasoning: VTimeCoT provides a training-free “thinking by drawing” approach that uses progress-bar tools to interact with video evidence.
Overall Direction: These methods extend explicit visual reasoning from images to videos by making temporal evidence retrieval part of the reasoning process.

A.3 More Details and Ablation on Training Data

The training corpus is assembled through filtering, sampling, and balancing choices intended to preserve annotation quality while supporting both grounding and general video question answering. The reported data mixture emphasizes grounding without discarding broader QA capability.

Data Preparation: The corpus filters temporal-grounding samples by chain-of-thought length and temporal-span coverage, while sampling spatial, general QA, and dense-captioning data from multiple sources.Temporal samples use length and duration constraints; 2.9k TVG-RL, 5k each from two spatial datasets, 15k Video-R1, and 2k filtered PLM-Rdcap samples are retained.
Data Mixture Ablation: 15k general VideoQA samples provide the best balance between grounding performance and general QA performance.
Data Mixture Ablation: The mixture study evaluates general VideoQA ratios on both V-STAR and VideoMME because emphasizing grounding can weaken general VideoQA performance.

A.4 Prompt for Data Annotation

The appendix details annotation, training, evaluation, and test-time procedures for grounded video reasoning. It also reports ablations, scaling results, and limitations spanning data quality, optimization, inference, and multimodal scope.

A.4 Prompt for Data Annotation: Annotation prompts convert video sources into structured JSON containing questions, answers, key frames, timestamps, objects, boxes, and grounded reasoning chains.PLM-Rdcap inputs use dense captions and frame counts, while temporal-grounding inputs use annotated segments, durations, and descriptions.
A.4 Prompt for Data Annotation: Validated boxes, aligned timestamps, and coherent reasoning are required, with inconsistent annotations discarded to support reliable spatio-temporal evidence.
A.5 More Ablation Studies on Hyper-parameters and Training Objectives: Adaptive temporal proximity outperforms fixed σ = 1 and σ = 4 by transitioning from coarse temporal alignment to precise localization.The schedule avoids sparse early rewards from small σ and weak refinement signals from large σ.
A.5 More Ablation Studies on Hyper-parameters and Training Objectives: Removing spatio-temporal evidence and thinking rewards consistently degrades performance across V-STAR, VideoMME, and WorldSense.
A.6 Results based on Qwen3-VL Models: +2.7%/+3.6% mAM/mLGM gains on Qwen3-VL 4B, +7.5%/+14.6% on 8B, and +4.0%/+8.1% on 32B show improvements across model scales.
A.7 Results on More Benchmarks: Open-o3-Video improves STAR accuracy by 3.2% and outperforms the baseline on both STAR and CameraBench.
A.8 Ablation Studies on Inference Frame Rate: 69.2% →69.4% accuracy from 16 to 64 frames indicates marginal gains, while key-frame sampling reaches 70.1% accuracy.
A.9 Details of GSPO Training: GSPO samples response groups, computes normalized advantages, and clips entire responses rather than individual tokens for long-chain reasoning optimization.The appendix states that response-level clipping aligns reward assignment with optimization granularity and produces more stable gradients.