PaperlayerView exact revision on arXiv

Source-linked AI summary

VideoLVR: Latent Visual Reasoning over Grounded Video Evidence

Shilong Li, Xingyuan Bu, Hao Shi, Yun Xing, Weixin Feng, Qiangpeng Yang, Shilei Wen

arXiv:2610.37613v1cs.CVcs.AI

TL;DR

Video reasoning traces can lose access to relevant visual evidence, while coordinate-based and tool-based approaches do not provide a compact way to revisit its visual content. VideoLVR generates grouped latent evidence from localized regions across video moments and trains its generation through visual token prediction, with reinforcement learning jointly optimizing text and visual actions. It improves benchmark performance over its backbone, with particular gains on tasks requiring fine-grained visual evidence.

  • Problem

    Language-based reasoning traces provide no reliable way to revisit relevant visual evidence, while coordinates encode locations rather than visual content and inserted crops or frames expand context.

    Source: §1 · p. 1

  • Method

    VideoLVR groups localized regions across video moments into latent evidence and trains visual token prediction over the input video’s vocabulary, with TVPO jointly optimizing text and visual-token actions.

    Source: §1 · pp. 1–2, §3.1 · p. 3, §3.3 · p. 4

  • Results

    VideoLVR-8B scores 63.95 on average, 2.64 points above Qwen3-VL-8B-Instruct, and improves over the backbone on every benchmark.

    Source: §4.2 · p. 6

  • Takeaways & Limitations

    VideoLVR shows particular gains on tasks requiring fine-grained visual evidence across multiple video understanding and reasoning benchmarks.

    Source: §1 · p. 2

Abstract

from arXiv · show

Chain-of-thought reasoning has substantially advanced video understanding in multimodal large language models (MLLMs). However, language-based reasoning traces do not provide a reliable mechanism for models to revisit relevant visual evidence, leading to visual forgetting and impairing their ability to reason over video content. Existing approaches maintain visual grounding during reasoning in two main ways: they either refer to evidence using textual coordinates, which specify symbolic locations rather than the visual content itself, or reinsert cropped regions or frames into the multimodal context, preserving visual details at the cost of substantial context expansion. To address these limitations, we introduce VideoLVR, which constructs latent visual evidence from local regions at relevant video moments, thereby providing visual support for intermediate reasoning steps. To supervise latent visual evidence generation, we introduce a visual token prediction objective that formulates visual token reconstruction as prediction over a vocabulary derived from the input video, thereby unifying visual evidence reconstruction and text generation within the same token prediction framework. This shared prediction framework enables MLLMs to reason over interleaved text tokens and latent visual states while integrating evidence from different moments. We further extend VideoLVR to reinforcement learning by treating sampled visual tokens as actions alongside text tokens and jointly optimizing the policy using final-answer rewards. By efficiently revisiting task-relevant visual information during reasoning, VideoLVR improves video understanding without substantially expanding the multimodal context. Experiments on multiple video benchmarks demonstrate its effectiveness, especially on tasks requiring fine-grained visual evidence.

1 Introduction

Video reasoning traces can lose access to sparse visual evidence, while existing grounding approaches either encode locations symbolically or add visual observations to context. VideoLVR instead generates latent evidence from localized regions across moments and trains it through visual-token prediction.

1 Introduction

  • Language-based reasoning can lose access to task-relevant visual evidence, making localized and temporally sparse evidence difficult to revisit in later reasoning steps.The paper frames this as visual forgetting that limits reasoning over video content.Source: §1 · p. 1
  • Textual coordinates are compact and traceable but encode evidence locations rather than visual content; crop- and frame-based tools preserve details but expand context as observations accumulate.Tool-based reasoning is also limited to a predefined set of visual operations.Source: §1 · p. 1
  • Video latent reasoning must combine localized evidence across moments, where a single moment may be insufficient and neighboring frames can contain similar visual tokens.The passage notes that MSE feature regression does not explicitly distinguish a target from visually similar tokens in the same video.Source: §1 · p. 1
  • VideoLVR groups local regions supporting the same reasoning step into latent visual evidence, allowing evidence from different video moments to be modeled jointly.The groups are supervised using visual-token sequences derived from spatio-temporal grounding annotations in evidence-grounded reasoning trajectories.Source: §1 · pp. 1–2
  • Visual token prediction over the input video’s visual vocabulary unifies latent evidence generation with text-token prediction, while TVPO jointly optimizes text and visual-token actions using final-answer rewards.The contributions report particular gains on tasks requiring fine-grained visual evidence.Source: §1 · p. 2

2 Related Work

Related work grounds video reasoning with explicit references, visual tools, or latent representations. VideoLVR differs by grouping localized evidence across moments and supervising its generation with the input video’s visual vocabulary.

Video Reasoning with Explicit Evidence

  • Explicit video grounding uses timestamps and bounding boxes to make evidence traceable, but these coordinates encode symbolic locations rather than visual semantics (Maaz et al. 2025; Meng et al. 2026).Related image reasoning methods also localize regions to ground intermediate reasoning (Fan et al. 2025; Wang et al. 2026a,b).Source: §2 · p. 2

Revisiting Visual Evidence during Reasoning

  • Visual tools inspect selected image regions or add video frames and retrieved items as multimodal observations to recover fine-grained evidence (Zheng et al. 2026; Gao et al. 2026).VITAL samples additional frames, while VideoSearcher combines temporal localization, spatial inspection, and multimodal search (Gao et al. 2026).Source: §2 · p. 2

Latent Visual Reasoning

  • Prior latent visual reasoning methods use approaches such as distillation, feature alignment, reinforcement learning, or latent bottlenecks, while diagnostic studies report bypassing, target deviation, or underuse of latent visual semantics (Viveiros et al. 2026).Coconut feeds hidden states back as input embeddings for intermediate computation (Hao et al. 2025).Source: §2 · p. 2
  • Mull-Tokens, UniVR, and VideoLatent use generic computation slots, discrete visual trajectories, or video latents without step-level localized spatio-temporal supervision (Ray et al. 2026; Ren et al. 2026; Hu et al. 2026b).VideoLVR addresses this gap by grouping temporally distributed local regions supporting the same reasoning step and predicting over the input video’s visual vocabulary.Source: §2 · p. 2

3 Method

VideoLVR synthesizes evidence-grounded reasoning trajectories, groups supporting video regions into latent targets, and jointly trains text and latent visual evidence generation. At inference, it feeds projected decoder states back as latent inputs for autoregressive reasoning without additional visual observations.

3.1 Evidence-Grounded Data Synthesis

  • Rejection sampling retains answer-correct trajectories with valid evidence groups, then polishing relocates each group before the reasoning statement that uses it.Polished trajectories must preserve the final answer and include every original group exactly once.Source: §3.1 · p. 3

3.2 Grouped Latent Visual Evidence

  • Regions supporting one claim are timestamp-ordered and concatenated into a latent target; training omits textual coordinates and avoids cropping, re-encoding, or extra observations.Regions sharing a timestamp retain their annotation order, and no separator tokens are inserted between regions.Source: §3.2 · p. 4

3.3 Visual Token Prediction

  • VTP trains each latent query to identify its grounded token among encoded input-video tokens, alongside next-token prediction for ordinary response text.Non-target tokens from the same video serve as competing vocabulary entries.Source: §3.3 · p. 4

3.4 Text–Visual Policy Optimization

  • TVPO treats sampled visual-token indices and text tokens as actions, jointly optimizing them with GRPO using final-answer rewards.The same group-normalized advantage is shared by sampled text and visual actions in each trajectory.Source: §3.4 · pp. 4–5

3.5 Inference with Latent Visual Evidence

  • At inference, each hidden state is projected and fed back as a latent input for fixed K steps, after which text generation resumes; the RL model instead feeds back sampled visual-token embeddings.The supervised model uses the learned projection head to produce each successive latent visual state.Source: §3.5 · p. 5

4 Experiments

VideoLVR is evaluated on five video-reasoning benchmarks against models using different reasoning paradigms. Its results and ablations examine overall performance, latent-evidence supervision and organization, reinforcement learning, and inference-time choices.

4.1 Experimental Setup

  • Evaluations cover five benchmarks, use zero-shot accuracy and a three-run average, and train on approximately 8.8K supervised trajectories plus 19K reinforcement-learning questions.The RL set excludes questions answered correctly in all eight supervised-model responses.Source: §4.1 · pp. 5–6

4.2 Main Results

  • VideoLVR-8B scores 63.95 on average, 2.64 points above Qwen3-VL-8B-Instruct and 1.24 above LVR-8B, and beats reproduced OneThinker-8B on all five benchmarks.The main comparison is reported in Table 1.Source: Table 1 · p. 6, §4.2 · p. 6

4.3 Ablation Studies

  • Replacing MSE with visual token prediction raises the five-benchmark average from 62.71 to 63.95, while evidence-grounded trajectories raise it from 62.53 to 63.95 versus rejection-sampled trajectories.VTP gains are largest on MMVU, VideoMathQA, and LongVideo-Reason; evidence-grounded training is not uniformly beneficial, with MMVU decreasing by 1.90 points.Source: §4.3 · pp. 6–7, Table 2 · p. 6, Table 3 · p. 7

4.4 Additional Analysis

  • Eight inference-time latent steps yield a 63.95 average, versus 62.12 with none; four steps score 62.82, while 16 and 32 score 62.00 and 62.31.Eight steps are best on four benchmarks and tie four steps on TempCompass.Source: Figure 2 · p. 7, §4.4 · pp. 7–8

4.3 Ablation Studies

  • Grouped latent blocks score 63.95 on average, exceeding First-Region by 0.80 and Per-Region by 0.84 points; grouping keeps temporally ordered regions together.The grouped strategy combines regions associated with one evidence group in a single latent block.Source: Table 4 · p. 7, §4.3 · p. 7
  • TVPO raises the five-benchmark average from 63.31 with GRPO to 64.39 and outperforms GRPO on all five benchmarks.Both variants use the same supervised VideoLVR-8B initialization and RL data.Source: Table 5 · p. 7, §4.3 · p. 7

4.4 Additional Analysis

  • Visual-token-prediction temperature 0.07 yields a 63.95 average and the best result on all five benchmarks; both zeroed and noise-intervened latent feedback lower the average.The intervention averages are 62.75 for zero feedback and 62.07 for noise feedback; TVPO also earns higher rewards than GRPO over most training steps.Source: §4.4 · p. 8, Table 6 · p. 8, Table 7 · p. 8

5 Conclusion

VideoLVR grounds video reasoning in latent visual evidence from temporally distributed local regions, grouped by the reasoning step they support. Across five benchmarks and two backbones it improves over base models, with a remaining reliance on synthesized spatio-temporal grounding annotations.

5 Conclusion

  • Across five video reasoning benchmarks and two backbone models, VideoLVR consistently improves over its base models.Controlled analyses support evidence-grounded training data, grouped evidence organization, and visual token prediction; latent-state interventions also show benefits from information carried by generated latent states.Source: §5 · p. 8
  • VideoLVR groups local regions by the reasoning step they support and generates corresponding latent visual states through visual token prediction.Textual reasoning and latent visual evidence are interleaved in a shared autoregressive process; TVPO extends the framework to jointly optimize text and visual token actions with GRPO.Source: §5 · p. 8
  • The framework represents localized video evidence during reasoning without inserting additional visual observations into the multimodal context.The authors identify reliance on synthesized spatio-temporal grounding annotations as a remaining limitation.Source: §5 · p. 8

A Supervised Training Data Statistics

The supervised training set comprises 8,782 evidence-grounded reasoning trajectories with multiple evidence groups and localized regions. Some groups cover multiple regions or video moments, providing supervision for grouped latent visual evidence.

A Supervised Training Data Statistics

  • The supervised set contains 8,782 trajectories, 25,710 evidence groups, and 32,180 localized visual regions.The reported averages are 2.93 evidence groups and 3.66 regions per trajectory.Source: A Supervised Training Data Statistics · p. 11, Table 8 · p. 11
  • 19.8% of evidence groups contain multiple regions, and 14.0% span multiple video moments.Some groups capture spatially distributed evidence within a frame, while others capture temporally distributed evidence across frames.Source: A Supervised Training Data Statistics · p. 11
  • Multiple localized regions within an evidence group provide supervision for grouped latent evidence.The larger average number of regions than groups reflects that some groups use multiple regions.Source: A Supervised Training Data Statistics · p. 11

B Prompts for Evidence-Grounded Trajectory Synthesis

The appendix describes two evidence-grounded trajectory synthesis prompts: one generates reasoning with visual evidence groups, and the other relocates groups in rejection-sampled trajectories. Their instructions specify evidence formatting, placement, and preservation of reasoning and answers.

B Prompts for Evidence-Grounded Trajectory Synthesis

  • The first synthesis prompt generates a reasoning trajectory interleaved with spatio-temporal evidence groups; the second polishes a rejection-sampled trajectory by relocating groups to the reasoning steps that use them.The polishing prompt asks for the original reasoning content to be preserved, rewriting surrounding text only as needed to integrate relocated evidence.Source: B Prompts for Evidence-Grounded… · p. 11, §B.2 · p. 11

B.1 Evidence-Grounded Trajectory Rejection Sampling

  • Each evidence group supports one claim, may combine boxes from the same or different moments, and is inserted into the reasoning sentence it supports.The prompt allows at most 10 groups and asks for evidence most relevant to the answer.Source: §B.1 · p. 11
  • Rejection-sampled trajectories are prompted to mark each short, specific visual claim with one or more timestamped bounding-box regions.The instructions require tight normalized boxes and timestamps copied from visible video-context markers.Source: §B.1 · p. 11
  • The rejection-sampling prompt requires multiple-choice responses to place reasoning inside a think block and end with exactly one option letter.The question is supplied as a video question, and the prompt requests careful reasoning based on visible evidence.Source: §B.1 · p. 11

B.2 Evidence-Guided Trajectory Polishing

  • The polishing prompt requires preserving substantive observations, option analysis, evidence interpretation, uncertainty, and self-correction without inventing visual facts or changing the final answer.It asks that each evidence group be placed before the first reasoning step that uses its visual content.Source: §B.2 · p. 12
  • The polishing prompt protects evidence placeholders for later restoration and requires each supplied placeholder to appear exactly once before the final answer.It requests JSON with separate think_content and answer fields; think_content contains the reasoning body, while answer is one uppercase option letter.Source: §B.2 · p. 12
  • The polishing instructions prohibit copying catalog coordinates, timestamps, IDs, or XML into prose and require preserving the reasoning order where possible.The prompt distinguishes Original Reasoning inside the think block from Original Content following it.Source: §B.2 · p. 12

C Case Study

The case study illustrates evidence-grounded reasoning for identifying deep-frying from visual cues distributed across a video. Two evidence groups connect the bubbling oil and the later removal and draining of food to separate reasoning steps.

C Case Study

  • The example asks which main activity is shown, and the trajectory concludes that the activity is deep-frying food.The listed alternatives are boiling water, deep-frying food, baking a cake, and making a salad.Source: C Case Study · pp. 12–13
  • The first evidence group uses regions at 2.3 and 6.4 seconds to support the claim that oil bubbles around food as it is added.The trajectory uses this cue to characterize the process as deep-frying and to rule out boiling water.Source: C Case Study · p. 13
  • The second group uses regions at 20.7 and 22.7 seconds to support the claim that the person removes and drains the fried food.The trajectory describes a skimmer lifting food from the oil and the food being placed on a paper towel.Source: C Case Study · p. 13
  • During training, regions within each group are timestamp-ordered and concatenated into one latent target, replacing textual evidence annotations in the generation target.The account describes latent blocks marked by <bot> and <eot>, with <latent> positions supervised to predict the corresponding visual tokens.Source: C Case Study · p. 13

Related papers

Loading 2610.37613v1…