PaperlayerView exact revision on arXiv

Source-linked AI summary

VLX-VR: An Agentic-Aware Video Reasoning Model

Sheng Li, Peng Liu, Qianqian Zhang, Tiancheng Zhao

arXiv:2609.09985v1cs.CLcs.CV

TL;DR

Real-world video understanding requires adaptive integration of multimodal evidence, but fixed-context, single-pass pipelines limit this process. VLX-VR learns evidence acquisition, memory use, and termination through a Think–Memory–Observation loop, achieving 78.79% accuracy on MINERVA while retaining evidence-grounded trace consistency and showing limitations on several reasoning skills.

  • Problem

    Fixed-context, single-pass video pipelines limit adaptive evidence acquisition when relevant observations are incomplete, ambiguous, or conflicting.

  • Method

    VLX-VR uses a Think–Memory–Observation loop and reinforcement learning over multimodal data and agent trajectories to learn evidence acquisition, memory use, and termination.

  • Results

    78.79% accuracy: VLX-VR achieves state-of-the-art performance on MINERVA among the models included in the comparison.

  • Takeaways & Limitations

    96.20% of correctly answered samples have reasoning traces consistent with MINERVA reference traces and described evidence, while counting, state changes, causal reasoning, and spatial perception remain challenging.

  • Takeaways & Limitations

    The framework cannot guarantee recovery of attributes absent from available visual evidence, and evidence-grounded trace agreement does not guarantee faithfulness to internal decisions.

Abstract

from arXiv · show

Real-world video understanding requires integrating visual, audio, textual, and temporal evidence distributed across a video. Yet many pipelines use a fixed video context and single-pass inference, limiting adaptive evidence acquisition when observations are incomplete, ambiguous, or conflicting. We present VLX-VR, an agentic-aware video reasoning model trained within a video reasoning framework defined by a Think--Memory--Observation loop. At each step, VLX-VR determines the needed evidence, invokes read_memory or write_memory, incorporates the returned Observation, and decides whether to continue or produce the task output. We train VLX-VR with multimodal data, including videos and agent trajectories, using reinforcement learning to learn evidence acquisition, memory use, and termination. On MINERVA, VLX-VR achieves state-of-the-art performance among the models included in our comparison, with 78.79% accuracy. Under the original three duration groups, its accuracies are 76.70%, 78.73%, and 80.92%, with a cross-duration accuracy variance of 2.97~$\mathrm{pp}^2$. On correctly answered samples, 96.20% of VLX-VR's reasoning traces are consistent with the MINERVA reference reasoning traces and the evidence described by them, while approximately 75.80% of all evaluated samples satisfy both answer correctness and this evidence-grounded trace criterion. These results show strong performance and broadly stable behavior across durations, while counting, state changes, causal reasoning, and spatial perception remain challenging.

1. Introduction

VLX-VR addresses the limits of fixed-input, single-pass video analysis by learning to acquire, retain, and reassess multimodal evidence within an agentic reasoning loop. It is trained with multimodal data and agent trajectories so evidence acquisition, memory use, and termination become learned reasoning behaviors.

  • Real-world video reasoning requires integrating visual, audio, textual, and temporal evidence distributed across events and modalities.Reliable conclusions may require locating moments, comparing states, reading scene text, and relating textual content to visual actions.
  • Fixed-input, single-pass pipelines cannot fully support video analysis that requires revisiting events, seeking evidence, or comparing information across time and modalities.
  • External agent loops may leave general-purpose VLMs without learned policies for evidence acquisition, memory use, and termination.Such models may request redundant evidence, fail to retain intermediate states, or stop before enough evidence is available.
  • VLX-VR is trained within a Think–Memory–Observation framework to acquire and reassess multimodal evidence for temporal localization, state tracking, and multi-step reasoning.The model has direct access to multimodal memory and targets both short and long videos.
  • 78.79% accuracy: VLX-VR achieves state-of-the-art performance on MINERVA among the models included in the comparison.The evaluation also examines cross-duration stability and evidence-grounded agreement between reasoning traces and reference traces.

2. Related Work

Prior work improves video perception, reasoning, memory, and adaptive evidence gathering, while VLX-VR targets learned control over what evidence to acquire, retain, and judge sufficient. Its reasoning process therefore controls multimodal-memory interaction rather than only generating an explanatory trace.

  • Prior VLMs strengthen image-video representation, temporal localization, long-video processing, continuous context maintenance, and semantic memory.VLX-Flow, for example, reuses previously observed content without reprocessing the complete video history.
  • VLX-VR differs by treating task-relevant evidence acquisition, intermediate information retention, and evidence sufficiency as a broader control problem.
  • Visual reasoning research extends models from recognizing content toward deriving conclusions through structured reasoning and reinforcement learning.
  • VLX-VR extends reasoning beyond intermediate-trace generation by controlling multimodal-memory reads and writes, incorporating Observations, and deciding whether more evidence is needed.Reasoning thus controls how the model observes and analyzes the video.
  • Video-agent systems use iterative visual tools, structured event and object memory, temporal localization, memory queries, and uncertainty-aware information gathering.
  • VLX-VR is trained as an agentic-aware VLM whose decisions condition on the task, reasoning state, accumulated memory, and returned Observations.The model directly selects read-memory or write-memory operations within the framework-defined loop.

3. The Proposed Method

VLX-VR performs video reasoning through a Think–Memory–Observation loop that updates a state with acquired evidence and intermediate information. Reinforcement learning trains the model to select useful memory operations, ground reasoning, and terminate when evidence is sufficient.

  • Think–Memory–Observation loop: VLX-VR first determines the needed evidence, invokes read memory or write memory, incorporates the returned Observation, and then gathers more evidence or produces an output.
  • Think–Memory–Observation loop: At each reasoning step, the model state includes the task, observed evidence, multimodal memory, output hypotheses, and unresolved uncertainty.The next evidence request can change as new Observations arrive.
  • Think–Memory–Observation loop: The loop comprises Think for proposing evidence needs, Memory for retrieving or retaining multimodal evidence, and Observation for guiding the next decision.When evidence is insufficient or conflicting, the model returns to Think; when sufficient, it produces the final output with supporting evidence.
  • Multimodal memory: Multimodal memory stores and exposes video, audio, supporting evidence, Observations, and intermediate states across reasoning steps.This makes the reasoning state external and inspectable rather than merely a cache of textual summaries.
  • Evidence and reasoning traces: A reasoning trace records how VLX-VR selects operations, incorporates Observations, connects evidence, and decides whether evidence is sufficient.Agreement between a trace and its supporting evidence indicates grounding but does not establish faithful access to internal decisions.
  • Memory operations: read memory retrieves relevant multimodal content, while write memory stores Observations, evidence, and state information in multimodal memory.Each operation returns an Observation containing requested evidence or memory-update status.
  • Inference procedure: The inference procedure initializes state, repeatedly generates a Think and memory call, executes the call, updates state with the Observation, and then outputs when stopping.
  • Training and termination: Premature termination, repeated reads, and unproductive loop extensions are discouraged through stopping criteria, costs, and redundancy penalties.

4. Dataset

MINERVA is the primary evaluation dataset because it combines short and long videos with detailed human reasoning traces and final answer labels. The protocol evaluates multiple-choice accuracy while enabling temporal localization and evidence-grounded reasoning analysis.

  • MINERVA combines broad temporal coverage with explicit reasoning supervision across short and long videos.It provides detailed human-annotated reasoning traces in addition to final answer labels.
  • MINERVA supports evaluating both answer accuracy and evidence-grounded reasoning, unlike complementary benchmarks focused on other video-understanding settings.
  • 92 words: MINERVA reference reasoning traces contain approximately 92 words on average, with 99.6% containing timestamps.The traces contain approximately four timestamps each and support temporal localization and reasoning-consistency analysis.
  • The task provides a video, a question, and five candidate answers, and evaluates the final choice using multiple-choice accuracy.VLX-VR also returns a reasoning trace for analysis.

5. Experimental Results

VLX-VR achieves strong MINERVA accuracy and comparatively stable performance across the original duration groups, while refined analysis reveals a mid-length peak and decline beyond 30 minutes. Its reasoning-skill profile and evidence-grounded trace results identify both strengths and remaining challenges.

  • 5.1. Overall Accuracy: 78.79% overall accuracy on MINERVA.VLX-VR is higher than the public comparison results in the current aggregate table but remains below the human reference level.
  • 5.2. Performance across Video Durations: 76.70%, 78.73%, and 80.92% accuracy across the three original duration groups, with no monotonic decrease as duration increases.Figure 2 reports the three-group accuracies, while the accompanying analysis compares VLX-VR against public baselines.
  • 5.2. Performance across Video Durations: 2.97 pp2 cross-duration variance, the lowest among five models, alongside 78.78% mean accuracy.Under the original grouping, VLX-VR also has a 1.72 pp standard deviation and 4.22 pp range; lower CDAV indicates less variation but does not itself imply higher accuracy.
  • 5.3. Performance Analysis on Different Durations: 83.40% accuracy at 15–30 minutes versus 74.75% above 30 minutes.Splitting the original Above 15 min group increases CDAV from 2.97 to 10.33 pp2, showing that the original grouping smooths over the peak and subsequent decline.
  • 5.4. Performance across Reasoning Skills: VLX-VR is strongest on reading, situational awareness, temporal reasoning, and numerical reasoning, while counting, state changes, cause and effect, and spatial perception are below overall accuracy.The skill distribution points to challenges involving object tracking, state modeling, local evidence retrieval, and causal judgment.
  • 5.5. Analysis of Reference-trace Agreement: 96.20% of traces on correctly answered samples are consistent with MINERVA reference traces and their described evidence.The estimated joint rate of answer correctness and trace consistency is approximately 75.80% across all evaluated samples.

6. Case Studies

The case studies show that VLX-VR can build coherent evidence chains for temporal order, numerical computation, and situational action understanding, but remains vulnerable to uncertainty in counting and state-transition selection.

  • 6.1. Correct case 1: Temporal reasoning: VLX-VR correctly identified the second speaker by combining coarse video review, timestamped observations, and later dialogue checks.The model localized the brunette woman's first utterance around 41 seconds and the blonde woman's response around 45.6–48 seconds.
  • 6.2. Correct case 2: Numerical reasoning: VLX-VR correctly solved the long-video numerical case by locating orange's takeover, reading 10 green territories worth 27, and computing 27 / 10 = 2.7.The reasoning chain connected event localization, state reading, and numerical computation rather than relying on a local frame.
  • 6.3. Correct case 3: Situational awareness: VLX-VR correctly interpreted several gaze directions as “looking around” before the kiss, using repeated observations to aggregate local actions into a situation.The model ruled out isolated “looks down” and “looks up” choices by considering the surrounding interaction.
  • 6.4. Incorrect case 1: Counting under occlusion and uncertainty: VLX-VR undercounted women in the first audience row because it excluded an occluded person whose gender was uncertain, while the benchmark counted five women.The error concerned target-set definition and uncertain-instance treatment rather than arithmetic.
  • 6.5. Incorrect case 2: State changes and temporal-chain tracking: VLX-VR selected a later green-to-red transition instead of the requested initial transition and produced an inconsistent final option marker.The case indicates difficulty maintaining the question's temporal boundary across multiple salient state changes.
  • 6.6. Cross-case analysis: Across the cases, VLX-VR formed coherent evidence chains in three correct examples but still failed when uncertainty, object-set identity, or state-transition boundaries required stricter tracking.The incorrect cases show that observing relevant frames does not ensure correct counting or selection of the transition requested by the question.

7. Conclusion

VLX-VR integrates evidence acquisition and memory use into a trained Think–Memory–Observation reasoning loop. It achieves strong MINERVA performance and evidence-grounded trace consistency, while remaining limited on several reasoning skills and by unresolved attribution among its components.

  • VLX-VR identifies needed evidence, executes read_memory or write_memory, integrates the returned Observation, and decides whether to continue or answer.Multimodal memory stores video, audio, supporting evidence, Observations, and intermediate states as an external reasoning state.
  • 78.79% overall accuracy makes VLX-VR state of the art among the models included in the MINERVA comparison.
  • 96.20% of correctly answered samples have reasoning traces consistent with MINERVA reference traces and their described evidence.Approximately 75.80% of all evaluated samples satisfy both answer correctness and the evidence-grounded trace criterion.
  • VLX-VR’s performance remains weaker for counting, state changes, cause and effect, and spatial perception.
  • Matched ablations are still needed to separate the contributions of the framework loop, direct multimodal-memory access, and reinforcement-learning training.
Loading 2609.09985v1…