PaperlayerView exact revision on arXiv

Source-linked AI summary

Joint Latent Reasoning and Observation for Agentic Video Understanding

Junzhang Liu, Hao Yuan, Qi Zhang, Yiming Wu, Long Chen, Zhecan Wang

arXiv:2610.35229v1cs.CVcs.AI

TL;DR

Video agents must use each observation both to answer and to decide what evidence to acquire next, but textual descriptions can omit temporal cues needed for that search. LARVA jointly trains latent observations and reasoning in a multi-round agent; across five benchmarks, it outperforms same-trajectory CoT students with both 4B and 7B backbones.

  • Problem

    Video agents need observations to support current reasoning and future evidence acquisition, while textual descriptions may omit temporal context needed to choose the next interval.

    Source: §1 · p. 1

  • Method

    LARVA jointly trains a reasoner–observer loop that passes latent observations and reasoning states, using action and answer supervision alongside observation reconstruction and CoT-distribution alignment.

    Source: §1 · p. 2

  • Results

    5.1, 2.5, and 3.7 percentage points are the 7B LARVA model’s gains over same-trajectory CoT students on VideoHolmes, Video-MME, and LVBench, respectively, with gains across all five benchmarks for both backbones.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Joint latent training is an effective approach to building accurate and efficient video agents, according to the paper’s three-benchmark conclusion.

    Source: §5 · p. 10

    LARVA processes sampled visual frames without audio, limiting its applicability to questions requiring audiovisual understanding.

    Source: §6 · p. 10

Abstract

from arXiv · show

Effective video agents must learn not only how to answer, but also where to look next. Transferring this capability to smaller models is challenging because each observation serves two purposes: supporting an answer and guiding the search for further evidence. Textual descriptions may capture what is currently visible while omitting cues needed for subsequent acquisition. We introduce Latent Agentic Reasoning for Video Analysis (LARVA), a framework that distills multi-round video agents through jointly learned latent reasoning and visual observations. Rather than communicating through captions, an observer passes compact continuous representations to a reasoner that selects the next acquisition action or produces the final answer. Crucially, action and answer supervision propagate through this interface, training the observer to communicate information useful for downstream decisions. Observation reconstruction and reasoning-distribution alignment provide auxiliary supervision from textual teacher trajectories. Across five video-understanding benchmarks, LARVA achieves strong performance with 4B and 7B backbones. On VideoHolmes, Video-MME, and LVBench, the 7B model outperforms a textual CoT student trained on the same teacher trajectories by 5.1, 2.5, and 3.7 percentage points, respectively. These results highlight the value of jointly learning how agents represent evidence and act on it when distilling video-understanding capabilities.

1 INTRODUCTION

The introduction frames agentic video understanding as a coupled observe–reason–acquire loop, where each observation must inform current interpretation and the next search step. LARVA replaces textual intermediates with jointly trained latent observations and reasoning, using downstream action and answer supervision.

1 INTRODUCTION

  • Textual descriptions may identify visible events yet omit temporal context needed to choose the next interval, while passing frames directly adds visual tokens to the accumulating context.The paper frames this as a challenge in preserving the observation–reasoning–action coupling when distilling to smaller models.Source: §1 · pp. 1–2
  • LARVA’s reasoner uses accumulated latent observations to choose a tool call or final answer, while an observer encodes requested frames to condition the next reasoning step.The framework avoids decoding intermediate reasoning or observations into text.Source: §1 · p. 2
  • Teacher trajectories provide fixed acquisition actions during replay, while action and answer losses train the observer through downstream use of its outputs, alongside observation reconstruction and CoT-distribution alignment.At inference, learned representations guide the student’s own acquisition loop.Source: §1 · p. 2
  • Joint training raises the latent/latent configuration by 5.3–7.2 percentage points across three diagnostic benchmarks, from below the textual baseline to the strongest evaluated representation configuration.The comparison is reported in Table 2.Source: §1 · p. 3
  • 5.1, 2.5, and 3.7 points are LARVA’s gains over same-trajectory CoT students on VideoHolmes, Video-MME, and LVBench, respectively, with both 4B and 7B models improving across all five benchmarks.The gains cited are for the 7B model; LARVA also improves over the CoT student on all five benchmarks with the 4B model.Source: §1 · p. 3

2 RELATED WORK

Prior agentic video systems vary in how they couple reasoning and perception, while latent-reasoning work explores continuous representations for reasoning, communication, and retrieval. LARVA’s distinction is jointly distilling latent reasoning and visual observations in a closed-loop video agent.

2.1 AGENTIC VIDEO UNDERSTANDING

  • Agentic video systems span unified agents, reasoner–observer designs, and decomposed tool or memory architectures; many are training-free.Their reasoning and perception interfaces differ, including whether acquired frames are interpreted in the reasoning model or returned by an observer.Source: §2.1 · p. 3
  • Existing modular systems commonly communicate reasoning and visual evidence through text, whereas LARVA retains the reasoner–observer split with a continuous interface.The paper characterizes language as interpretable and convenient, but requiring sequential verbalization and projecting visual evidence into a discrete representation.Source: §2.1 · p. 3

2.2 LATENT REASONING AND COMMUNICATION

  • Prior latent approaches cover hidden-state reasoning, visual thoughts, compact communication, and iterative retrieval, while LARVA jointly distills latent reasoning and visual observations in a closed-loop video agent.Its action and answer losses shape both the observation interface and reasoning states; Figure 2 depicts the inference loop and joint training objectives.Source: §2.2 · p. 3, Figure 2 · p. 4

3 METHOD

LARVA jointly trains a latent reasoner–observer loop in which observations inform subsequent decisions without being decoded into text. Training combines teacher-trajectory supervision with reconstruction and reasoning-distribution alignment.

3 METHOD

  • LARVA jointly trains latent observations and reasoning so acquired evidence supports both interpretation and subsequent decision-making.The reasoner selects an action or answer; the observer’s latent output conditions the next reasoning step.Source: §3 · p. 4

3.1 SEQUENTIAL VISUAL EVIDENCE ACQUISITION

  • At each round, the reasoner uses accumulated context to choose an acquisition action or answer, and acquired frames enter the context as timestamped latent observations.The final answer follows the completed acquisitions; tool calls and answers remain explicit text while intermediate states are continuous.Source: §3.1 · p. 4

3.2 LATENT REASONER–OBSERVER INTERFACE

  • The observer encodes selected frames, tool type, and any query into continuous tokens that enter the reasoner’s context without decoding.Each observation adds a fixed number of latent positions, while the context still grows across rounds and the observer still processes the selected visual input.Source: §3.2 · p. 5

3.3 TRAJECTORY DISTILLATION AND JOINT OPTIMIZATION

  • Teacher trajectories initialize the latent models and provide replayed acquisition actions; action and answer losses update the observer through later decisions, not the already-selected action.Observation reconstruction preserves information needed to recover teacher descriptions, while CoT-distribution alignment encourages latent contexts to support next-reasoning predictions like those induced by teacher text.Source: §3.3 · pp. 5–6

3.4 INFERENCE

  • At inference, latent reasoning selects tools or answers, and each acquired observation conditions the following decision until answering or reaching the acquisition limit.The main configuration uses eight latent tokens per reasoning and observation block and permits at most eight acquisition rounds.Source: §3.4 · p. 6

4 EXPERIMENTS

The experiments compare LARVA with controlled textual students and evaluate its benchmark performance, latent-interface ablations, inference cost, and evidence acquisition. Results favor LARVA across benchmarks and show benefits from joint training and its selected evidence.

4.1 EXPERIMENTAL SETUP

  • Experiments use 20,000 filtered teacher trajectories, 4B and 7B LARVA backbones, and controlled CoT and Direct Policy students trained from the same source trajectories.The main configuration uses eight latent tokens per reasoning and observation block and allows at most eight acquisition rounds.Source: §4.1 · p. 6

4.2 MAIN RESULTS

  • LARVA outperforms the same-trajectory CoT student on all five benchmarks with both backbones, including 5.1, 2.5, and 3.7-point gains on VideoHolmes, Video-MME, and LVBench for 7B.It also improves 7B accuracy by 1.3 points on LongVideoBench and 0.4 on MLVU; mean gains are 2.6 points for 7B and 2.8 for 4B.Source: §4.2 · p. 7, Table 1 · p. 7

4.3 REPRESENTATION AND TRAINING ABLATIONS

  • Joint training raises the latent-reasoning/latent-observation variant by 6.9, 7.2, and 5.3 points on VideoHolmes, Video-MME, and LVBench, making it the strongest evaluated configuration.Without joint training, that variant scores below the fully textual student on all three benchmarks; increasing latent capacity from one to eight generally improves accuracy, with smaller gains from six to eight.Source: §4.3 · pp. 7–8, Table 2 · p. 8

4.4 INFERENCE COST

  • LARVA averages 8.2 seconds per question, versus 48.3 for CoT Policy and 39.0 for Direct Policy, while processing fewer frames and acquisition turns than CoT Policy.These are complete-configuration measurements; the observer still incurs the cost of encoding selected frames, and systems differ in frame counts, token types, architectures, and API overhead.Source: §4.4 · p. 8, Table 3 · p. 8

4.5 EVIDENCE ACQUISITION ANALYSIS

  • LARVA acquisition trajectories improve accuracy in all nine fixed-answerer comparisons, including gains of 1.7, 4.2, and 2.6 points for a 27B visual answerer.The swap experiment compares complete trajectories, which may differ in realized lengths and frame counts.Source: §4.5 · p. 9, Table 4 · p. 9
  • With the same Overview and additional-frame budget, full LARVA yields new answer-relevant evidence on 35% of questions, versus 15% for latent reasoning with textual observation and 11% for text reasoning with latent observation.This measures the first acquisition after a shared Overview, not evidence coverage over an entire trajectory; uncertain cases remain in the denominator.Source: §4.5 · p. 9, Table 5 · p. 9
  • LARVA exceeds both textual students in tool-sequence similarity, temporal IoU, and optional-tool agreement across all three diagnostic benchmarks.Relative to CoT, tool-sequence scores rise by 4.6, 0.8, and 1.3 points, while temporal IoU rises by 1.2, 2.3, and 2.0 points.Source: §4.5 · pp. 9–10

5 CONCLUSION

The conclusion presents LARVA as a jointly trained latent-reasoning and visual-observation framework for multi-round video agents, reporting stronger evidence gathering and teacher-acquisition agreement alongside lower latency.

5 CONCLUSION

  • Across three benchmarks, LARVA outperforms existing methods while reducing inference latency and shows stronger evidence gathering and closer agreement with teacher acquisition behavior.The framework combines action supervision, observation reconstruction, and CoT-distribution alignment to train compact representations for multi-round acquisition and answer generation.Source: §5 · p. 10

6 LIMITATIONS

LARVA is limited by its frame-only input, dependence on teacher trajectories, and less interpretable latent representations. These constraints affect audiovisual questions, student histories after incorrect acquisitions, and some questions requiring fine detail.

6 LIMITATIONS

  • LARVA processes sampled visual frames but not audio, limiting its applicability to questions that require audiovisual understanding.It cannot directly use dialogue, environmental sounds, or other acoustic cues.Source: §6 · p. 10
  • Teacher-generated trajectories and a frozen CoT student provide supervision, but may contain errors or suboptimal acquisition decisions that LARVA inherits.Teacher trajectories may also cover student-execution histories poorly, particularly after an incorrect acquisition.Source: §6 · p. 10
  • Latent representations are less directly interpretable than textual reasoning and observations, making it harder to diagnose whether failures arise from compression, reasoning, or evidence selection.Compression may discard fine-grained details needed for some questions.Source: §6 · p. 10

AI USE STATEMENT

The authors used generative AI to assist with manuscript and research work, and teacher models to generate training trajectories. The authors reviewed AI-assisted content and accept responsibility for the manuscript, methodology, and results.

AI USE STATEMENT

  • Generative AI tools assisted with manuscript editing, figure preparation, and discussions of research framing and experimental design.Source: AI USE STATEMENT · p. 10
  • Teacher models generated the training trajectories described in the paper.Source: AI USE STATEMENT · p. 10
  • The authors reviewed AI-assisted content for accuracy and consistency and take responsibility for the manuscript, methodology, and reported results.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

The work studies efficient video evidence acquisition using existing research datasets. The authors flag risks from incorrect interpretations, inherited biases, and overlooked visual evidence, and call for oversight and compliance with data restrictions.

ETHICS STATEMENT

  • The work studies efficient evidence acquisition for video understanding using existing research datasets.Source: ETHICS STATEMENT · p. 10
  • Video agents may produce incorrect interpretations, inherit biases from training data and teacher models, or overlook relevant visual evidence.Source: ETHICS STATEMENT · p. 10
  • Consequential applications require appropriate human oversight, while video data and derived trajectories should respect dataset licenses, privacy requirements, and usage restrictions.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The paper describes its architecture, training, trajectory generation, and evaluation protocols. The authors plan to release reproducibility materials and code, subject to underlying dataset and model restrictions.

REPRODUCIBILITY STATEMENT

  • The paper describes LARVA’s architecture, training objectives, teacher-trajectory generation, and evaluation protocols.Source: REPRODUCIBILITY STATEMENT · p. 10
  • Planned reproducibility materials include implementation details, model configurations, prompts, hyperparameters, and evaluation scripts.Scripts include protocols for trajectory similarity, answerer-swap evaluation, and inference-time measurement.Source: REPRODUCIBILITY STATEMENT · p. 10
  • The authors plan to release code and permitted training artifacts, subject to underlying dataset and model licenses and usage restrictions.Source: REPRODUCIBILITY STATEMENT · p. 10

A APPENDIX

The appendix adds implementation and training details, describes the observation tools and evaluation protocol, and reports further latent-capacity and teacher-trajectory analyses.

A APPENDIX

  • The appendix supplements Sections 3 and 4 with implementation and training details, observation tools, evaluation protocols, and analyses of latent-token capacity and teacher-trajectory similarity.Source: A APPENDIX · p. 14

A.1 OBSERVATION TOOLS

The appendix specifies LARVA’s observation and training procedures, evaluation protocol, and implementation, then analyzes token capacity and similarity to teacher acquisition trajectories.

A.1 OBSERVATION TOOLS

  • Overview uniformly samples 64 frames, while Skim and Focus each sample 16 frames from a reasoner-specified interval for progressively targeted inspection.Skim searches for query-relevant cues; Focus targets a concrete sub-question.Source: §A.1 · p. 14

A.2 PROJECTION AND EXPLICIT ACTION DECODING

  • Role-specific parameters θ and ϕ include a projector implemented as layer normalization followed by a linear layer.Source: §A.2 · p. 14

A.3 TRAJECTORY REPLAY AND TEACHER FORCING

  • During training, teacher actions determine which frames are acquired, while generated latent reasoning and observations are learned and used for later decisions.At inference, the reasoner generates acquisition actions itself.Source: §A.3 · p. 14

A.4 ALIGNMENT CONTEXTS AND TOKEN DISTRIBUTIONS

  • The alignment objective compares next-reasoning predictions from textual and latent contexts under the same frozen reference model.Both branches receive the same teacher-forced prefix, while gradients from the latent-context branch reach the generated reasoner and observer states.Source: §A.4 · p. 15

A.5 TRAINING AND EVALUATION SETUP

  • The training set is generated from 20,000 interaction trajectories drawn from Video-Marathon, VideoHolmes training set, and LongVideo-Reason; teacher trajectories with an incorrect final answer or any invalid tool call are filtered.LARVA is instantiated with Qwen2.5-VL-7B and Qwen3-VL-4B. Its main configuration uses eight latent tokens per reasoning and observation block and permits at most eight acquisition rounds.Source: §A.5 · p. 15

A.6 HUMAN-EVALUATION PROTOCOL

  • The human evaluation compares the first acquisition after a shared Overview, judging whether sampled frames reveal a new answer-relevant clue absent from that Overview.It uses 100 held-out questions and reports evidence-gain rate as adjudicated Yes judgments divided by all questions, with Uncertain cases retained in the denominator.Source: §A.6 · pp. 15–16

A.7 LATENT-TOKEN CAPACITY

  • Accuracy generally improves as observation or reasoning latents increase from one to eight, with smaller changes between six and eight tokens.Eight tokens provide a strong accuracy setting within the evaluated range; the figure does not establish an optimal latency–accuracy trade-off.Source: §A.7 · p. 16, Figure 3 · p. 16

A.8 TEACHER-TRAJECTORY SIMILARITY

  • LARVA outperforms both textual students on all three teacher-trajectory metrics across benchmarks, with gains over CoT varying by metric and benchmark.Gains over CoT are 4.6/0.8/1.3 points for Tool Sequence, 1.2/2.3/2.0 for Temporal IoU, and 1.2/5.0/11.9 for Optional Tool Agreement, in benchmark order.Source: §A.8 · p. 16, Table 6 · p. 16

Related papers

Loading 2610.35229v1…