PaperlayerView exact revision on arXiv

Source-linked AI summary

LensWalk: Agentic Video Understanding by Planning How You See in Videos

Keliang Li, Yansong Li, Hongze Shen, Mengdi Liu, Hong Chang, Shiguang Shan

arXiv:2603.24558v1cs.CVcs.AI

TL;DR

Fixed and preprocessed video inputs limit a reasoner’s ability to seek raw evidence as its understanding evolves. LensWalk lets an LLM plan observations through a reason–plan–observe loop, and improves o3 accuracy on long-video benchmarks while supporting adaptive observation budgets.

  • Problem

    Fixed sampling and preprocessed video representations limit the ability to shift visual focus or revisit raw evidence as intermediate hypotheses change.

    Source: §1 · p. 1

  • Method

    LensWalk uses a reason–plan–observe loop in which an LLM selects observation tools, temporal scope, and sampling density from its evolving reasoning state.

    Source: §1 · p. 2

  • Results

    Over 11.5% on LVBench and 6.7% on VideoMME (long split): LensWalk improves o3 accuracy on these long-video benchmarks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    LensWalk’s reasoning traces exhibit strategic reflection, progressive zoom-in, and integrative verification alongside performance and efficiency gains.

    Source: §1 · pp. 2–3

    Observer or memory-updating errors can still introduce inconsistencies in the evidence grounding process.

    Source: §3.4 · p. 5

Abstract

from arXiv · show

The dense, temporal nature of video presents a profound challenge for automated analysis. Despite the use of powerful Vision-Language Models, prevailing methods for video understanding are limited by the inherent disconnect between reasoning and perception: they rely on static, pre-processed information and cannot actively seek raw evidence from video as their understanding evolves. To address this, we introduce LensWalk, a flexible agentic framework that empowers a Large Language Model reasoner to control its own visual observation actively. LensWalk establishes a tight reason-plan-observe loop where the agent dynamically specifies, at each step, the temporal scope and sampling density of the video it observes. Using a suite of versatile, Vision-Language Model based tools parameterized by these specifications, the agent can perform broad scans for cues, focus on specific segments for fact extraction, and stitch evidence from multiple moments for holistic verification. This design allows for progressive, on-demand evidence gathering that directly serves the agent's evolving chain of thought. Without requiring any model fine-tuning, LensWalk delivers substantial, plug-and-play performance gains on multiple model recipes, boosting their accuracy by over 5\% on challenging long-video benchmarks like LVBench and Video-MME. Our analysis reveals that enabling an agent to control how it sees is key to unlocking more accurate, robust, and interpretable video reasoning.

1. Introduction

Video’s dense, unfolding information challenges finite analytical capacity, while fixed or one-shot observation cannot adapt when hypotheses change. LensWalk lets a reasoner plan observations from evolving evidence and reports accuracy gains alongside adaptive observation costs.

1. Introduction

  • Video’s dense, temporally unfolding information challenges finite analytical resources, making uniform exhaustive analysis intractable and inefficient.The introduction frames purposeful information seeking and actively scheduled observation under limited capacity as a response to this challenge.Source: §1 · p. 1
  • Fixed, one-time sampling and preprocessed retrieval cannot shift focus or revisit raw video as intermediate hypotheses change.Such approaches may drop sparse decisive events or bury them in redundancy.Source: §1 · pp. 1–2
  • LensWalk uses a multi-round reason–plan–observe loop to choose observation tools, video regions, and sampling levels as evidence accumulates.Its tools support broad cue discovery, focused inspection, and verification across multiple segments.Source: §1 · p. 2
  • LensWalk maintains a round-updated global cast table and reallocates frame budgets across regions and time scales as hypotheses change.Its reasoning traces include strategic reflection, progressive zoom-in, and integrative verification.Source: §1 · pp. 2–3
  • Over 11.5% on LVBench and 6.7% on VideoMME (long split): o3’s accuracy improves with LensWalk.LensWalk also uses minimal turns for easy queries and allocates more budget to uncertain cases, with total token usage far below retrieval agents consuming millions of tokens.Source: §1 · p. 2

2. Related Work

Related work selects or retrieves video context in several ways, but generally leaves observation fixed before reasoning or relies on static preprocessed representations. LensWalk instead makes context selection a step-wise process driven by evolving hypotheses.

2. Related Work

  • Context-selection methods choose keyframes, clips, or pruned representations before reasoning, whereas LensWalk makes selection step-wise and hypothesis-driven.The cited approaches include query-aware selection, task-specific alignment, and redundancy pruning.Source: §2 · p. 3
  • Video test-time scaling either strengthens reasoning over fixed visual inputs or uses agents to retrieve and aggregate preprocessed clips and captions.LensWalk instead scales inference by letting agents choose temporal ranges, sampling densities, and segment compositions.Source: §2 · p. 3
  • Video tool-use agents commonly query static caption or embedding repositories, leaving temporal coverage, sampling density, and segment composition largely fixed by preprocessing.More broadly, multimodal tool-use work includes agentic search and agentic visual processing.Source: §2 · p. 3

3. Method

LensWalk treats video understanding as active observation: a reasoner plans what to inspect and at what granularity, while an observer extracts evidence from the selected context. Its tools and lightweight grounding mechanisms support iterative exploration and multi-turn coherence.

3. Method

  • LensWalk addresses fixed observation stages by letting an agent dynamically choose where to look and how densely to sample as its thoughts evolve.The method frames video understanding as an active observation process coordinated by an LLM reasoner and VLM observer.Source: §3 · p. 3, §3.1 · p. 3

3.2. Reason-Scheduled Video Observation

  • The reasoner uses the query, video metadata, and prior actions and observations to produce a parameterized plan for the next observation.Each plan specifies a tool, guiding sub-question, scope and budget, and tool-specific arguments; the resulting frames and question go to the observer, and the plan–observation pair enters the history.Source: §3.2 · p. 4

3.3. Multi-Granularity Observation Tool Suite

  • The toolkit offers Scan Search for broad localization, Segment Focus for dense inspection of a continuous interval, and Stitched Verify for integrating non-contiguous segments.Stitched Verify can assign different sampling rates to different segments and present them together for checking.Source: §3.3 · pp. 4–5

3.4. Evidence Grounding for Long-Horizon Coherence

  • Timestamp anchors ground observations in time, while a global Subject Memory Table tracks entities and attributes across reasoning turns.The table supports consistent entity references and planning without repeatedly reidentifying subjects from the tool history.Source: §3.4 · p. 5
  • Observer or memory-updating errors can still introduce inconsistencies despite the lightweight grounding and reobservation approach.This evidence grounding ties observations to temporal scopes and entities and allows ambiguous regions to be revisited.Source: §3.4 · p. 5

4. Experiments

LensWalk is evaluated across long-video understanding and video-reasoning benchmarks, with analyses of model choices, component ablations, efficiency, and tool-call behavior. Results show improved benchmark performance alongside adaptive, multi-turn observation strategies.

4.1. Setting

  • Evaluation covers LVBench, LongVideoBench, VideoMME, MMVU, Video-MMMU, and EgoSchema, against VLM and agentic-video baselines.LensWalk uses combinations of language and vision-language models, with a maximum of 20 tool invocations and per-call frame budgets for its observation tools.Source: §4.1 · pp. 5–6

4.2. Main Results

  • LensWalk exceeds prior agentic methods by over 4% accuracy on VideoMME-Long and improves o3 accuracy by 2% on MMVU and 3% on Video-MMMU.It also enhances powerful base models and supports complementary Reasoner–Observer pairings and single-model self-play without fine-tuning.Source: Table 1 · p. 6, §4.2 · p. 6, Table 2 · p. 7
  • On MMVU, o3 reasoning over GPT-4.1 observations surpasses either model alone, while o3 self-play provides plug-and-play gains without fine-tuning.LensWalk also boosts o3 accuracy on MMVU and Video-MMMU by 2% and 3%, respectively.Source: §4.2 · p. 6

4.3. Ablation and Analysis

  • Qwen3-235B-A22B improves Qwen2.5-VL-7B by 4.3%, but yields only +0.1% on GPT-4.1 and -0.6% on Qwen2.5-VL-72B.The authors report that the open-source Reasoner struggles to enhance stronger Observers.Source: §4.3 · p. 6
  • Removing Scan Search, Stitch Verify, and Segment Focus drops accuracy by 4.6%, 3.2%, and 1.9%; Timestamp Anchor and Subject Memory raise it to 69.7% and 70.0%.The ablation is on VideoMME’s long split; the authors link the modules to cue discovery, cross-scene causal analysis, fine-grained facts, temporal alignment, and entity tracking.Source: Table 4 · p. 7, §4.3 · p. 7
  • LensWalk uses total tokens comparable to single-forward baselines while lowering peak per-turn tokens; it also achieves competitive accuracy with fewer frames and less inference time.Its observation budget converges quickly for simple queries and scales for longer or reasoning-intensive cases.Source: §4.3 · p. 7, Figure 3 · p. 7

4.4. Analysis on Tool-Call Behaviour

  • Tool calls fall into six modes: Direct Inquiry, Progressive Zoom-in, Scope Partitioning, Strategic Reflection, Integrative Verify, and Static Repetition.Figure 4 encodes each mode’s ratio by sector angle, average frame cost by dashed polygon, and accuracy by sector radius.Source: Figure 4 · p. 8, §4.4 · p. 8
  • Harder, lower-accuracy queries correlate with longer observation traces; strong Observers favor low-cost Direct Inquiry, while weaker ones prompt costlier scans or verification.Under uncertainty, Strategic Reflection and Integrative Verify are used instead of most Static Repetition, though rare looping failures remain.Source: §4.4 · p. 8

5. Conclusion

LensWalk actively schedules video observations from reasoning, reports plug-and-play gains and emergent behaviors, and argues for visual cognition orchestrated by reasoning intent.

5. Conclusion

  • LensWalk schedules video observations from reasoning in each iteration, transforming understanding into an adaptive reason–plan–observe loop.The framework lets a language model dynamically control what to see and at what granularity.Source: §5 · p. 8
  • The design yields plug-and-play performance gains across diverse multimodal models and long-video benchmarks, alongside progressive zoom-in, reflective replanning, and integrative verification.The passage characterizes these behaviors as emergent.Source: §5 · p. 8
  • The authors conclude that scalable video intelligence involves visual cognition orchestrated by reasoning intent, beyond ever-larger frozen representations.Source: §5 · p. 8

A. Method Details

The appendix details LensWalk’s reason–plan–observe algorithm, prompts, and tool schemas. Its pseudocode traces planned observations through VLM-visible contexts and the multi-turn loop.

A. Method Details

  • The appendix documents the overall reason–plan–observe algorithm, module prompts, and tool schemas used in the LensWalk agent.Source: A. Method Details · p. 12

A.1. Algotithm Demonstration

  • The pseudocode translates each Reasoner plan into VLM-visible context groups, queries the Observer, and threads time-anchored evidence and subject-table state through the loop.Source: §A.1 · p. 12

A.2. Prompt Design

  • The appendix presents the Reasoner’s THINK–PLAN–OBSERVE and termination prompts, plus separate Observer prompts for Scan Search, Segment Focus, and Stitched Verify.It also describes an optional subject-memory update prompt.Source: §A.2 · p. 12

A.3. Tool Schema, Format, and Signatures

  • Reasoner tool calls use the OpenAI SDK function-call structure, while each tool’s JSON schema specifies its name, description, and argument fields.The full tool-schema instantiation is shown in Figure 11.Source: §A.2 · p. 12, §A.3 · p. 12

B.1. Implementation Details

The implementation details identify the experiment’s models, observation-tool budgets, and deployment setup.

B.1. Implementation Details

  • Table 5 lists the experiment’s Reasoner and Observer models, their first public release dates, and the exact aliases used.Source: §B.1 · p. 12
  • Observation is limited to 20 tool-calling turns, with distinct per-tool frame budgets and sampling defaults for Scan Search, Segment Focus, and Stitch Verify.The requested interval-and-fps settings are converted to frame budgets and proportionally rescaled to each tool’s maximum.Source: §B.1 · p. 12
  • Closed-source models and Qwen3-235B-A22B use official APIs; other open-weight models are served with vLLM on four NVIDIA H100 80 GB GPUs.Source: §B.1 · p. 12

B.2. Benchmark Datasets

The appendix describes benchmark datasets and baseline reproduction, then provides qualitative Video-MME trajectories and visualizations of observation-tool use.

B.2. Benchmark Datasets

  • LVBench contributes 1,549 multiple-choice questions from 103 curated hour-long movies, while LongVideoBench-long uses 564 questions from 188 validation videos in its 900–3,600-second bucket.Automatic transcripts are used only for LongVideoBench-long; LVBench has no transcripts, so the agent relies on frames.Source: §B.2 · pp. 12–13
  • Algorithm 1 initializes query history and an empty subject table, obtains a Reasoner plan each turn, and returns its answer when the plan selects Finish.Source: §B.2 · p. 13
  • The loop maps tool arguments into VLM-visible context groups and joins per-group replies in temporal order; intervals carry sampling settings such as fps or frame budget.Source: §B.2 · p. 13
  • Evaluations include 900 Video-MME long-split[13] MCQs across 300 videos lasting 30–60 minutes, 625 MMVU[67] validation questions, 900 Video-MMMU[17] MCQs, and 500 EgoSchema[27] validation questions.Video-MMMU questions are evenly split across Perception, Comprehension, and Adaptation; MMVU spans 40+ academic subjects.Source: §B.2 · p. 13

B.3. Baseline Reproduction

  • Most baseline results come from prior papers or public leaderboards; the authors rerun GPT-4.1, GPT-5, and selected Qwen2.5-VL-72B evaluations.GPT-5 uses minimal reasoning effort in both settings; frame budgets are 256 for GPT-4.1/5 and 128 for local Qwen2.5-VL-72B.Source: §B.3 · p. 13

C. More Results

  • Qualitative case studies use Video-MME’s long split, with o3 acting as both Reasoner and Observer; trajectories highlight crucial visual evidence in red and mark sampled tool contexts green, blue, and yellow.Green indicates ScanSearch, blue Segment Focus, and yellow Stitched Verify.Source: C. More Results · pp. 13, 15

C.1. Additional Quantitative Analyses

LensWalk’s additional analyses examine its efficiency, adaptive observation budget, and performance across model configurations. They also test whether gains come merely from reusing selected frames.

C.1. Additional Quantitative Analyses

  • 14–32× fewer frames per query while outperforming MR. Video and VideoAgent; compared with DVD, LensWalk trades some accuracy for zero preprocessing and much lower total cost.Retrieval-based agents’ end-to-end cost rises when full-video preprocessing is counted.Source: §C.1 · pp. 15–16
  • LensWalk typically converges in 2.6–2.8 steps on short or simple benchmarks, increasing interaction steps and frame use as videos lengthen or questions demand more reasoning.The analysis attributes efficiency to avoiding full-video preprocessing and allocating observation budget when additional evidence is needed.Source: §C.1 · p. 16
  • A single-pass VLM given LensWalk’s visited frames gains only +0.8 to +2.6 points, indicating that frame selection alone does not account for the improvement.The same analysis reports LensWalk remains effective with GLM-4.7 and DeepSeek-V3, so it is not tied to one planner.Source: §C.1 · p. 16

C.2. Behaviour Analysis

The behavior analysis describes distinct observation patterns, from quick answers to progressively focused or cross-segment investigation, alongside strategic adjustment and unproductive repetition.

Behaviour Analysis

  • Direct Inquiry resolves straightforward temporal-localization or overall-video-type questions with one or two tool calls.Examples include asking what happens at a specified time or at the beginning of a video.Source: §C.2 · p. 16
  • Progressive Zoom-in starts with a coarse global scan, then narrows candidate segments through increasingly fine-grained observations.The example locates relevant cues before focusing on the queried 10m platform-diving competition.Source: §C.2 · p. 16
  • Integrative Verify formulates a hypothesis after preliminary observations, then tests it by jointly inspecting related segments.In the example, the agent reviews clips for earthquake evidence and concludes none is reported when it finds none.Source: §C.2 · p. 16
  • Strategic Reflection recognizes an unhelpful observation strategy, backtracks to broader temporal regions, and refines the search again.The example shifts away from subtitles that do not explain game rules and finds the relevant strategy elsewhere in the video.Source: §C.2 · p. 16
  • Scope Partitioning divides a video into coarse intervals, checks them for useful clues, then inspects the most promising region more finely.The example selects 1200–1800s after scanning three broad intervals.Source: §C.2 · p. 17
  • Static Repetition traps the agent in nearly identical calls on the same segment, even after repeated failures to find supporting evidence.Unlike Strategic Reflection, it does not explore alternative temporal regions or adjust the query.Source: §C.2 · p. 17

C.3. Failure Modes and Analysis

The failure analysis describes premature stopping, excessive or misdirected observation, and repeated calls as ways the agent can fail to obtain or preserve decisive evidence.

Failure Modes and Analysis

  • Premature Conclusion can occur within 1–2 turns when the agent accepts a frequent initial cue without checking distractors or other video segments.The example’s “pyramid complex” cue triggers unwarranted confidence; the passage suggests stronger internal criticism or verification.Source: §C.3 · p. 17
  • Evidence Dilution occurs when continued exploration buries strong early evidence under irrelevant or noisy context, inviting recency bias or confusion.The described agent eventually hallucinates an answer based on later, weaker observations.Source: §C.3 · p. 18
  • Persistent Ambiguity arises when the agent follows an uninformative trajectory and never obtains decisive visual evidence, leaving a semantics-only guess.The example focuses on related subtitles rather than frame-wise visual changes and misses the key opening event.Source: §C.3 · p. 18
  • Static Repetition is a degenerate loop of identical calls for the same timestamp, reflecting failures of meta-cognition or state tracking.The passage identifies detecting stagnation and triggering a strategic shift as needed mechanisms.Source: §C.3 · p. 18

Related papers

Loading 2603.24558v1…