PaperlayerView exact revision on arXiv

Source-linked AI summary

Active Video Perception: Iterative Evidence Seeking for Agentic Long Video Understanding

Ziyang Wang, Honglu Zhou, Shijie Wang, Junnan Li, Caiming Xiong, Silvio Savarese, Mohit Bansal, Michael S. Ryoo, Juan Carlos Niebles

arXiv:2512.05774v2cs.CVcs.AIcs.CL

TL;DR

Long-video queries rely on sparse, dispersed cues, while caption-based agentic pipelines can spend computation on irrelevant content and lose fine-grained grounding. AVP actively plans and executes targeted observations, then reflects on whether the resulting evidence is sufficient. Across five benchmarks, it achieves the best overall accuracy among agentic frameworks and surpasses DVD by 5.7% in average accuracy.

  • Problem

    Long-video understanding must locate sparse, temporally dispersed cues, while query-agnostic captioning can waste computation and blur fine-grained temporal and spatial information.

    Source: §1 · p. 1

  • Method

    AVP treats video as an interactive environment, iteratively planning targeted observations, collecting compact timestamped evidence, and reflecting on whether it suffices to answer the query.

    Source: §1 · p. 2, §3.1 · p. 4, §3.2 · p. 4

  • Results

    AVP achieves the best overall accuracy among agentic frameworks across five long-video benchmarks, surpassing DVD by 5.7% in average accuracy.

    Source: §6 · p. 8

  • Takeaways & Limitations

    The results support active, query-directed evidence seeking as an approach that improves accuracy and efficiency on long-video understanding benchmarks.

    Source: §1 · p. 2, §6 · p. 8

    AVP is primarily evaluated in offline video QA with the full video available; embodied or online streaming settings remain future-work directions.

    Source: A. Limitations · p. 13

Abstract

from arXiv · show

Long video understanding (LVU) is challenging because answering real-world queries often depends on sparse, temporally dispersed cues buried in hours of mostly redundant and irrelevant content. While agentic pipelines improve video reasoning capabilities, prevailing frameworks rely on a query-agnostic captioner to perceive video information, which wastes computation on irrelevant content and blurs fine-grained temporal and spatial information. Motivated by active perception theory, we argue that LVU agents should actively decide what, when, and where to observe, and continuously assess whether the current observation is sufficient to answer the query. We present Active Video Perception (AVP), an evidence-seeking framework that treats the video as an interactive environment and acquires compact, queryrelevant evidence directly from pixels. Concretely, AVP runs an iterative plan-observe-reflect process with MLLM agents. In each round, a planner proposes targeted video interactions, an observer executes them to extract time-stamped evidence, and a reflector evaluates the sufficiency of the evidence for the query, either halting with an answer or triggering further observation. Across five LVU benchmarks, AVP achieves highest overall accuracy with significant improvements. Notably, AVP outperforms the best agentic method by 5.7% in average overall accuracy while only requires 18.4% inference time and 12.4% input tokens.

1. Introduction

Long-video queries depend on sparse, temporally dispersed cues, while caption-based agentic pipelines can waste computation and blur fine-grained grounding. AVP instead treats video as an interactive environment and iteratively seeks query-relevant evidence.

1. Introduction

  • Long-video understanding requires integrating sparse, temporally dispersed cues, while dense full-length MLLM processing is costly and can dilute or overlook relevant evidence.Most video tokens are redundant, but brief, localized evidence may determine the answer.Source: §1 · p. 1
  • Agentic methods that rely on query-agnostic captioners spend computation on irrelevant content and may lose fine-grained temporal and spatial cues.Caption-based localization may also weaken causal tracing.Source: §1 · p. 2
  • Inspired by active perception, AVP chooses what, when, and where to observe and checks whether accumulated evidence is sufficient for the query.The approach draws on the idea of planning observation around a question rather than watching every frame.Source: §1 · p. 2
  • AVP uses an iterative plan–observe–reflect process to acquire compact, time-stamped evidence and refine its focus when the current evidence is insufficient.The planner proposes targeted interactions, the observer executes them, and the reflector assesses query sufficiency.Source: §1 · p. 2
  • AVP surpasses the leading agentic method DVD by 5.7% in average accuracy across five benchmarks, using 18.4% of its inference time and 12.4% of its input tokens on LVBench.The benchmarks are MINERVA, LVBench, Video-MME, MLVU, and LongVideoBench.Source: §1 · p. 2

2. Related Work

Prior LVU work addresses long-video inputs through context extension, token reduction, keyframe selection, and video reasoning strategies. Agentic frameworks also use captions, visual programming, or text-space retrieval, whereas AVP reasons directly over selected visual inputs.

2. Related Work

  • Prior LVU methods extend context, reduce video tokens, or select keyframes; VAP frames keyframe selection as active-perception data acquisition using a text-conditioned video-generation model.AVP instead frames LVU as query-driven evidence seeking in video environments.Source: §2 · p. 2

Agentic Frameworks for Long Video Understanding

  • Agentic LVU methods include captioner–LLM pipelines, visual programming, reflection, and text-space retrieval, while AVP observes visual inputs directly through an iterative process.Compared with visual chain-of-thought approaches, AVP is query-adaptive rather than using fixed FPS/resolution settings, and it is training-free.Source: §2 · pp. 2–3, Agentic Frameworks for Long Video… · p. 3

3. Method

AVP formulates long-video understanding as iterative, query-driven evidence seeking. It plans targeted observations, records timestamped evidence, and reflects on whether that evidence supports an answer or requires another observation round.

3. Method

  • AVP treats the video as an interactive environment and iteratively decides how to observe it based on the query and previous observations.Its plan–observe–reflect cycle continues until the evidence is sufficiently confident or the round limit is reached.Source: §3 · p. 3

3.1. Query-Conditioned Action Planning

  • The planner specifies what evidence to seek, which temporal region to inspect, and the sampling granularity for each observation.It can use query timestamps or textual cues, begin with a low-cost sweep when no prior is available, and tighten or shift the region based on reflector feedback.Source: §3.1 · p. 4

3.2. Targeted Video Observation

  • The observer executes the plan and records compact, timestamp-aware descriptions of visual events in an evidence list accumulated across rounds.The cumulative list serves as working memory for assessing evidence sufficiency and updating later plans.Source: §3.2 · p. 4

3.3. Evidence Reflection and Re-Planning

  • The reflector evaluates whether accumulated evidence supports an answer; if confidence is below threshold, its justification and observation history guide replanning toward unresolved cues.The process repeats until evidence is sufficient or the round limit is reached.Source: §3.3 · p. 4

4. Experimental Setup

AVP is evaluated on five diverse long-video benchmarks using multiple-choice QA and standard accuracy, with Gemini-2.5-Pro as the default MLLM agent.

4.1. Datasets

  • The five benchmarks cover hand-crafted video reasoning, hour-long videos, multitask QA, and referred reasoning questions.They include MINERVA (1,515 questions), LVBench (1,549 questions across 103 hour-long videos), MLVU (2,175 QA samples), Video-MME (2,700 samples), and LongVideoBench (1,337 questions).Source: §4.1 · p. 5

4.2. Evaluation Metrics

  • Evaluation uses multiple-choice QA and standard accuracy, without auxiliary subtitles.Source: §4.2 · p. 5

4.3. Implementation Details

  • Gemini-2.5-Pro is the default agent; experiments fix the input-token limit at 128K, maximum rounds at 3, and confidence threshold at 0.7.For spatial tokens, the setup uses low and medium scales of 66 and 258 tokens per frame.Source: §4.3 · p. 5

5. Results

AVP achieves the best overall accuracy across five benchmarks and outperforms agentic baselines, while using less inference time and fewer input tokens on LVBench. Ablations report gains from planning and reflection.

5.1. Main Results on Long Video Benchmarks

  • 5.7% average improvement over DVD accompanies AVP’s best overall accuracy on all five benchmark datasets.Source: Table 1 · p. 6, §5.1 · p. 6
  • 4.5% average accuracy gain over Gemini-2.5-Pro and 4.4% over Gemini-2.5-Flash are reported for AVP with those respective backbones.AVP also significantly outperforms video-specific MLLMs.Source: §5.1 · p. 6

5.2. Quantitative Analysis

  • The quantitative analysis examines AVP’s inference efficiency and the effects of its design choices.Source: §5.2 · p. 6

5.2.1. Efficiency Analysis

  • 5.44× faster inference (81.6% reduction) and 12.4% of DVD’s input tokens accompany improved LVBench accuracy.DVD takes 790.5 seconds per video, including 637.2 seconds for captioning; AVP takes 145.3 seconds.Source: Table 2 · p. 7, §5.2.1 · p. 7

5.2.2. Ablation Study

  • Adding the Planner and then the Reflector to the Observer baseline consistently improves MINERVA and LVBench accuracy.The reported component ablation attributes gains to query-conditioned planning and reflection.Source: Table 3 · p. 7, §5.2.2 · p. 7
  • Table 4 examines the impact of varying model selection across the Planner, Observer, and other AVP components.Source: §5.2.2 · p. 7

REFLECTOR MINERVA LVBench

AVP benefits from stronger agent components and a few iterative observation rounds; its visualization shows how a targeted revisit resolves evidence left insufficient by a coarse scan.

REFLECTOR MINERVA LVBench

  • Stronger Observer models yield the largest gains across both benchmarks, while the best configuration strengthens Planner, Observer, and Reflector together.The ablation attributes the Observer’s importance to acquiring evidence that planning and reflection depend on.Source: REFLECTOR MINERVA LVBench · p. 7, Table 4 · p. 7
  • Performance improves from one to three rounds on MINERVA and LVBench, then saturates beyond three rounds.The gains are more pronounced on MINERVA, where multi-hop reasoning benefits from repeated reflection and targeted re-observation.Source: REFLECTOR MINERVA LVBench · p. 7, Table 5 · p. 7

5.3. Visualization

  • In the Tombstone example, a coarse scan localizes [1:00, 1:10] but is insufficient; a targeted revisit reveals the monument and supports answer D.The follow-up uses 2 FPS at medium resolution and identifies the monument in the upper-left background.Source: §5.3 · p. 8, Figure 3 · p. 8

6. Conclusion

AVP frames long video understanding as iterative, query-driven evidence seeking and reports accuracy and efficiency gains over the leading agentic method. The appendix also includes limitations, further results and analysis, qualitative examples, and implementation details.

6. Conclusion

  • AVP treats video as an interactive environment, using MLLM agents in an iterative plan–observe–reflect loop to gather compact, time-stamped evidence directly from pixels.The agents decide what, where, and at what granularity to inspect.Source: §6 · p. 8
  • Across five long video benchmarks, AVP has the best overall accuracy among agentic frameworks and beats DVD by 5.7% in average accuracy while using 18.4% of its inference time and 12.4% of its input tokens.Source: §6 · p. 8

Appendix

  • The appendix contains limitations, additional quantitative and qualitative analyses, and implementation details.Source: Appendix · p. 13

A. Limitations

AVP is primarily evaluated in offline video QA with the full video available; online or embodied real-time settings and learned sensing policies remain future directions. Its reasoning-trace analysis reports the highest overall MiRA score and stronger temporal localization, logical reasoning, and correctness than single-pass MLLMs.

A. Limitations

  • AVP is primarily evaluated in offline video QA with the full video available; online streaming or embodied real-time settings remain future work.The authors also suggest learning policies to optimize long-horizon sensing efficiency under resource and latency constraints, as a complement to prompting.Source: A. Limitations · p. 13

B.1. Reasoning Trace Analysis

  • MiRA evaluates generated step-by-step video-QA reasoning against a ground-truth trace across perceptual correctness, temporal localization, logical reasoning, and completeness.Source: §B.1 · p. 13
  • AVP achieves the highest overall MiRA score and outperforms baselines across key reasoning dimensions.Compared with single-pass MLLMs, it delivers stronger temporal localization, logical reasoning, and correctness; temporal and completeness gains are highlighted for complex multi-hop queries.Source: §B.1 · p. 13

B.2. Full Results for LVBench

On LVBench, AVP achieves the best overall accuracy among prior systems, including DVD. Its strongest split results are on tasks requiring integration across long temporal ranges.

B.2. Full Results for LVBench

  • AVP achieves the best overall accuracy on LVBench, outperforming all prior systems, including DVD.Source: §B.2 · p. 13, Table 7 · p. 13
  • Table 6 reports MINERVA multiple-choice accuracy and MiRA scores normalized between 0 and 1, with four reasoning dimensions.The dimensions are Perceptual Correctness, Temporal Localization, Logical Reasoning, and Completeness.Source: Table 6 · p. 13
  • AVP’s strongest split scores are on Event Understanding, Temporal Grounding, and Summarization, which require integrating information across long temporal ranges.Source: §B.2 · p. 13, Table 7 · p. 13

B.3. Additional Ablation Study

The additional studies test AVP under controlled backbones, alternative evidence formats and stopping thresholds, and different query and video-domain settings. They report gains in accuracy and efficiency across these settings.

B.3. Additional Ablation Study

  • On Video-MME’s long split, AVP achieves the highest overall accuracy and lowest inference time among compared agentic methods with OpenAI-o3.Table 8 reports long-split accuracy and average inference time.Source: §B.3 · pp. 13–14, Table 8 · p. 14
  • On MINERVA, AVP accuracy rises from 41.2% with Qwen3-VL-8B to 65.6% with Gemini-2.5-Pro, with consistent gains across the tested MLLMs.The passage reports 56.9% with Gemini-2.5-Flash and 59.0% with OpenAI-o3; 41.2% is 2.0% above direct inference.Source: §B.3 · p. 14, Table 9 · p. 14
  • Replacing the structured, time-aligned evidence list with a flat list degrades performance on both benchmarks.The ablation attributes the difference to the importance of temporal and semantic organization for planning and reflection.Source: §B.3 · p. 14, Table 10 · p. 14
  • A moderate confidence threshold performs strongest on MINERVA and ties for best on LVBench; with OpenAI-o3, adding the Planner and Reflector raises MINERVA accuracy by 3.2% and 1.6%.Using only the Observer achieves 54.2%; lower thresholds risk premature halting, while stricter thresholds offer no additional gains.Source: §B.3 · p. 14, Table 11 · p. 14

B.4. Amortized Multi-Query Efficiency

  • In LVBench’s multi-query setting averaging 15 queries per video, AVP reduces total inference time by 26.5% compared with DeepVideoDiscovery.DeepVideoDiscovery spends 637.2 seconds on database construction and 2329.5 seconds on per-query localization and answering; AVP answers all queries in 2179.5 seconds.Source: §B.4 · p. 14

B.5. Token Efficiency Across Backbones

  • On LVBench with OpenAI-o3, AVP averages 62.5K input tokens per query, 5.8% of DeepVideoDiscovery’s usage and below Gemini-based AVP’s 132.5K.The passage attributes token efficiency to adaptive temporal and spatial token planning, rather than backbone-specific visual compression.Source: §B.5 · pp. 14–15

B.6. Robustness Across Video Domains

  • On MINERVA, AVP outperforms its Gemini-2.5-Pro backbone across sports, instructional videos, and films.Table 12 reports domain-level accuracy percentages.Source: §B.6 · p. 15, Table 12 · p. 15

C. Additional Qualitative Results

The examples show AVP using successive observations to resolve a multi-hop query, and illustrate how a coarse scan can miss a brief event and yield a wrong answer.

C.1. Additional Visualization

  • For a numerical query, AVP first reads timestamped measurements, then broadens its search to find a narrated hatchling-length range before combining both evidence sources.The reflector marks the first round’s evidence insufficient and triggers the second observation round.Source: §C.1 · p. 15

C.2. Failure Case

  • A coarse 0.5 FPS scan misses a three-pointer at 00:20, leaving a coherent but incomplete evidence list and a confident incorrect count.Among 50 sampled LVBench failures, 76% are attributed to observer perception, versus 8% to planning and 16% to reflection.Source: §C.2 · p. 15

D. Additional Implementation Detail

The implementation prompts specify how the planner selects observations, how the observer extracts evidence, and how the reflector decides whether that evidence can answer the query. Examples illustrate targeted and broader searches, as well as a missed event under coarse sampling.

D.1. Prompts

  • The planner specifies what to observe, where to look, and how to sample, using exact timestamp ranges for factual queries and added context for reasoning queries.For single timestamps, factual queries use a one-second forward window; reasoning queries add 15–30 seconds of context, while approximate timing uses ±15 seconds.Source: §D.1 · p. 15
  • When timing is unknown, the planner uses a coarse uniform scan at 0.25–1 FPS; explicit timestamps use region analysis at about 2 FPS.The prompt sets low or medium spatial token rates; unknown timing defaults to low or medium resolution.Source: §D.1 · p. 15
  • The prompt outputs are structured as JSON: the planner returns reasoning and observation plans, while the observer returns a detailed response, timestamped key evidence, and reasoning.Planner fields specify the sub-query, uniform or region search, FPS, spatial token rate, and time spans.Source: §D.1 · pp. 15–17
  • In the MINERVA example, AVP reads measurements at 09:58, then scans the full video at low FPS to find the missing narrated length range.The reflector judges the first observation insufficient and combines the measurements with the newly found range to choose the correct option.Source: Figure 4 · p. 16
  • The observer analyzes a specified segment to extract query-relevant, timestamped evidence, using prior-round context to avoid redundant descriptions.If a region contains no relevant information, the prompt directs the observer to say so and suggest a uniform scan or additional regions.Source: §D.1 · p. 16
  • In the basketball example, a low-resolution 0.5 FPS full-video scan misses a three-pointer, producing a confident count of two instead of three.The prompt’s fallback advises expanding the search when a region contains no relevant information.Source: §D.1 · p. 16, Figure 5 · p. 17
  • The reflector evaluates cumulative evidence for sufficiency and either gives a direct answer or identifies what remains missing or uncertain.It reports a confidence score compared with the halting threshold; the prompt describes scores of at least 0.7 as clear entailment and scores below 0.4 as key cues unaddressed.Source: §D.1 · p. 17
Loading 2512.05774v2…