Source-linked AI summary
From Intent to Evidence: Policy-Steered Multi-Strategy Retrieval for Long-Video Agents
Can Zhang, Baofeng Zhang, Xiaotian Han, Junyuan Shang, Yuchen Ding, Shuohuan Wang, Dianhai Yu, Ruirui Li
TL;DR
Existing long-video agents often acquire evidence uniformly despite question-specific requirements, motivating VESTA’s policy-steered alternative. VESTA routes acquisition and evidence accounting, preserves autonomous verification and exploration, and improves performance especially as temporal search spans grow.
Problem
Existing agents use uniform evidence acquisition despite requirements for concentrated evidence, broad occurrence coverage, or competing-hypothesis discrimination, while fine-grained procedures restrict exploration autonomy.
Method
VESTA is a training-free acquire–verify–consolidate agent whose router selects focused, recall, or contrastive retrieval and an evidence-accounting policy, while the Reasoner remains autonomous.
Results
VESTA improves across Video-MME-v2 and long-video benchmarks, with gains scaling from +1.5 on LVBench to +6.9 on the LongVideoBench long subset and parity on EgoSchema.
Takeaways & Limitations
Policy-steered acquisition and temporal evidence accounting provide benefits that are strongest when relevant evidence must be located across longer temporal spans.
Takeaways & Limitations
Retrieved scenes remain provisional references and do not constrain later search, but recall retrieval does not guarantee recovery of every relevant occurrence.
Abstract
from arXiv · showhide
Existing long-video agents acquire evidence through one uniform behavior, ignoring whether the required evidence is concentrated, requires broad occurrence coverage, or must discriminate competing hypotheses---which can cause failure before substantive reasoning begins. Prescribing a fine-grained solution procedure for every question is not a satisfactory remedy, as it restricts autonomous exploration. We propose VESTA, a training-free long-video agent organized as a route-conditioned acquire--verify--consolidate loop. Before exploration, an intent router infers an evidence-acquisition policy---focused, recall, or contrastive retrieval over a shared visual--speech scene index---together with an evidence-accounting policy that configures the evidence view maintained during exploration. Policy-steered retrieval yields provisional references that multimodal evidence operations convert into observations, while the Reasoner remains free to verify them, re-query using intermediate findings, or inspect regions outside the retrieved set. A temporal evidence ledger consolidates observations into an adaptive, compressed view of temporal location, provenance, coverage, conflicts, verification outcomes, and hypothesis support, exposing missing and unresolved evidence to guide subsequent acquisition; finalization prioritizes verified observations. On Video-MME-v2, VESTA improves average accuracy by 2.7 points over VideoARM and gains across all six reported metrics. On LongVideoBench, EgoSchema, and LVBench under shared query-time models, it improves by 6.9 points on the LongVideoBench long subset and 1.5 on LVBench, and matches VideoARM on EgoSchema.
1 Introduction
Long-video agents must acquire question-relevant evidence under limited computational budgets, but existing approaches often use uniform or overly prescribed acquisition behaviors. VESTA instead routes evidence requirements while preserving autonomous exploration, and reports gains that grow with temporal span.
- Relevant evidence may be concentrated, sparse, recurrent, or distributed across distant intervals, making acquisition a prerequisite for long-video reasoning.
- Uniform acquisition can collapse candidates onto one region, omit relevant occurrences, or imbalance evidence for competing answers.
- Fine-grained tutorial procedures would constrain reasoning, hypothesis formation, search revision, and inspection of unanticipated regions.
- VESTA routes focused, recall, or contrastive acquisition and configures evidence accounting while leaving subsequent reasoning steps open.
- VESTA improves Video-MME-v2 accuracy by 2.7 points over VideoARM and gains +6.9 on LongVideoBench’s long subset and +1.5 on LVBench, while matching VideoARM on EgoSchema.
2 Related Work
Long-video research combines model-centric context expansion and agentic systems that actively select evidence through tools or external substrates. VESTA’s framing builds on this agentic direction while organizing evidence acquisition around differing requirements.
- Model-centric methods extend long-video processing through larger context windows, frame sampling, visual-token compression, or integrated long-range information.
- Video agents use ReAct-style control, task decomposition, multi-round prompting, and external evidence substrates to revise trajectories from intermediate observations.
3 Method
VESTA uses strategy-level routing to structure initial evidence acquisition without constraining the Reasoner’s later exploration. Its shared scene index, multimodal operations, and ledger support adaptive retrieval, verification, and coverage.
- Given a video, question, candidates, and interaction budget, VESTA provides strategy-level acquisition guidance while leaving reasoning, tool use, and exploration adaptive.
- The intent router outputs acquisition policy π and accounting policy µ before exploration, without accessing video content, predicting answers, or updating evidence state.
- Focused retrieval targets concentrated evidence, recall retrieval broadens coverage across separated temporal regions, and contrastive retrieval supports competing-hypothesis comparison.
- All policies share a scene cache, encoders, similarity function, evidence operations, and interaction budget while changing query construction and candidate formation.
- Recall retrieval broadens temporal coverage through bounded, time-separated expansion but does not guarantee recovery of every relevant occurrence.
- Retrieved scenes remain provisional references, while the Reasoner can verify, re-query, inspect omitted intervals, and finalize using verified observations.
4 Experiments
Experiments evaluate VESTA across complementary temporal regimes and mechanism configurations. Results show strongest gains when evidence must be located across long timelines, while ablations support contributions from retrieval, routing, and ledger-based consolidation.
- Video-MME-v2: VESTA improves Video-MME-v2 average accuracy from 47.6% to 50.3% over VideoARM, with gains across all six reported metrics.The largest gain is Level 1 (+6.6), while Capability Consistency increases by 3.5 points and Reasoning Coherence by 1.4.
- Video-MME-v2: VESTA improves Video-MME-v2 accuracy in seven of ten question categories, covering 74.6% of the benchmark.The fixed ablation subset differs from the full benchmark by less than one percentage point in every category.
- Video-MME-v2: Frame-Only questions gain +7.0 points, while Frames & Audio and Order gain +4.6 and +4.4 points, respectively.These categories require focused visual localization, audiovisual alignment, or ordering observations from separated moments.
- Long-video benchmarks: LongVideoBench improves by 6.9 points over the VideoARM∗ rerun, while LVBench rises from 67.2% to 68.7% and EgoSchema matches VideoARM∗ at 73.8%.The cross-benchmark pattern follows the temporal span over which evidence must be located: largest on LongVideoBench, present on LVBench, and absent on compact EgoSchema clips.
- Efficiency: VESTA’s actual token consumption remains in the same range as agentic baselines because its reusable scene cache is local and retrieval consumes no remote tokens.Its nominal step budget counts more decisions than baselines because only remote decisions are paid.
- Ablation study: On the fixed 400-question Video-MME-v2 ablation subset, the full configuration outperforms the no-mechanism variant by 15.8 points.Retrieval contributes 9.3 points over no mechanisms, routing adds 3.0 points over retrieval, and the ledger adds 3.5 points to route-conditioned retrieval.
5 Conclusion
VESTA addresses the mismatch between question-specific evidence requirements and uniform acquisition by combining policy-steered retrieval with structured evidence accounting. Its gains span four long-video benchmarks, while fixed policies and proprietary query-time models remain scope boundaries.
- VESTA uses a route-conditioned acquire–verify–consolidate loop with focused, recall, or contrastive acquisition and a temporal evidence ledger.The ledger tracks observations, verification outcomes, coverage, conflicts, and unresolved requirements while retrieved scenes remain provisional references.
- Across Video-MME-v2 and three long-video benchmarks, gains scale with the temporal span over which evidence must be located.Controlled ablations attribute complementary benefits to policy-steered retrieval and structured evidence accounting.
- The current instantiation uses a fixed policy set and proprietary models for query-time reasoning and visual verification.Extending the policy space and evaluating fully open-model configurations remain future work.
A.1 Ablation Subset Composition
The fixed 400-question Video-MME-v2 ablation subset closely preserves the full benchmark’s question-type composition across all ten official categories.
- The ablation subset covers all ten official Video-MME-v2 question categories.Its category shares differ by less than one percentage point from the shares among all 3,200 questions.
- The composition alignment reduces the risk that ablation conclusions reflect over-representation of a small set of question types.
- Figure 4 presents the question-type distribution for the fixed 400-question Video-MME-v2 ablation subset.
A.2 Routing and Budget Trends
VESTA’s router adapts acquisition breadth and interaction effort to anticipated evidence demands across Video-MME-v2 question categories.
- Categories requiring repeated-event coverage, temporal comparison, or separated-region evidence predominantly use broad, recall-oriented exploration.Frame-Only questions use a more balanced mixture of focused and broad routes.
- Order, temporal, physical-world, and other globally grounded categories tend to require longer interaction trajectories.Localized visual and social questions more often terminate after shorter exploration.
- Contrastive routing remains selective rather than becoming a default response to difficulty.
A.3 Additional Benchmark Evaluations
The main paper reports the shared-model evaluation configuration and results for LongVideoBench, EgoSchema-500, and LVBench, while the appendix adds LVBench ledger case studies.
- Results on the LongVideoBench long subset, EgoSchema-500, and LVBench are reported in the main paper’s Section 4.3.
- Appendix A.6 provides two LVBench ledger case studies as mechanism examples.
A.4 Qualitative Case Studies
VESTA’s case studies show route-specific acquisition helping the agent recover evidence that uniform search misses, through targeted localization and cross-scene verification.
- Contrastive acquisition compares separated candidate scenes and combines speech transcription with clip analysis to verify a narrative tonal transition.The baseline selects dialogue from the climax but misses the transition toward a calmer exchange.
- Focused acquisition retrieves the target sign near 540–546 seconds, whereas the unguided baseline repeatedly inspects overlapping regions from 15–133 seconds.The diagnostic question asks for the complete name on a circular golden sign in an 844.8-second video.
A.5 Evidence-Acquisition Cost Analysis
The cost analysis shows that VESTA’s interaction burden depends primarily on the downstream search trajectory and evidence consolidation, while on-demand processing avoids DVD’s much larger fixed visual-input burden.
- All three Video-MME-v2 UID 015-1 configurations answer correctly while distributing effort differently across reasoning, retrieval, and visual inspection.The diagnostic isolates how interaction budgets are allocated rather than whether the answer is correct.
- Routing with the Temporal Evidence Ledger but without Retrieval spends the most, Retrieval only spends the least, and Routing with Retrieval but without the Ledger lies between them.The cost differences arise mainly from downstream Reasoner and visual-analysis calls.
- Semantic Scene Retrieval adds little direct latency; its meaningful cost effect comes from changing Temporal Visual Inspection calls and evidence consolidation.Retrieval and the ledger shape cost indirectly through the search trajectory rather than making retrieval itself expensive.
- Figure 5 reports focused, broad, and contrastive route distributions alongside their corresponding mean interaction budgets across Video-MME-v2 question categories.The figure compares routing prevalence with interaction-budget trends by question category.
- DVD’s fixed preprocessing pipeline imposes an order-of-magnitude larger visual-input burden than on-demand VESTA trajectories.VESTA pays for evidence when the question requires it instead of frontloading a dense whole-video scan.
- LVBench examples illustrate the ledger’s organization of provisional references, verified observations, temporal coverage, and task-specific evidence relations rather than additional benchmark results.One example consolidates repeated numeric observations; another organizes several verified intervals for a non-counting question.
B.1 Implementation Details
VESTA uses a local visual–speech scene cache, one-shot intent routing, route-specific retrieval, and persistent ledger state to support adaptive evidence acquisition and verification.
- The local scene cache uses PySceneDetect scenes, up to four frames per scene, aligned ASR, 2,048-dimensional Qwen3-VL-Embedding representations, and NanoVectorDB.Cache construction, encoding, and retrieval run locally without closed-model API calls.
- All retrieval policies start with K = 6: focused returns six scenes, recall disperses candidates across time, and contrastive distributes references across hypotheses.Recall expands to at most 15 candidates and favors scenes separated by at least 20 seconds.
- VESTA maintains detailed tool memory and a Temporal Evidence Ledger that normalizes observations with provenance and exposes coverage, relations, conflicts, verification, and unresolved requirements.Ledger compaction changes the Reasoner’s snapshot without removing detailed observations.
- The intent router uses only the question and optional candidate answers to select evidence-acquisition and evidence-accounting policies, without perceiving video or predicting the answer.A deterministic linguistic prior can be retained when the model estimate is unavailable or insufficiently confident.
- Acquisition policy π constructs initial temporal references, while accounting policy µ determines which coverage, relational, and unresolved-evidence information is emphasized.Neither policy prescribes a fixed reasoning procedure or answer; subsequent retrieval is formulated by the Reasoner.
- Algorithm 1 alternates ledger compaction, Reasoner action selection, evidence execution, tool-memory updates, and ledger updates before finalization.The loop permits re-querying and inspection outside the initial candidate set.
- Retrieval returns provisional temporal references, while visual inspection, speech transcription, and targeted clip analysis convert selected regions into observations for reasoning.Visual inspection can verify or reject retrieved locations, and speech aligns utterances with the global timeline.
- Persistent ledger updates are monotone and policy-free, whereas compaction recomputes a view whose emphasis can change without losing evidence.This separation allows ledger ablations to retain the full structured tool memory while changing only what is surfaced next.