Source-linked AI summary
Routing Before Looking: Query-Adaptive Evidence Acquisition for Long-form Video Understanding
Tianyue Wang, Xuying Wu, Yuxiang Ma, Ruiming Liang, Jiaxuan Kang, Yanchao Hao, Zheng Wei, Leigang Qu, Haiyun Guo, Jinqiao Wang
TL;DR
Long-form video understanding suffers from a mismatch between query demands and evidence acquisition strategies, especially when systems rely on one dominant strategy. Route2Look learns query-adaptive routing through contrastive trajectory analysis and applies it across three evidence tools. Experiments report state-of-the-art accuracy with strong frame efficiency across datasets and query types, while oracle results show further room for adaptive routing.
Problem
Long-form video systems lack evidence acquisition strategies that systematically adapt to diverse query demands, particularly when relevant evidence is not explicitly localized.
Method
Route2Look uses a Route-Look-Memorize loop with Global Browse, Temporal Ground, and Semantic Retrieve, while distilling routing skill through contrastive strategy analysis.
Results
Route2Look improves accuracy and efficiency across datasets and query types, with oracle routing reaching 81.5% and revealing substantial room for adaptive routing.
Takeaways & Limitations
Query-adaptive evidence acquisition is supported as a promising direction for long-form video understanding, including for implicit temporal queries.
Takeaways & Limitations
The routing skill is distilled from a finite evolution set, and verification can still require moderate frame usage, especially for multiple retrieved candidates.
Abstract
from arXiv · showhide
Long-form video understanding remains challenging for video agents due to the mismatch between query demands and evidence acquisition strategies. Although recent planning-before-perception methods outperform query-agnostic pipelines, they often rely on a single dominant strategy, either generation-based strategy or retrieval-based strategy, limiting their ability to handle diverse query demands. We propose Route2Look, a lightweight and model-agnostic framework for query-adaptive evidence acquisition in long-form video understanding. Route2Look operates in a Route-Look-Memorize loop with three tools: Global Browse for holistic context, Temporal Ground for explicit temporal cues, and Semantic Retrieve for semantic search. The core component is a routing policy that dynamically selects evidence acquisition tools based on the query. To build this policy, Route2Look adopts a two-stage design: first distilling the routing skill from differential contrastive analysis between generation-based and retrieval-based trajectories, and then applying the distilled skill with hard routing rules and continue-or-stop criteria during inference. Experiments on challenging long-video benchmarks show that Route2Look achieves state-of-the-art performance while maintaining strong frame efficiency across datasets and query types. Oracle routing analysis further reveals the potential of query-adaptive evidence acquisition for future long-form video understanding.
1 Introduction
Long-form video queries require different evidence acquisition strategies, but existing systems often use fixed or single-paradigm pipelines. Route2Look addresses this mismatch with query-adaptive routing among complementary tools and strategies.
- Motivation: Query-agnostic pipelines waste computation or miss key evidence, while query-conditioned systems commonly retain a single dominant inspection strategy.Generation-based methods support global reasoning but may miss localized events; retrieval-based methods capture local evidence but can struggle with long-range causal chains.
- Motivation: 13.0% absolute gain: oracle selection improves LVBench accuracy from 68.5% for the best single strategy to 81.5%.The oracle result indicates substantial value in selecting generation-based or retrieval-based strategies per query.
- Route2Look: Route2Look uses a Route-Look-Memorize loop that routes queries, inspects selected evidence, and memorizes verified evidence for subsequent routing.Its tools are Global Browse, Temporal Ground, and Semantic Retrieve.
- Motivation: Different query types require distinct evidence acquisition strategies rather than one fixed pipeline.Global understanding and explicit temporal queries can often use global or temporal guidance, while implicit temporal queries require inferring where relevant evidence lies.
- Route2Look: Route2Look distills routing skill through differential contrastive analysis and applies it with hard routing and continue-or-stop criteria.The framework is designed to be lightweight, model-agnostic, and transferable.
- Results: Experiments show improved accuracy and efficiency across long-form video benchmarks, datasets, and query types.Oracle routing results further indicate potential for future query-adaptive research.
2 Related Works
Related work spans query-agnostic pipelines, query-conditioned single-strategy systems, and methods for evolving reusable agent skills. Route2Look extends trajectory-based skill distillation to adaptive evidence acquisition in long-form video.
- Long-form video agents: Long-form video agents commonly use query-agnostic pipelines or query-conditioned systems with one dominant evidence acquisition strategy.Examples include global browsing with local refinement and retrieval with verification.
- Long-form video agents: Query-agnostic methods use fixed inspection pipelines, whereas query-conditioned methods adapt inspection based on the query.The latter still typically rely on a single strategy rather than selecting among complementary strategies.
- Skill self-evolution: Automatic skill self-evolution extracts generalizable skills from trajectories, interactions, or execution feedback for reuse in future tasks.Prior approaches include exploration, user-agent interactions, structured skill memory, and failure-driven refinement.
- Skill self-evolution: Route2Look extends trajectory-based skill distillation by learning routing skills from contrastive generation-based and retrieval-based trajectories.The resulting skill supports query-adaptive evidence acquisition in long-form video understanding.
3 Method
Route2Look combines iterative evidence acquisition with query-adaptive routing, selecting among tools for global, explicit temporal, and semantic inspection. Its routing skill is distilled from competing trajectories and applied during frozen inference with stopping criteria.
- Evidence acquisition loop: Route2Look uses a Route-Look-Memorize loop in which the agent routes, inspects returned frames, updates memory, and decides whether to continue.The backbone agent remains frozen throughout inference.
- Evidence acquisition tools: Global Browse samples frames across the video, Temporal Ground samples within explicit intervals, and Semantic Retrieve ranks multimodal clip relevance for parallel retrieval.These tools target holistic context, explicit temporal grounding, and query-aware semantic search, respectively.
- Semantic retrieval: Semantic Retrieve diversifies the current query, embeds non-overlapping clips, pools relevance curves across reformulations, and extracts candidate segments above a dynamic threshold.The top-K extracted segments are ranked by fused relevance and passed to detailed verification.
- Skill evolution: Stage 1 compares generation-based and retrieval-based trajectories, preferring correct trajectories and then fewer inspected frames, before distilling reusable skill patches.Patches encode triggers, recommended strategies, and lessons, then are hierarchically merged into a compact skill without updating agent parameters.
- Routing skill formalization: The routing skill is a human-readable document specifying when to use generation-based or retrieval-based strategies on disjoint evolution and evaluation sets.The initial skill is a fixed strategy, while the target skill should outperform the better fixed-strategy baseline.
4 Experiments
Route2Look is evaluated across three long-form video benchmarks, query types, backbone modules, and ablations, achieving strong accuracy with efficient frame use. Results indicate that adaptive routing and differential trajectory comparison are central to its performance.
- Experimental Setup: Route2Look is evaluated on LVBench, VideoMME, and LongVideoBench, covering diverse long-form scenarios for accuracy and efficiency.The benchmarks contain 1,549, 900, and 564 questions, respectively.
- Main Results: Route2Look achieves state-of-the-art performance across all three benchmarks while maintaining strong frame efficiency.It reaches 75.4% accuracy with 202.3 frames on LVBench, 76.1% with 126.6 frames on VideoMME, and 77.8% with 168.7 frames on LongVideoBench.
- Query-Type Analysis: Route2Look performs especially strongly on implicit temporal queries, which comprise 82.4% of LVBench.It achieves 92.0% on global understanding queries and improves explicit temporal accuracy from 77.4% to 82.7% over VideoSeek with 66.0 frames on average.
- Ablation Studies: Removing differential contrastive analysis drops accuracy from 70.0% to 61.5%, while Oracle routing reaches 81.5%, showing the value and remaining potential of adaptive routing.Retrieval-only exceeds Generation-only by 5.0% but uses 2.7× more frames, indicating a local-recall versus verification-cost trade-off.
- Compatibility and Cases: Route2Look remains compatible with varied LLM and VLM backbones, with stronger visual inspection and reasoning modules associated with higher accuracy and fewer frames.Accuracy rises from 41.0% with Qwen2.5-VL to 54.5% with Qwen3.5-VL; GPT-5 reaches 70.0% as the varied LLM.
- Compatibility and Cases: Case studies show hard routing handles global and explicit temporal queries with as few as 50 frames, while implicit queries flexibly combine tools.Semantic Retrieve addresses localized evidence, Global Browse persistent attributes, and multiple tools complex movement reasoning.
5 Conclusion
Route2Look is a lightweight, model-agnostic framework that adapts evidence acquisition to query-specific needs. Across three long-video benchmarks, it combines state-of-the-art accuracy with strong frame efficiency, especially for implicit temporal queries.
- Conclusion: Route2Look dynamically selects among three evidence acquisition tools using reusable routing skills distilled from contrastive trajectories.The framework routes before looking and applies the distilled skill without updating the backbone model.
- Conclusion: Experiments across three long-video benchmarks demonstrate state-of-the-art accuracy with strong frame efficiency.The largest gains occur on implicit temporal queries, supporting query-adaptive evidence acquisition.
Limitations
Route2Look’s limitations concern the finite evolution set, remaining verification cost, and its current focus on visual evidence.
- Limitations: The finite evolution set leaves room to narrow the gap between learned and oracle routing through more data or improved skill optimization.This limits how fully the routing skill can approach oracle performance.
- Limitations: Route2Look still requires moderate frame verification, especially when Semantic Retrieve returns multiple candidates.More selective verification policies are proposed to reduce frame usage while maintaining or improving accuracy.
- Limitations: The framework mainly focuses on visual evidence, leaving audio cues as a potential complementary signal.The paper identifies multimodally grounded long-video agents as a direction for future work.
A.1 Hyperparameter Analysis
The analysis examines how evolution-set size, merge batch size, and retrieved-segment count affect Route2Look’s accuracy and efficiency on LVBench.
- Evolution set size: Increasing the evolution set from 200 to 500 samples improves accuracy from 67.0% to 70.0%.The result indicates that more diverse training queries help distill a more robust routing skill.
- Merge batch size: A moderate merge batch size B = 32 achieves the best accuracy, while extremely small or large batches degrade performance.With 500 samples, increasing batch size from 32 to 200 drops accuracy from 70.0% to 64.0%.
- Number of retrieved segments: Reducing retrieved segments from K = 5 to K = 3 lowers frame usage from 249.5 to 171.7 but reduces accuracy from 67.0% to 61.5%.Fewer candidates improve efficiency but increase the risk of missing relevant evidence.
- Number of retrieved segments: K = 5 provides a favorable balance between evidence coverage and frame cost.The analysis identifies five retrieved segments as the preferred setting under the reported trade-off.
A.2 Efficiency Analysis
On LVBench, Route2Look improves accuracy while using substantially fewer frames and tokens than the GPT-5 base model.
- Overall comparison: The efficiency analysis reports fewer frames and fewer tokens together with substantially higher accuracy for Route2Look.The comparison is summarized in Table 5 for LVBench.
A.3 Algorithm Demonstration
Route2Look repeatedly selects a tool, inspects sampled frames, stores verified evidence, and either answers the query or continues until the evidence is sufficient or the iteration budget is reached.
- Route: The Route-Look-Memorize loop updates the query state and selects a toolkit operation using the routing policy.Selection uses the current context memory and query state.
- Look: The selected tool produces sampled frames that the VLM inspects to extract verified evidence.A temporary working memory is initialized for each tool call.
- Memorize: Only verified evidence and routing information are written back to context memory.This controls what information persists across iterations.
- Stopping criterion: The agent generates the final answer when accumulated evidence is sufficient; otherwise, it continues until the maximum iteration budget H.The workflow therefore includes an explicit continue-or-stop decision.
A.4 Subset Analysis in the Ablation Study
The ablation subset preserves LVBench’s distributional and accuracy trends, while analyses and examples illustrate Route2Look’s adaptive routing behavior and remaining failure cases.
- Subset construction: The 200-sample LVBench subset is constructed from the joint domain–task distribution to reduce evaluation cost while preserving benchmark proportions.Sampling prioritizes domain-level proportions and maintains task-level proportions within each domain.
- Subset validity: The resulting subset closely matches the full LVBench domain composition and preserves similar accuracy trends.This supports its use as a representative, efficient testbed for controlled ablation analysis.
- Oracle routing: Oracle routing answers 81.5% of samples, with generation-only succeeding on 13.0% and retrieval-only on 18.0% where the other strategy fails.Both strategies correctly answer 50.5% of samples.
- Distilled routing skill: The distilled routing skill favors generation-based strategies for global context, stable attributes, contiguous workflows, and clear temporal boundaries.The learned preferences are presented as interpretable routing guidance.
- Query-adaptive routing: Global Browse handles holistic and long-range queries, while Temporal Ground verifies explicit timestamps, ranges, or temporal descriptions.Examples cover global understanding and explicit temporal query types.
- Query-adaptive routing: Semantic Retrieve targets localized visual cues, and ambiguous implicit-temporal queries can trigger switching between retrieval and global browsing.Representative cases include reactions, object states, target-event existence, and insufficient initial routes.
- Failure cases: Route2Look may fail on subtle spatial details, OCR-like numerical reading, and reasoning beyond directly observable visual evidence.These limitations are shown in representative failure cases.