VideoSeek: Long-Horizon Video Agent with Tool-Guided Seeking

1. Introduction

• Most video agents densely parse videos at 0.2–2 FPS, although over 80% of LVBench questions can be answered from less than 5% of the video.The paper argues that exhaustive multi-granular annotation is therefore inefficient and unnecessary.Source: §1 · p. 2
• Human viewers use temporal and causal structure to form a storyline, inspect promising intervals, and zoom in only when details matter.This motivates seeking informative evidence rather than greedily parsing the full video.Source: §1 · p. 2
• VideoSeek uses a think–act–observe loop and overview, skim, and focus tools to seek answer-critical evidence at multiple temporal granularities.The agent reasons over accumulated observations to adapt subsequent tool calls.Source: §1 · p. 2
• VideoSeek achieves strong accuracy across four challenging benchmarks while using far fewer frames than peer video agents.Its analysis highlights video logic flow, reasoning capability, and toolkit design.Source: §1 · p. 2

Video-Language Models

• Early video-LMM work extended image-language architectures to video [10] [64] [70], while later work developed synthetic instruction data and caption or QA generation [18] [24].Long-form understanding and complex reasoning have drawn attention, but these methods largely retain a single-pass paradigm.Source: §2 · p. 2

Video Agentic Models

• Figure 2’s toolkit progresses from overview’s global storyline to skimmed candidate intervals and focus’s dense inspection of short clips.Red, blue, and green boxes identify frames viewed by overview, skim, and focus; gray boxes are unseen frames.Source: Figure 2 · p. 3
• Video agents evolved from handcrafted workflows and segment search toward adaptive tool use, but many still rely on prebuilt video databases [32] [33] [41] [51] [71] or greedy full-video scans [57] [58].The cited discussion says expensive preprocessing limits practicality and scaling to long videos.Source: §2 · p. 3

3.1. Problem Formulation

• VideoSeek models a task as a trajectory of think–act–observe steps followed by answer generation conditioned on the video, query, and accumulated trajectory.The formulation factorizes trajectory seeking from generating an answer based on that trajectory.Source: §3.1 · p. 3

3.2. Toolkit Design

• The toolkit narrows from a global overview to candidate-segment skimming and then high-rate focus inspection when precise details are needed.Overview forms a storyline; skim samples candidate intervals; focus inspects short clips at 1 FPS for details such as text, faces, counts, or brief actions.Source: §3.2 · p. 4

3.3. Agentic Workflow of VideoSeek

• At each turn, the agent reasons over its trajectory, chooses an answer or a video-tool action, and appends returned observations for the next turn.It repeats while uncertainty remains, stopping when evidence is sufficient or the turn limit is reached.Source: §3.3 · pp. 4–5
• The authors describe the resulting trajectory as compact and informative, with query-aware observations supporting accurate answers while processing fewer frames than dense-parsing baselines.They relate this to estimating the evidence-seeking trajectory and generating the answer from accumulated evidence.Source: §3.3 · p. 5

4.1. Experimental Setting

• Evaluation spans four long-form and reasoning benchmarks, using GPT-5 by default, specialized overview/skim/focus sampling, and at most 20 reasoning turns.The toolkit parameters are tuned with α = 4 for LVBench and α = 2 for the other benchmarks.Source: §4.1 · p. 5

4.2. Main Results

• Across LVBench, Video-MME, and LongVideoBench, VideoSeek improves over GPT-5 while processing 76–96% fewer frames.On LVBench with subtitles, it reaches 76.7% with 27.2 frames; on Video-MME long without subtitles, it reaches 70.1% with 60.9 frames.Source: §4.2 · pp. 5–6, Table 1 · p. 6
• On Video-Holmes, VideoSeek achieves 47.3% accuracy with 42.7 frames, surpassing GPT-5’s 44.1% with 384 frames.It attains the highest accuracy on SR, TA, MHR, and CTI, and ranks second on TCI.Source: §4.2 · p. 6, Table 2 · p. 7

4.3. Empirical Analysis

• With subtitles, VideoSeek gains performance and sharply reduces frames on LVBench and Video-MME, consistent with subtitles exposing scene logic for localization.The analysis reports that subtitles provide a textual storyline that helps localize answer-critical segments and avoid redundant scanning.Source: §4.3 · p. 7
• With the full toolkit, LVBench (w/o subtitles) reaches 68.4%; replacing GPT-5 with o4-mini drops accuracy to 58.5% despite processing more frames and taking more think-act-observe turns.The authors suggest reduced reasoning capability damages judgment, so additional computation does not necessarily improve performance.Source: §4.3 · p. 7
• Leaving out <overview>, <skim>, and <focus> drops accuracy by 13.3, 6.0, and 4.7 points, respectively.The analysis identifies <overview> as most crucial because it captures video logic flows through a global timeline pass.Source: §4.3 · p. 7, Table 4 · p. 7
• The case study shows adaptive sequencing—overview, focus, then skim if evidence is insufficient—rather than a fixed coarse-to-fine rule.The agent progressively refines the search space and selects tools based on its current state.Source: §4.3 · pp. 7–8, Figure 3 · p. 8

• VideoSeek uses a lightweight multi-granular toolkit and a think–act–observe loop to navigate toward answer-critical evidence by reasoning over accumulated observations.Source: §5 · p. 8
• Across long-form video understanding and complex reasoning benchmarks, VideoSeek achieves strong accuracy while using far fewer frames than prior agents and standalone LMMs.Further analysis highlights video logic flow, strong reasoning, and complementary toolkit design.Source: §5 · p. 8
• VideoSeek may be less suitable for anomaly detection and other tasks involving unexpected or highly localized moments that logic-driven navigation cannot anticipate easily.Source: §5 · p. 8

• The appendix provides additional empirical analysis and implementation details for VideoSeek.Source: A. Appendix · p. 12
• Its listed topics include token consumption and runtime, tool frame budget α, and intermediate reasoning.Source: A. Appendix · p. 12
• The appendix also lists prompts and an additional case study.Source: A. Appendix · p. 12

• 92.3 vs 384 frames and 49K vs 83K tokens without subtitles, and 27.2 vs 384 frames and 57K vs 97K tokens with subtitles (VideoSeek vs GPT-5).Source: §A.1 · p. 12
• Without subtitles, VideoSeek takes 135.9s versus GPT-5’s 66.1s; with subtitles, it takes 89.7s versus 71.3s.Source: §A.1 · p. 12
• Runtime is reported for completeness but is not treated as a reliable efficiency metric because network latency, hardware, scheduling, and tokenization or batching can vary.Source: §A.1 · p. 12

• The base configuration uses α to set overview sampling to 16α frames, skim sampling to 4α frames on segments of at least 4α seconds, and focus analysis to clips at most 4α seconds.LVBench videos average 67 minutes, versus 44 minutes for VideoMME-long, 27 minutes for LongVideoBench-long, and 3 minutes for VideoHolmes.Source: §A.2 · p. 12
• α = 2 provides a strong accuracy–efficiency tradeoff on VideoMME, LongVideoBench, and Video-Holmes; LVBench shows a similar dominant gain from α = 1 to 4.The selected settings are α = 4 for LVBench and α = 2 for the other three benchmarks.Source: §A.2 · p. 12
• Table 5 is labeled as an analysis of the effect of intermediate reasoning.Source: Table 5 · p. 12

A.4. Effect of Intermediate Reasoning

• GPT-5* is evaluated on the same frames selected by VideoSeek, isolating a comparison using matched visual observations.This setup tests performance with VideoSeek’s selected evidence rather than a separate frame sample.Source: §A.3 · p. 12
• GPT-5* outperforms vanilla GPT-5 while using substantially fewer frames.The passage attributes the more informative visual observations to VideoSeek’s evidence selection.Source: §A.3 · p. 12
• A clear gap remains between GPT-5* and VideoSeek, suggesting gains are not solely due to additional visual evidence but also to intermediate reasoning.The passage presents this as an interpretation of the remaining performance gap.Source: §A.3 · p. 12

A.4. Prompts

• The system instruction has six parts: Role, Environment, State, Workflow, Toolkit, and Operational Rules.Together, these specify agent behavior, available inputs, trajectory memory, iterative operation, tools, and practical guidance.Source: §A.4 · p. 12
• The workflow checks whether the trajectory supports an answer, gathers missing evidence with tools, and incorporates observations until sufficient support or the step limit.State is represented as a list of prior thought–action–observation tuples.Source: §A.4 · p. 12
• The toolkit defines overview, skim, focus, and answer tools for coarse summaries, locating relevant moments, checking details, and responding.The roles are complementary, moving from whole-video context to localized inspection and a final answer.Source: §A.4 · p. 12
• The initial query includes video duration, optional subtitles, and the question; round instructions invoke predefined policies, while tool calls specify spans, timestamps, optional subtitles, and tool-specific instructions.Figures 4 and 5 show the initial-query and per-round prompts, respectively.Source: §A.4 · p. 13, Figure 4 · p. 13, Figure 5 · p. 13

A.5. Additional Case Study

• The case studies highlight reasoning before observing, following video logic flow to seek answer-critical evidence, and reasoning over accumulated observations.The examples are presented as representative agent behavior.Source: §A.5 · p. 13
• The overview prompt uniformly samples a video segment and requests timestamp-matched frame descriptions, optional subtitles, and JSON output.It asks for concise, detailed descriptions and requires timestamps to match the supplied timestamp list.Source: §A.5 · p. 13
• The skim prompt asks for timestamped descriptions of viewed frames and highlights relevant timestamps with explanations when query-related content appears.The requested descriptions are about 25 words per frame, with roughly 50 additional words for each relevant timestamp.Source: §A.5 · p. 13
• The focus-tool prompt supplies the video clip with its start and end times and frames.The prompt template identifies the clip span and its video frames as inputs.Source: §A.5 · p. 13
• One case-study query asks what is written on a high-rise after the camera turns from the B1 sign; the supplied answer is option D.The answer is “Happy Chinese New Year to all the people in the city” in Chinese.Source: §A.5 · p. 17