PaperlayerView exact revision on arXiv

Source-linked AI summary

Online Video Agent Harness for Long Video Understanding

Sen Yang, Boqiang Duan, Jing Yang, Weihao Bo, Jie Liu, Boyuan Tong, Ze Feng, Wenkang Zhang, Jingdong Wang, Hua Wu

arXiv:2609.12818v1cs.CVcs.AI

TL;DR

Long-video systems face sparse query-relevant evidence alongside the cost of packing many frames into model context, while existing agents may depend on offline preprocessing or ad hoc tools. VideoXAgent uses online, query-conditioned tool orchestration guided by a mined capability taxonomy; it reports competitive benchmark performance with a smaller context footprint.

  • Problem

    Long-video methods face sparse query-relevant evidence, costly frame-packed contexts, and agent designs that may rely on query-agnostic preprocessing or ad hoc tool sets.

    Source: §1 · p. 2

  • Method

    VideoXAgent starts from the raw video and query, decomposes the task, invokes taxonomy-guided expert tools on demand, and aggregates multimodal evidence.

    Source: §1 · p. 3, §3.3.2 · p. 8

  • Results

    85.1% on Video-MME-Long, 76.5% on LVBench, and 65.7% on MINERVA with about 50k agent context, competitive with frontier LMMs.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The harness remains effective with visually weak or text-only orchestrators, and its performance is reported alongside a smaller visual footprint than prior video agents.

    Source: §1 · p. 3

    The mined MINERVA annotations provide only a high-level capability taxonomy, not ground-truth instance data.

    Source: §3.2 · p. 6

Abstract

from arXiv · show

Long video understanding often behaves like a visual needle-in-a-haystack problem: query-relevant evidence is sparsely distributed across long temporal spans, while packing dense frames into a single VLM context incurs \textit{context rot} and high cost. Existing video agents often rely on query-agnostic offline preprocessing or ad hoc tool sets, which can miss query-specific details and waste computation. In this work, we present VideoXAgent, a purely online video-agent harness for long video understanding that starts from the given video file and user query, plans and decomposes the task, invokes specialized expert tools on demand, and aggregates multimodal evidence to produce a final answer while resolving conflicts among observations. To support this on-demand invocation, we design a suite of heterogeneous expert tools guided by a data-driven taxonomy of atomic capabilities, spanning scripts, VLMs, and domain models (e.g., detection, OCR, ASR, face recognition). The harness further enforces objective evidence prompting and budget-aware control to curb hallucination and non-termination. Across Video-MME-Long, LongVideoBench-Long, LVBench, and MINERVA, VideoXAgent is competitive with frontier LMMs and video agents under a smaller context footprint---about 50k tokens of agent context per sample, even on hour-long videos. In particular, on complex video-reasoning benchmarks such as MINERVA, it matches this level while using only about 15\% of the context of a 1,024-frame dense-packing baseline. Notably, the harness remains effective with a visually weak or even text-only orchestrator, suggesting that strong long-video understanding can emerge from progressive agentic evidence seeking rather than from packing the full video into a single context. Project page: https://go-agent-x.github.io/video_agent_harness/

1 Introduction

The introduction frames long-video understanding as finding sparse, query-relevant evidence without overwhelming model context. VideoXAgent addresses this with online, query-conditioned tool use and reports competitive benchmark accuracy with a compact context footprint.

1 Introduction

  • Long videos can contain thousands of frames even at 1 FPS, creating million-token contexts that exceed many models’ effective context and risk degradation.Uniform downsampling may miss relevant frames while including unrelated segments; recent targeted methods still mostly pack frames into one context.Source: §1 · p. 2
  • Existing video agents may preprocess query-agnostic fixed segments, overlooking query-specific evidence and adding computational overhead.The introduction also identifies ad hoc tool selection as a source of redundant, missing, or ambiguous capabilities.Source: §1 · p. 2
  • VideoXAgent starts from the raw video and query, then decomposes the task, invokes tools on demand, and aggregates multimodal evidence without building a query-agnostic index.Its orchestrated models progressively disclose evidence rather than loading the entire video context at once.Source: §1 · pp. 2–3
  • A taxonomy mined from MINERVA expert traces guides 60+ tools spanning scripts, VLMs, and specialist models for capabilities such as OCR, ASR, detection, and face recognition.The taxonomy is intended to cover fine-grained and long-tail cases.Source: §1 · p. 3
  • 85.1% on Video-MME-Long, 76.5% on LVBench, and 65.7% on MINERVA with about 50k agent context, competitive with frontier LMMs.The reported context budget applies even to hour-long videos.Source: §1 · p. 3

2 Related Work

Prior work makes long videos tractable through long-context frame processing, intermediate representations, or agentic tool use. VideoXAgent is positioned as a query-conditioned online alternative that derives its tool space from expert reasoning traces.

2.1 Large Multimodal Models for Long Video Understanding

  • Long-context LMMs process sampled or low-resolution frames directly, but performance remains coupled to frame-sampling strategy and model context capabilities as video duration grows.Examples include Qwen-VL [2] [1], Kimi-K2.5/K2.6 [17], Seed models [7] [5] [6], and Gemini [11].Source: §2.1 · pp. 3–4

2.2 Long Video Representation and Retrieval

  • Long-video representation methods use retrieval over graphs, documents, keyframe trees, clip summaries, databases, or audiovisual hierarchies, shifting substantial perception and organization into preprocessing.VideoXAgent instead acquires evidence from the raw video on demand for the current query, without a pre-built database or hierarchical memory.Source: §2.2 · p. 4

2.3 Tool-Augmented Video Agents

  • Video agents combine planning and visual tools or internalized temporal grounding, while many bind actions to system-level tool choices or indexed representations.VideoXAgent distinguishes itself by deriving an atomic-capability taxonomy from expert traces and mapping it to heterogeneous tools for online evidence acquisition.Source: §2.3 · p. 4

3 Method

The method formulates long-video question answering as budgeted, query-time interaction with a raw video file and a suite of expert tools. It mines human reasoning traces for a capability taxonomy that guides tool design and orchestration.

3.1.1 Problem formulation

  • The agent receives a video-file path, query, available tools, and execution budget, then maintains a trajectory of calls, observations, evidence, and remaining budget.At each step it either invokes a tool with query-specific arguments or produces the final answer.Source: §3.1.1 · pp. 4–5
  • The harness combines a data-driven taxonomy, 60+ expert tools and compositional workflows, and budget-controlled orchestration that adapts to the query and video duration.Its iterative loop plans, decomposes, invokes tools, reflects, and refines until sufficient evidence is collected or another round exceeds the remaining budget.Source: §3.1.1 · p. 5

3.2 Capability Mining and Expert Tool Design

  • Because coarse benchmark labels offer limited guidance for atomic capabilities, the authors mine capabilities from MINERVA expert reasoning traces rather than defining them subjectively.The traces are (Q, R, A, O) tuples: question, reasoning, answer, and candidate options when available.Source: §3.2 · pp. 5–6
  • Capability extraction prompts an LLM to identify trace-grounded reasoning steps in structured JSON, then normalizes and merges low-frequency capabilities using embedding similarity.The mined taxonomy is used as a design blueprint; only high-level capabilities are extracted, not ground-truth instance data.Source: §3.2 · p. 6, §2 · p. 6

2. Merging & Filtering:

The method consolidates mined capabilities into a taxonomy, then uses it to structure tool workflows and an iterative agent loop. Tool selection, evidence checks, and temporal coverage adapt to the query and available budget.

2. Merging & Filtering:

  • Capability names are embedded and clustered with K-means, with K selected by silhouette score and cluster boundaries refined manually.The resulting taxonomy guides focused tools for perceptual or retrieval tasks and multi-tool workflows for comparison, cross-modal fusion, and logical inference.Source: §2 · pp. 6–7
  • The consolidated taxonomy contains 22 atomic capabilities across five categories: Visual Perception, Temporal Relation, Spatial Relation, Audio-Visual, and High-Level Reasoning.The authors argue that capabilities should be discovered from actual reasoning processes rather than inferred from coarse labels or subjective assumptions.Source: §2 · pp. 6–7
  • Each tool declares its media scope, inputs, and outputs; observations align to video, time, and space so workflows can pass localized evidence to subsequent tools.Explicit empty results and errors let the agent revise a call, switch tools, or answer from available evidence.Source: §2 · p. 7

3.3.1 ReAct-style Agent Loop

  • The ReAct-style loop plans from the query, gathers preliminary cues, decomposes subproblems, invokes scoped tools, reflects on observations, and continues or answers within budget.The plan can be revised from tool outputs, and unresolved evidence or a budget limit determines whether another round is useful.Source: §3.3.1 · p. 7

3.3.2 Tool/Workflow Selection and Evidence Aggregation

  • Tool choice follows the subproblem’s modality and media scope; specialized tools handle OCR, speech, detection, tracking, and temporal retrieval, while dependent workflows narrow later inputs.Complementary observations are judged by query relevance and specificity; conflicts prompt checks across times, regions, entities, or interpretations and may trigger targeted verification.Source: §3.3.2 · pp. 7–8

3.3.3 Adaptive Strategy for Video Length

  • For compactly inspectable content, the agent favors direct VLM analysis; for long videos, it maps sparse temporal cues, localizes intervals, and densifies sampling only there.It can expand adjacent intervals or switch modality when needed, avoids repeatedly resampling the full video, and stops when subproblems resolve or the budget requires an answer.Source: §3.3.3 · pp. 8–9

4 Experiments

VideoXAgent is evaluated across long-video benchmarks and ablations of its tools, models, evidence rules, and budget controls. Results show competitive accuracy with selective evidence acquisition, while the case study illustrates resolving conflicting observations through targeted verification.

4.1 Experimental Setup

  • The evaluation covers long-video benchmarks, including full official splits for LongVideoBench-Long, Video-MME-Long, and LVBench, plus MINERVA’s public release and a stratified Video-MME-v2 subset.The Video-MME-v2 comparison uses the same 118-video / 472-question subset for the agent and LMM baselines; LVBench uses visual frames only.Source: §4.1 · pp. 9–10

4.3 Comparison with State-of-the-Art Large Multimodal Models and Video Agents

  • 85.1% VideoMME-Long accuracy with 52.0 frames / ∼51k context is close to the strongest direct LMM rows (85.5%–86.2%).On MINERVA, limited-vision and text-only orchestrators reach 65.7% and 65.4%, respectively, versus 62.4–66.5% for frontier LMMs.Source: Table 1 · p. 10, §4.3 · p. 11, Table 2 · p. 11

4.2 Implementation

  • VideoXAgent uses a ReAct-style orchestrator and 60+ taxonomy-organized tools, extracting only query-relevant media and returning structured observations for evidence aggregation.The default trajectory limit is L=125 node executions, with escalating warnings and forced answering when another tool round cannot fit.Source: §4.2 · pp. 10–11

4.4 Ablation study

  • Tool, model, and objective-rule ablations show complementary tool contributions, accuracy–context trade-offs, measurable value from evidence rules, and capability-dependent Video-MME-v2 performance.Removing sequence-level VLM or audio/subtitle tools lowered LongVideoBench-Long accuracy from 75.33% to 68.67%; disabling objective rules lowered it from 75.33% to 72.60% while context rose from 31.6k to 32.2k tokens. The Gemini agent with Qwen-3.6 tools reached 80.67% accuracy at 50.1k context, while Video-MME-v2 leadership varied by metric and capability group.Source: §4.4 · pp. 12–14, Figure 6 · p. 13

4.4.1 Effect of Budget Control

  • 43.8% accuracy is reached at L=125 on the targeted stress set; progressive warnings eliminate unanswered runs and raise end-to-end accuracy by 8.8 points.At fixed L=125, warnings reduce unanswered runs from 14.7% (32/217) to 0/217 and mean token usage from 47.5k to 42.0k.Source: Table 4 · p. 14, §4.4.1 · p. 14, Table 5 · p. 16

4.5 Cost and Evidence Efficiency

  • Selective evidence acquisition keeps VideoXAgent competitive without uniformly dense video ingestion, while ablations show that lower context alone does not guarantee better accuracy.Removing visual expert tools reduces context from 32.2k to 25.4k tokens but lowers accuracy from 75.33% to 72.00%; progressive warnings also reduce usage from 47.5k to 42.0k tokens while eliminating unanswered runs.Source: §4.5 · p. 16

4.6 Case Study

  • A real trace shows the agent narrowing a video interval, gathering subtitle and visual evidence, then resolving competing hypotheses through targeted frame verification.The subtitle track suggested wind, a fall, and a mechanical issue; frame-level inspection resolved the timing conflict before the final answer.Source: §4.6 · p. 16

5 Limitations and Conclusion

VideoXAgent uses an online, budget-bounded harness guided by a taxonomy mined from MINERVA reasoning traces, and is competitive with frontier models on the reported benchmarks. The authors note that its efficiency measure excludes total token cost and that the taxonomy may reflect MINERVA-specific distributions or omit capabilities salient elsewhere.

5 Limitations and Conclusion

  • VideoXAgent is competitive with frontier LMMs on Video-MME-Long, LongVideoBench-Long, and LVBench, and brings weak or text-only orchestrators to the same range as direct frontier LMMs on MINERVA.The harness starts from the raw video and query, invokes expert tools on demand, and aggregates multimodal evidence under a bounded budget.Source: §5 · p. 17
  • The efficiency analysis measures retrieved frames and maximum agent-model context, not total token cost; the MINERVA-mined taxonomy may be dataset-specific or omit capabilities salient in other domains.The authors suggest investigating more compact taxonomies that reduce redundancy while preserving cross-domain coverage or adapting to a specific domain.Source: §5 · p. 17

Appendix

  • Appendix sections cover the capability taxonomy, tool mapping and specifications, tool-usage analysis, capability-mining procedure, and agent system prompt.The listed tool-mapping tables describe 22 mined atomic capabilities grouped into five macro categories and map them to concrete tools and observed multi-step chains.Source: Appendix · p. 21, Table 6 · p. 22, Table 7 · p. 23

A.3 Tool Specifications

The tool-specification tables describe backend technologies and functions for 70 tools.

A.3 Tool Specifications

  • The tool-specification tables list backend technologies and functions for 70 tools.The supplied captions identify Table 9 as continued tool specifications.Source: Table 9 · pp. 24–26

B.1 Usage Distribution by Capability Category

The usage analysis counts whether a trajectory exercises a capability, rather than how many times it calls a tool, to characterize capability coverage. The resulting distribution is head-heavy, but frequency does not measure a capability’s utility.

B.1 Usage Distribution by Capability Category

  • The analysis credits a capability once per trajectory that uses any of its tools, yielding 60,474 trajectory-capability pairs across 33,544 of 34,364 partially collected trajectories.Generic operators without capability semantics are excluded; the counting unit avoids letting repeated calls inflate a capability’s share.Source: §B.1 · p. 27, Figure 8 · p. 27
  • Visual Content Understanding accounts for 49.5% of capability pairs and appears in 87.1% of trajectories; Speech Transcription accounts for 25.3% and appears in 44.6%.Temporal localization accounts for 11.5% and OCR for 6.6%; the remaining ten capabilities together contribute under 8%.Source: §B.1 · p. 27
  • A small usage share does not imply low utility: removing visual expert tools, which account for roughly a tenth of pairs, costs 3.33 accuracy points.Some capabilities have no mass because they are implemented as ReAct chains, their tools are credited under another capability, or the statistics are partially collected.Source: §B.1 · p. 27, Figure 8 · p. 27

B.2 Emergent Tool-Chaining Behaviour

The transition analysis reveals recurring tool workflows in the high-traffic call traces, including visual inspection and subtitle-grounding cycles. Its scope is typical traffic, not tool importance.

B.2 Emergent Tool-Chaining Behaviour

  • The common visual workflow funnels thumbnail grids, extracted frames, and concatenated images into inspect_image, which often self-loops or hands off to query_segment_detail.The subtitle-grounding workflow links video-duration checks and scene or subtitle checks to cycles between get_video_subtitle and locate_event_by_text.Source: §B.2 · p. 28, Figure 9 · p. 28
  • The transition matrix covers the twenty most-used tools, which account for 99.3% of calls, so it describes common paths rather than ranking tool importance.A specialist tool may be useful for a particular question without forming a high-probability edge.Source: §B.2 · p. 28, Figure 9 · p. 28

B.3 Tool Usage and Answer Correctness

Tool-use accuracy patterns are observational and reflect which questions summon which tools. Subtitle and temporal-grounding tools are positively associated with accuracy, while visual-tool comparisons are confounded by question difficulty and, for very common tools, atypical control groups.

Tool Usage and Answer Correctness

  • Visual-tool use is negatively associated with accuracy largely because those tools are called on the hardest questions, so the correlations are not causal effects.The analysis notes that visual-tool baseline accuracy can be inflated by comparison with short sessions that stop after only a few calls.Source: §B.3 · pp. 28–30
  • Subtitle and temporal-grounding tools show positive associations: locate_event_by_text +6.6pp, get_video_subtitle +4.5pp, and check_subtitle_exists +3.6pp.The passage attributes these correlations partly to text-answerable questions being easier and text-tool-driven.Source: §B.3 · pp. 28–29
  • Tool pairs using subtitle and temporal-grounding tools have the largest positive lift, while pairs with scene_change_detect or ocr_video_text_timeline mark harder slices.Pair-level Δ is measured against the overall baseline, not a leave-one-pair-out control.Source: §B.3 · p. 29, Table 10 · p. 29
  • For get_video_duration, used in 97% of sessions, the 3% non-use group is atypical, making its Δ statistically fragile.The paper cautions against over-interpreting comparisons for tools used in almost every session.Source: §B.3 · p. 30
  • get_video_subtitle is associated with +4.8pp (76.7% vs. 71.8%) and locate_event_by_text with +7.3pp (78.2% vs. 70.9%) among 4,539 sessions that probed for subtitles.The text-tool lift remains when comparing transcript readers with non-readers within this matched stratum.Source: §B.3 · p. 30
  • Overall, tool-accuracy correlations largely reflect which questions summon which tools, not the tools’ marginal value.Visual tools are associated with harder items; subtitle tools are associated both with text-answerable questions and with a clean evidence channel.Source: §B.3 · p. 30

B.4 Accuracy by Task Type

Accuracy varies substantially by task type, with stronger categories associated with subtitle and temporal tools and weaker categories with dense frame inspection and more calls.

Accuracy by Task Type

  • 91% Spatial Reasoning and 88% Information Synopsis are among the strongest categories, while State Changes is weakest at 42%.The strongest categories use mainly subtitle/temporal tools and few calls; weaker categories rely on dense frame inspection and issue the most calls.Source: §B.4 · p. 30, Figure 11 · p. 31

B.5 Provenance of the diagnostic set

This material describes a targeted diagnostic set, a trace-grounded procedure for discovering atomic capabilities, and the deployed prompt’s investigation rules. Together, the passages specify how capabilities are extracted, consolidated, and used to guide video-agent work.

B.5 Provenance of the diagnostic set

  • The 217 loop-prone questions form a targeted stress test, selected from cases where Gemini-3.1-Pro exhausted its step budget without predicting; ablation runs use Kimi-K2.6.The cases span five benchmarks, and benchmark-wide results are reported separately in Table 1.Source: §B.5 · p. 31

C Mining Procedure for Atomic Capabilities

  • Each instance is represented as (Q, R, A, O), and required capabilities are mapped to executable tools under budget B for the agent to answer from video V and query Q.Q is the query, R the expert reasoning trace, A the ground-truth answer, and O the candidate options.Source: C Mining Procedure for Atomic… · p. 31

C.1 Step-by-Step Discovery Pipeline

  • A MINERVA-style record illustrates how a broad “Sports Analysis” label decomposes into scoreboard reading, player identification, action recognition, and linking actions to score change.The example is schematic; quoted reasoning-trace fragments are representative rather than a reproduced benchmark item.Source: §C.1 · p. 32, Table 11 · p. 32

C.1.1 LLM prompting

  • The prompting phase reverse-engineers expert reasoning traces into atomic abilities, using empirical macro-categories and requiring direct trace quotations as evidence for each ability.The structured input includes the query, reasoning trace, answer, and options; extracted abilities are returned in a strict JSON schema.Source: §C.1.1 · p. 32

C.1.2 Merging & Filtering

  • Filtering normalizes naming variants, then maps 458 terms occurring fewer than five times to their most similar high-frequency terms using embeddings.Macro-category weights from merged terms are retained.Source: §C.1.2 · p. 34

C.1.3 Embedding & Clustering

  • The embedding sweep selects 86 clusters at silhouette score 0.138, which are manually reviewed and distilled into 22 atomic capabilities.Cluster sizes from 10 to 100 were tested in steps of five.Source: §C.1.3 · p. 34

C.1.4 Final Taxonomy Construction

  • Canonical capability names prioritize global frequency, then source diversity, then the shortest name; examples merge 13 OCR variants and 12 attribute-related terms.The resulting 22-item taxonomy is reported in Table 6.Source: §C.1.4 · p. 34

D Agent System Prompt

  • The deployed prompt directs planning before tool use, cross-modal verification when results are uncertain, and stopping when evidence suffices rather than fabricating details.It also instructs the agent to state limitations when the video lacks enough information.Source: D Agent System Prompt · p. 35
Loading 2609.12818v1…