Source-linked AI summary
VidHarness: Evolving Agent Harnesses for Cost-Efficient Long Video Understanding
Susan Liang, Jianmin Wu, Daxiang Dong
TL;DR
Long-video understanding with VLMs is costly because processing every frame is inefficient, while existing agent harnesses require extensive manual engineering. VidHarness evolves harnesses using MCTS and uncertainty-aware multi-fidelity validation, then routes questions to budget-specialized harnesses. It reports state-of-the-art results on three benchmarks and stronger out-of-domain performance than uniform sampling with fewer frames.
Problem
Processing every frame in long videos is costly, while existing agent harnesses rely on extensive expert trial and error.
Source: §1 · p. 1Method
VidHarness iteratively evolves harnesses using MCTS with uncertainty-aware multi-fidelity validation and routes questions to harnesses specialized for their frame-budget tiers.
Source: §1 · p. 2, §3.5 · p. 6Results
Takeaways & Limitations
The reported results show that automatically evolved, budget-specialized harnesses can support cost-efficient long-video understanding across in-domain and out-of-domain benchmarks.
Source: §1 · p. 2, §3.5 · p. 6, §5 · p. 9
Abstract
from arXiv · showhide
Vision-language models (VLMs) can answer questions about hour-long videos, but processing every frame is prohibitively expensive, even though the evidence for a question usually spans only a few seconds. Video agents, i.e., harness programs wrapped around a frozen VLM, address this by observing the video selectively, yet existing harnesses are hand-crafted by experts through slow build-and-test cycles. We propose VidHarness, a framework that automates harness design for cost-efficient long video understanding, in which a harness proposer iteratively evolves harnesses based on execution feedback from an evolution environment. To escape the local optima of greedy refinement, we organize the evolution as Monte Carlo tree search (MCTS), and to reduce the evaluation cost, we integrate uncertainty-aware multi-fidelity validation, which screens new harnesses on a few questions and promotes only the promising ones. Since the best harness varies with the frame budget, we further introduce a mixture-of-harness that routes each question to a harness specialized for its budget. VidHarness sets new state-of-the-art results on LongVideoBench, Video-MME, and Video-Holmes, outperforms the strongest hand-crafted video agent by up to $11.2$ points, and generalizes to the knowledge-intensive benchmarks Video-MMMU and MMVU with fewer than half of the frames of uniform sampling.
1 INTRODUCTION
Long-video QA wastes computation on irrelevant frames, while existing agent harnesses require extensive expert engineering. VidHarness automates harness evolution and combines tree search, cost-aware validation, and budget-specific routing to improve performance across long-video benchmarks.
1 INTRODUCTION
- Long-video QA is costly because a one-hour video sampled at one frame per second yields about 1.84 million visual tokens, although answer-relevant evidence usually spans only seconds.The cost grows linearly with video length while answer-relevant evidence does not.Source: §1 · p. 1
- VidHarness automates the expert-intensive cycle of proposing, implementing, testing, diagnosing, and revising agent harnesses for cost-efficient long-video QA.The framework iteratively evolves harnesses around a frozen VLM.Source: §1 · pp. 1–2
- MCTS searches multiple harness lineages to balance exploration and exploitation, while uncertainty-aware multi-fidelity validation screens candidates on a few questions and promotes promising ones.The method also penalizes accuracies measured on few questions during node selection.Source: §1 · p. 2
- The mixture-of-harness routes each question to a harness specialized for its requested frame-budget tier because the best harness varies with budget.The strongest harness under a small budget may differ from the strongest under a large budget.Source: §1 · p. 2
- VidHarness sets state-of-the-art results on LongVideoBench (72.0%), Video-MME (73.1% without and 83.2% with subtitles), and Video-Holmes (60.8%), beating VideoSeek by up to 11.2 points.On Video-MMMU and MMVU, it also surpasses uniform frame sampling with fewer than half the frames.Source: §1 · p. 2
2 RELATED WORK
Prior work scales long-video understanding through more frames, keyframe selection, or hand-crafted agent harnesses, and applies harness optimization and tree search across other domains. VidHarness differs by automatically evolving video harnesses with multi-fidelity MCTS and assembling budget-specialized harnesses.
Long Video Understanding
- Long-video methods scale VLM input, select query-relevant keyframes, or use hand-crafted evidence-gathering harnesses; VidHarness evolves harnesses automatically.The cited approaches include (Team et al., 2024; Zhang et al., 2024; Shu et al., 2025), (Tang et al., 2025; Yu et al., 2025), and (Wang et al., 2024; 2025b; Zhang et al., 2026b; Li et al., 2026; Lin et al., 2026), respectively.Source: §2 · p. 2
Agents and Harnesses
- Harness optimization has been studied for text, code, games, and video, but prior methods largely follow one improvement trajectory and deploy one agent.VidHarness instead explores a harness tree and assembles a mixture-of-harness.Source: §2 · p. 2
Evolution Algorithms
- LLM-driven evolution and tree search have discovered programs, prompts, and agentic systems, but most methods evaluate every candidate fully; VidHarness adds multi-fidelity evaluation to MCTS.The cited examples include (Romera-Paredes et al., 2024; Novikov et al., 2025; Hu et al., 2025; Zhang et al., 2025b; Jiang et al., 2025).Source: §2 · p. 2
- VidHarness’s evolution environment returns execution feedback to a proposer, while its multi-fidelity MCTS validates new harnesses on a subset and revalidates promising ones.The figure contrasts this with single-lineage refinement and full validation of every harness in vanilla refinement and standard MCTS.Source: Figure 1 · p. 3
3 VIDHARNESS
VidHarness frames long-video QA as harness design under a frame budget, then evolves candidate programs using execution feedback. Its method combines tree-based search, staged validation, and budget-tier routing.
3 VIDHARNESS
- VidHarness evolves agent harnesses through an environment that supports candidate design, execution, and diagnosis, then selects budget-specialized harnesses for deployment.The method overview describes multi-fidelity MCTS as its low-evaluation-cost search mechanism.Source: §3 · p. 3
3.1 PROBLEM FORMULATION
- Long-video QA under a frame budget seeks a harness that uses a frozen VLM to predict the answer while decoding no more than the requested number of frames.A harness is an executable program orchestrating multi-turn interaction among the video, transcription, and model.Source: §3.1 · p. 3
3.2 EVOLUTION ENVIRONMENT
- The evolution environment provides guidance and interaction APIs, enforces frame budgets, and returns accuracy, frame-use, error, and trace feedback to guide proposer revisions.Trace inspectors examine observed frames to diagnose failures such as missed coverage, insufficient resolution, or flawed reasoning.Source: §3.2 · p. 4
3.3 VALIDATION SET CONSTRUCTION
- The 1,000-pair validation set spans multiple video sources, durations, transcription availability, and question formats, and does not overlap with the test benchmarks.It includes videos from under five minutes to over one hour, with both transcribed and non-transcribed samples and multiple-choice and open-ended questions.Source: §3.3 · p. 4
3.4 MULTI-FIDELITY MCTS
- MCTS organizes revised harnesses in a tree and iterates selection, expansion, validation, and backpropagation, balancing promising branches with exploration of less-visited branches.Expansion encourages exploration at the virtual root and exploitation when revising a selected harness; backpropagation tracks the best harness in each subtree.Source: §3.4 · p. 5
- Uncertainty-aware multi-fidelity validation screens each new harness on a small question subset, then promotes promising candidates through larger nested rungs while penalizing uncertain accuracy estimates.Promotion re-evaluates only unanswered questions; the penalty is a lower confidence bound that discounts results based on few questions.Source: §3.4 · p. 6
3.5 MIXTURE-OF-HARNESS
- Mixture-of-harness selects harnesses suited to budget tiers from their behavior and validation feedback, then routes each question by requested budget without a learned router or runtime cost.Selected harnesses may come from different tree lineages, and one harness may serve multiple tiers.Source: §3.5 · pp. 6–7
4 EXPERIMENTS
Experiments compare VidHarness with multimodal and agentic baselines across long-video benchmarks, ablate its evolution strategy and budget routing, test transfer to knowledge-intensive benchmarks, and track accuracy over evolution iterations.
4.1 EXPERIMENTAL DETAILS
- The evaluation covers six long-video benchmarks, with comparisons against uniformly sampled multimodal models and video agents including VideoSeek.The benchmark suite includes LVBench, Video-MME, LongVideoBench, Video-Holmes, Video-MMMU, and MMVU; VideoSeek is reproduced with Qwen-3.6-27B for backbone-controlled comparison.Source: §4.1 · p. 7
4.2 COMPARISON WITH STATE-OF-THE-ART METHODS
- At xhigh, VidHarness reaches 72.0% on LongVideoBench, 73.1% on Video-MME without subtitles, 83.2% with subtitles, and 60.8% on Video-Holmes, beating VideoSeek by 11.2 points.The Video-Holmes comparison uses the same backbone; VidHarness also surpasses the uniform-sampling baseline by 1.3 points.Source: §4.2 · p. 8, Table 2 · p. 8
- On LVBench, VidHarness trails DVD, which uses 8,074 frames versus VidHarness’s 126, while VidHarness exceeds same-backbone VideoSeek in several settings.With subtitles, VidHarness reaches 63.5% using 43 frames versus VideoSeek’s 63.3% using 45.9; without subtitles, VidHarness reaches 56.0% using 70 frames versus VideoSeek’s 54.7% using 111.5.Source: §4.2 · p. 8
4.3 ABLATION STUDIES
- MCTS + multi-fidelity validation achieves 59.4% with 56.8 frames, while reducing validation samples 5.1× and evolution time 2.1× versus MCTS.Vanilla iterative refinement reaches 57.3% after 75.5 hours, below uniform sampling at 58.9%; MCTS reaches 58.4% after 71.9 hours.Source: §4.3 · pp. 8–9, Table 3 · p. 9
- MoH beats the top-1 harness at low (55.5% vs 55.2%), medium (58.7% vs 57.5%), and xhigh (60.8% vs 59.3%) tiers, tying at high (59.4%).At xhigh, the MoH member uses 126.0 frames on average versus 82.0 for the top-1 harness.Source: §4.3 · p. 9, Table 3 · p. 9
4.4 GENERALIZATION TO OUT-OF-DOMAIN BENCHMARKS
- On Video-MMMU and MMVU, the low tier matches or exceeds uniform sampling by 0.5 and 2.4 points while using less than a third and less than half its frames, respectively.The low tier uses 20 frames on Video-MMMU and 22 on MMVU, and exceeds same-backbone VideoSeek by 6.7 and 7.8 points.Source: §4.4 · p. 9, Table 4 · p. 9
4.5 EVOLUTION DYNAMICS
- Best validation accuracy rises from 74.0% at iteration one to 76.0% at iteration six and 77.0% at iteration seven.From iteration three onward, each new harness reaches at least 70%; Figure 3 marks harnesses by average frame use and tracks the best accuracy.Source: §4.5 · p. 9, Figure 3 · p. 9
5 CONCLUSION
VidHarness combines MCTS with uncertainty-aware multi-fidelity validation and budget-specialized harness routing; the paper reports state-of-the-art results on three long-video benchmarks. Claude Code served as the harness proposer and a core component of the experimental method.
5 CONCLUSION
- VidHarness uses MCTS with uncertainty-aware multi-fidelity validation to escape local optima at low evaluation cost, then routes questions to budget-specialized harnesses.The conclusion reports state-of-the-art results on three long-video benchmarks.Source: §5 · p. 9
AI USE STATEMENT
- Claude Code served as the harness proposer that proposed and edited harnesses; authors evaluated those proposals through the experimental pipeline and against held-out data.The authors also used generative AI to create or edit figures and to edit prose, then reviewed the resulting work.Source: AI USE STATEMENT · p. 10
A EXTENDED RELATED WORK
Prior long-video methods scale context, select keyframes, strengthen reasoning, or use fixed expert-designed agents. VidHarness instead automatically evolves harnesses and uses multi-fidelity MCTS to explore harnesses specialized for different frame budgets.
Long Video Understanding
- VidHarness automatically evolves harnesses and routes among budget-specialized harnesses, unlike prior video agents built as fixed expert designs for particular models.The mixture-of-harness selects among harnesses built around one frozen VLM according to the requested frame budget.Source: A EXTENDED RELATED WORK · p. 15
- Prior long-video methods extend context, compress tokens, select query-relevant keyframes, strengthen reasoning through reinforcement fine-tuning, or use agentic evidence-gathering loops.VideoSeek uses a multi-granular seeking toolkit, while LensWalk plans observation scope and sampling density.Source: A EXTENDED RELATED WORK · p. 15
Evolution Algorithms
- VidHarness combines tree search with multi-fidelity evaluation because long-video evaluation is expensive, and its mixture selects among harnesses for one frozen VLM by frame budget.Related approaches often follow one improvement trajectory and deploy one agent; VidHarness explores a harness tree and assembles budget-specialized harnesses.Source: A EXTENDED RELATED WORK · p. 15
B ALGORITHM OF UNCERTAINTY-AWARE MULTI-FIDELITY MCTS
VidHarness iteratively expands and evaluates a harness tree using MCTS and uncertainty-aware multi-fidelity validation. Candidates advance through validation rungs, and the tree is ultimately ranked by penalized accuracy.
B ALGORITHM OF UNCERTAINTY-AWARE MULTI-FIDELITY MCTS
- Each iteration selects an expandable tree node by UCT, expands it with proposed harnesses, validates new candidates on the first rung, and updates ancestor statistics.The proposer adds each new harness as a child of the selected node; backpropagation updates visit counts and exploitation terms.Source: B ALGORITHM OF UNCERTAINTY-AWARE… · pp. 15–16
- The promotion stage chooses the most accurate harness not yet fully validated, evaluates it on the next rung, and updates its penalized accuracy and ancestor exploitation terms.Promoted harnesses are tested on unanswered questions from the next rung.Source: B ALGORITHM OF UNCERTAINTY-AWARE… · pp. 15–16
- After all iterations, harnesses in the tree are ranked by penalized accuracy.The algorithm initializes the tree with a virtual root and returns the ranked harnesses.Source: B ALGORITHM OF UNCERTAINTY-AWARE… · pp. 15–16
C STANDARD ERROR OF LOW-FIDELITY ACCURACY
The low-fidelity accuracy estimate is treated as a sample mean over independent validation-question outcomes. Its worst-case standard-error bound depends only on the number of questions and is used to penalize measured accuracy.
C STANDARD ERROR OF LOW-FIDELITY ACCURACY
- Assuming independent validation questions, each correctness indicator is Bernoulli with mean p and variance p(1 − p), and low-fidelity accuracy is their sample mean.The sample mean has expectation p.Source: C STANDARD ERROR OF LOW-FIDELITY… · p. 16
- The standard-error bound is 5.0, 3.2, 2.2, and 1.6 percentage points at 100, 250, 500, and 1,000 questions, respectively.The bound holds for any harness regardless of its accuracy and depends only on the number of questions.Source: C STANDARD ERROR OF LOW-FIDELITY… · p. 16
- Penalized accuracy subtracts λp times the worst-case standard-error bound from measured accuracy, yielding a lower confidence bound.It lies λp worst-case standard errors below measured accuracy.Source: C STANDARD ERROR OF LOW-FIDELITY… · p. 16
D IMPLEMENTATION DETAILS
The implementation evolves 40 candidate harnesses using a 128-frame budget and evaluates them across four validation rungs. Its mixture combines four harnesses centered on distinct frame-budget tiers.
D IMPLEMENTATION DETAILS
- The evolution uses 20 iterations with two child harnesses per iteration, producing 40 candidates under a 128-frame budget.Claude Code with Claude Opus 5.0 proposes harnesses, while Claude Sonnet runs trace inspections.Source: D IMPLEMENTATION DETAILS · p. 16
- Multi-fidelity validation uses rungs of 100, 250, 500, and 1,000 questions, penalty coefficient λp = 1.5, and one promotion per iteration.The MCTS exploration weight is λx = 0.5.Source: D IMPLEMENTATION DETAILS · pp. 16–17
- The mixture-of-harness ensembles four harnesses centered around 16, 32, 64, and 128 frames for low, medium, high, and xhigh tiers.The VLM backbone uses temperature 1.0, top-p = 0.95, and top-k = 20.Source: D IMPLEMENTATION DETAILS · p. 17