Video-RSI: Recursive Self-Improvement of Video Understanding Agents via Harness Evolution

• Execution traces contain only current-harness observations, so a missing event may reflect a sampling gap, perceptual error, or failure to use available evidence.Source: §1 · pp. 1–2
• Across four benchmarks, the evolved harness improves accuracy while processing fewer frames than its initial version; MLVU investigation beats trace-only revision at similar frame use.Source: §1 · p. 2
• Video-RSI uses the same frozen language model to investigate failures, revisit training videos, and revise the executable harness offline.Source: §1 · p. 2
• The improvement loop links failure diagnosis to code revision and selects changes by accuracy and visual cost for subsequent executions.Source: §1 · p. 2

2.1 AGENTIC VIDEO UNDERSTANDING

• Training-based observation-policy methods such as FrameThinker and EVA contrast with Video-RSI’s fixed-weight harness evolution (He et al., 2026; Zhang et al., 2026d).Source: §2.1 · p. 3

2.2 AUTOMATED AGENT DESIGN AND SELF-IMPROVEMENT

• Code-level agent search and Meta-Harness optimize programs around language models; Video-RSI adds original-video failure investigation to guide the Editor’s revisions (Hu et al., 2025; Zhang et al., 2025a; Lee et al., 2026; Zhang et al., 2026a).Source: §2.2 · p. 3

2.3 SELF-EVOLVING VIDEO AGENTS

• Compared with training-based video self-evolution and VideoHarness-RSI, Video-RSI investigates original videos and selects revisions by accuracy and visual cost (Jung et al., 2026; Zhang et al., 2026c; Huang et al., 2026; Xu & Chen, 2026).Source: §2.3 · p. 3

3 VIDEO-RSI

• Video-RSI evolves a video question-answering agent’s executable harness while keeping its underlying models fixed, with accepted revisions starting the next attempt.The Editor revisits original videos to investigate failures, diagnoses them, and submits candidate harness revisions for selection.Source: §3 · p. 4, §3.2 · p. 4

3.1 PROBLEM FORMULATION AND EVOLVABLE HARNESS

• The harness connects evidence acquisition to decisions through prompt construction, tools, observation processing, memory, and execution control.Investigation and revision use training data alongside a fixed video-disjoint selection set whose examples and per-item outcomes remain private to the evaluator.Source: §3.1 · p. 4

3.2 FRAMEWORK OVERVIEW

• The same frozen language model serves as Solver and Editor: it inspects traces, investigates training videos, proposes revisions, and uses independent selection-set evaluation to choose whether they persist.Investigation is offline; subsequent questions are answered using their own inputs.Source: §3.2 · p. 4

3.3 ACTIVE VIDEO INVESTIGATION

• Active investigation revisits training videos to distinguish whether failures reflect missed events, misperception, or failure to use existing evidence.Queries vary temporal region, sampling density, modality, or perceptual question; perception requests omit the gold answer.Source: §3.3 · p. 5
• New observations can support, contradict, or leave explanations unresolved, guiding further queries and a diagnosis-linked reusable revision hypothesis.The Editor compares cases to identify applicable conditions and behaviors to preserve, while ensuring revisions rely on cues available on new questions.Source: §3.3 · p. 5

3.4 COST-AWARE HARNESS EVOLUTION

• Diagnosis-guided revisions specify when changes apply and which successful behaviors to preserve, and may coordinate edits across tools and consuming code.The Editor can debug candidates through training-side trials before selection.Source: §3.4 · p. 5
• Matched-condition selection retains candidates that improve accuracy within bounded visual-cost growth or reduce visual cost with bounded accuracy loss.Visual cost is measured as mean frames processed by successful visual-model calls over complete question-answering executions.Source: §3.4 · pp. 5–6

4.1 EXPERIMENTAL SETUP

• The evaluation evolves the harness on 216 LVBench questions and selects candidates with a video-disjoint 72-question development set, then reports accuracy and mean frames per question.The Solver and Editor use frozen model weights; frame usage counts frames processed during successful visual-model calls and excludes offline Editor observations.Source: §4.1 · p. 6

4.2 MAIN RESULTS

• 4.8 percentage points higher accuracy on MLVU than VideoSeek, with 27.2% fewer frames; Video-RSI also exceeds both reproduced baselines in accuracy across all four benchmarks.Compared with VideoHarness-RSI, it uses fewer frames on every benchmark; against VideoSeek, frame usage is lower except on Video-MME.Source: Table 1 · p. 7, §4.2 · p. 7

4.3 ABLATION STUDY

• Active investigation yields higher final accuracy than trajectory-only revision on every benchmark, including an 8.2-percentage-point gain on MLVU with similar inference-time frame usage.The investigation-enabled run retains five of twenty revisions, compared with three for trajectory-only revision.Source: §4.3 · pp. 7–8, Table 2 · p. 8
• Video-RSI achieves higher accuracy and fewer frames than the accuracy-only gate on every benchmark, while the gate increases frame usage on three benchmarks.On Video-MME, Video-RSI uses roughly half as many frames as the accuracy-only variant.Source: §4.3 · p. 8, Table 2 · p. 8

4.4 EVOLUTION DYNAMICS

• On the development set, the first accepted revision produces most of the net frame reduction, while later revisions continue improving accuracy with smaller, nonmonotonic cost changes.Figure 3 tracks accuracy and mean frames per question for the retained harness across 20 attempts on the development set and MLVU.Source: §4.4 · p. 8, Figure 3 · p. 9
• On MLVU, the first accepted revision reduces frames but lowers accuracy; accuracy recovers later, and the final two accepted revisions account for most of its net gain.From HS2 onward, MLVU accuracy rises alongside frame usage, yet the final harness remains more accurate and uses fewer frames than HS0.Source: §4.4 · p. 8

4.5 ANALYSIS OF EVOLVED HARNESSES

• Retained changes add timestamp-linked structured observations, relation graphs that surface evidence gaps, and selective observation routines; the final harness uses 10.8–44.1% fewer frames than HS0 across four benchmarks.The reported savings measure visual frames, not total model computation.Source: §4.5 · p. 9

5 CONCLUSION

• Video-RSI recursively improves a video agent’s harness with fixed model weights, using the same language model as Solver and Editor.The model revisits training videos, tests failure explanations, and turns diagnoses into reusable code changes retained by accuracy and visual-usage criteria.Source: §5 · p. 9
• Across four benchmarks, the evolved harness improves accuracy while using fewer frames than its initial version.The ablations support contributions from active investigation and cost-aware selection.Source: §5 · p. 9
• Generative AI tools assisted with code editing, figure and table preparation, and manuscript clarity review; the authors reviewed the assisted content and take responsibility for the final work.Source: AI USE STATEMENT · p. 10