PaperlayerView exact revision on arXiv

Source-linked AI summary

VideoHarness-RSI: Recursive Harness Self-Improvement for Long-Video Understanding with Frozen Vision-Language Models

Guoyang Xu, Hao Chen

arXiv:2608.24302v1cs.AI

TL;DR

Long-video understanding depends on constructing a useful bounded context, but prior systems often change multiple components simultaneously. VIDEOHARNESS-RSI recursively searches executable context constructors around a frozen VLM and fixed interface. The search improves performance from uniform and stronger hand-crafted starting points, transfers without further search, and exposes selection and generalization boundaries.

  • Problem

    Existing long-video systems often combine context mechanisms with changes to representations, retrieval, tools, workflows, or models, making the contribution of executable context construction difficult to isolate.

  • Method

    VIDEOHARNESS-RSI uses an outer-loop proposer to generate, execute, evaluate, and retain executable harness programs while the VLM and answering interface remain fixed.

  • Results

    Recursive harness search improves long-video understanding from uniform and stronger hand-crafted baselines, and the selected harness transfers to additional benchmarks without further search.

  • Takeaways & Limitations

    Executable context construction is a distinct optimization layer and a reproducible setting for studying harness discovery, transfer, and selection around frozen VLMs.

  • Takeaways & Limitations

    The experiments use one frozen VLM, one data seed, and limited search trajectories, so they do not establish cross-model behavior or search variance.

Abstract

from arXiv · show

Long-video understanding depends critically on how a limited model context is constructed from a much longer video. Existing approaches improve this process through compression, retrieval, memory, and agentic evidence acquisition, but these mechanisms are typically introduced as part of a manually designed inference system or optimized together with other components. This makes it difficult to isolate a simpler question: how much can be gained by improving the executable context-construction program alone? We study this question through VIDEOHARNESS-RSI, a controlled baseline for recursively searching executable context constructors around a frozen vision-language model (VLM). An outer-loop proposer uses prior programs, evaluation outcomes, and execution traces to generate candidate harnesses, which are executed and evaluated end to end before successful variants are retained for further search. This makes long-video understanding a controlled instance of automated harness design: the searchable object is executable program structure, while the answering model and interface remain fixed. Starting from uniform sampling, recursive harness search consistently finds room for improvement and surpasses several weaker hand-crafted baselines. Starting instead from a stronger hand-crafted baseline, the same RSI process yields a further improvement. The selected harness also transfers to additional long-video benchmarks without further search. Together, these results establish executable context construction as a distinct optimization layer and provide a reproducible baseline for studying harness discovery and transfer around frozen VLMs.

1 Introduction

Long-video understanding is partly a context-construction problem because relevant evidence is sparse and distant. VIDEOHARNESS-RSI isolates executable harness design around a frozen VLM and evaluates recursively selected programs.

  • Motivation: Long-video VLM performance depends on which bounded context-construction system exposes, not only on the model’s inference ability.Relevant evidence can be sparse, temporally distant, and surrounded by irrelevant observations.
  • Motivation: Existing systems often modify evidence representation, retrieval, tools, reasoning workflows, or models together, obscuring the contribution of executable context construction alone.
  • Approach: VIDEOHARNESS-RSI recursively proposes, executes, evaluates, and retains harness programs while keeping the VLM and final context interface fixed.The harness organizes, retrieves, and packs evidence before the frozen model answers.
  • Study design: The controlled setting separates model, context-construction, and search capabilities and supports studying discovered strategies, transfer, saturation, and development-feedback overfitting.
  • Contributions: The paper formalizes automated harness design and provides a propose–execute–evaluate–retain baseline with an auditable archive of programs, traces, outputs, and scores.

2 Related Work

Prior work uses designed or learned mechanisms such as compression, retrieval, memory, and agentic evidence acquisition. VIDEOHARNESS-RSI instead searches the executable program that constructs the final context while keeping the answering model fixed.

  • 2 Related Work: Long-video systems expose smaller question-relevant contexts through adaptive compression, frame selection, caption or embedding indexes, and retrieval.
  • 2 Related Work: Agentic methods actively acquire evidence, while other systems explore multimodal memory, hierarchical reasoning, planner–inspector control, and pipeline evolution.
  • 2 Related Work: The paper’s narrower focus is outer-loop search over the executable final-context construction program with the answering model fixed.
  • 2 Related Work: Unlike general-purpose search proposals, VIDEOHARNESS-RSI provides a controlled long-video instantiation connected to program-level harness synthesis and LLM-guided program search.

3 VideoHarness-RSI

VIDEOHARNESS-RSI formalizes a harness as executable context construction under fixed model and visual-capacity constraints. Its recursive search proposes candidates from frontier and archive information, evaluates them on development data, and tracks accuracy–cost trade-offs.

  • 3.1 Executable Context Constructors: A harness maps each video–question pair to bounded multimodal context for a frozen VLM under a final-context constraint K.
  • 3.1 Executable Context Constructors: The controlled variable is executable context construction rather than model training or final visual capacity.
  • 3.1 Executable Context Constructors: WRITE builds an addressable representation, READ retrieves question-conditioned evidence, and PACK formats the bounded context supplied to the model.These operations are an analytical decomposition of one constructor, not separately optimized objectives.
  • 3.2 Recursive Harness Search: The outer-loop proposer uses frontier and archive artifacts to generate executable harness candidates, which are smoke-tested and evaluated end to end on development data.
  • 3.2 Recursive Harness Search: Promotion uses strict development-accuracy improvement, while held-out data remain excluded from proposal, selection, promotion, early stopping, and rollback.A separate Pareto reporting set tracks development accuracy and estimated input-side context cost.
  • 3.2 Recursive Harness Search: Candidates may alter representation, retrieval, navigation, selection, packing, prompting, or compositions while the VLM, answer interface, metric, and evaluator data remain fixed within a protocol.

4 Experimental Setup

Experiments evaluate controlled harnesses around a frozen Qwen3-VL-8B-Instruct model with a shared K = 40 visual context and answer protocol. Searches use LVBench development feedback, while selected harnesses are frozen for held-out and cross-benchmark evaluation.

  • Datasets: The local LVBench collection contains 83 videos and 1,232 question–answer pairs after excluding 20 inaccessible source videos and their 317 questions.
  • Evaluation: The harness is searched only on LVBench and then directly reused on full Video-MME and MLVU without proposal, selection, or adaptation feedback.
  • Controlled protocol: The main experiments freeze Qwen3-VL-8B-Instruct, use temperature 0 with thinking disabled, and cap final visual context at K = 40 observations.
  • Search budget: The main search starts from UNIFORM-40, evaluates three candidates per generation for five generations, and promotes only strict point-estimate development-accuracy improvements.
  • Baselines: Reference constructors include UNIFORM-40, CLIP-KNN, CAPTION-KNN, and AKS under the same VLM, K = 40, and answer protocol.
  • Metrics: Accuracy is primary, with paired McNemar tests and mechanism measures including evidence visibility, conditional accuracy, in-window frames, parse failures, and input-side context cost.

5 Results

Search improves both uniform and strong hand-crafted starting points, with selected harnesses transferring across benchmarks. The results also identify complementary navigation–retrieval mechanisms, context-organization effects, accuracy–cost trade-offs, and limits to held-out selection.

  • Main results: Search improves both starting points: EMBEDNAVIGATE-HYBRID is strongest from uniform sampling, while TIMESTAMPED-AKS achieves the best held-out result under frozen Qwen.These gains persist after harnesses are frozen, suggesting transferable context-construction policies rather than development-only fitting.
  • Main results: Direct image retrieval outperforms uniform sampling, while caption retrieval is competitive in development but fails to transfer to held-out questions.The hybrid remains stronger than the image-retrieval reference, combining temporal navigation with visual similarity.
  • Main results: EMBEDNAVIGATE-HYBRID incurs substantial text overhead, whereas TIMESTAMPED-AKS reaches the stronger endpoint with a smaller answer-time input footprint.Input-cost figures estimate visual-plus-text context volume, not end-to-end latency or monetary cost.
  • Cross-benchmark reuse: The selected frozen harness improves over uniform sampling on Video-MME and MLVU without target-specific search or adaptation, supporting direct reuse rather than universal transfer.Per-question predictions were not retained, so paired significance testing is unavailable.
  • Recursive search: The uniform-start search accepts two strict frontier updates—CAPTIONNAVIGATE first and EMBEDNAVIGATE-HYBRID second—then plateaus despite varied later mutations.The hybrid combines a navigation prior, question-conditioned visual similarity, and chronological packing; the Pareto view separates maximum accuracy from lower-cost candidates.
  • Mechanism: CAPTIONNAVIGATE improves accuracy even when annotated evidence is missed, indicating that context organization and local packing density matter beyond exposing labeled intervals.The diagnostics use evidence windows only as post-hoc annotations.
  • Mechanism: The hybrid broadens evidence visibility and reduces misses, but Figure 4 shows that reaching an annotated interval is not sufficient when grouping and presentation remain inadequate.Navigation is especially useful for temporal and reasoning questions, while visual retrieval contributes additional evidence access.
  • Boundary and stress tests: A high-capacity stress protocol reveals a search–evaluation gap: the development-selected PRF candidate underperforms the matched uniform baseline on test.That protocol uses a different split and much larger answer-time context, so it is not directly comparable to the main setting.

6 Discussion

Executable context construction is a distinct optimization layer: search can improve bounded evidence organization around a frozen VLM, but capacity alone does not ensure selection quality and generalization requires separate evaluation.

  • Executable context construction: VIDEOHARNESS-RSI optimizes the executable mapping from a long video and question to bounded model context, rather than a named sampler or retriever.Its Write–Read–Pack view accommodates indexes, captions, temporal navigation, retrieval, and packing within one controlled interface.
  • Capacity and selection: K = 40 gains and high-capacity failures show that larger evidence pools can increase exposure while enlarging the selection problem.A smaller budget can work well when complementary access paths are organized effectively.
  • Evaluation discipline: Search archives, traces, and proposer hypotheses guide mutations, but development promotion is not evidence of generalization.The main search plateaus after two accepted updates, while the high-capacity protocol reveals a development–test selection gap.
  • Scope and limitations: The experiments use one frozen VLM, one data seed, and limited search trajectories, so they do not establish cross-model behavior or search variance.LVBench also uses a locally available subset and a question-level rather than video-disjoint split.

7 Responsible Use

Automated harness search introduces risks including evaluator overfitting, accidental task exploitation, and costly execution paths; the paper mitigates but does not eliminate them.

  • Risks and mitigations: Automated harness search can overfit evaluator feedback, exploit accidental task regularities, or introduce costly, difficult-to-audit execution paths.The paper uses a frozen downstream model and answer interface, bounded context, smoke testing, archived trajectories, and held-out evaluation as mitigations.

8 Conclusion

VIDEOHARNESS-RSI provides a controlled, reproducible setting for recursively searching executable context constructors around frozen long-video VLMs, with a Write–Read–Pack taxonomy organizing the methods.

  • 8 Conclusion: The framework studies how changing evidence organization, retrieval, and packing affects long-video understanding without modifying the underlying model.It also exposes reusable behavior and search–evaluation failure through transfer and stress tests.
  • 8 Conclusion: The appendix supplies the context-constructor taxonomy, complete search archive, mechanism diagnostics, replay details, complementary protocols, adaptation notes, and reproducibility audit.
  • 8 Conclusion: Table 4 distinguishes mechanisms through the Write–Read–Pack taxonomy, including caption-kNN mapping caption hits back to frames for visual answering.Dense caption RAG instead sends retrieved text without images.
  • 8 Conclusion: Main and strong-seed methods finish with K = 40 visual observations, while exploratory methods use different budgets and are not directly ranked against the main hybrid.

B Search Archive and Cost Accounting

The archive records the full K = 40 search and its accounting metadata, while preserving rejected candidates and distinguishing development search from final evaluation.

  • Search archive and cost accounting: 15 candidates from five Claude Code sessions are archived with source, parentage, proposer interactions, traces, smoke tests, scores, duration, costs, and per-question results.The search used 17,514 proposer input tokens, 114,738 output tokens, 35.8 minutes, and approximately $7.79 in proposer API cost.
  • Search archive and cost accounting: Table 6 reports LVBench development accuracy by question tag, and a question may carry multiple tags.
  • Search archive and cost accounting: Table 7 presents a frozen channel ablation of the iteration-2 program rather than independently searched restricted spaces.
  • Search archive and cost accounting: Table 5 lists all 15 candidates in the main K = 40 search, with three programs proposed per generation and bold entries marking development-frontier updates.
  • Search archive and cost accounting: Controlled K = 40 visual contexts render approximately 10,355–10,361 visual tokens per question, while the hybrid adds roughly 6,500 text tokens for caption navigation.These measurements define the archive’s logged input-side cost boundary.
  • Search archive and cost accounting: Rejected candidates remain archived; later motion-aware, verification, diversity, and multi-stage-reasoning attempts did not replace the hybrid frontier.These failures delimit the observed search trajectory without becoming separate paper claims.

C Fine-Grained Mechanism Analysis

The mechanism analysis compares multimodal harness components and literature-inspired adaptations under a frozen-model interface. Results indicate that composition can outperform individual channels, but visible evidence does not guarantee correct interpretation.

  • Channel Composition: The hybrid significantly exceeds both single-channel variants, while navigation-only and image-only do not differ reliably.This supports composition rather than a uniformly superior individual channel.
  • Channel Composition: Navigation locates captions containing “green cup,” enabling hybrid packing to supply in-interval frames and produce the correct answer.The repair depends on subsequent visual reading because captions alone were insufficient.
  • Budget and Search Controls: The visual-budget sweep compares uniform sampling with a non-deployable evidence-window oracle across budgets on 350 development questions.The remaining gap indicates that context selection matters even as K increases.
  • Channel Composition: Image retrieval inserts an in-window frame after navigation fails, but the frozen VLM still repeats the incorrect answer.The failure separates evidence visibility from successful interpretation.
  • Budget and Search Controls: The development-selected ADAPTIVEDENSITY-PRF candidate uses a larger answer context but falls below the matched uniform baseline on test.This protocol is distinct from the development-only visual-budget sweep.
  • Literature-Inspired Harnesses: WorldMM-style, Homer-style, and VideoSEAL-matched rows are controlled harness adaptations, not strict reproductions of their source systems.Comparisons are restricted to fixed evaluations under the common Write–Read–Pack interface and frozen answerer.

G Reproducibility and Data Audit

The reproduction package documents dataset splits, immutable identifiers, and processing rules, while disclosing gaps in historical context logging. Some artifacts were reconstructed or replayed, but caption-navigation timestamps remain unavailable without rerunning the VLM.

  • Data Audit: The package fixes seed-42 indices, records 83 accessible and 20 inaccessible LVBench source IDs, and documents the 350/882 split.The repository’s generic config uses a different 200/1,032 split; the main paper uses config_k40.yaml.
  • Reproducibility: Historical logging gaps prevent reconstruction of caption-navigation frame timestamps without rerunning the VLM.Uniform timestamps were reconstructed deterministically, while hybrid development contexts came from a later replay agreeing on 348/350 questions.
Loading 2608.24302v1…