PaperlayerView exact revision on arXiv

Source-linked AI summary

VideoHarness-SRSI: Structured Recursive Self-Improvement for Video Understanding Harness

Zihao Lin, Haoyu Dong, Haobo Xu, Chi-Chih Chang, Xuying Ning, Yinglong Xia, Shuang Yang, Zhuokai Zhao, Xinying Song, Dongfang Liu, Lifu Huang, Yang Song, Qifan Wang

arXiv:2610.19424v1cs.CVcs.AI

TL;DR

Recursive harness evolution must discover generalizable video-understanding systems despite limited, noisy, repeatedly reused evidence and interacting mechanisms. VideoHarness-SRSI governs what evidence informs search and how mechanisms are explored, improving held-out scores while reducing the validation–test gap across four benchmarks.

  • Problem

    How recursive harness evolution can reliably find generalizable designs from limited, noisy, repeatedly reused evidence in a search space with interacting mechanisms remains unclear.

    Source: §1 · p. 1

  • Method

    VideoHarness-SRSI restricts the proposer to content-free feedback, uses instrumented warm-up experiments, and develops mechanisms in isolated branches before factorial composition and joint refinement.

    Source: §6 · p. 15

  • Results

    6.2% relative improvement over VideoHarness-BRSI in equally weighted mean held-out score and a 37.9% reduction in mean validation–test gap across four benchmarks.

    Source: §5.2 · p. 12

  • Takeaways & Limitations

    Reliable harness evolution depends on how search is organized and informed, not only on validation optimization.

    Source: §1 · p. 2

    Both evolved frameworks still score higher on validation than held-out data on every benchmark, and repeated selection on the same validation set can still favor specialized harnesses.

    Source: §5.2 · p. 12

Abstract

from arXiv · show

LLM system capabilities depend not only on model parameters but also on the executable harnesses that organize information acquisition, tool use, reasoning, and memory. Recursive harness evolution can automate their design, but how it can reliably discover generalizable systems from limited, noisy, and repeatedly reused feedback remains unclear. We study this question in agentic video understanding, where rich multimodal evidence and coupled mechanisms make harness optimization challenging. We first introduce VideoHarness-BRSI, a recursive self-improvement framework that evolves video harnesses around a frozen vision-language model. Analyzing over 4,000 evolved candidates, we find it competitive with hand-engineered systems but identify three interconnected failure modes: validation overfitting, search-budget dilution, and the early-experience trap. We trace all three to a common root: the same imperfect validation evidence drives both harness revisions and the proposer's search strategy. We therefore propose VideoHarness-SRSI, an evidence-governed framework in which every search decision rests on evidence that is transferable, sufficient, and gathered before the search commits. It restricts the proposer to content-free feedback, informs exploration with instrumented warm-up experiments, and develops mechanisms in isolated probe-race branches before factorial composition and joint refinement. Across four video understanding benchmarks, VideoHarness-SRSI improves the equally weighted mean held-out score by 6.2% relative to VideoHarness-BRSI and reduces the mean validation-test gap by 37.9%. Reliable harness evolution thus depends on how search is organized and informed, not only on how well it optimizes validation performance.

1 Introduction

The introduction asks how recursive harness evolution can find generalizable designs from limited, noisy, repeatedly reused evidence in video understanding. It diagnoses shared validation evidence as the source of three failures and proposes an evidence-governed alternative that improves held-out performance across four benchmarks.

1 Introduction

  • The motivating question is how recursive evolution can reliably discover generalizable harnesses from limited, noisy, repeatedly reused evidence in a search space with interacting mechanisms.Source: §1 · p. 1
  • Video harness optimization is challenging because frame selection, retrieval, tools, reasoning, reflection, memory, and collaboration interact, while finite repeated evaluations invite specialization to seen examples.Source: §1 · pp. 1–2
  • VideoHarness-BRSI’s shared validation verdict drives validation overfitting, search-budget dilution, and the early-experience trap because it selects harnesses and steers subsequent search.Early noisy outcomes persist in proposer memory, causing useful mechanisms to be abandoned prematurely and early successes to be over-explored.Source: §1 · p. 2
  • In a retrieval-only Video-MME run, VideoHarness-BRSI matches the hand-designed AKS variant’s 68.4% held-out accuracy with 38.7% fewer frames.Source: §1 · p. 2
  • VideoHarness-SRSI limits proposer feedback to aggregate scores and content-free execution statistics, then uses warm-up experiments, isolated mechanism branches, factorial composition, and joint refinement.Warm-up experiments measure mechanisms while keeping under-tested ones open; each stage supplies evidence for the next.Source: §1 · p. 2
  • VideoHarness-SRSI improves the equally weighted mean held-out score by 6.2% relative and cuts the mean validation–test gap by 37.9% over VideoHarness-BRSI across four benchmarks.The evaluation covers Video-MME, LongVideoBench, OVO-Bench, and MomentSeeker.Source: §1 · p. 2

2 Related Work

Prior work progresses from prompt optimization to executable workflows and harness evolution, while video harnesses use varied mechanisms to organize long-video evidence. This paper treats validation evidence and mechanism interactions as central objects of harness evolution.

2 Related Work

  • Prior automated optimization spans prompts, multi-stage programs, reflective prompt evolution, executable workflows, and agent architectures.Source: §2 · pp. 2–3
  • Recent harness-evolution methods refine proposal strategies but generally take the validation signal used to select candidates as given.The cited directions include modular search, edit regularization, harness updating, continual adaptation, and skill compilation.Source: §2 · p. 3
  • VideoHarness-SRSI makes the validation signal itself an object of study, controlling proposer information, factorizing mechanism search, and using warm-up experiments to address noisy early verdicts.Source: §2 · p. 3
  • Hand-designed video harnesses use compression, keyframe selection, hierarchical representation, and evidence retrieval, while agentic systems coordinate retrieval, tools, and memory.This work treats those mechanisms as a search space whose components’ value depends on context.Source: §2 · p. 3

3 Investigation of Video Understanding Harness Evolution

VideoHarness-BRSI evolves competitive video harnesses around a frozen vision-language model, but its validation verdicts both select candidates and steer subsequent search. The section diagnoses three resulting failure modes and motivates evidence-governed evolution.

3.1 Can VideoHarness-BRSI Evolve Effective Video Harnesses?

  • 68.4% held-out accuracy with 38 frames per question lets the evolved retrieval harness match the strongest AKS variant using 38.7% fewer frames.The evolved harness is 2.2 points above AKS-BLIP with 32 frames and on par with the strongest AKS variants at 68.4% and 68.1%, which use 62 frames.Source: §3.1 · p. 4

3.2 Diagnosing VideoHarness-BRSI in Video Understanding

  • Validation fit, verdict noise, and champion inflation can each overwhelm true gain, producing overfitting, budget dilution, and early-experience failures.The same validation verdict determines both which candidate is kept and where the proposer searches next.Source: §3.2 · p. 5

3.2.1 Validation Overfitting

  • The validation–test gap rises from 2.0 points with Blindfolded feedback to 3.2 with Multimodal feedback, while validation-fit share rises from 31% to 56%.True-gain share falls from 54% to 30%, while noise share stays at 14–16%.Source: §3.2.1 · p. 6, Figure 3 · p. 6

3.2.2 Search-Budget Dilution

  • With six mechanism families rather than one, evaluations per combination fall from 3.5 to 1.7 and verdict noise rises from 1.0 to 1.5 points, pushing detection power below 0.2.At 87 candidates, retrieval-only evolution gains 6.4 validation points versus 5.1 for the full harness, whose deployed champions are no better on held-out data.Source: Figure 3 · p. 6, §3.2.2 · p. 7

3.2.3 The Early-Experience Trap

  • Only 8% of early verdict variation reflects true differences, yet VideoHarness-BRSI continues about 77% of winning candidates and only 8% of the rest.Across 12 retrieval runs, champions that expand the frame budget reach 67.6% held-out accuracy versus 64.9% for those staying near 16 frames; 37% of dropped candidates with held-out evaluations beat the champion.Source: §3.2.3 · p. 8

3.3 Implications for Harness Evolution

  • Changing parent selection or starting from AKS does not resolve the problem because verdicts still compare candidates with a selected champion on the same validation questions.The three failures share the use of imperfect verdicts to determine both which harness is kept and where search proceeds; the proposed principles are content-free feedback, isolated mechanism development, and controlled warm-up comparisons.Source: §3.3 · pp. 8–9

4 VideoHarness-SRSI

VideoHarness-SRSI keeps the propose–evaluate–accumulate loop but changes the evidence guiding its decisions. It isolates proposer feedback and stages warm-up, mechanism-specific search, and factorial composition before joint refinement.

4.1 Information Isolation

  • The proposer receives validation accuracy and content-free per-question traces, but never validation videos, questions, options, or answers.The harness itself still executes on validation instances; a deny list on proposer tools enforces the isolation.Source: §4.1 · p. 10

4.2 Three-Stage Evolution

  • Three-stage evolution preserves the full mechanism space while concentrating evidence on one mechanism at a time before measuring interactions among selected mechanisms.The stages are instrumented warm-up, isolated branch evolution, and factorial composition with joint refinement.Source: §4 · p. 9, §4.2 · p. 10

4.2.1 Stage I: Instrumented Warm-up

  • Warm-up adds a removal and an intensity change to each round’s free proposals, then aggregates per-mechanism evidence before selecting a seed pool and backbone H0.Under-explored mechanisms are flagged rather than discarded; when accuracy plateaus, the six best warm-up harnesses form the seed pool.Source: §4.2.1 · p. 10

4.2.2 Stage II: Isolated Branch Evolution

  • Stage II evolves six mechanism axes plus a free-form axis in isolated branches from H0, with fixed templates keeping edits local and attributable.Every axis receives three probes; the top half race to fifteen candidates, and at most three axes proceed to composition.Source: §4.2.2 · pp. 10–11

4.2.3 Stage III: Factorial Composition and Joint Refinement

  • Stage III distills each selected axis into a portable modification, evaluates every combination to measure transfer and interactions, then jointly refines the selected mechanisms.With three axes, the factorial evaluation costs seven evaluations, and each modification’s effect is averaged over four comparisons; final validation-best selection can reintroduce some fit and champion inflation.Source: §4.2.3 · p. 11

5 Experiments

Across four video-understanding benchmarks, VideoHarness-SRSI improves held-out performance over VideoHarness-BRSI while reducing the validation–test gap. Experiments also test whether gains reflect larger inference budgets and examine the framework’s components.

5.1 Experimental Setup

  • The evaluation uses four video benchmarks with shared validation and held-out splits; only the executor sees video or answers questions, while the proposer writes harness code.The benchmarks cover video question answering and temporal grounding; subtitles and metadata are withheld from harness code.Source: §5.1 · p. 11, Models · p. 11

5.2 Main Results

  • Both evolved frameworks beat every fixed baseline on held-out data across all four benchmarks, while no single fixed design leads everywhere.Adding mechanisms by hand did not reliably help: debate_pair used five times the frames of clip_topk on Video-MME but was less accurate on every benchmark.Source: §5.2 · p. 12
  • VideoHarness-SRSI raises mean held-out score from 54.83 to 58.23 (6.2% relative) and cuts mean validation–test gap from 4.95 to 3.08 points (−37.9%).The score gain is relative to VideoHarness-BRSI; both figures are averaged across benchmarks.Source: §5.2 · p. 12, Table 1 · p. 12
  • Both frameworks still score higher on validation than held-out data on every benchmark, so repeated selection can continue to favor validation-specific harnesses.The reported structured search mitigates overfitting but does not eliminate it.Source: §5.2 · p. 12

5.3 Inference Cost and Budget-Matched Comparison

  • At matched frame budgets, VideoHarness-SRSI remains ahead of VideoHarness-BRSI on every benchmark, with a mean held-out gain of 1.66 points versus 3.40 points for deployed systems.The matched harnesses use fewer frames and at most 0.09 more executor calls per question.Source: §5.3 · p. 13, Table 3 · p. 13

5.4 Ablations

  • Full instrumented warm-up transfers better than free-only warm-up: deployed held-out accuracy is 70.1 versus 67.3 despite nearly identical warm-up validation accuracy.The full warm-up provides more removal contrasts and more complete coverage increments, supplying evidence for later search.Source: §5.4 · p. 14, Figure 6 · p. 14
  • On Video-MME, fixed VideoHarness-SRSI branch axes generalize better than automatic or manual axes, while manual axes score higher on validation but lower on held-out data.Five of nine fixed-axis branches improve on their backbones, compared with none of the automatic branches and all nine manual branches.Source: §5.4 · p. 14, Figure 7 · p. 14
  • With nearly equal validation accuracy, withholding question content raises held-out accuracy by 1.50 points over VideoHarness-BRSI feedback; VideoHarness-SRSI also remains effective with weaker proposers.The additional 0.37 held-out points from the remaining stages are described as directional, given differences in candidate counts and tool access.Source: §5.4 · p. 15, Table 4 · p. 15

6 Conclusion

VideoHarness-BRSI finds competitive harnesses, but its validation gains do not reliably generalize because the same imperfect evidence shapes both harness revisions and search strategy. VideoHarness-SRSI organizes and governs that evidence, improving held-out results across four benchmarks.

6 Conclusion

  • VideoHarness-SRSI raises mean held-out score by 6.2% and cuts the mean validation–test gap by 37.9% relative to VideoHarness-BRSI.About half of the gain remains at a matched frame budget.Source: §6 · p. 15
  • Analysis of over 4,000 VideoHarness-BRSI harnesses identifies validation overfitting, search-budget dilution, and the early-experience trap as consequences of reusing imperfect validation evidence.The same evidence adapts both harnesses and the proposer’s search strategy.Source: §6 · p. 15
  • VideoHarness-SRSI uses content-free proposer feedback, an instrumented warm-up, and isolated mechanism branches before factorial composition.The appendices document failure-mode protocols, results, propositions, proofs, robustness checks, SRSI implementation, and additional analyses.Source: §6 · p. 15, Overview of the Appendix · p. 19

A Experimental Setup

The experiments use a frozen vision-language executor, fixed benchmark splits, and constrained video-only harnesses. VideoHarness-BRSI iteratively proposes and evaluates harnesses on validation data, while held-out results remain outside the proposer’s workspace.

A.1 Models

  • The executor is a frozen Qwen3-VL-8B-Instruct model, while a separate proposer writes harness code but never answers questions or runs benchmarks.Harness roles and agents are prompts to the same executor; retrieval uses CLIP and open-vocabulary detection uses Grounding DINO.Source: §A.1 · p. 20

A.2 Benchmarks and Data Splits

  • Evolution and selection use fixed validation splits, and held-out results never reach the proposer; MomentSeeker splits by source video while the other benchmarks split by question.The experiments cover Video-MME, LongVideoBench, OVO-Bench, and MomentSeeker.Source: §A.2 · p. 20

A.3 Inference Budgets

  • Harnesses operate video-only under per-question frame, pixel, and output limits, and must still return an answer if a tool fails or exhausts its budget.A harness may override default frame count, pixel budget, and output length; temporal grounding defaults to more, lower-resolution frames.Source: §A.3 · p. 20, Table 8 · p. 21

A.4 Seeded Harnesses

  • Full-harness runs start from six simple hand-written seeds, while spaces with one or two mechanism families start from five simpler, space-specific seeds.Seeded mechanisms are designed to keep frame budgets comparable for selected baseline contrasts.Source: §A.4 · p. 20, Table 9 · p. 21

A.5 Evaluation Metrics

  • Multiple-choice tasks use accuracy and MomentSeeker uses mIoU; harness cost is mean frames per question, and evolved results average three seeds.Paired harness comparisons generally use McNemar’s test, while group contrasts use Welch’s t-test or exact permutation tests.Source: §A.5 · p. 21

B.1 Evolution Loop

  • VideoHarness-BRSI candidates are Python answer-method classes that access video, tools, executor, and memory through a constrained context, and each iteration adds candidates to a validation-scored archive.The proposer declares each candidate’s parent, hypothesis, and mechanisms; leak-guard failures and crashes cannot become champion harnesses.Source: §B.1 · pp. 21–22

B.2 Study Configurations

  • The validation-best harness is deployed after the loop, with held-out evaluation kept outside the proposer’s workspace; the main Video-MME comparison uses 450 validation and 2,250 held-out questions.The 50-iteration retrieval run evaluates 150 candidates, balancing evaluation cost with a large held-out split for testing generalization.Source: §B.1 · p. 22, §B.2 · p. 22, §C.1 · p. 23

C.1 Experimental Setting

  • The retrieval-only VideoHarness-BRSI comparison with Adaptive Keyframe Sampling (AKS) isolates the same mechanism on the same frozen executor, so differences compare design rather than mechanism choice.Evolving additional mechanism families would make a win impossible to attribute to evolution alone.Source: §C.1 · p. 23

C.2 Evolution Across Mechanism Families and Datasets

VideoHarness-BRSI improves seeded harnesses across expanded mechanism spaces and exceeds the strongest fixed baseline on each of four benchmarks. These results extend its evolving capability beyond retrieval-only Video-MME experiments.

Other datasets

  • The mechanism-family results show gains over seeded harnesses in each tested space, while the four-benchmark results extend the comparison to general, long-form, and streaming question answering and temporal grounding.The retrieval-only comparison isolates one mechanism for a fair test against AKS, but the capability is not specific to retrieval or Video-MME.Source: §C.2 · p. 23

Other mechanism families

  • Across expanded Video-MME search spaces, evolution improves validation accuracy by 5.1–7.6 points over the best seeded harness within 87 candidates, and champions reach 65.7–66.8% held-out accuracy.These spaces add frame selection, perception tools, and eventually reflection, role-based collaboration, and working memory.Source: §C.2 · p. 23

Other datasets

  • 66.5 versus 63.4% on Video-MME, 66.6 versus 62.1% on LongVideoBench, 67.2 versus 55.4% on OVO-Bench, and 19.0 versus 18.0 mIoU on MomentSeeker are the held-out scores for VideoHarness-BRSI and the strongest fixed baselines.The average advantage is 5.12 points, though validation progress overstates held-out gains.Source: §C.2 · p. 23

D A Model of Validation Verdicts

The verdict model separates general harness accuracy from validation-specific fit and noise, explaining why validation wins can misrepresent true gains. Information-isolation experiments find that richer proposer feedback coincides with larger validation–held-out gaps and more instance-specific fit.

D A Model of Validation Verdicts

  • The model decomposes validation score into population accuracy, validation-specific fit, and zero-mean noise; held-out accuracy estimates population accuracy because it does not influence search.Validation score can be systematically optimistic for harnesses built using validation feedback.Source: D A Model of Validation Verdicts · p. 24
  • Edits that change more answers have noisier paired verdicts, and the model treats |z| < 1.96 only as a nominal marker of an unresolved comparison.The calculation ignores adaptive reuse of validation data and within-video dependence.Source: D A Model of Validation Verdicts · p. 24
  • A validation verdict compares candidate and incumbent scores, but the incumbent’s selection as the current validation maximum makes its noise upward-biased and contributes −ω to later comparisons.The verdict’s estimated gain therefore combines true change with candidate fit, noise, and incumbent selection bias.Source: D A Model of Validation Verdicts · p. 24
  • The model links validation overfitting to content-revealing feedback, search-budget dilution to larger spaces and fewer evaluations per mechanism combination, and early-experience traps to unrevisited low-signal verdicts.It is a lens rather than a claim of exact Gaussianity, and each failure-mode analysis states its assumptions.Source: D A Model of Validation Verdicts · p. 24

E Validation Overfitting

  • The mean validation–held-out gap rises from 2.0 points with Blindfolded feedback to 2.9 with Trace and 3.2 with Multimodal feedback, while hand-written baselines show none.Validation scores are equally reliable across arms, but the transferable fraction β falls from 0.54 to 0.30 as instance-specific fit grows.Source: E Validation Overfitting · p. 25
  • Question-text-visible proposers quote validation questions in code 1.55× as often as chance with Trace and 1.27× with Multimodal feedback, whereas Blindfolded proposers do not.The three feedback arms use the same benchmark, executor, full-harness space, and iteration budget, differing in what the proposer observes.Source: E Validation Overfitting · p. 25, §E.1 · p. 25

E.2 Results

Across E.2–F.1, validation gains do not consistently transfer to held-out performance, and richer feedback and broader search spaces expose distinct sources of overfitting and dilution. The sections quantify transfer, document content reuse, and test robustness and search-space consequences.

E.2 Results

  • Winners’ validation accuracy rose from 70.2% to 71.1% while held-out accuracy fell from 66.5% to 65.7%; pooled gaps grew, but per-ladder orderings were noisy.Each per-ladder cell had three runs.Source: §E.2 · p. 25

E.3 Theory: Selection on a Partially Transferable Signal

  • Theory predicts winner held-out gains are β times validation gains; observed winners imply at least 1.9, 3.3, and 4.1 nats of validation information across feedback arms.The lower bounds correspond to Blindfolded, Trace, and Multimodal feedback, respectively.Source: §E.3 · pp. 26–27

E.4 Measuring Transfer and Instance-Specific Fit

  • Transfer estimates fall from 0.54 [0.33, 0.74] (Blindfolded) to 0.30 [0.15, 0.41] (Multimodal), as the instance-specific variance share rises from 0.31 to 0.56.The pooled ordering Blindfolded > Trace > Multimodal is robust across alternative estimators.Source: §E.4 · pp. 27–28, §E.6 · p. 29

E.5 Direct Evidence of the Content Channel

  • Proposers seeing question text quote validation questions beyond chance, while Blindfolded proposers do not; matched Trace quotations appear mostly in docstrings and comments.Of 168 matched Trace questions, 144 were quoted in docstrings and 25 in comments; a question could appear in both.Source: §E.5 · pp. 28–29

E.6 Robustness

  • The Blindfolded-over-Multimodal transfer ordering survives every leave-one-ladder-out fit and several alternative estimators, while adjacent orderings can reverse.Changing the catastrophic-proposal threshold to 5 or 15 points changed variance shares by at most 0.03.Source: §E.6 · p. 29

F Search-Budget Dilution

  • From one to six mechanism families, evaluations per mechanism combination fall 2.05× and verdict variance rises 2.24×, reducing power for a 1-point gain from 0.45 to 0.14.Deployed champions reach 66.8%, 65.8%, 66.0%, and 65.7% held-out accuracy with one, two, three, and six families, respectively.Source: F Search-Budget Dilution · p. 29

F.1 Protocol

  • The Video-MME search-space ladder ranges from retrieval-only (one family, eight primitives) to the full harness (six families, 33 primitives).The full space adds reflection, role-based collaboration, and working memory; one- and two-family runs were excluded after proposer access to outside files was detected.Source: §F.1 · p. 29

F.2 Results

Across the search-space ladder, validation gains did not translate into better held-out accuracy, while broader spaces spread evidence across more mechanism combinations. The analysis links this dilution to coupled edits and noisy, path-dependent decisions.

F.2 Results

  • No search space improved held-out accuracy; the three-family space gained 7.6 validation points after 87 candidates but only 3.5 held-out points over the seed.Held-out gain per evaluation hour fell from 0.79 points for one family to 0.17 for six; candidate counts alone therefore favor larger spaces.Source: §F.2 · p. 30

F.3 Theory: Evidence per Mechanism Combination

  • Power decreased as the number of mechanism families grew (Spearman ρ = −0.74, p = 0.02 across nine nested runs), and the three full-harness runs had the three lowest values.The ordering held under alternative summaries of verdict noise; the analysis does not explain the two-family space’s slower validation progress, which was within seed variation.Source: §F.3 · p. 32

F.4 Behavioral Footprint of Edits

  • The median edit footprint was 50 answers in the full harness, versus 10, 23, and 14 for one, two, and three families.Even edits touching at most one family flipped 47.6 answers on average in the full harness versus 19.8 with retrieval alone, because downstream stages receive changed inputs.Source: §F.4 · p. 32, Figure 10 · p. 32

G The Early-Experience Trap

  • Six of 12 retrieval-family runs never expanded the frame budget, consistent with a model estimating a 0.54 probability of losing a weak beneficial direction after one trial.Seeds diverged more than independent sampling predicts, and runs staying at 16 frames lost 2.7 held-out points.Source: G The Early-Experience Trap · p. 33

G.1 Corpus and Definitions

  • The 44 analyzed VideoHarness-BRSI runs included 14 Video-MME runs with 2,229 candidates and 30 LongVideoBench runs with 2,577 candidates.Held-out evaluations existed for every candidate in three Video-MME runs and for frontier or best candidates elsewhere.Source: §G.1 · p. 33

G.2 A Verdict Model with Incumbent Optimism

  • On Video-MME, verdict reliability was 0.18 and only 0.08 in iterations 1–10, while incumbent optimism was about 1.0 point.LongVideoBench verdicts were more reliable (0.48) and less biased (bω = 0.44), but the policy was the same.Source: §G.2 · p. 34, Table 17 · p. 34

G.3 Continuation Policy versus Evidential Value

  • The continuation policy was nearly a step at zero verdict, although held-out win rates changed gradually and 37% of tested non-winners beat their incumbent on held-out data.On Video-MME, winners were continued with probability 0.77 and non-winners with probability 0.076; false losers were abandoned as often as true losers.Source: §G.3 · p. 34, Figure 11 · p. 35

G.4 Why Early Errors Persist

A rejected direction can disappear after one noisy verdict because rejections are rarely revisited, while continued directions receive further evidence. The analyses connect this persistence to seed divergence, memory, and the limits of simple search adjustments.

G.4 Why Early Errors Persist

  • With incumbent optimism, a weak beneficial direction is more likely to be lost after one trial; at the frame-budget calibration, the estimated loss probability was 0.54.Reconsidering half of all rejections lowers the modeled loss probability to 0.29, while quadrupling validation questions lowers it to 0.40.Source: §G.4 · pp. 35–36

G.5 Path Dependence across Seeds

  • Candidate shares diverged across seeds beyond independent-sampling expectations, with median ratios of 2.8 on Video-MME and 1.7 on LongVideoBench.The held-out accuracy was 64.9% for runs staying at the baseline budget versus 67.6% otherwise (p = 5 × 10−5).Source: §G.5 · p. 36

H Why Simple Remedies Are Insufficient

  • Changing parent selection reshaped the search but did not improve held-out accuracy; stronger AKS initialization sped early validation progress but did not produce a better endpoint.Neither remedy changes the verdicts on which the loop acts, so neither removes the diagnosed failure modes.Source: H Why Simple Remedies Are Insufficient · p. 36

H.1 Parent Selection

Parent-selection rules changed which lineages the proposer used, but none yielded a consistent held-out advantage. A stronger AKS start accelerated early progress without establishing a reproducible final gain.

H.1 Parent Selection

  • No parent rule yielded a consistent held-out advantage; on Video-MME, deployed accuracies ranged from 65.7% to 67.4%, and rule-to-rule differences were at most 0.5 points.The manipulation changed search shape, but no search-shape statistic predicted held-out gain across the 18 runs.Source: §H.1 · p. 37, Figure 12 · p. 37, Figure 13 · p. 38

H.2 Stronger Initialization

  • The AKS-seeded run reached 70% validation accuracy at iteration 2, but its deployed 68.2% held-out accuracy lay within the range of the six simple-seed runs.The single AKS run does not establish a reproducible final gain; it mainly shortened the search to competitive performance.Source: §H.2 · p. 38

I Implementation and Statistical Details of VideoHarness-SRSI

VideoHarness-SRSI separates evidence gathering, mechanism testing, composition, and refinement, while isolating proposer access to content-free information. Its warm-up ablation reports better held-out outcomes than a free-only warm-up despite similar validation accuracy.

I.1 Information Isolation

  • Proposer calls are confined to their own workspaces, and question text, frames, gold answers, held-out outputs, and evaluation code are kept out of reach.Because records pair predictions with correctness, a proposer could reconstruct answers for some questions; the leak guard prevents harness code from accessing question identity.Source: §I.1 · p. 39

I.2 Stage I: Instrumented Warm-up

  • WARM grows the seed archive with free proposals, family removals, and intensity changes, then selects six strong candidates and a best-performing backbone.The warm-up report includes removal-based family importance, observational dose response, usage share, oracle-union gain, and a baseline ladder.Source: §I.2 · pp. 39–40

I.3 Stage II: Nomination, Probe, and Race

  • Each mechanism axis is probed in an isolated branch from a shared backbone; surviving axes continue to 15 candidates and are admitted using a cluster-bootstrap effect estimate and thresholds.Fewer than three lineage clusters leaves an axis unjudged, and at most three axes enter composition.Source: §I.3 · pp. 40–41

I.4 Stage III: Composition and Joint Refinement

  • COMPOSE writes all nonempty combinations of the selected axis modifications to measure main and interaction effects; MERGE then jointly refines from a ledger holding every cell.Final selection is the validation argmax of the refinement ledger.Source: §I.4 · p. 41

I.5 Budget and Configuration

  • The configured budget is at most 156 candidates per seed, with the three Video-MME runs evaluating 138–141.A tie at the probe cutoff can add one raced axis and 12 candidates beyond the stated bound.Source: §I.5 · p. 41

J Ablation: Instrumented Warm-up

  • Every full-warm-up run beat every free-only run on held-out data, with advantages of 1.85 percentage points for the backbone and 2.8 points for the deployed system.Both arms reached the same validation accuracy.Source: J Ablation: Instrumented Warm-up · p. 42, Table 19 · p. 42
  • The full warm-up produced 2.7 times as many removal contrasts and fewer undefined family estimates than the free-only arm.A within-run replay attributed 28% of the backbone gap to selecting structured candidates; the remainder was consistent with slot results informing free proposals.Source: J Ablation: Instrumented Warm-up · p. 42

J.1 Protocol

  • With three runs per arm, the smallest attainable permutation-test p-value was 0.05, requiring complete separation; results indicate direction rather than precise effect sizes.The arms otherwise shared seeds, splits, caches, isolation, stopping rules, and later stages.Source: §J.1 · p. 42

J.2 Results

The full warm-up arm produced stronger held-out systems than free-only proposals despite indistinguishable validation scores. Frame-adjusted comparisons and selection replay suggest that candidate selection explains only part of the held-out gap.

Warm-up

  • The free-only arm rose faster after six rounds (71.18 versus 69.26), but the full arm caught up and finished at 71.85 versus 71.70.Eight of 11 full-arm champion replacements were structured, compared with five of 13 in the free-only arm; requested removals more often kept frame counts within 10%.Source: §J.2 · p. 42, Figure 15 · p. 43

Evidence

  • The full arm yielded 11.7 versus 4.3 removal contrasts per run and 13.0 versus 9.3 intensity contrasts, covering six versus five change families.Five versus eleven of 21 family–run coverage increments remained undefined; retrieval removals averaged −7.15 pp, but the largest drop also reduced frames from 80 to 16.Source: §J.2 · p. 42

Held-out contrasts

  • 68.95 versus 67.10 for backbones and 70.07 versus 67.29 for deployed systems: every full-arm system beat every free-only system on held-out data, while validation did not separate the arms.The held-out comparisons were significant for both backbones and deployed systems (p = 0.05); validation comparisons had p = 0.65 and 0.25.Source: §J.2 · p. 42, Table 20 · p. 43
  • After adjusting for log2 frames, the full-arm effect remained +2.0 pp for backbones and +2.2 pp for deployed systems.Frames were the main confound; the adjusted effects had t = 2.38 and 2.55, respectively.Source: §J.2 · pp. 42–43

Attribution

  • Selection replay accounted for +0.52 pp, or 28% of the backbone gap; the remaining 1.33 pp advantage was consistent with archive information transfer but could reflect run variation.Only three runs per arm were available, with a standard error of about 0.7 pp.Source: §J.2 · p. 43

K Held-out Questions on Unseen Videos

Re-scoring questions from videos absent from validation preserves the reported information-isolation effect and the evolved-versus-AKS comparison. The interpretation is bounded by how benchmarks split questions and source videos.

Unseen-video evaluation

  • MomentSeeker held out whole videos, but the other three benchmarks split by question, leaving some held-out questions from validation videos.The unseen-video re-score included 1,548 Video-MME and 588 LongVideoBench questions; results otherwise retained the original harnesses and predictions.Source: K Held-out Questions on Unseen Videos · p. 43

Information isolation

Evidence per decision

  • Table 21 reports Video-MME evaluations per configuration, verdict noise, and power to detect a 1-point gain; its VideoHarness-BRSI row uses the first 57 candidates per run.The caption identifies these as evidence-per-decision measures.Source: Table 21 · p. 44

Evolved versus hand-designed harnesses

  • On unseen-video questions, the evolved retrieval harness stayed within one point of both AKS variants without significant differences, using 38 rather than 62 frames per question.This preserves the reported competitiveness of VideoHarness-BRSI with the hand-designed method on videos never seen during evolution.Source: K Held-out Questions on Unseen Videos · p. 44

L Failure-Mode Diagnostics of VideoHarness-SRSI

The diagnostics examine how VideoHarness-SRSI uses warm-up evidence and isolated branch exploration to narrow validation–test gaps, concentrate search, and keep early noisy losses from closing off mechanisms. They also trace how Stage III combines probe branches in a factorial lattice so the gate can assess axis effects and interactions before deciding what to ship.

L Failure-Mode Diagnostics of VideoHarness-SRSI

  • On LongVideoBench, information isolation raises VideoHarness-BRSI held-out accuracy from 65.4 to 66.9% at equal validation accuracy, close to VideoHarness-SRSI’s 67.3%.VideoHarness-SRSI narrows the validation–test gap on three of four benchmarks; the LongVideoBench gain also holds on held-out questions whose videos never appear in validation.Source: L Failure-Mode Diagnostics of… · p. 44
  • On Video-MME, the probe–race raises evaluations per axis decision from 1.55 to 9.80 and detection power for a 1-point gain from 0.13 to 0.50.Stage III also has the lowest verdict noise of any stage (σ = 1.12).Source: L Failure-Mode Diagnostics of… · p. 44
  • No VideoHarness-SRSI run stays at the seeds’ frame budget on any benchmark, versus four of twelve VideoHarness-BRSI runs.The warm-up gathers removal and intensity contrasts across mechanism families regardless of earlier verdicts, so an early noisy loss cannot close off a mechanism for the rest of the run.Source: L Failure-Mode Diagnostics of… · p. 44

M Proposer Prompts of VideoHarness-SRSI

  • The WARM prompt assigns five candidates per iteration to three free explorations, one family removal, and one family intensity sweep.The added removal and intensity candidates manufacture controls and dose ranges for measurement without reducing free exploration; the removal candidate is meant to measure, not necessarily win.Source: M Proposer Prompts of VideoHarness-SRSI · pp. 46–47
  • NOMINATE fills every fixed class-1 axis slot and may add one class-2 charter, while branches proceed through three-candidate blind trials before survivors continue to 15.A prior version let the proposer exclude a family after reading a null union-ceiling increment as no headroom, although the family was unmeasurable without a control group; the prompts treat null as untestable, not refuted. Only the frame-budget axis may change frame count, keeping that confound on its own axis.Source: M Proposer Prompts of VideoHarness-SRSI · pp. 47–48
  • Branch mechanism estimates use mean paired validation deltas against the backbone with lineage-clustered bootstrap SEs, counting same-parent descendants as one draw.COMPOSE receives each raced axis’s estimate and SE along with the gate verdict.Source: M Proposer Prompts of VideoHarness-SRSI · p. 49
  • The gate controls what may ship, not what is measured: COMPOSE builds every combination, including those containing rejected axes.It extracts one axis-specific diff before composition and evaluates the full lattice.Source: M Proposer Prompts of VideoHarness-SRSI · p. 49
  • Each axis uses a byte-identical diff across lattice cells, so factorial interactions compare the same modifications and are not confounded by rewrites.If diffs need rewriting to compose, shrink them until they compose without editing each other.Source: M Proposer Prompts of VideoHarness-SRSI · p. 50
  • Singleton cells are required to compute interactions, and their one-diff effects are distinct from the branch means; a weaker singleton signals a poor diff choice.Each branch’s mean reflects about 15 candidates, whereas its singleton measures the selected diff.Source: M Proposer Prompts of VideoHarness-SRSI · p. 50
  • If a branch contains more bespoke code off-axis than on-axis, its estimate cannot support its named mechanism and its diffs must be treated as unlabelled.This checks whether a branch stayed within its charter.Source: M Proposer Prompts of VideoHarness-SRSI · p. 50

Related papers

Loading 2610.19424v1…