PaperlayerView exact revision on arXiv

Source-linked AI summary

Harness Compilation: Which Decisions Should a Small Vision-Language Model Keep?

Minhao Fan, Yinyi Liu, Jiayu Zhao, Zihan Teng, Song Chen, Weichen Liu

arXiv:2610.11231v1cs.AIcs.CV

TL;DR

Small VLMs can use external evidence but may struggle to obtain it, leaving open which runtime decisions a harness should handle. Harness Compilation uses student execution traces and validation to adapt reusable content and control around a frozen model. Across seven settings, it raises macro-average task score from 48.3 to 62.9.

  • Problem

    Small VLMs may answer from supplied evidence yet struggle to request, select, or generate what they need, raising the question of which decisions a harness should leave to them.

    Source: §1 · p. 1

  • Method

    Harness Compilation uses a large teacher offline to revise reusable content and control from student execution traces, then uses disjoint validation to select the deployed harness.

    Source: §1 · pp. 1, 3

  • Results

    Across seven settings, HC raises macro-average task score from 48.3 to 62.9, improving over bare students in all 21 main builds.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Interventions indicate that evidence access and open query generation can be costly, while bounded choices and evidence reading can remain useful student work.

    Source: §7 · p. 11

    Measurements cover task scores and generation costs, not end-to-end industrial deployment or hardware-specific latency, and offline teacher access to practice traces is still required.

    Source: §6 · p. 11

Abstract

from arXiv · show

Small vision-language models may be able to read external evidence yet struggle to obtain it. We introduce Harness Compilation (HC), an offline procedure that adapts the division of work between a frozen small VLM and its external harness. A large teacher uses student execution traces to revise reusable content and control, while a separate validation set selects the deployed harness. Deployment requires neither weight updates nor teacher calls. Across seven visual question-answering settings with students of at most 9B parameters, HC improves scores over bare students by 9.9-23.9 points, averaged over three independent builds per setting. Interventions on five runtime decision types (invocation, selection, argument generation, evidence integration and abstention) show why this allocation matters: requesting evidence and generating open queries can be costly, whereas bounded choices and reading supplied text can remain useful student work. Fact cards benefit all ten evaluated students, but decision policies transfer unevenly. Recompilation for a new student model helps when the transferred interface no longer fits the student. With 100 practice items, HC exceeds answer-only LoRA on three tasks. Larger training budgets can match or surpass a fixed harness, while combining the two improves SlideVQA beyond either alone. These findings support allocating work from measured student behavior rather than uniformly removing decisions.

1 INTRODUCTION

Small VLMs may read supplied evidence yet struggle to obtain it, so a harness can impose decisions beyond the student’s capabilities. Harness Compilation adapts reusable content and control to a frozen student using execution traces and validation.

1 INTRODUCTION

  • A student may answer correctly from a supplied fact card yet fail earlier when it must request a lookup, select an entry, or generate a useful query.The harness can therefore impose demands that exceed the student’s capabilities.Source: §1 · p. 1
  • Across seven settings, HC raises macro-average task score from 48.3 to 62.9, improving over bare students in all 21 main builds.Six settings use the same frozen Qwen3-VL-4B with task-specific harnesses.Source: §1 · p. 2
  • HC uses student execution traces to revise reusable content and control, then selects the deployed harness on disjoint validation data.Deployment uses the frozen student without teacher calls or weight updates.Source: §1 · pp. 1–3
  • Interventions distinguish costly evidence access and open argument generation from bounded choices and evidence reading that can remain useful student work.The allocation can vary by task and student even when external content stays the same.Source: §1 · pp. 2–3
  • With 100 practice items, HC outperforms answer-only LoRA on three tasks and can complement weight updates.Combining HC with tuned students improves document and multi-page reading.Source: §1 · p. 3

2 RELATED WORK

Prior approaches optimize inference-time work, prompts and context, or the harness around a model. HC focuses on how runtime decisions are allocated between the model and its harness.

Test-time scaling

  • Test-time scaling increases per-query work through reasoning-path voting, adaptive rollouts, transformed-input aggregation, or tool-using programs and agents.HC instead fits reusable content and control offline, with controls that compare alternatives using target students and available resources.Source: §2 · p. 3

Prompt and context optimization

  • Prompt and context optimizers revise instructions, demonstrations, critiques, or standing context, while HC can edit memory, routing, and executable control as well as prompts.The paper distinguishes the search procedure from the object it edits.Source: §2 · p. 3

Harness optimization

  • Harness optimization revises systems around models using execution evidence; HC’s focus is runtime decision allocation, studied through interventions, transfer, and recompilation.Related work includes harness-code search, component analyses, and reusable tools or controllers.Source: §2 · p. 3

3 PROBLEM: WHICH DECISIONS SHOULD THE STUDENT KEEP?

The problem is to decide which work a frozen student should retain when answering under an external harness. Useful resources are not enough unless the student can reliably access and use them.

3 PROBLEM: WHICH DECISIONS SHOULD THE STUDENT KEEP?

  • The setup pairs a frozen student and harness on labelled practice data and a held-out test set, with probe feedback and disjoint validation selecting a deployable harness.A large teacher may revise the harness offline but cannot be called at deployment.Source: §3 · p. 3
  • The five decision types are Decide whether to invoke a resource, Select a tool or entry, Generate an open argument, Integrate supplied evidence, and Persist by stopping or abstaining.Tasks need not expose all five types.Source: §3 · p. 4
  • These decision categories are not independent capability axes: changing one interface can affect another, and a task need not expose every type.Generating a final answer does not itself require an open lookup argument.Source: §3 · p. 4
  • Compilation tests whether a resource improves answers and whether the student can reliably make the decisions needed to use it, rather than assuming fewer student decisions are always better.Retrieved fact cards and rule-based verdicts offer distinct forms of help and must suit the inputs routed to them.Source: §3 · p. 4

4 HARNESS COMPILATION

Harness Compilation lets a teacher fill a developer-specified runtime with reusable content and control, revises proposals from student execution feedback, and uses disjoint validation to select what deploys.

4 HARNESS COMPILATION

  • The compiler prompt specifies the task, bare-student probe score, resource examples, allowed observations, editable fields, and hook contracts for a complete harness.The teacher returns harness JSON with hook source code.Source: §4 · p. 4
  • Table 1 gives fixed-harness examples from diagnostic studies, but the allocations are not universal policies for each task.Policies can differ across builds.Source: Table 1 · p. 4
  • Harness content sets prompt framing, answer format, and memory use, while a router selects evidence and invokes the student using observable inputs.The router can supply cards directly or ask the student to choose or name an entity; BLINK also permits code verdicts with student fallback.Source: §4 · p. 4
  • The teacher revises the latest proposal from probe execution reports, which connect failures to decisions such as routing, selection, evidence sufficiency, or reading instructions.Main builds use three reflection rounds; rationale does not establish causality when multiple components change together.Source: §4 · p. 4
  • Disjoint validation selects the deployed harness, retaining a proposal only when its unrounded mean score strictly improves and not below bare validation performance.Test items never reach the compiler, and deployment runs locally without teacher calls.Source: §4 · p. 5
  • A resource path must improve accuracy on an observable routed subset, and a student decision is worth retaining only when its precision benefit outweighs accuracy lost to contract-induced abstentions.The edge compares the path and bare student on the same routed items; HC selects complete harnesses by validation rather than optimizing these criteria directly.Source: §4 · p. 5

5 EXPERIMENTS

Across seven settings, Harness Compilation improves over bare students, and the experiments examine feedback-guided revision, decision interfaces, transfer, adaptation, and generation costs.

5.1 OVERALL EFFECTIVENESS

  • HC raises macro-average test score from 48.3 to 62.9, improving over bare students in all seven settings and all 21 builds.The macro-average weights all seven task-score scales equally; HC scores average three independent builds per setting.Source: §5.1 · p. 5, Table 2 · p. 6
  • On LiveVQA, the human recipe scores 25.5 versus 36.9 for student routing, while an unoptimized typed-input/output DocVQA program reaches 86.2 versus HC’s 85.0 mean.The LiveVQA arms differ in prompts and output formats; the DocVQA program is distinct from the Table 2 human recipe.Source: §5.1 · p. 6

5.2 WHAT DOES EXECUTION FEEDBACK CONTRIBUTE?

  • Feedback-guided revision and validation raise initial macro-average test score from 60.4 to 62.9, though one accepted SlideVQA edit loses 1.8 test points.Validation selects among proposals, so the gain measures search and selection together; matched optimizer controls do not establish a universally best search algorithm.Source: §5.2 · pp. 6–7, Table 3 · p. 6

5.3 WHICH DECISIONS SHOULD THE STUDENT KEEP?

  • Requesting evidence costs 9.0 points on InfoSeek and 2.7 on DocVQA, while removing SlideVQA’s evidence-slide selector costs 12.1 points.Requiring an entity name costs 13.3 points on InfoSeek, whereas removing a bounded pick changes the score by only −0.5 points, nonsignificantly.Source: §5.3 · pp. 7–8, Table 4 · p. 7
  • Raw BLINK signals cost 14.2 reflectance points and 22.6 depth points; restoring InfoSeek card lines improves scores by 0.27–2.87 points, while output-contract effects vary by task.Raw-signal comparisons also remove compiled verdict control; InfoSeek controls show the contract, not the extractor, carries the reported effect.Source: §5.3 · p. 8, Table 5 · p. 8

5.4 STUDENT-DEPENDENT DECISIONS AND RECOMPILATION

  • Fact cards benefit all ten tested students, but fresh InfoSeek builds average 0.4 points below transferred harnesses and help only three of nine recipients.Qwen3.5-0.8B gains 11.3 points; fresh DocVQA builds turn OCR off or retain the bare model, staying within one point of bare.Source: §5.4 · p. 8, Figure 3 · p. 9

5.5 DATA EFFICIENCY AND WEIGHT ADAPTATION

  • With 100 practice items, HC exceeds answer-only LoRA on all three tasks; on SlideVQA, combining the harness with 1,300-label tuning scores 63.7 versus 58.1 and 53.2.With enough reflectance labels, LoRA surpasses the frozen harness, whose control reduces tuned-student performance; recompilation recovers the tuned student’s score.Source: §5.5 · pp. 9–10, Table 6 · p. 9, Table 7 · p. 10

5.6 GENERATION EFFICIENCY AND OFFLINE COST

  • On InfoSeek, DocVQA, and SlideVQA, HC scores higher than chain-of-thought sampling with voting while generating about 69–189 times fewer student tokens.These counts exclude input and image prefill, tool computation, and offline teacher use, so they do not establish end-to-end speedups.Source: §5.6 · p. 10, Table 8 · p. 10

6 DISCUSSION AND LIMITATIONS

The evaluation is bounded by benchmark and deployment scope, and its validation and transfer checks do not establish performance on a newly collected test population.

6 DISCUSSION AND LIMITATIONS

  • Three builds only partly characterize variation, BLINK test pools are small, and validation can retain poor decisions.The accepted SlideVQA regression and harmful InfoSeek card compression illustrate this risk.Source: §6 · p. 10
  • The validation floor and branch-support audits do not guarantee improvement on new items.Appendix G collects branch-support, overlap, and evaluation audits.Source: §6 · p. 10
  • Formal evaluation uses observation-derived routing and an extraction-only parser; InfoSeek uses a fixed entity inventory and a practice-informed reused store.The passage also notes benchmark lineage for the reused store.Source: §6 · p. 10
  • Formal runs repeat compilation on benchmark splits already used in the research, measuring build variation rather than confirmation on a newly collected population.Transfer diagnostics characterize studied behavior rather than establish an independent prospective prediction test.Source: §6 · p. 11
  • Measurements cover task scores and generation costs, not end-to-end industrial deployment or hardware-specific latency, and offline teacher access to practice traces remains required.Residual teacher-student gaps may reflect missing tools, retrieval failures, or interface limits rather than establish that reasoning cannot be compiled.Source: §6 · p. 11

7 CONCLUSION

Harness Compilation allocates work around a frozen small VLM using offline execution feedback and validation, improving scores across seven VQA settings. The conclusion presents it as a practical option for repeated local assistants and reports benefits alongside weight adaptation; appendices collect implementation, comparisons, task evidence, and reliability checks.

7 CONCLUSION

  • 48.3 to 62.9: HC raises the macro-average task score across seven VQA settings.On InfoSeek, DocVQA, and SlideVQA, it also scores higher than chain-of-thought sampling with voting while generating about 69–189 times fewer student tokens, a comparison that does not establish end-to-end speedups.Source: §7 · p. 11
  • With 100 practice items, HC outperforms answer-only LoRA on three tasks and improves tuned students on SlideVQA and DocVQA beyond either component alone.The conclusion presents the combined benefit as applying when weight adaptation is available.Source: §7 · p. 11
  • HC’s interventions support allocating work from measured student behavior and reassessing that allocation as the model improves.The conclusion identifies repeated local document assistants and image-query services as practical settings for offline adaptation to reduce online generation.Source: §7 · p. 11

Appendix Contents

  • The appendices group implementation, controlled comparisons, task-specific evidence, and reliability checks.Source: Appendix Contents · p. 14

A IMPLEMENTATION, PROMPTS AND EVALUATION PROTOCOLS

This appendix documents the evaluation protocols, student measurements, and runtime and compiler interfaces used to build and compare harnesses, including a formal InfoSeek harness example. It also examines selected execution paths and frozen-artifact controls involving output instructions, extraction hooks, card rewriting, and question-type notes.

A.1 TASKS, SPLITS AND EVALUATION

  • Seven primary settings use 150 probe items, 300 disjoint validation items, and three reflections per build; final test items are withheld from the compiler.BLINK uses cross-build folds, and each item is evaluated only by a harness built on the other folds.Source: §A.1 · pp. 16–17
  • Five BLINK tasks form three settings chosen to expose distinct measurement–decision relationships, not to represent coverage of all BLINK capabilities.The selection considered available external signals; a large teacher–student gap alone was not sufficient.Source: §A.1 · p. 17
  • Binary correctness contrasts use paired exact McNemar tests, ANLS contrasts use paired item-bootstrap intervals, and repeated-build variation is distinguished from test-item uncertainty.Average rows weight named settings or students equally rather than pooling accuracy.Source: §A.1 · p. 17

A.2 HARNESS LANGUAGES

  • The harness separates prompts, tools, routing, retrievable content, and bounded-candidate construction, with task-specific languages defining editable controls and available signals.The documented languages cover InfoSeek knowledge retrieval, DocVQA document reading, and BLINK perception signals.Source: §A.2 · pp. 17–18

A.3 STUDENT PROFILE PROTOCOL

  • Student profiles measure contract-specific resource requests, evidence presentation, and response behavior; no single profile score is treated as a general reasoning-capability scale.Activation rates depend on the interface contract, and the compiler-input ablation tests whether supplying these measurements helps.Source: §A.3 · pp. 18–19

A.4 CURRENT COMPILER INTERFACE AND A FORMAL EXECUTION EXAMPLE

  • The runtime restricts routers to observation-derived inputs and parsers to contiguous spans of student output, while the teacher edits harnesses offline and validation retains the best eligible artifact.Under reflection A, the teacher edits the latest proposal—even if rejected—so proposal lineage can differ from the deployed validation incumbent.Source: §A.4 · p. 19
  • InfoSeek retrieval uses a fixed 1,380-entity inventory covering sampled practice and test items, and its reused fact-card store is practice-informed despite question-free card-generation calls.The setup is retrieval over a fixed inventory, not open-world recognition.Source: §A.4 · pp. 19–20
  • A formal InfoSeek example shows the selected harness using a two-pass bounded candidate choice for one item and direct-card answering for another, with both outputs scored correct.The requests omit images from the listing and do not provide gold answers to the student.Source: §A.4 · pp. 20–22
  • Card compression keeps the first line, adds 2 per requested-term match and 5 for a digit-containing line when the question is numeric, selects seven highest-scoring lines, then truncates to the limit.Source: §A.4 · p. 24
  • Question-specific notes direct taxonomy answers to the Genus field, numeric answers to one requested number, named-after answers to the eponym or root, and other answers to the requested part.The taxonomy instruction falls back to the first word of the scientific name when the Genus field is absent.Source: §A.4 · p. 25
  • The example router answers with a compressed top card when confidence criteria are met; otherwise, it asks the student to discriminate among candidate names.Its confidence checks include candidate count, top similarity, and the gap between the top two similarities.Source: §A.4 · p. 25
  • InfoSeek questions ask for short answers about the entity in an image and use official relaxed exact match, with numeric answers required to fall in the gold range.The runtime example describes questions such as asking a mountain’s elevation or a plant’s family.Source: §A.4 · p. 26
  • With no large model available at runtime, the router uses question text and candidate names, similarities, and cards to return answer, discriminate, or name modes.Its input contains the top-three CLIP candidates sorted by similarity, and question type must be inferred from the text.Source: §A.4 · p. 26
  • At inference, CLIP ViT-B-32 ranks entity names from a 1,380-entity store containing the gold entity: top-1 is gold about 52% of the time, and top-3 includes it 68%.Source: §A.4 · p. 26
  • The optional harness JSON configures prompt formatting, memory retrieval, decoding, discrimination and naming prompts, and sandboxed routing or extraction code.The typed-fields format is a fixed serialization contract, while parse_py may only extract a contiguous span of the student’s raw output.Source: §A.4 · p. 26
  • 36.67 probe accuracy for the selected harness exceeded the bare student’s 19.33 on the same 150 items; its validation accuracy was 40.0.The harness used the regenerated fact-card store and trusted CLIP top-1 only when comparatively confident, otherwise asking the student to discriminate among candidates.Source: §A.4 · p. 28
  • Accuracy was 46.94 when the shown card matched the gold entity, 31.58 with no card, and 9.09 with a wrong card.Discriminate mode scored 37.0 versus 36.0 for answer mode; 19.0% of discrimination passes picked none, and linked correctness when a candidate was picked was 65.4%.Source: §A.4 · pp. 28–29
  • The next harness added a type-mismatch fallback, question-specific guidance, a derived genus line for taxonomy questions, and tighter parsing of single numeric answers with units.These changes addressed failures involving wrong-kind candidate cards and answers that named an entity or type rather than the requested fact.Source: §A.4 · p. 29
  • Replaying the default extractor on the extraction-only parser’s archived outputs changed no test score.Source: §A.4 · p. 30

A.5 WHAT THE FORMAL ROUTERS DO, AND FROZEN-ARTIFACT CONTROLS

  • The six formal InfoSeek routers all use keyword-scored card-line selection and none inserts a target-answer value; routing and added question guidance vary across artifacts.Three derive a genus line for taxonomy questions, three attach question-type notes, and card lines are selected under a character budget.Source: §A.5 · p. 30, Table 18 · p. 30
  • Controls remove derived lines or question-type notes, restore original card-line order, or disable the category abort; exact McNemar tests mark paired changes at p < 0.05.R0 replays the selected artifact, while R1–R5 vary one named component at a time; removing a component is not equivalent to compiling without it.Source: §A.5 · pp. 30–31
  • R0 replay scores were within 0.1 point of archived scores across 6,000 test items; the controls compare paired changes against R0.Each control replaces one named function while keeping the store, prompts, and parse hook byte-identical.Source: §A.5 · p. 30, Table 19 · p. 31
  • Keyword-scored card rewriting costs 0.3–2.9 test points on each of six artifacts, and R4 is positive and significant everywhere.The largest effect is on replicate 3, where routers compress each of three cards to 560 characters; teacher line selection more often drops needed lines than distractors.Source: §A.5 · p. 31
  • Removing question-type notes costs 2.2 points on replicate 1 (A) and 1.2 on replicate 3 (A), but has no measurable effect on replicate 2 (A).Replicate 2’s notes are shorter.Source: §A.5 · p. 31
  • The category abort of replicate 1 (A) costs 0.7 points, while the derived genus line changes scores by at most 0.3 points.These are validation-chosen artifact gaps on held-out items, limited to the frozen artifacts and named interventions.Source: §A.5 · p. 31
  • Replaying the default extractor changes no artifact’s score by more than 0.1 points when the contract is present, indicating extraction hooks do no semantic work.Without the contract, the extractor matters by up to 1.5 points.Source: §A.5 · p. 31
  • Removing output contracts costs 1.8–4.6 points on A artifacts with the default extractor, and the extractor then matters by up to 1.5 points as outputs become longer.The contract-sentence rule was frozen before A evaluation and selected the same sentences on B artifacts as the earlier rule.Source: §A.5 · p. 31
  • Removing output-contract sentences costs 1.8–6.1 points on A artifacts and 1.7–2.6 points with artifact hooks or 1.3–3.2 with the default extractor on B artifacts.The contract accounts for the measured output-interface benefit on both protocols; removing it makes student outputs longer.Source: Table 20 · p. 31, §A.5 · p. 31

B.1 TEACHER, PROFILE AND FEEDBACK ABLATIONS

The formal results favor the reported reflection procedure over its incumbent-anchored variant across most settings, while validation-selected improvements and decision allocation remain task- and student-dependent. The ablations also show why execution traces, build safeguards, and careful interpretation of controls matter.

B.1 TEACHER, PROFILE AND FEEDBACK ABLATIONS

  • Score-only feedback kept a flawed DocVQA proposal, while profile formats and teacher capacity produced task-dependent outcomes.The score-only DocVQA build removed OCR after an undiagnosed parse failure; the profile-withheld condition retained two profile facts, and smaller teachers stayed near the InfoSeek name-card default.Source: §B.1 · p. 32

B.2 ONE-SHOT CONSTRUCTION

  • The one-shot harness matched the selected build on depth and was statistically tied on reflectance and DocVQA, but fell below the loop on InfoSeek, correspondence, LiveVQA, and SlideVQA.Its InfoSeek score was 25.0 versus 33.8 for the public-content human recipe; changed guidance and an independent initial draw prevent attributing the difference specifically to iteration.Source: §B.2 · pp. 32–33, Table 22 · p. 33

B.4 INITIAL-BUILD SAFEGUARD AND EXECUTION FEEDBACK

  • The below-bare safeguard keeps bare as a validation candidate but cannot guarantee improvement on unseen data, and execution reports localize failures without proving which changed component caused a score change.Reports identify stages including linking, candidate selection, evidence use, and output extraction; multiple components may change together.Source: §B.4 · p. 33

B.3 BUILD VARIATION AND VALIDATION SELECTION

  • Validation may select a proposal that loses on held-out items, as illustrated by a reflectance fold whose test accuracy fell despite a gate improvement.The gate selects rounds using validation; held-out test scores do not select the retained round.Source: §B.3 · p. 33, Figure 4 · p. 34

B.5 FORMAL REPLICATES, THE FROZEN GATE AND THE REFLECTION-PROTOCOL COMPARISON

  • Protocol A’s gains over H0 were significant in every InfoSeek, DocVQA, and LiveVQA build, while protocol B was empirically weaker on five of seven settings and better only on SlideVQA.SlideVQA still had one protocol-A test regression, illustrating that a validation-accepted edit can lose on test.Source: Table 24 · p. 35, §B.5 · pp. 35–36

B.6 REFLECTION A AND THE INCUMBENT-ANCHORED VARIANT B

  • Protocol B changes the proposal procedure while holding runtime interface, initial harness, practice items, and strict gate fixed; neither procedure uses test outcomes to accept or reverse a rejection.B edits the incumbent and requests one coherent mechanism change, whereas A edits the last proposal.Source: §B.6 · p. 36

C.1 STUDENT-ROUTED AND SINGLE-DECISION CONTROLS

  • Single-decision interventions are non-independent and non-additive, and InfoSeek’s select ablation removes an explicit two-pass choice rather than simply returning a code decision to the student.Each ablation compares one unselected variant with a validation-selected harness.Source: §C.1 · p. 37, Table 26 · p. 37

C.2 DECISION OWNERSHIP AND STUDENT-SPECIFIC POLICIES

  • Decision ownership varies by task: InfoSeek retains bounded picks and content tasks retain reading, while BLINK source rules answer 55–93% of items.LiveVQA and SlideVQA retain similar broad roles but have different control gains, so counting compiled decisions does not predict their value.Source: §C.2 · p. 37, Figure 5 · p. 38, Figure 6 · p. 38

C.3 CROSS-STUDENT TRANSFER AND RECOMPILATION

Harness transfer is uneven across students, and fresh builds do not always outperform a transplant. The diagnostics also show that results depend on the evidence store and that the tested interfaces do not establish an irreducible reasoning limit.

C.3 CROSS-STUDENT TRANSFER AND RECOMPILATION

  • Under the cleaned interface, own builds beat the transplant on 3/9 students and lost on 5/9, with an average difference of −0.4.The comparison used the same protocol, but each own build was a single teacher draw.Source: Table 28 · p. 39, §C.3 · p. 39
  • The correspondence rule answered the same 243/441 items at 83.1% precision for every student, while a separate Qwen3-VL-2B build reached 63.3 versus 58.1 for the transplant.Short decode budgets can prevent weaker students from returning an answer letter on deferred items.Source: §C.3 · p. 40
  • Fresh DocVQA builds under the current protocol were not significantly different from bare, and MiniCPM-V-4.6 retained the bare model after its proposals were rejected.The two Qwen3.5 builds turned OCR off.Source: Table 32 · p. 40, §C.3 · p. 40

C.4 EVIDENCE FOR DECISION CONSTRAINTS AND REMAINING REASONING GAPS

  • The tested-channel residual cannot be read as an irreducible reasoning fraction, because missing tools, retrieval, output contracts, or limited search may also account for it.IQ Test was screened but did not receive a comparable HC build.Source: §C.4 · p. 41

D.1 INFOSEEK: CONTENT, PLACEMENT AND STORE LINEAGE

  • Build 1 rewrote all 1,380 cards, and linking-condition accuracy when the shown card was gold rose from 36.4 for name cards to 59.9 under the discriminate harness.On wrongly linked items, accuracy stayed at 9–11; the transplant matrix also includes store differences.Source: §D.1 · p. 42

D.2 DOCVQA: CONSOLIDATED BASELINE COMPARISON

  • Plain chain-of-thought scored 75.2 on DocVQA, while adding it to plain OCR scored 78.8 against 80.1 for plain OCR alone.The figures are ANLS scores on 3,568 test items.Source: §D.2 · p. 43

D.3 BLINK: VERDICT COVERAGE AND TASK-DEPENDENT RESIDUALS

The correspondence results leave substantial teacher–student gaps, and the compiled build does not significantly exceed its fixed human rule. Exploratory tool access also varies sharply by task, while build scores vary across folds.

D.3 BLINK: VERDICT COVERAGE AND TASK-DEPENDENT RESIDUALS

  • For visual, semantic, and functional correspondence, HC scored 92.4 / 54.0 / 49.2 against the teacher’s 95.3 / 74.1 / 61.5, and no compiled build significantly exceeded its fixed human rule.These comparisons use identical items; correspondence subsets are not additional independent benchmarks.Source: Figure 8 · p. 43, §D.3 · p. 44
  • In an exploratory tool comparison, the 4B requested signals on 95–100% of depth, counting, and spatial items but only 2% of visual-correspondence items; tool-only accuracy was below the student on four of five screened tasks.These contract-specific observations motivated controlled activation interventions and were excluded from the main cross-build average.Source: §D.3 · p. 44
  • Fold scores varied across builds: reflectance included 52.3 and 71.1, correspondence 61.2 and 72.7, and depth 90.2 and 97.6.The selected round was round 0 in all six depth folds.Source: §D.3 · p. 44

D.4 LIVEVQA: RETRIEVAL, CONTRACTS AND SCORING

On LiveVQA, the compiled harness beats the human recipe and top-1 retrieval, but scoring and abstention choices affect the comparison. Letting students inspect all candidate article cards avoids an early information bottleneck, though the evidence does not isolate that factor alone.

D.4 LIVEVQA: RETRIEVAL, CONTRACTS AND SCORING

  • Showing all three candidate cards directly scored 36.9, compared with 25.5 when the student first chose one headline, an 11.4-point gap plausibly tied to the early information bottleneck.The arms also differ in prompting and output handling, so the gap is not an isolated selection ablation.Source: §D.4 · p. 44
  • The reported judge agreed with GPT-5.5 on 96–99.8% of items per arm, and the positive HC-versus-human and HC-versus-top-1 conclusions held under all three metrics.Claude Sonnet 5 was 0.2–1.9 points more lenient on every arm.Source: §D.4 · pp. 44–45
  • The selected 4B harness and student-routed arm differed by 1.2 points, while handing the output contract back to the student changed four items (+0.2).Both arms require the student to read the same three cards; the diagnostic is consistent with much of the advantage arising before output extraction.Source: §D.4 · p. 45
  • The harness left 166 items not attempted versus 627 for student routing, but student routing had the higher LiveVQA F-score: 43.7 against 39.8.The gate optimized containment, where a wrong guess costs no more than abstention; a gate using the task’s F-score would select differently.Source: §D.4 · p. 45
  • HC beat the human recipe and top-1 retrieval for each student on LiveVQA, with the table reporting paired test comparisons.The four HC rows are single builds; the Qwen3-VL-4B row is a diagnostic reference rather than the three-build main-table mean.Source: Table 37 · p. 45

D.5 SLIDEVQA: PAGE SELECTION AND IMAGE BUDGET

For SlideVQA, selected HC beats bare and the human recipe on every reported row, while student-specific builds show that OCR and page selection do not help every model equally. The accompanying comparisons also find mixed returns from test-time scaling and retrieval or prompt optimization.

D.5 SLIDEVQA: PAGE SELECTION AND IMAGE BUDGET

  • Selected HC exceeded both the human recipe and bare at p < 10^-8 and p < 10^-3, respectively, on every SlideVQA row.Bare used the full deck at 448 pixels; paired exact McNemar tests were used.Source: §D.5 · p. 46
  • SlideVQA gains were student-dependent: OCR was worth +4 to +16 for three students but −3.2 for Gemma, while the 4B numerical subset rose from 16.0 to 41.2 without compiled arithmetic.The 4B harness transferred to the 9B for +11.9 but harmed Qwen3.5-4B and MiniCPM.Source: §D.5 · pp. 46–47

E.1 FIXED AND TUNED HUMAN HARNESSES

  • The tuned DocVQA human recipe scored 80.11 test ANLS, and its best validation variant was 4,000-character OCR with a plain suffix; every exact-span variant scored 77–79.The fixed BLINK rules produced verdict-only scores of 59.0 / 64.2 / 94.4.Source: §E.1 · p. 47

E.2 TEST-TIME SCALING

  • Voting moved no setting by more than 0.9 and moved nothing atop a harness, while TTAug’s paired changes ranged from +4.0 on depth to −1.5 on reflectance, both nonsignificant.TTAug required N prefills per item; one 3,568-item pass took five GPU-hours on one 3090.Source: §E.2 · p. 48, Table 41 · p. 48

E.3 RETRIEVAL AND PROMPT OPTIMIZATION

  • RAG-wiki scored 30.87 and gold-entity RAG-wiki scored 48.75, while on the same subset RAG-web scored 22.0 versus 30.0 for store-only and 38.3 for HC.GEPA’s accepted candidates stayed within the gate’s standard error of the seed, and its InfoSeek instructions memorized probe facts.Source: §E.3 · p. 49

E.4 GEPA OVER THE SAME EXECUTABLE ARTIFACT

Across these experiments, harness performance depends on the editable artifact, interface, and training regime: GEPA and HC were similar within build variation, while student-driven tool controls often lagged compiled harnesses.

E.4 GEPA OVER THE SAME EXECUTABLE ARTIFACT

  • On two matched runs, HC and GEPA differed by only −0.2 points on average, with build variation exceeding that difference; the results do not establish search superiority.Both procedures made similar edits, while GEPA cost about twice as much in this setting.Source: §E.4 · pp. 49–50, Table 43 · p. 49

E.5 TYPED INPUT AND OUTPUT FIELDS

  • GEPA raised the DSPy seed program’s validation score from 87.3 to 90.7 but changed its test score by 0.0 (86.24 → 86.19); typed-field serialization itself scored 81.1 without OCR and 85.3 with OCR.The serialization effect varied by student and task, ranging from +9.0 for Gemma to −6.0 on reflectance.Source: §E.5 · p. 50, Table 44 · p. 50

E.6 STUDENT-DRIVEN CODE AND TOOL-USE CONTROLS

  • Student-written controls stayed below compiled harnesses; BLINK measurement calls were 0/699, while DocVQA OCR was called on 81–97% of items but fetched lines were integrated worse (Gemma 75.2 vs 84.9).A separate DocVQA program arm crashed and fell back to self-query; the compiled harness fixed both activation and integration decisions.Source: §E.6 · p. 51, Table 45 · p. 51

F.1 PRACTICE BUDGETS AND THE 100-ITEM REGIME

  • At 100 practice items, HC averaged 33.6 / 84.7 / 54.7 on InfoSeek / DocVQA / SlideVQA, above the corresponding answer-LoRA means.One InfoSeek HC build scored 25.9 on test after selecting a round-1 harness, versus 26.8 for its initial harness; ten-epoch InfoSeek trajectory-SFT reached 35.3, above HC’s 33.6.Source: Figure 9 · p. 52, §F.1 · p. 53, Table 47 · p. 53

F.2 TRAJECTORY SUPERVISION AT MATCHED ITEM BUDGETS

  • The trained InfoSeek agent scored 36.9 versus 23.7 for answer-only LoRA and 34.1 / 35.6 for the original HC builds, but the 450-item comparison applies only to those source harnesses.At 1,200 practice items, the agent scored 40.7 versus 39.7 for HC, a statistically tied difference (+0.9, 95% interval [−0.1, +2.1], p = .10).Source: §F.2 · p. 53

F.3 ANSWER ADAPTATION AND TASK-SPECIFIC TRAINING PROTOCOLS

Weight adaptation and harnesses had task-dependent effects: their combination improved SlideVQA, while outcomes varied across reading, image-question, and BLINK settings. Additional reflectance labels could overtake a fixed harness, and recompilation altered the harness after tuning.

F.3 ANSWER ADAPTATION AND TASK-SPECIFIC TRAINING PROTOCOLS

  • On DocVQA, matched-label answer-LoRA scored 84.7 ANLS versus 85.4 for the compiled harness; running that harness on the adapter scored 86.4.The harness-on-adapter result was +1.8 over the adapter and +1.0 over the harness on the base student.Source: §F.3 · pp. 53–54
  • BLINK answer-LoRA scored 53.7 on reflectance, 82.3 on depth, and 70.5 on correspondence; only correspondence significantly exceeded the compiled harness (+2.9, p = 0.21, not significant).Reflectance and depth remained below HC by 11.2 and 12.9 points, respectively; the stated gains over bare were not significant on those two settings.Source: §F.3 · p. 54
  • On the same practice items, LiveVQA trajectories scored 33.4, 18.6 points above bare and 7.7 above the recipe, but 3.8 below Hval (p = 5 × 10−5).Source: §F.3 · p. 54
  • On SlideVQA, the 1,300-label adapter plus the frozen harness scored 63.7, exceeding the adapter alone (53.2) and the harness on the base student (58.1).With the earlier round-2 harness, the full-train adapter added nothing further (64.1 vs 63.9, p = 0.86).Source: §F.3 · p. 54

F.4 LABEL SCALING AND RECOMPILATION AFTER TUNING

  • Reflectance LoRA rose from 52.2 with 44 labels to 80.6–81.3 with 20,000, above HC’s 64.9 using 89 probe-and-gate items; after tuning, a fresh harness scored 82.1 versus 66.4 for transplanted harnesses.The fresh build removed the tool in two folds and used a conservative router in the third; its +1.5 over the tuned bare student was not significant.Source: §F.3 · p. 54, Figure 10 · p. 55, Figure 11 · p. 55

G.1 SUPPORT, STORE AND EVALUATION AUDITS

The audits identify support limits for router rules, a corrected fact-store contamination issue, and safeguards used in evaluation.

G.1 SUPPORT, STORE AND EVALUATION AUDITS

  • Several selected SlideVQA router rules matched probe phrases but fired on zero validation and zero of 2,215 test items; a later safeguard drops branches with fewer than ten validation firings.Removing the round-3 verdict rules and parser special cases left the score essentially unchanged (56.70 versus 56.66).Source: §G.1 · p. 56
  • A store audit found earlier name-card contamination for 120/1,380 entities; rerunning names-only controls with a clean store changed scores by 0.3–0.6 points.Teacher fact-card generation used entity names, not evaluation questions or answers.Source: §G.1 · p. 56
  • Evaluation safeguards held image inputs, reasoning defaults, and answer-letter normalization fixed across compared arms; fresh compiler probes used items held out from adapter training.The LiveVQA judge comparison and scoring distinctions are documented separately, and failed builds and unfavorable transfer results remain with their experiments.Source: §G.1 · p. 56

G.2 EXPLORATORY VLA BOUNDARY CASE: LIBERO-PLUS

In this exploratory LIBERO-plus evaluation, interface changes and external control yielded limited benefits, but rollout noise constrained protocol comparison. The authors distinguish this boundary case from the seven-setting VQA results and identify additional measurements needed for stronger robotics claims.

G.2 EXPLORATORY VLA BOUNDARY CASE: LIBERO-PLUS

  • Providing X-VLA decisions as appended instruction text scored 60.2; the base SmolVLA validation-selected round scored 34.5 on test versus 37.0 for its first candidate, not significantly different.Canonicalization recovered most of SmolVLA’s gain, while X-VLA showed little canonicalization effect and a few points from transplanted external-control settings.Source: §G.2 · pp. 56–57, Table 48 · p. 56
  • Selected or transplanted harnesses had at most six correct refusals, contributing at most 1.0 point to the mixed score; the base SmolVLA selected round falsely aborted 62 valid episodes.Those false aborts cost 3.7 completion points against its own H0; the evaluation did not log or assess partial-progress stages.Source: §G.2 · p. 57, Table 49 · p. 57
  • Across three builds, protocol A averaged 34.2 and protocol B 32.3; the comparison was inconclusive because rollout noise reached three points and both protocols mainly gained by removing false aborts.The authors do not use this comparison for a main-text claim.Source: §G.2 · p. 57, Table 50 · p. 58
  • The exploratory results do not establish long-horizon planning, subtask recovery, or industrial deployment; stronger robotics claims require task-level sequencing and recovery measurements.The results are kept outside the seven-setting VQA average.Source: §G.2 · p. 57

G.3 COMPILATION AND DEPLOYMENT COST

  • InfoSeek costs differ by store regime: compilation with an existing store cost about $1, while regenerating the store raised the build cost to about $7; deployment used zero teacher calls.These are GPT-5.5 charges for the reported configurations, not model-independent prices; the reported token counts exclude image-prefill and tool-computation costs.Source: §G.3 · p. 58
  • The token-cost comparison pairs scores and generated-token counts from the same executions, but those counts do not measure latency.HC token counts include all student passes; code-only verdicts contribute zero student tokens.Source: §G.3 · p. 58, Table 51 · p. 58
Loading 2610.11231v1…