AI4AI at Test-Time: Strong-to-Weak Capability Transfer via Harnesses

• Scaffolding transfers capability through a harness that shapes how a fixed weaker target receives, reasons, and responds, without updating its parameters.The builder designs reusable task structure using components such as routing, prompts, solvers, exemplars, verification, and format enforcement, without access to the full test set.Source: §1 · pp. 1–2
• The study uses Theory-of-Mind benchmarks to examine scaffold gains, stability, validation effort, techniques, platform and target dependence, mechanisms, cognitive-load reduction, and failure modes.The introduction motivates a systematic account of harnesses because their impact, stability, and design choices remain poorly understood.Source: §1 · p. 2
• 0.49 to 0.91 macro-average accuracy: the best scaffold raises GPT-5.4-mini by 0.42, and every builder configuration yields positive net uplift.The gains come more from structure externalization, deterministic offloading, and strict format enforcement than from using more validation data, sampling more, or eliciting longer reasoning.Source: §1 · p. 2
• Findings vary by benchmark and target: BigToM gains can compile into skills and rules, while other benchmark gains reflect reduced reasoning burden.Residual errors cluster in higher-order Hi-ToM cases with recursion depth at least two and Bayesian goal-inference subtypes; builder reasoning effort improves scaffold quality monotonically, platform effects are modest, and the weaker GPT-5.4-mini benefits more than Gemini-3.5-flash.Source: §1 · p. 2
• Successful scaffolds favor deterministic offloading, benchmark-aware routing, format control, and targeted decomposition over brute-force validation search.These are the paper’s actionable design principles for harnesses.Source: §1 · p. 2
• Scaffolding offers a way to improve weaker or cheaper models at inference time and to evaluate how well builders externalize reasoning into procedures, tools, feedback loops, and constraints.The paper presents harness evolution as a platform for studying what structures builders invent, revise, and fail to construct.Source: §1 · p. 2

Capability transfer and distillation

• Unlike distillation and alignment that update model parameters (Hinton et al., 2015; Ouyang et al., 2022), this work transfers task structure through a reusable inference-time harness for a weaker target.Prior approaches also transfer rationales or reasoning supervision and use on-policy teacher feedback (Hsieh et al., 2023; Agarwal et al., 2024), while weak-to-strong generalization still studies transfer through model updating (Burns et al., 2023).Source: §2 · p. 3

Inference-time reasoning, prompting, and decomposition

• Prompting, sampling, decomposition, and iterative search improve inference-time reasoning, but typically optimize how one model solves each instance rather than building a persistent scaffold for another model (Wei et al., 2022; Wang et al., 2022; Zhou et al., 2022; Madaan et al., 2023).Related approaches include modular delegation and structured search over intermediate states (Khot et al., 2022; Yao et al., 2023; Besta et al., 2024).Source: §2 · p. 3

Tool use, programmatic reasoning, and deterministic offloading

• Tool use and program-aided reasoning motivate moving fragile execution and symbolic work into external tools or code; this study finds deterministic solvers, routing, and format enforcement outperform simply encouraging longer target reasoning.Related examples include Toolformer and ReAct (Schick et al., 2023; Yao et al., 2022), PAL and Program-of-Thoughts (Gao et al., 2023; Chen et al., 2022), and faithful chain-of-thought (Lyu et al., 2023).Source: §2 · p. 3

Harness engineering and automated scaffold construction

• Harness engineering optimizes the system around an LLM, and this work isolates strong-to-weak transfer while examining how builder capability, reasoning effort, platform, validation budget, target headroom, and offloadability shape harness quality.Prior examples include declarative LM pipelines in DSPy (Khattab et al., 2023), interface design in SWE-agent (Yang et al., 2024), and scaffold search or evaluation in ADAS, Meta-Harness, and Harness-Bench (Hu et al., 2025; Yao et al., 2026); survey work describes code as an agent harness substrate (Ning et al., 2026).Source: §2 · pp. 3–4

Theory-of-Mind evaluation and mental-state scaffolds

• Theory-of-Mind benchmarks test tracking beliefs, intentions, hidden information, and nested perspectives; this work meta-evaluates automatic reusable scaffolds, not a new benchmark or manually designed solver.Its stated failure cases include belief recursion and Bayesian goal inference that resist compilation into rules and skills; examples of benchmark coverage include recursive beliefs, household goals, embodied interaction, and broader social cognition (Jin et al., 2024; Chen et al., 2024).Source: §2 · p. 4

Setup

• Each benchmark is split into a randomly sampled 5% validation set and a hidden test set for evaluating the completed scaffold.The builder can access only the validation set during construction; the test set is held out for separate evaluation.Source: §3 · p. 4
• Builders work inside an existing agentic coding harness and receive task instructions, a target-model calling demonstration, and labeled validation data.The initial workspace includes a rule file describing task instructions and submission format, a demonstration file showing how to call the target, and the validation set.Source: §3 · pp. 4–5

Scaffold design

• The builder may implement any inference-time procedure, provided the final scaffold exposes an entry point applicable to unseen test examples.Possible procedures include prompting, benchmark routing, deterministic preprocessing or postprocessing, answer-format enforcement, verification, few-shot retrieval, and symbolic solvers.Source: §3 · p. 5

Evaluation

• After building, a human evaluator runs the exported entry point on the hidden test set without further builder intervention.Because the builder cannot access the test set, success depends on identifying reusable task structure from validation examples and transferring it to held-out examples.Source: §3 · p. 5

Task and metric

• The hidden test set aggregates four ToM datasets into 3,900 items: BigToM (1,200), Hi-ToM (1,200), MMToM-QA (600), and MuMA-Tom (900).Questions cover beliefs, goals, actions, nested beliefs, and multi-agent social reasoning.Source: §4 · pp. 5–6
• Each builder receives a fixed-seed 195-item validation sample; the primary metric is the unweighted macro average of full-set accuracies across the four benchmarks.The number of validation evaluations is a secondary efficiency criterion.Source: §4 · p. 6

Experiment design

• The design varies builder platform, builder and target model, and repeats each setting three times, yielding 72 experiment runs.Platforms are Cursor, Claude Code, and GPT Codex; targets are GPT-5.4-mini and Gemini-3.5-flash, with GPT-5.4-mini the common control for most comparisons.Source: §4 · p. 6

Baselines

• Vanilla macro-average accuracy is 0.488 for GPT-5.4-mini and 0.761 for Gemini-3.5-flash; UserHarness yields 0.939 and 0.941, respectively.Vanilla measures the performance scaffolding is expected to improve upon, while UserHarness is a human-designed harness reference.Source: §4 · p. 6

5 Results and Analysis

• Builder model was the dominant source of variation, and many scaffolded configurations surpassed raw GPT-5.4, although automated scaffolding still fell short of the human-inspired scaffold overall.Every evaluated builder–platform configuration substantially exceeded the GPT-5.4-mini no-scaffold baseline.Source: §5 · p. 6

5.1 Aspect 0: Main Results

• Across 57 scaffolded GPT-5.4-mini runs, mean macro-average accuracy was 0.763, an uplift of +0.275 over baseline, and 100% of runs exceeded baseline.The best individual run, GPT-5.5 on GPT Codex, reached 0.912 (+0.423, 87% relative).Source: Figure 3 · p. 7, §5.1 · p. 7
• The best automated scaffold outperformed both vanilla baselines on all four benchmarks; against UserHarness, it scored 1.00 vs. 0.95 on BigToM, but trailed on the other three.The remaining UserHarness comparisons were Hi-ToM (0.80 vs. 0.87), MMToM-QA (0.84 vs. 0.98), and MuMA-ToM (0.88 vs. 0.96).Source: Figure 3 · p. 7, §5.1 · p. 7

5.2 Aspect 1: Run-to-Run Stability

• Mean run-to-run standard deviation was 0.036, about an order of magnitude below the +0.275 mean uplift, but the widest setting had a repeat range of 0.201.The largest spreads arose when deterministic-solver strategies introduced benchmark-specific logic errors; building two or three scaffolds and selecting the best validation performer was suggested as a low-cost way to capture stronger performance.Source: §5.2 · pp. 7–8, Figure 4 · p. 8

5.4 Aspect 3: Scaffolding Techniques

• Across 72 runs coded with a fixed twelve-technique taxonomy, robust format enforcement and greedy decoding were nearly universal, while routing and forced chain-of-thought were also common.Deterministic solvers, self-consistency, verification, and few-shot prompting appeared less often.Source: §5.4 · pp. 8–9

5.3 Aspect 2: Refinement on Validation

• With a mean of 4.9 validation passes, builders improved validation accuracy by 0.216; best validation accuracy tracked full-set accuracy (r=0.96), while iteration count did not (r=0.17).The mean validation–full optimism gap was 0.021, suggesting the 5% validation sample guided refinement without substantial overfitting; builder quality mattered more than probing frequency.Source: §5.3 · p. 8, Figure 5 · p. 9

5.4 Aspect 3: Scaffolding Techniques

• Technique choice depended on the benchmark: BigToM often used deterministic or hybrid rules, whereas MuMA-ToM was almost always handled by the model itself.Format enforcement, greedy decoding, routing, and forced chain-of-thought acted as reliability controls; deterministic solvers, structured state extraction, and polarity logic required exploitable task regularity.Source: §5.4 · p. 9

5.5 Aspect 4: Harness Platform’s Impact

• Native-platform runs change macro accuracy by only +0.013 on average across eight matched configurations, with no reliable general advantage (p= 0.484).The native platform wins in 5 of 8 cells, and GPT-5.4-mini and Sonnet-4.6 perform worse on their native platforms after averaging parallel runs.Source: Figure 7 · p. 10, §5.5 · p. 10
• For Opus-4.7, Claude Code trails Cursor at low effort (−0.034) but leads at medium (+0.045), high (+0.038), and extra-high (+0.032) reasoning effort.This indicates a platform×effort interaction: native tooling helps when the builder has sufficient reasoning budget to exploit it.Source: Figure 7 · p. 10, §5.5 · p. 10
• Claude Code’s pooled mean is 0.799 versus 0.754 for GPT Codex and 0.750 for Cursor, but builder-roster differences confound this platform comparison.The platforms host different builder rosters (K= 11, 5, and 3 builders), and broad score overlap indicates builder identity dominates platform identity.Source: Figure 7 · p. 10, §5.5 · p. 10

5.6 Aspect 5: Analysis Across Target Models

• GPT-5.4-mini gains +0.262 (0.488 →0.750), while Gemini-3.5-flash gains +0.110 (0.761 →0.871); all five builders improve both targets.The weaker target benefits more from scaffolding across the matched builder comparisons.Source: Figure 8 · p. 11, §5.6 · p. 11
• Across builder×benchmark×target settings, uplift correlates with available benchmark headroom (Pearson r= 0.75), consistent with scaffolding recovering latent competence.Correctable failures include answer-format errors, missed observation cues, and failures to maintain recursive state; payoff is greatest where more headroom remains.Source: Figure 8 · p. 11, §5.6 · pp. 11–12
• Gemini-3.5-flash’s uplift is concentrated on BigToM, which accounts for 96% of its macro uplift (0.42), while GPT-5.4-mini gains across all four benchmarks.BigToM retains meaningful headroom for Gemini-3.5-flash and is especially amenable to rule-based compilation.Source: Figure 8 · p. 11, §5.6 · p. 11
• Builders use less deterministic machinery for Gemini-3.5-flash than GPT-5.4-mini, increasing model-only handling on every task and most on MuMA-ToM (40% →73%).The strategy shifts heavier rule-based interventions toward tasks where the stronger target still has compilable headroom.Source: Figure 8 · p. 11, §5.6 · pp. 11–12
• Gemini-3.5-flash regresses in 9/20 matched cases, especially on Hi-ToM (−0.04 on average) and MuMA-ToM (−0.02 on average), while GPT-5.4-mini never regresses (0/20).When the target is near ceiling, additional prompts, routing, or rules can disrupt correct behavior; strong-target scaffolding should be selective and headroom-gated.Source: Figure 8 · p. 11, §5.6 · pp. 11–12

5.7 Aspect 6: Builder Reasoning Effort

• 0.711 pooled macro accuracy at low effort rose to 0.793 at medium, 0.807 at high, and 0.856 at extra-high effort (Spearman ρ=0.77).The extra-high tier significantly outperformed high effort (0.856 vs. 0.807; permutation test p=0.013) and low effort (p=0.002); macro accuracy increased monotonically on both platforms.Source: Table 3 · p. 12, §5.7 · p. 12

5.8 Aspect 7: Attribution of Why Does Improvement Happen?

• 1717 baseline errors were fixed and 105 previously correct items broken by the best scaffold across 3,900 items, a highly significant paired McNemar improvement over GPT-5.4-mini’s no-scaffold baseline (p < 10^-4).The item-level change was overwhelmingly in the direction of correction.Source: Figure 10 · p. 13, §5.8 · p. 13
• GPT-5.4-mini self-scaffolding improved over its no-scaffold baseline by +0.17 to +0.22, while stronger builders gained +0.31 versus +0.17 for self-scaffolding on GPT Codex.Self-scaffolding shows that a target model can use validation feedback and task structure to build a useful harness, but stronger builders unlock the highest gains on both platforms.Source: Figure 10 · p. 13, §5.8 · p. 13
• The union of top scaffolds repaired 97% of baseline errors, more than any individual scaffold, as they overlapped on easy fixes but diverged on harder cases.Different strong scaffold designs therefore captured partly distinct mechanisms for repairing the target model’s reasoning.Source: Figure 10 · p. 13, §5.8 · pp. 13–14

5.7 Aspect 6: Builder Reasoning Effort

• Reasoning-effort gains were largest from low to medium and remained positive at higher tiers, with scaffold size growing from ∼510–650 LOC at low effort to ∼1000–1300 at extra-high.Moderate deliberation helps identify structural devices such as routing, format enforcement, and deterministic solving; higher effort adds refinements such as polarity logic and belief-state extraction. The analysis attributes useful compute to deeper task-structure hypothesis formation, unlike additional validation evaluations, which did not predict final quality.Source: §5.7 · pp. 12–13

5.8 Aspect 7: Attribution of Why Does Improvement Happen?

• The strongest better-powered technique associations were polarity/negation logic (+0.09), structured extraction (+0.06), hybrid fallback (+0.04), and deterministic solving.These comparisons are associational because techniques co-occur; apparent negative associations for near-universal greedy decoding (56/1) and benchmark routing (54/3) reflect tiny, unusually weak non-user groups rather than evidence of harm.Source: §5.8 · p. 13
• Nearly universal format enforcement, routing, and greedy decoding establish a reliability floor, while polarity logic, structured belief-state extraction, and deterministic solving distinguish the strongest scaffolds.The first layer prevents avoidable malformed-output, format, and sampling errors; the second compiles benchmark regularities into inference-time machinery, leaving residual hard cases across diverse designs.Source: §5.8 · p. 14

5.9 Aspect 8: Cognitive-Load Reduction

• Builders reduce cognitive load through offloading, which answers some items directly with code or rules, and structuring, which constrains prompts when the target model is still needed.The harness encodes part of the task’s decision procedure once, reserving the target model for cases that resist rule or skill compilation.Source: §5.9 · p. 14
• Determinism fraction correlated with final accuracy at r=0.72, while scaffold code size had only a weak relationship with accuracy (r≈0.22).The result suggests that converting the right work into executable structure matters more than simply writing more code.Source: §5.9 · p. 14, Table 4 · p. 14, Figure 11 · p. 15
• Offloadability varied by benchmark: mean determinism was 0.94 for BigToM, approximately 0.51 for Hi-ToM, 0.44 for MMToM-QA, and 0.36 for MuMA-ToM.BigToM questions often expose observed/unobserved distinctions, while Hi-ToM supports symbolic belief-state tracking; free-form dialogue reasoning makes MuMA-ToM less reducible to deterministic code.Source: §5.9 · p. 14, Figure 11 · p. 15

5.10 Aspect 9: Remaining Error Analysis

• BigToM accuracy was at least 0.95 on every slice, while Hi-ToM accuracy fell from 0.999 at recursion order 0 to 0.700 at order 4.Deception further reduced Hi-ToM performance; MMToM-QA errors clustered in Bayesian goal-inference questions, and MuMA-ToM errors persisted on social-goal and belief-of-goal labels.Source: §5.10 · p. 15, Figure 12 · p. 16
• Residual errors mark what current scaffolds cannot readily compile away: deep belief tracking under deception and Bayesian goal inference remain difficult, despite top-scaffold fixes covering 97% of baseline errors in union.The deepest recursion and goal-inference failures likely require a stronger explicit belief-tracking mechanism than more variants of the same scaffold design.Source: §5.10 · pp. 15–16
• 83% of baseline-wrong items were repaired on average, while only 7% of baseline-correct items were broken by the strongest scaffolds.This fix/break asymmetry indicates that scaffolding improves performance rather than merely redistributing errors.Source: §5.10 · p. 15, Figure 12 · p. 16

Synthesis and Conclusion

• The mean uplift over the GPT-5.4-mini no-scaffold baseline was +0.275, and all 72 runs and 11 builder configurations exceeded that baseline.The best scaffold reached 0.912 (+0.423), surpassing the Gemini-3.5-flash no-scaffold baseline on multiple tasks and approaching the human-inspired harness reference on the same backbone (0.939), without human ToM-specific engineering.Source: §6 · p. 16
• The procedure was reproducible but not deterministic: mean within-cell standard deviation was 0.036, roughly an order of magnitude smaller than the main uplift.Remaining variance concentrated in deterministic-solver strategies, where one implementation error could substantially affect a benchmark.Source: §6 · p. 16
• Builders used a median of 5 validation evaluations, with a mean validation–full gap of 0.021 and no clear benefit from additional probing (𝑟= 0.17).Builder quality mattered more than validation budget.Source: §6 · p. 16
• Accuracy was strongly associated with the fraction of items answered by deterministic codes, rules, and scaffolds (𝑟= 0.72).Strong scaffolds paired a reliability floor—format enforcement, routing, and greedy decoding—with task-structure exploitation such as polarity logic, structured extraction, and deterministic solving.Source: §6 · p. 16
• Builder identity and reasoning effort were primary drivers of scaffold quality, while platform effects were second-order and conditional.For Opus-4.7, performance often improved monotonically with reasoning effort (Spearman 𝜌= 0.77).Source: §6 · p. 16
• Scaffolding helps most when the target leaves correctable errors on the table; as target strength rises, headroom shrinks and scaffolding must be more selective.The goal is to avoid disturbing or breaking behaviors the stronger target already performs correctly.Source: §6 · p. 16
• Residual failures concentrated in deep belief recursion under deception and Bayesian goal inference, though top scaffolds fixed about 83% of baseline errors.These less-compilable cases resist reduction to explicit rules or skills.Source: §6 · p. 16
• A capable builder can compile task competence into an inference-time scaffold, letting a weaker, cheaper target perform closer to stronger models on structured subproblems.The practical recipe is to use the strongest builder with high reasoning effort, modest validation, cognitive offloading for provable subtasks, and—when budget permits—multiple independent scaffolds selected or ensembled for complementary repairs; residual cases still require model-dependent reasoning.Source: §6 · pp. 16–17

Benchmark selection

• Theory-of-Mind benchmarks combine scaffoldable structure with residual reasoning difficulty, and future work should test scaffolding on broader benchmark families.The discussion highlights families with different mixtures of symbolic structure, ambiguity, and open-ended reasoning.Source: §7 · p. 17

Harness self-evolution

• Automatic scaffold construction can reveal which harness features help and how builders adapt them, informing human harness design and enabling infrastructure to co-evolve with agents.The discussion distinguishes improving a target model from improving the harness that structures its reasoning.Source: §7 · p. 17

Toward scaffolding as a benchmark

• A builder benchmark could test whether a model improves a weak target on a hidden test set using only validation data to refine a scaffold.Possible secondary metrics include validation usage, inference cost, scaffold complexity, code length, robustness across repeats, and work offloaded from the target.Source: §7 · p. 17

Two complementary routes to stronger systems

• Harness design and model improvement are complementary routes: scaffolds reshape inference, and models and harnesses may be optimized together.The paper presents strong-to-weak scaffolding as a step toward studying the co-evolution of models and inference environments.Source: §7 · p. 17

A Instruction Details

• Builders inspect a random 2% validation sample, test and analyze scaffold failures, then recursively refine the harness before final full-set evaluation.The instructions frame scaffold construction as a coding competition and call for strategic validation runs.Source: A Instruction Details · p. 21
• The primary objective is average performance across four benchmarks; validation evaluations are a secondary criterion, and each run costs one.The scaffold should also improve target-model task performance while reducing its cognitive load.Source: A Instruction Details · p. 21
• Builders must record every validation run and maintain a recursive refinement log documenting each change, its rationale, results, lessons, and next step.The performance record includes scores for all four benchmarks, their average, and notes.Source: A Instruction Details · pp. 21–22
• Permitted harness techniques include task routing, answer-format enforcement, few-shot examples, verification, and deterministic preprocessing or postprocessing.Prompt templates and other methods that improve target-model performance are also allowed.Source: A Instruction Details · p. 22
• The instructions prohibit hard-coding validation answers and direct builders to optimize for hidden-set generalization rather than overfitting.Builders should stop when performance plateaus or further changes risk overfitting.Source: A Instruction Details · p. 22
• The final harness must run on a supplied full-set path, save predictions and results, and append benchmark results to performance.csv.The prescribed script uses parallel target-model calls with a maximum of 16 workers and saves final evaluation output in final_eval.Source: A Instruction Details · p. 22

B Complete Per-Run Results

• Table 6 lists all 72 runs by target, builder, platform, and repeat, with factor coordinates, benchmark accuracies, macro-average accuracy, and validation-evaluation counts.“Avg.” is the macro average of the four benchmark accuracies and the primary metric; “Val.” is the number of validation evaluations used.Source: Table 6 · p. 22, B Complete Per-Run Results · p. 22