PaperlayerView exact revision on arXiv

Source-linked AI summary

AI4AI at Test-Time: Strong-to-Weak Capability Transfer via Harnesses

Cheng Qian, Wenting Zhao, Liangwei Yang, Heng Wang, Jielin Qiu, Heng Ji, Silvio Savarese, Huan Wang, Shelby Heinecke

arXiv:2608.12307v1cs.LGcs.AIcs.CL

TL;DR

The paper asks whether strong models can transfer capability to weaker models at test time through inference-time harnesses rather than parameter updates. It builds such harnesses automatically and finds large, reliable gains across Theory-of-Mind benchmarks, nearly doubling average target performance from 0.49 to 0.91.

  • Problem

    Existing work lacks a systematic account of why inference-time harnesses help, when they are stable, and which design choices matter.

  • Method

    A strong builder model iteratively constructs an inference-time scaffold for a fixed weaker target using 5% validation splits, then evaluates it on held-out test data.

  • Results

    0.49 to 0.91: strong-to-weak scaffolding produces large, reliable gains across Theory-of-Mind benchmarks, driven mainly by deterministic offloading, routing, and format enforcement.

  • Takeaways & Limitations

    Inference-time harness design can partially substitute for target-model capability by compiling task structure into deterministic scaffolds, while leaving genuinely difficult reasoning to the target.

  • Takeaways & Limitations

    Scaffolding remains weaker on nested higher-order belief tracking, deception, and Bayesian goal inference, where the target must perform difficult residual reasoning.

Abstract

from arXiv · show

Recent work on distillation transfers the capabilities of large models to smaller ones often by updating the latter's parameters, through teacher forcing, on-policy distillation, and related training-time methods. In this paper, we ask whether such transfer can instead occur at test time. We study strong-to-weak scaffolding: whether a stronger builder model can construct inference-time harnesses that help a weaker target model solve tasks more reliably without any parameter updates. Using four representative Theory-of-Mind benchmarks, each builder model uses 5% of the data as a validation set to iteratively refine its harness over multiple rounds, after which the finalized harness is evaluated on the full test set. Empirically, this form of test-time capability transfer is highly effective, nearly doubling average target-model performance from 0.49 to 0.91. Our analysis shows that the gains come primarily from offloading unstable model reasoning into deterministic code, benchmark-specific routing, and strict answer-format enforcement, rather than from encouraging the target model to reason more extensively or sample more broadly. We further find that builder-model reasoning effort improves harness quality monotonically, platform effects are modest relative to the builder model's own capability, and weaker target models receive the largest gains. These results suggest that inference-time harness design is an important complement to conventional training-time distillation, enabling strong models to transfer cognitive structure to weaker models without retraining.

1 Introduction

The paper introduces strong-to-weak scaffolding as inference-time capability transfer through harnesses rather than target-model weight updates. Across Theory-of-Mind benchmarks, scaffolding substantially improves weaker models by externalizing structure, offloading deterministic reasoning, routing tasks, and enforcing answer formats.

  • Setting: Strong-to-weak scaffolding transfers capability through a harness that shapes a fixed weak model’s inputs, reasoning, and outputs, without changing its parameters.The builder constructs task routing, prompts, solvers, exemplars, verification, or format enforcement without observing the full test set.
  • Empirical results: 0.42 absolute gain: the best scaffold raises GPT-5.4-mini macro-average accuracy from 0.49 to 0.91, while every builder configuration yields positive net uplift.The improvement is large and robust across builder configurations.
  • Mechanisms: Scaffolding gains depend more on structure externalization, deterministic offloading, and strict format enforcement than on more validation data, broader sampling, or longer reasoning.The paper identifies benchmark-aware routing and targeted decomposition as actionable design principles.
  • Benchmark dependence: On BigToM, scaffolds can convert structured reasoning into compilable skills and rules, whereas other benchmarks primarily benefit from reduced reasoning burden.Residual errors remain concentrated in higher-order Hi-ToM cases with recursion depth at least two and Bayesian goal-inference subtypes.
  • Implications: The framework treats harness construction as both a deployment strategy for improving weaker or cheaper models without retraining and an evaluation of stronger models’ ability to create solving conditions.This reframes evaluation from solving tasks directly to constructing conditions under which weaker models solve them effectively.

2 Related Work

The related work situates this paper at the intersection of training-based capability transfer, inference-time reasoning strategies, programmatic tool use, and harness engineering. Theory-of-Mind benchmarks provide the motivating evaluation setting because they require tracking hidden information and nested mental states.

  • Capability transfer and distillation: Training-based distillation transfers larger models’ capabilities to smaller models by matching teacher outputs or transferring rationales, traces, and task-specific reasoning.This paper contrasts those parameter-updating approaches with test-time transfer through harnesses.
  • Inference-time reasoning, prompting, and decomposition: Inference-time methods improve reasoning by eliciting intermediate steps, sampling multiple paths, or decomposing difficult problems into sequential subproblems.Examples include chain-of-thought prompting, self-consistency, and least-to-most prompting.
  • Tool use, programmatic reasoning, and deterministic offloading: Programmatic reasoning and tool-use methods address execution, verification, and state-tracking failures by combining language-model reasoning with external APIs, environment actions, or executable programs.The passage identifies Toolformer, ReAct, PAL, and Program-of-Thoughts as related approaches.
  • Harness engineering and automated scaffold construction: Harness engineering treats prompts, tools, memory, context, interfaces, routing, and validation as optimizable components surrounding an LLM.DSPy compiles language-model pipelines against task metrics, while SWE-agent demonstrates that the agent-computer interface can affect coding-agent performance.
  • Theory-of-Mind evaluation and mental-state scaffolds: Theory-of-Mind benchmarks test scaffolding systems on observations, beliefs, intentions, hidden information, and nested perspectives.BigToM and Hi-ToM represent related benchmarks emphasizing social reasoning and higher-order recursive belief reasoning.

3 Method

The method has a strong builder model construct an inference-time scaffold for a fixed weaker target using 5% validation data, then evaluate the exported scaffold on a hidden test split. Scaffolds may combine prompting, routing, deterministic processing, format enforcement, verification, retrieval, or symbolic solving, with validation performance serving as the selection signal.

  • Data split: 5% of each benchmark is randomly sampled for validation, while the remaining examples form a hidden test split for separate evaluation.The builder uses the validation subset during construction, and the hidden test set is reserved for post-build assessment.
  • Builder setup: The builder accesses only the validation set while constructing the scaffold inside an existing agentic coding harness.The target model remains fixed, and the test split is held out throughout scaffold construction.
  • Scaffold design: The scaffold architecture is unconstrained and may include prompt templates, benchmark routing, deterministic processing, answer-format enforcement, verification, retrieval, or symbolic solvers.The method requires only an inference-time procedure that uses the specified task instructions, target-model demonstrations, and labeled validation data.
  • Evaluation: Validation performance acts as a proxy for selecting reusable task structure because the builder cannot access the hidden test split.The finalized scaffold is exported and run on the hidden test set by a human evaluator without further builder intervention.

4 Experimental Setup

The experiments evaluate strong-to-weak scaffolding on a 3,900-item hidden test set spanning four Theory-of-Mind benchmarks, using fixed validation data and macro-average accuracy as the primary metric. They vary platforms, builder and target models, reasoning effort, and repetitions, while comparing scaffolded targets with vanilla and human-inspired harness baselines.

  • Task and metric: The benchmark combines four ToM datasets into a 3,900-item hidden test set: BigToM, Hi-ToM, MMToM-QA, and MuMA-Tom.The datasets contain 1,200, 1,200, 600, and 900 data points, respectively, with binary, nested-belief, Bayesian, and 3-choice social-inference questions.
  • Task and metric: Each builder receives a fixed-seed 195-item validation sample, while the primary metric is the unweighted macro average of full-set accuracies across four benchmarks.Validation evaluation uses are tracked secondarily for efficiency, and only text-format questions are used.
  • Experiment design: The study varies platform, builder model and reasoning effort, target model, and three repeats, yielding 72 experiment runs.GPT-5.4-mini is the dominant target in the main setting, while Gemini-3.5-flash is used to contrast target-model effects.
  • Baselines: Scaffolded targets are compared against direct vanilla prompting and the human-designed UserHarness reference.Vanilla uses the same naive prompt without task-specific scaffolding; UserHarness provides a human-inspired harness for ToM problems.
  • Baselines: 0.488 is the vanilla macro-average accuracy for GPT-5.4-mini, compared with 0.761 for Gemini-3.5-flash.These values establish the direct-prompting performance expected to improve through scaffolding.
  • Baselines: 0.939 is the human-inspired-harness macro-average accuracy for GPT-5.4-mini, compared with 0.941 for Gemini-3.5-flash.UserHarness serves as a human-designed reference point for harness effectiveness.

5 Results and Analysis … 5.4 Aspect 3: Scaffolding Techniques

Strong-to-weak scaffolding substantially improves GPT-5.4-mini through reliable, builder-designed harnesses, while validation refinement and technique analysis indicate that disciplined task engineering matters more than extensive inference. The gains are robust but imperfectly reproducible, with builder capability dominating platform effects and deterministic task-specific mechanisms driving the strongest scaffolds.

  • 5 Results and Analysis: Builder identity is the dominant variation source: builders form a clear vertical ordering, whereas platform differences within a builder are comparatively small.This pattern recurs throughout the analysis, and every evaluated builder–platform configuration substantially exceeds the GPT-5.4-mini no-scaffold baseline.
  • 5.1 Aspect 0: Main Results: 0.763 mean macro-average accuracy across 57 scaffolded runs represents a +0.275 uplift over GPT-5.4-mini’s 0.488 no-scaffold baseline, with 100% of runs exceeding it.The best individual run, produced by GPT-5.5 on GPT Codex, reaches 0.912.
  • 5.1 Aspect 0: Main Results: The best automatically built scaffold outperforms raw no-scaffold GPT-5.4 and GPT-OSS-120B on all four benchmarks, while matching UserHarness on structured BigToM.Performance relative to UserHarness remains task-dependent.
  • 5.2 Aspect 1: Run-to-Run Stability: 0.036 mean macro-average standard deviation is roughly an order of magnitude below the +0.275 mean uplift, although the widest setting has a 0.201 repeat range.Reproducibility is therefore strong but not perfect; deterministic-solver strategies can amplify instability when benchmark-specific rules contain errors.
  • 5.3 Aspect 2: Refinement on Validation: 4.9 mean validation passes, with median 5 and range 2–15, accompany a 0.216 mean increase from first logged to best logged validation accuracy.Best validation accuracy strongly predicts final full-set performance, whereas refinement-iteration count is largely unrelated, with correlations r=0.96 and r=0.17 respectively.
  • 5.3 Aspect 2: Refinement on Validation: A 5% validation slice provides a faithful hidden-set proxy, while the flat relationship between validation budget and final performance suggests hypothesis quality, not feedback quantity, is limiting.The small optimism gap supports refinement without substantial overfitting.
  • 5.4 Aspect 3: Scaffolding Techniques: Robust format enforcement and temperature-0 greedy decoding are nearly universal, followed by benchmark routing and forced chain-of-thought; deterministic solvers, voting, verification, and few-shot prompting are minority or rare techniques.These common controls address malformed outputs, format confusion, and sampling variance, while deeper task analysis differentiates stronger scaffolds.

5.5 Aspect 4: Harness Platform’s Impact · 5.6 Aspect 5: Analysis Across Target Models

Platform effects are small and conditional, while target-model headroom largely determines scaffolding gains. Weaker targets benefit more, but strong targets can experience benchmark-specific regressions when additional scaffolding overcomplicates already reliable behavior.

  • 5.5 Aspect 4: Harness Platform’s Impact: Native platforms change macro accuracy by only +0.013 on average across eight matched configurations, with no statistically reliable general native advantage.The native platform wins in 5 of 8 cells, with paired permutation test p=0.484.
  • 5.5 Aspect 4: Harness Platform’s Impact: For Opus-4.7, Claude Code trails Cursor at low effort (−0.034) but leads at medium (+0.045), high (+0.038), and extra-high (+0.032) effort.This indicates a platform×effort interaction rather than a uniform native-platform effect.
  • 5.5 Aspect 4: Harness Platform’s Impact: Claude Code’s pooled mean is 0.799 versus 0.754 for GPT Codex and 0.750 for Cursor, primarily reflecting different builder rosters rather than causal platform quality.The platforms host K=11, 5, and 3 builders, respectively, so marginal averages conflate platform effects with builder selection.
  • 5.6 Aspect 5: Analysis Across Target Models: Scaffolding improves GPT-5.4-mini by +0.262 (0.488 →0.750) but Gemini-3.5-flash by only +0.110 (0.761 →0.871), consistently favoring the weaker target.The comparison uses Cursor and matched builder runs across both targets.
  • 5.6 Aspect 5: Analysis Across Target Models: Across builder×benchmark×target settings, uplift correlates strongly with available headroom, 1 −baseline, with Pearson r=0.75.The finding frames scaffolding as recovery of latent competence that targets fail to deploy reliably.
  • 5.6 Aspect 5: Analysis Across Target Models: Gemini-3.5-flash gains are concentrated on BigToM, which accounts for 96% of its macro uplift (0.42), whereas GPT-5.4-mini gains span all four benchmarks.BigToM combines meaningful remaining headroom with structure that is readily compiled into rules.
  • 5.6 Aspect 5: Analysis Across Target Models: Builders use less deterministic machinery for Gemini-3.5-flash, with model-only handling rising on every task and most sharply on MuMA-ToM (40% →73%).They reserve heavier rule-based interventions for sub-tasks where the stronger target retains headroom.
  • 5.6 Aspect 5: Analysis Across Target Models: Strong-target scaffolding can backfire: Gemini-3.5-flash regresses in 9/20 cases, including −0.04 on average for Hi-ToM and −0.02 for MuMA-ToM.GPT-5.4-mini regresses in 0/20 matched cases, illustrating the risk of over-scaffolding near performance ceilings.

5.7 Aspect 6: Builder Reasoning Effort · 5.8 Aspect 7: Attribution of Why Does Improvement Happen?

Builder reasoning effort monotonically improves scaffold quality, while attribution analyses show that gains chiefly arise from compiling task structure and enforcing reliable execution. Stronger builders and complementary scaffold designs increase uplift, but even self-scaffolding helps substantially.

  • 5.7 Aspect 6: Builder Reasoning Effort: 5.7 Aspect 6: Builder Reasoning Effort — Controlled sweeps isolate builder deliberation across four effort tiers on Cursor and Claude Code.Opus-4.7 builds for GPT-5.4-mini while model identity and platform are held constant.
  • 5.7 Aspect 6: Builder Reasoning Effort: 5.7 Aspect 6: Builder Reasoning Effort — 0.711 at low effort rises to 0.856 at extra-high effort, with monotonic gains across platforms and Spearman ρ=0.77.Intermediate pooled performance is 0.793 and 0.807 at medium and high effort, respectively.
  • 5.7 Aspect 6: Builder Reasoning Effort: 5.7 Aspect 6: Builder Reasoning Effort — More deliberation strengthens scaffolds without evidence of over-engineering, whereas additional validation evaluations did not predict final quality.The useful compute is deeper hypothesis formation about task structure rather than more probing.
  • 5.7 Aspect 6: Builder Reasoning Effort: 5.7 Aspect 6: Builder Reasoning Effort — Gains diminish but remain positive: moderate effort discovers routing, format enforcement, and deterministic solving, while higher effort adds polarity logic and belief-state extraction.Scaffold size grows from ∼510–650 LOC at low effort to ∼1000–1300 at extra-high effort.
  • 5.8 Aspect 7: Attribution of Why Does Improvement Happen?: 5.8 Aspect 7: Attribution of Why Does Improvement Happen? — The strongest technique associations are polarity/negation logic (+0.09), structured extraction (+0.06), hybrid fallback (+0.04), and deterministic solving.These techniques compile task structure into the scaffold rather than acting as generic prompting tricks.
  • 5.8 Aspect 7: Attribution of Why Does Improvement Happen?: 5.8 Aspect 7: Attribution of Why Does Improvement Happen? — 1717 baseline errors are fixed versus 105 previously correct items broken, producing an overwhelmingly significant gain over 3900 items with χ2≫104 and p<10^-4.The paired McNemar result indicates that aggregate uplift is not driven by noise or a small error redistribution.
  • 5.8 Aspect 7: Attribution of Why Does Improvement Happen?: 5.8 Aspect 7: Attribution of Why Does Improvement Happen? — Self-scaffolding improves performance by +0.17 to +0.22, stronger builders achieve larger gains, and the union of top scaffolds covers 97% of baseline errors.Different builders capture partly distinct ToM skills, while format enforcement, routing, and greedy decoding provide a shared reliability floor.

5.9 Aspect 8: Cognitive-Load Reduction · 5.10 Aspect 9: Remaining Error Analysis

Cognitive-load reduction is central to strong-to-weak scaffolding: deterministic offloading and structured prompts improve target performance, with benchmark-dependent compilability. Remaining errors are concentrated in difficult, poorly compilable reasoning cases, although the strongest scaffolds repair far more baseline mistakes than they introduce.

  • 5.9 Aspect 8: Cognitive-Load Reduction: Deterministic offloading reduces cognitive load by answering items directly, while structured prompts constrain inputs, formats, and reasoning when target-model calls remain necessary.These are the two mechanisms identified as the mechanistic core of strong-to-weak scaffolding.
  • 5.9 Aspect 8: Cognitive-Load Reduction: Offloadability varies sharply by benchmark: BigToM is almost fully compilable into code, whereas MuMA-ToM remains substantially model-dependent.Figure 11(b) reports mean determinism around 0.94 for BigToM.
  • 5.10 Aspect 9: Remaining Error Analysis: The 8 strongest GPT-5.4-mini scaffolds repair 83% of baseline-wrong items on average while breaking 7% of baseline-correct items.The analysis pools the best repeat from each platform×builder setting with mean accuracy above 0.80.
  • 5.10 Aspect 9: Remaining Error Analysis: BigToM is essentially solved, reaching accuracy at least 0.95 on every fine-grained metadata slice.Residual errors instead cluster in harder benchmark regions.
  • 5.10 Aspect 9: Remaining Error Analysis: Hi-ToM accuracy declines with recursion depth, falling from 0.999 at order 0 to 0.700 at order 4, with deception further reducing performance.The scaffold advantage narrows on the deepest cases.
  • 5.10 Aspect 9: Remaining Error Analysis: Residual errors mark the boundary of compilation: scaffolds struggle with nested higher-order belief tracking under deception and Bayesian goal inference from ambiguous action traces.These cases require leaving more reasoning to GPT-5.4-mini; combining diverse scaffolds may help, but deeper errors likely need explicit belief tracking.

6 Synthesis and Conclusion

Across 72 runs, strong-to-weak scaffolding produced large, reliable, and mechanistically interpretable gains. The results frame scaffolding as one-time competence compilation by a capable builder, with strongest benefits for structured sub-problems whose procedures can be made explicit.

  • Synthesis and Conclusion: The mean uplift over the GPT-5.4-mini no-scaffold baseline is +0.275, with all 72 runs and 11 builder configurations exceeding the baseline.The best scaffold reaches 0.912 (+0.423).
  • Synthesis and Conclusion: A capable builder acts as a compiler of task competence, encoding task structure into an inference-time scaffold that a weaker target model can execute.This substitution is strongest for structured sub-problems whose decision procedures can be made explicit.
  • Synthesis and Conclusion: The practical recipe is to use the strongest available builder, allocate high reasoning effort, use modest validation, offload provable sub-tasks, and combine independent scaffolds when budget permits.Selection or ensembling can capture complementary repairs.

7 Discussion and Future Work

The discussion frames strong-to-weak scaffolding as a complementary route to improving systems: reshaping inference can partially substitute for target-model capability, while harness self-evolution and standardized evaluation offer promising directions.

  • Benchmark selection: Theory-of-Mind benchmarks serve as a representative testbed because they are well studied, contain diverse question types, and span a wide range of difficulty.Their mixture includes genuinely hard reasoning items and items with exploitable regularities that can reduce the target model’s cognitive load.
  • Harness self-evolution: Models can automatically improve the harness around a target model through tools, prompts, workflows, and evaluation loops, rather than answering within a fixed harness.This setting provides an empirical lens for studying harness self-evolution in modern agentic coding environments.
  • Toward scaffolding as a benchmark: Strong-to-weak scaffolding could become a benchmark that evaluates builders on refining a scaffold with validation data and scoring it on a hidden full test set.The proposed setup includes a workspace, weak target model, fixed downstream task, validation set, and hidden test set.
  • Two complementary routes to stronger systems: Harness design offers a second route to stronger systems by making tasks easier to execute rather than improving the model’s internal capability.The paper positions post-training primarily on the first route and harness or scaffold design on the second.

Appendix · A Instruction Details

The appendix instructs the builder to iteratively develop and validate a scaffold that improves downstream benchmark performance while reducing target-model cognitive load. It also specifies evaluation records, refinement logs, generalization constraints, and a reproducible full-set execution procedure.

  • A Instruction Details: The builder must inspect validation cases, design and test a scaffold, analyze failures, and recursively refine it before final full-set evaluation.The instructions frame scaffold construction as an iterative process using a small random validation sample.
  • A Instruction Details: The primary grading criterion is average performance across four benchmarks, while validation-evaluation count is secondary and each validation run costs 1.Builders are told to avoid repeatedly testing small or unprincipled changes.
  • A Instruction Details: The scaffold should improve target-model task performance and reduce its cognitive load.
  • A Instruction Details: Each validation iteration must record the scaffold change, rationale, validation result, learned lesson, and next planned improvement.The performance record includes four benchmark scores, an average, run metadata, and notes.
  • A Instruction Details: The workflow proceeds from workspace inspection and format understanding through baseline construction, meaningful validation, recursive failure analysis, plateau-based stopping, and final-script preparation.The instructions specifically prioritize meaningful scaffold changes and stopping when further changes risk overfitting.
  • A Instruction Details: Permitted scaffold mechanisms include prompt templates, task routing, answer-format enforcement, few-shot examples, verification, and deterministic preprocessing or postprocessing.The scaffold must avoid hard-coding validation answers and optimize for generalization to the hidden full set.
  • A Instruction Details: The final bash harness must accept a full-set path, run the scaffold, save predictions and results in the workspace, append benchmark results to performance.csv, parallelize target-model calls with max worker equals 16, and write final outputs to final_eval.The evaluator is intended to insert or pass only the full-set path, with one final hidden-set evaluation chance.

B Complete Per-Run Results

This section presents the complete set of 72 experimental runs, organized by factor coordinates and reporting benchmark-level, macro-average, and validation-evaluation results. The macro average across four benchmarks is the primary metric.

  • Complete Per-Run Results: 72 runs are listed with factor coordinates, per-benchmark accuracies, macro-average accuracies, and validation-evaluation counts.The runs are sorted by target, builder, platform, and repeat.
  • Complete Per-Run Results: “Avg.” is the macro average of four benchmark accuracies and serves as the primary metric, while “Val.” counts validation evaluations used by the builder.These definitions specify how the table’s aggregate performance and validation-budget columns should be interpreted.
Loading 2608.12307v1…