Source-linked AI summary
Learning Meta-Skills for Agent Harness Design in Test-Time AI4AI
Cheng Qian, Kunlun Zhu, Beibin Li, Zhenhailong Wang, Heng Ji
TL;DR
This paper asks how a Builder can learn to construct better execution environments for a Target while both models’ weights remain fixed. It learns reusable meta-skills from Target feedback and uses a frozen bank to build harnesses for unseen tasks. Across Harness-Bench and NewtonBench, full-bank meta-skills improve macro-average performance over both no-skill construction and direct delivery of the same bank to the Target.
Problem
Method
The Builder learns meta-skills from Target execution feedback and uses the frozen principles to construct task-specific harnesses while both models’ weights remain fixed.
Source: §1 · p. 1Results
8.95 percentage points over no-skill construction and 12.02 points over direct delivery of the same bank to the Target: full-bank meta-skills improve macro-average performance across Harness-Bench and NewtonBench.
Source: §6 · p. 12Takeaways & Limitations
The comparisons highlight the value of translating experience into executable support, and same-model studies suggest that learning to build better harnesses can improve a model’s own test-time execution.
Source: §1 · p. 2, §6 · p. 12Evidence spans one main Builder, two benchmarks, and one adopted execution per task and condition; broader applicability and gains relative to construction cost remain to be evaluated.
Source: §6 · p. 12
Abstract
from arXiv · showhide
Agent performance depends on both reasoning ability and the environment in which it acts. We study test-time AI-for-AI, asking how a Builder can learn to construct better execution environments for a Target while both models' weights remain fixed. To make the Builder's experience reusable, we introduce Meta-Skill: principles specifying when support is needed and what resources to provide. The Builder learns these principles from Target's execution feedback on the development set, then uses the frozen skill bank to construct harnesses for unseen tasks. Across Harness-Bench and NewtonBench, full-bank meta-skills improve macro-average performance by 8.95 percentage points over no-skill construction, and 12.02 points over direct delivery of the same bank to the Target. These results highlight the value of translating experience into executable support. Gains when the same model serves both roles further suggest a path to system level self-improvement through learning to build better environments.
1. Introduction
The paper asks how a Builder can turn Target execution outcomes into reusable support for future tasks, rather than teaching the Target directly. It introduces meta-skills for harness design and reports gains across two benchmarks, including when the same model serves as Builder and Target.
1. Introduction
- Agent performance depends on both reasoning ability and the environment in which the agent acts [11], motivating support that helps existing capabilities become more effective [3] [13].The paper frames this as a complementary direction for improving agents.Source: §1 · p. 1
- Meta-skills specify when support is needed, what capability or resource to provide, and how the Target should use it while retaining responsibility for judgment.They distinguish the Builder's support-design principles from the Target's task skills [9].Source: §1 · p. 1
- The Builder reviews Target execution records and scores on development tasks, revises a skill bank, then freezes it to guide harnesses for held-out tasks while both models’ weights remain fixed.This makes the Builder's experience reusable without updating either model's weights.Source: §1 · p. 2
- 65.31% macro-average score: the full-bank meta-skill Builder exceeds no-skill construction by 8.95 percentage points and full-bank independently learned Target skills by 10.93 points.It also outperforms direct delivery of the same bank to the Target in all six settings.Source: §1 · p. 2
- Across three same-model settings, meta-skills improve scores by 18.71 points on average over no-skill construction.The paper presents this as a possible path to system-level self-improvement through better harness construction.Source: §1 · p. 2
2. Related Work
Related work covers AI systems that design agent workflows and harnesses, reusable task skills, and meta-level learning. This paper places its contribution at the Builder level: learning principles from harness outcomes and implementing them as task-specific support for a separate Target.
AI for AI in Agentic Systems
- Test-time agent-design work searches agent programs, workflows, or harnesses [1] [3] [15]; this paper learns reusable construction principles from Target executions and freezes them for test-time harness construction.Strong-to-weak harness construction uses a stronger Builder to supply executable support for a weaker Target [5].Source: §2 · p. 3
Agent Skills and Meta-Skills
- Agent-skill methods preserve procedural knowledge or textual insights for reuse [7] [16] [17], while skill-improvement methods co-evolve skills and policies or compile experience into solver skills [9] [10].SkillsBench measures skill utility using curated packages and deterministic verifiers [4].Source: §2 · p. 3
- At the meta level, prior work learns skills for constructing context files and code or evolves policies for improving task skills [8] [13]; here, the Builder learns principles from its own harness outcomes and implements support for a separate Target.The principles specify when support is needed, what capabilities to supply, and which responsibilities remain with the Target.Source: §2 · p. 3
3. Problem Setting and Method
The method treats harness design as a policy that augments a baseline environment to improve expected test performance. A Builder learns reusable support principles from development executions, freezes the bank, and uses it to construct fresh task-specific harnesses for test tasks.
3. Problem Setting and Method
- A harness policy maps each public task input to an environment that augments the baseline environment with executable support, with the objective of maximizing expected test performance.The Builder learns on development instances and is evaluated on held-out test instances; the Target execution budget excludes Builder construction compute.Source: §3 · p. 3
3.1. Learning Meta-Skills from Target Execution During Development
- A meta-skill records when observable conditions call for support, what capability or resource the environment should provide, and how the Target should use it while retaining responsibility for judgment.The Builder begins with an empty bank, reviews generated harnesses and execution feedback, and adds or revises at most one evidence-backed skill per batch, or leaves the bank unchanged.Source: §3.1 · pp. 3–4
3.2. Building New Harness for Every Task at Test-Time
- After learning, the Builder freezes the bank and constructs a new harness in a fresh environment for each test task, using either the full bank or retrieval of at most two positively relevant skills.The harness refinement space includes instructions, memory, context organization, composed tools, execution control, verification and recovery, and workspace preparation; benchmark tools, budgets, and scoring remain fixed.Source: §3.2 · pp. 4–5, Table 1 · p. 4
- The Target operates in the Builder-created harness, while the Builder works within a designed framework with shared interfaces, execution isolation, guidance, and resource limits.For example, a meta-skill can call for version tracking and revalidation when an artifact is edited after validation, implemented through file-version memory and a validation tool.Source: §3.2 · p. 5
4. Experiments
Across Harness-Bench and NewtonBench, the full meta-skill bank improves Builder performance over no-skill construction and direct delivery to the Target. The experiments also compare full-bank access with sparse retrieval and examine how gains vary across benchmarks.
4.1. Experimental setup
- Harness-Bench covers agent workflows and NewtonBench interactive scientific law discovery; 10% of tasks are reserved for learning, with the remainder used for evaluation.GPT-5.6-Sol is the Builder, and the bank is updated over two development-set passes before being frozen for test-task harness construction.Source: §4.1 · p. 5
- Harness-Bench allows 30 turns, 30 tool calls, and 96K cumulative tokens per task; NewtonBench allows 12 turns, 10 tool calls, and 192K tokens.Harness-Bench uses a deterministic completion-oracle score over 95 test tasks, while NewtonBench uses symbolic-structure accuracy over 292; both apply end-to-end evaluation.Source: §4.1 · p. 5
4.2. Main results
- 8.95 points is the full-bank Builder’s average gain over the no-skill Builder; giving the bank to the Builder instead of directly to the Target improves all six settings by 12.02 points on average.Table 3 reports test performance across tasks and settings; the Builder translates meta-skills into executable support rather than relying on the Target to apply advice at inference time.Source: Table 3 · p. 6, §4.2 · pp. 6–7
4.1. Experimental setup
- Five baselines compare native solving, empty-bank construction, direct Builder skills, independently learned structured Target skills, and mined free-form Target skills.The no-skill baseline controls for construction capability, while direct delivery of the full bank controls for semantic knowledge; Builder conditions compile guidance into task-specific harness components.Source: §4.1 · pp. 5–6
4.2. Main results
- NewtonBench gains average 10.96 points across the three Targets, versus 6.95 points on Harness-Bench, where workflows are more heterogeneous.The paper associates NewtonBench’s recurring coordination failures with scaffolding opportunities, while Harness-Bench support depends more on Target-specific planning and tool use.Source: §4.2 · p. 7
- The full skill bank beats BM25 top-2 retrieval in five of six settings, with an average gain of 7.19 points on NewtonBench.The paper notes one exception, so broader access is not uniformly beneficial.Source: §4.2 · p. 7
5. Analysis
The analyses show that meta-skill refinement can yield delayed gains but is not monotonic, while same-model improvement and transfer depend on how Builders implement support. Component and outcome analyses connect performance to execution control, selective state exposure, and task-specific effects.
5.1. Effect of Meta-skill Refinement
- Across settings, zero-to-two refinement passes average 10.42 points; on NewtonBench, the second pass adds 13.01 points for Gemini and 12.33 for Qwen after first-pass changes below 1.1 points.Manual inspection suggests early updates gather local observations, while later revision connects them into reusable interventions.Source: §5.1 · pp. 7–8
5.2. Self-Improvement through Harness Design
- Across three same-model settings, Builder meta-skills yield average gains of 18.71 points over no-skill construction and 14.14 over skills delivered directly to the Target.The comparison identifies support design as a distinct route to self-improvement without weight updates or a stronger teacher.Source: Figure 3 · p. 8, §5.2 · p. 8
5.1. Effect of Meta-skill Refinement
- Refinement is not monotonic: Qwen drops 4.06 points from its first-pass peak on Harness-Bench, motivating development-only confidence checks and possible rollback.Gemini improves with each pass on Harness-Bench.Source: §5.1 · p. 8
5.3. Meta-Skill Transfer across Builders and Targets
- Cross-Builder transfer varied: the Sol-to-Qwen bank gained 13.36 points with Sol, fell 4.45 on NewtonBench with Gemini-Pro, and gained 2.72 on Harness-Bench; all confidence intervals include zero.The differing outcomes suggest that transfer partly depends on how the receiving Builder translates shared guidance into a harness.Source: §5.3 · p. 9
- Gemini-Pro’s own bank beat its imported Sol-to-Qwen bank by 3.69 points on Harness-Bench and 9.93 on NewtonBench; cross-Target transfer still gained 4.58 and 5.48 points, respectively.On NewtonBench, the cross-Target bank scored 0.68 points below Gemini-Pro’s own Qwen-derived bank but outperformed the imported Sol-to-Qwen bank.Source: §5.3 · p. 9
5.4. Harness Component Ablations
- Removing the controller lowers scores by 13.36 points for Gemini, 4.79 for Qwen, and 1.03 for GPT-OSS, though only Gemini’s confidence interval excludes zero.Controller functions may differ across Targets, so external coordination should be tailored to execution needs.Source: Table 4 · p. 10, §5.4 · p. 10
- Retaining memory plus context has small positive estimated effects for Gemini and Qwen and a negative effect for GPT-OSS, but all three confidence intervals contain zero.The authors suggest selective state exposure to preserve relevant evidence and avoid redundant or outdated information.Source: §5.4 · p. 10
5.5. Outcome Transitions and Error Analysis
- On NewtonBench, no-valid-submission cases fell from 394 to 254 while correct discoveries rose from 439 to 535; Harness-Bench full successes rose from 59 to 67 amid task-dependent effects.NewtonBench transitions included 145 recoveries from no valid submission to a correct law versus 46 regressions; Harness-Bench gains occurred for every Target on SRE/release tasks, while GPT-OSS declined 2.23 points on vertical workflows.Source: §5.5 · pp. 10–11, Figure 4 · p. 11
6. Discussion and Conclusion
The discussion frames Builder learning as a way to translate experience into executable support and as a complementary axis of agent improvement. The authors also delimit the evidence and identify broader evaluation needs.
6. Discussion and Conclusion
- Builder enactment gains 12.02 points over direct delivery on average, suggesting experience becomes useful through decisions about which burdens to externalize as state, tools, or control.Meta-skills guide those support-allocation decisions so Targets can devote limited interaction budgets to remaining judgment.Source: §6 · p. 11
- Observed transfer gains across Builders and Targets suggest support principles can generalize, while variable effects and stronger recipient-learned-bank performance motivate adapting transferred guidance.The authors propose preserving a skill’s support intent while using recipient-side feedback to refine stored information, tools, and timing.Source: §6 · p. 11
- Evidence spans one main Builder, two benchmarks, and one adopted execution per task and condition; broader applicability and performance relative to construction cost remain to be evaluated.The authors identify more diverse Builders and tasks as necessary for establishing broader applicability.Source: §6 · p. 12
- Full-bank meta-skills improve macro-average performance by 8.95 percentage points over no-skill construction and 12.02 points over direct delivery of the same bank to the Target.The conclusion connects these comparisons to translating experience into tools, state, and control mechanisms that help the Target apply its capabilities during execution.Source: §6 · p. 12
Ethics Statement
The ethics statement places automated harness design in existing agent-workflow and simulated scientific-discovery benchmarks and calls for safeguards beyond those settings. The appendix describes Builder learning as reusable support design whose guidance is implemented in fresh harnesses.
Ethics Statement
- Experiments use isolated execution environments, bounded resources, separated development and test tasks, and frozen meta-skill banks before evaluation.For applications beyond these benchmarks, the authors call for restricted permissions, validation of generated components, and application-appropriate human oversight.Source: Ethics Statement · p. 12
A.1. Why Builder Learning Matters
- Meta-skills specify when support is needed, what to provide, and how the Target should use it, guiding fresh harness construction on subsequent tasks.The Builder carries design knowledge forward while adapting implementation to each task.Source: §A.1 · p. 14
- Giving the full skill bank to the Builder yields higher scores than giving it directly to the Target in all six main-study settings.The authors connect this result to translating guidance into tools, state, and execution control; same-model experiments illustrate self-improvement with unchanged weights.Source: §A.1 · p. 14
A.2. Clarifications of Results
The clarifications distinguish reusable support-design principles from Target task skills and explain how controlled comparisons assess learned Builder experience. They bound the conclusions to the evaluated tasks, models, banks, and execution settings while identifying broader transfer and longer learning histories as future directions.
A.2. Clarifications of Results
- The contribution is a protocol in which the Builder extracts support-design principles from harness outcomes and uses them to construct fresh harnesses for held-out tasks.This differs from learning solver guidance, searching harness implementations, or evolving context and code strategies.Source: §A.2 · p. 14
- Full-bank macro-average gains are 8.95 points over no-skill construction and 12.02 points over direct delivery, supporting learned and implemented support principles within the shared framework.Target execution limits are fixed, and construction allowances are matched across Builder conditions.Source: §A.2 · p. 14
- The evaluation freezes banks learned from development episodes before testing, and documents partitions, information access, model choices, limits, failure accounting, and paired task-bootstrap procedures.All declared test tasks remain in the denominator; code and prompts are slated for release upon acceptance.Source: §A.2 · pp. 14–15
- Banks learned from approximately 10% of each benchmark improve over no-skill construction in all six settings on held-out instances within known categories and mechanisms.The evaluated domains include workflow execution and scientific discovery; cross-dataset reuse remains future work.Source: §A.2 · p. 15
- Cross-Builder-and-Target transfer has positive point estimates on both benchmarks, but realized utility depends on the receiving configuration and is qualified by uncertainty.Recipient-side learning is presented as a complementary way to align principles with a Builder’s implementations.Source: §A.2 · p. 15
- A two-pass, evidence-grounded keep/revise/add procedure yields useful held-out support principles, and same-model experiments extend the result to self-improvement without a stronger external teacher.Refinement effects vary with Target and task, motivating selective revision and retention.Source: §A.2 · p. 15
A.3. Scope and Future Directions
- The study concerns system-level performance improvements from changing support around fixed models, with findings conditional on evaluated task inventories, models, banks, and recorded executions.The authors suggest combining Builder learning with improvements to the Target as a complementary direction.Source: §A.3 · pp. 15–16
- Generalization evidence covers new instances in known workflow categories and physical mechanisms; reuse across datasets, unseen task families, and tool interfaces remains future work.The authors hypothesize that separating reusable support needs from adaptable implementations could clarify when direct transfer or recipient-side adaptation helps.Source: §A.3 · p. 16
B.1. Data partitions and information access
The protocol separates development-based skill learning from held-out testing and restricts construction to information available before Target execution. It also specifies shared resource limits, fixed-denominator failure accounting, and the scope of uncertainty and component comparisons.
B.1. Data partitions and information access
- Approximately 10% of each benchmark is reserved for skill learning, with proportional allocation across categories and domains; the frozen bank is not revised using test outcomes.There is no separate validation partition.Source: §B.1 · p. 16
- Each skill has when, provide, and use fields and a 192-token combined limit; updates use execution evidence to keep, revise, or add one skill, without private answers or researcher diagnoses.Superseded skill versions remain archived.Source: §B.1 · pp. 16–17
- At deployment, BM25 uses the initial public task prompt to retrieve at most two positive-scoring skills, and the Builder sees no test trajectories, scores, reference answers, or evaluator internals.Harness construction uses the public task, neutral component interfaces, and assigned skill pack.Source: §B.1 · p. 17
B.2. Models and resource limits
- GPT-5.6-Sol serves as Builder, with Gemini-3.6-Flash, Qwen3.8-Flash, and GPT-OSS-120B as Targets; construction allows 16K output tokens and two repairs, while skill updates allow 8K tokens.Target limits are identical across settings; NewtonBench additionally caps an experiment call at 20 queried items.Source: §B.2 · p. 17, Table 5 · p. 17
B.3. Outcome and uncertainty accounting
- Every declared test task stays in the fixed denominator, and runs without a valid result count as zero; paired 20K-task bootstrap intervals describe variation across the fixed test inventory.The intervals do not represent variability across repeated stochastic model generations.Source: §B.3 · p. 17
- Frozen-harness ablations remove a component without rebuilding other support, while transfer comparisons use the concurrent no-skill control and cross-campaign differences are descriptive only.Ablation contrasts are conditional on one official execution per task.Source: §B.3 · pp. 17–18
C. Baseline configurations and interpretation
The baselines distinguish learned support implemented by a Builder from native task solving, construction without learned skills, and direct delivery of Target-oriented advice. The section also summarizes retrieved-versus-full-bank access and introduces selected case studies as qualitative illustrations.
C. Baseline configurations and interpretation
- Retrieved access uses the same learned content as full-bank access but supplies at most two positive BM25 matches, enabling a comparison of selective and comprehensive delivery.No positive match yields an empty pack; free-form notes are retrieved as paragraphs.Source: C. Baseline configurations and… · p. 18
- The native-environment baseline gives the Target no learned guidance or Builder-generated components, while allowing it to create its own working aids with native tools.Gains over this baseline reflect the combined contribution of harness construction and learned meta-skills.Source: C. Baseline configurations and… · p. 18
- No-skill construction preserves the Builder’s support-design permissions but removes learned meta-skills; Gemini’s Harness-Bench score rises from 37.35 natively to 53.29 with construction and 67.81 with the method.The latter comparison isolates learned experience under the same construction interface.Source: C. Baseline configurations and… · p. 18
- Full-bank construction scores 67.81 versus 42.38 for Gemini on Harness-Bench, and the method scores higher in all six original settings than direct delivery of the same meta-skill pack.The comparison holds learned content fixed while changing whether a Builder implements it as harness support.Source: C. Baseline configurations and… · p. 18
- Independent structured Target skills learn task-solving guidance separately from Builder support principles; Qwen scores 72.89 on Harness-Bench versus 74.81 with the method, while Gemini scores 67.81 on NewtonBench versus 68.84.The extraction model is the same, but the experience and learned bank are separate.Source: C. Baseline configurations and… · pp. 18–19
- Free-form Target notes consolidate one development pass without prescribed fields; their relative effectiveness versus structured skills varies by model and benchmark.On Harness-Bench GPT-OSS scores 66.01 with free-form notes versus 56.63 with structured Target skills; on NewtonBench Gemini scores 41.44 with free-form notes versus 67.81 with structured skills and 68.84 with the method.Source: C. Baseline configurations and… · p. 19
D. Case Studies: From Meta-Skills to Executable Support
- Two successful episodes illustrate task-specific checker construction and coordination of a scientific interface through memory and control, without estimating typical success rates or isolating component causality.Each traces a learned principle into harness support and Target execution.Source: D. Case Studies: From Meta-Skills to… · p. 19
D.1. Making an artifact contract executable
In the recorded metric-migration episode, a Builder turns checkpointing guidance into executable validation and submission control. The episode illustrates how mechanical consistency checks can support Target analysis while leaving semantic judgment to the Target.
D.1. Making an artifact contract executable
- The Builder implements checkpointing guidance with task-specific instructions, an auditor, controller, and submission gate, while the Target retains evidence interpretation, drafting, and repair.The auditor checks coverage, source-value fidelity, ledger coverage, consistency across deliverables, artifact requirements, schemas, and numerical formatting; semantic justification remains the Target’s responsibility.Source: §D.1 · pp. 19–20
- The controller prompts an audit after all four deliverables are observed, and edits invalidate a clean audit so the submission gate requires revalidation before completion.A clean audit establishes a concrete stopping condition.Source: §D.1 · p. 20
- The Target achieved a native score of 1.0 with 14 model calls and 15 counted tool calls after a zero-error, zero-warning audit; no-skill construction and Direct-all each scored 0.0435 without producing artifacts.The latter runs terminated from token-budget exhaustion and model failure, respectively.Source: §D.1 · p. 20
- The episode’s support design assigns deterministic bookkeeping to generated code so the Target can focus on substantive migration decisions and their consistent expression across artifacts.A single episode illustrates the integrated workflow but does not isolate each component’s contribution.Source: §D.1 · p. 20
D.2. Preserving scientific progress after computational failure
Despite two failed fitting attempts, the harness preserved measurements and redirected the Target to further experiments and submission. The Target completed the NewtonBench task with a law accepted as equivalent to the reference.
D.2. Preserving scientific progress after computational failure
- The task required inferring pre-loss elastic potential from measured velocity-derived kinetic energy, while designing experiments and finishing within ten tool calls.Its support challenge combined recovering latent energy with preserving enough evidence and budget after computational analysis failed.Source: §D.2 · p. 20
- Three learned principles guided support: retain evidence while bounding analysis, reserve budget for submission, and distinguish the requested quantity from measured observables.Instructions, memory, a controller, and recovery and submission checks structured the existing interfaces, while Target retained responsibility for inference.Source: §D.2 · pp. 20–21
- Memory kept experiment inputs and measurements separate from computation results, making both earlier batches available immediately after the second failure.Later requests could retrieve the latest analysis result and up to four recent experiment batches.Source: §D.2 · pp. 20–21
- After two fitting failures, the controller removed computation; Target gathered ten more measurements and proceeded to submission with three tool calls remaining.The run used eight model calls and eight counted tool calls.Source: §D.2 · p. 21
- Symbolic accuracy 1 and RMSLE 0 on the native evaluator, versus 0 and 0.092 for no-skill construction; Direct-all failed without a submission.The submitted law was U(x) = max{0, 1241.9012 x + 462.282 x2}.Source: §D.2 · p. 21
E. AI Use Statement
- The authors report using LLMs for framework results and language refinement, while retaining authorship of the ideas, logic, results, and technical contributions and reviewing refined text.They say the review was intended to confirm no hallucinated content was introduced.Source: E. AI Use Statement · pp. 21–22