PaperlayerView exact revision on arXiv

Source-linked AI summary

Harness Learning Enables Generalizable Test-Time Adaptation

Alvin Zhang, Xuecheng Liu, Zixuan Wang, Fahim Tajwar, Daman Arora, Ruslan Salakhutdinov, Daniel Khashabi, Yuda Song, Andrea Zanette

arXiv:2609.35738v1cs.CLcs.LG

TL;DR

Harness design shapes how language models use tools and process information, yet effective revisions may require repeated experimentation and may not transfer across tasks. This paper trains a proposer to revise executable harnesses from execution feedback, then reports transfer to unseen tasks and continued improvement over multiple rounds. The reported results include higher single-step scores on unseen reasoning families and transfer in multi-hop question answering.

  • Problem

    Effective harness revisions can require repeated test-time experimentation, and revisions that work on one task may not transfer to another.

    Source: §1 · p. 1

  • Method

    Harness learning trains a proposer with reinforcement learning to revise a fixed solver’s executable harness from execution feedback, without parameter updates at test time.

    Source: §1 · p. 2

  • Results

    On 21 unseen Reasoning Gym families, SFT followed by RL raises mean single-step revision scores from 0.32 with the base proposer to 0.62, while trained proposers also transfer in multi-hop QA and improve harnesses over successive rounds.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    Policies trained on individual revisions can remain useful as harnesses change, continuing to improve them over multiple test-time rounds.

    Source: §1 · p. 2, §7 · p. 9

    Training uses prescribed revision prompts and a fixed solver, with one teacher for Reasoning Gym supervision.

    Source: §7 · p. 9

Abstract

from arXiv · show

A language-model agent is jointly defined by its model and its harness, the executable program that organizes model calls, tool use, and information flow. Because different tasks call for different ways of organizing these operations, the harness needs to be adapted using feedback from the task at hand. We introduce harness learning, which trains a proposer model to revise a solver's harness using execution feedback. We formulate this process as meta-learning over executable programs, with harness revisions playing the role of weight updates in gradient-based adaptation. We train the proposer with reinforcement learning, using the task performance of revised harnesses as the reward. At test time, the proposer uses feedback from successive executions on a new task to refine the harness, without performing any parameter-space update. Experiments on reasoning and multi-hop question answering show that harness learning improves revision quality and that the ability to adapt at test time transfers to unseen tasks. Policies trained on individual revisions can continue improving harnesses over multiple rounds, while the benefits of training on revision sequences vary across settings. These findings suggest a path towards continually learning agents that turn accumulated experience into generalizable improvements.

1 Introduction

Harness learning treats harness revision as meta-learning: a proposer learns from execution outcomes to adapt a solver’s executable harness. Experiments report transfer to unseen tasks and continued improvement over successive revisions.

1 Introduction

  • Harness learning trains a proposer to revise executable harness code from task descriptions, current harnesses, and execution reports, using revised-harness performance as reinforcement-learning reward.At test time, the proposer can apply successive revisions using feedback from each execution.Source: §1 · p. 1, Figure 1 · p. 2
  • The approach frames harness revision as meta-learning over executable programs, adapting how a model uses its capabilities without changing its parameters.The harness organizes model calls and tool interactions, shaping what the model sees and how execution proceeds.Source: §1 · pp. 1–2
  • Trained proposers transfer harness-revision capability to unseen reasoning tasks and question-answering benchmarks.In multi-hop QA, a proposer trained on HotpotQA transfers to MuSiQue and 2WikiMultihopQA.Source: §1 · p. 2
  • Proposers trained on individual revisions can continue improving harnesses over successive test-time rounds.The paper also analyzes revision reliability, auxiliary structure, and generated-tool use.Source: §1 · p. 2

2 Related Work

Related work spans automated agent and harness optimization, dedicated models for harness construction, and meta-learning approaches to workflows or harness evolution. This paper focuses on learning an adaptation rule that revises executable harnesses from feedback across successive rounds.

2 Related Work

  • Automated agent-design work optimizes language-model pipelines and workflows, while harness-specific methods revise programs governing agent behavior.Concurrent systems train dedicated harness construction or revision models, including Harness-R1 and JIT-Agent.Source: §2 · p. 2
  • The paper studies whether successive execution-feedback revisions improve performance on task families excluded from training, alongside related frozen-model adaptation using unlabeled traces.The latter describes TTHE (Nie et al., 2026).Source: §2 · pp. 2–3
  • Unlike shared workflow initializations or harness-evolution blueprints, this formulation adapts executable harness code and trains the proposer as the adaptation rule.Changing code can alter control flow, tool interfaces, and the sequence of model calls.Source: §2 · p. 2

3 Preliminaries

The preliminaries define a harness as the program orchestrating model calls and environmental interaction, then introduce reinforcement learning with verifiable rewards and GRPO’s clipped objective.

3.1 Harness

  • A harness orchestrates a language model’s calls, constructs call contexts, executes tools, and incorporates environmental feedback; it can change while model parameters stay fixed.In the experiments, the solver is kept fixed and omitted from the notation.Source: §3.1 · p. 3

3.2 Reinforcement learning with verifiable rewards

  • A policy is an autoregressive language model mapping context to a token sequence, and verifiable-reward training maximizes expected reward over prompts.A programmatic grader evaluates responses or their execution outcomes.Source: §3.2 · p. 3
  • GRPO normalizes rewards within groups of responses, assigns token-level advantages, and uses a clipped surrogate objective.The token-level importance ratio compares current and old-policy probabilities for each sampled token.Source: §3.2 · p. 3

4 Harness Learning as Meta-Learning

Harness learning treats adaptation of a fixed solver’s executable workflow as meta-learning: a proposer learns to revise harnesses from execution feedback, then applies that revision rule to new tasks.

4.1 Tasks, harnesses, and execution feedback

  • A harness organizes solver calls, tool use, and direct computation for a task, while the solver itself remains fixed.Tasks include Reasoning Gym families and multi-hop QA benchmarks.Source: §4.1 · p. 3

4.2 Harness adaptation and the meta-learning objective

  • At each adaptation round, the proposer uses the task description, current harness, and execution report to generate edits, then a selection protocol chooses the next harness.Execution of the retained harness on the next feedback set supplies the next report.Source: §4.2 · p. 4
  • The meta-learning objective is to train proposer parameters that improve the held-out performance of a harness adapted over multiple rounds.Training instead uses each revision’s immediate performance as a surrogate objective and assigns credit from that candidate’s outcome.Source: §4.2 · p. 4

4.3 Training the proposer

  • The proposer is trained on task, harness, and report contexts using revision outcomes, with optional supervised initialization and reinforcement learning from candidate performance.Scoring questions are disjoint from report-feedback questions, and rewards include auxiliary credit for valid edits and successful execution; training can also use successive revision contexts.Source: §4.3 · pp. 4–5

5 Experiments

Experiments test whether learned revisions transfer to unseen reasoning and QA tasks, improve through repeated adaptation, and benefit from sequence-based training. Single-step revision improves on unseen reasoning families, while the effects of sequence training vary by setting.

5. Experiments

  • The experiments compare generalization to unseen tasks, successive test-time revision, and sequence-based training across Reasoning Gym families and multi-hop QA benchmarks.Unseen tasks are excluded from supervised fine-tuning and reinforcement learning; Reasoning Gym also tests canonical and format-varied question formats.Source: §5 · p. 5, §5.1 · pp. 5–6

5.2 Learning transferable revisions on Reasoning Gym

  • 0.62 after RL versus 0.32 for Base on 21 unseen Reasoning Gym families, with the mean proposal already exceeding the seed after RL.The improvement appears across the mean of all proposals, rather than depending on selecting the best candidate.Source: Figure 3 · p. 6, §5.2 · p. 6
  • On 21 unseen families, the trained 4B proposer averages 0.62 versus 0.56 for its 35B teacher under the same revision protocol.The teacher scores higher under oracle best-of-eight selection.Source: §5.2 · p. 6

5.3 Adapting retrieval workflows on unseen QA benchmarks

  • On unseen QA benchmarks, both RL proposers improve harnesses over successive rounds and finish above Base, while Base ends below the seed on HotpotQA and MuSiQue.On MuSiQue, Single-step RL averages 0.15 exact match for independent revisions versus 0.27 after ten rounds; each run uses 80 proposals, but total compute and feedback are unmatched.Source: §5.3 · p. 7

5.4 Effect of training on revision sequences

  • Training on revision sequences does not consistently outperform individual-revision training: Multistep RL is higher on canonical but lower on format-varied reasoning questions, and QA scores are nearly identical.Proposers trained on individual revisions also improve harnesses over successive test-time rounds.Source: §5.4 · p. 7

6 Analysis and Discussion

Training improves proposal reliability and yields distinct reasoning and QA harness structures; repeated revision can transfer across changing harnesses, though some explanations and benefits remain qualified.

6.1 Training improves proposal reliability

  • Training reduces proposals that fail to run or score zero, supplying more viable candidates for selection in each revision round.The authors suggest this may help sustain progress across successive revisions.Source: §6.1 · p. 7, Figure 6 · p. 9

6.2 Qualitative analysis of learned harnesses

  • On Reasoning Gym, SFT and Single-step RL mostly produce interpreter loops; helper integration often fails to contribute to answers when surrounding interfaces break.Unsuccessful compositions can pair a correct helper algorithm with failures in parsing, tool communication, or answer extraction.Source: §6.2 · p. 8, Figure 6 · p. 9
  • In QA, RL preserves multi-hop retrieval while Base sometimes collapses the harness to one solver call or removes the solver.A common RL revision keeps summaries for retrieval guidance and passes retrieved passages directly to the answering call; other revisions add retrieval hops or search triggers.Source: §6.2 · p. 8, Figure 7 · p. 8, Figure 6 · p. 9

6.3 Repeated revision

  • The QA seed’s existing retrieval workflow may explain why gains accumulate over more rounds than on Reasoning Gym, but training and selection differences prevent attributing the contrast solely to the seed.Reasoning Gym revisions often introduce interpreter loops, where parsing and tool-interface failures can limit further gains.Source: §6.3 · p. 8, §6.2 · p. 8
  • Single-step RL trained only on seed-harness inputs continues improving revised harnesses on unseen benchmarks, suggesting its revision skills extend to intermediate harnesses and reports.Figure 5 presents held-out exact match over ten rounds and compares independent and sequential QA revision under a matched 80-proposal budget on unseen benchmarks.Source: §6.3 · p. 8, Figure 5 · p. 8
  • Sequence-training experiments use local rewards for single revisions, leaving open whether cross-round credit assignment or different intermediate-state training would improve longer revision chains.The authors also identify supervised demonstrations of successive revisions as a possible alternative.Source: §6.3 · p. 9

7 Conclusion

Harness learning trains a proposer to revise a frozen solver’s executable harness from execution feedback, and the learned procedure transfers to unseen tasks. The conclusion reports strong unseen-family results while noting constraints in the training setup.

7 Conclusion

  • On 21 unseen Reasoning Gym families, SFT followed by RL raises mean single-step revision scores from 0.32 with the base proposer to 0.62.The trained 4B proposer outperforms the 35B teacher on average.Source: §7 · p. 9
  • A HotpotQA-trained proposer transfers to unseen QA benchmarks, and policies trained on individual revisions continue improving harnesses over multiple rounds.The paper frames this as evidence that learned revision skills remain useful as the harness changes.Source: §7 · p. 9
  • Training uses prescribed revision prompts and a fixed solver, with one teacher providing Reasoning Gym supervision.The authors identify stronger teachers, tool-using proposers, and joint proposer-solver optimization as directions for future work.Source: §7 · p. 9

A Extended Related Work

The related work situates harness learning among learned harness editors, evaluation of harness evolution, continual-learning agents, meta-learning, and self-improvement methods. It distinguishes this work by its revision procedure, transfer evaluation, and fixed-solver setup.

A Extended Related Work

  • Harness optimization includes modular pipeline optimization, code-workflow search, feedback-driven harness editors, failure aggregation, and policies that manage external state.WHALE alternates model-weight updates with harness optimization, whereas harness learning trains a separate proposer with a fixed solver.Source: A Extended Related Work · p. 14
  • Contemporary harness-learning work differs in training signals and adaptation scope: Harness-R1 uses same-benchmark partitions and does not refine the same patch successively, while this paper tests family- and benchmark-level transfer and repeated revision.Both use supervised initialization followed by RL in the Reasoning Gym setting; the QA proposer instead starts RL from the base proposer without teacher-generated revision demonstrations.Source: A Extended Related Work · p. 14, Table 2 · p. 14
  • JIT-Agent adapts around task instances and an expanding harness archive, whereas this work revises a current harness across instances within a task family and measures transfer to excluded families or benchmarks.JIT-Agent combines teacher SFT, execution-based preference optimization, supervised repair, and evolutionary optimization.Source: A Extended Related Work · p. 14
  • Ornith jointly improves scaffolding and the task-solving policy, while this work updates only the proposer and evaluates harness changes with a fixed solver.At test time, both models remain fixed as execution feedback guides code revision.Source: A Extended Related Work · p. 14
  • Meta-learning approaches range from gradient-based initialization and recurrent policies to shared workflows and interaction-history strategies; this proposer maps a harness and execution feedback to a code revision.Training uses immediate revision rewards as a greedy surrogate for final harness quality.Source: A Extended Related Work · p. 15
  • Self-improvement methods revise outputs, memory, code, or parameters; harness learning instead trains a separate proposer from revision outcomes and keeps both models frozen at test time.Graded development feedback guides harness edits.Source: A Extended Related Work · p. 15
  • Harness-evolution evaluations examine matched budgets, useful updates, constrained edits, held-out suites, and continual learning; this paper fixes each benchmark’s solver and tests transfer across excluded families or benchmarks.Its reported measures include average proposal scores, best-of-eight scores, and successive-revision performance.Source: A Extended Related Work · p. 15

B.1 Evaluation protocol

The appendices specify evaluation and training protocols, then report additional reasoning results across unseen families, proposal scoring rules, and multistep revisions. The results favor trained proposers overall, with comparisons dependent on evaluation setting and question format.

B.1 Evaluation protocol

  • Evaluation separates disjoint development feedback and harness selection from hidden held-out scoring, and compares independent seed revisions with sequential revisions of the best retained harness.Sequential-versus-independent comparisons do not isolate feedback’s contribution because parent harnesses and candidate-selection protocols also differ.Source: §B.1 · pp. 15–16

B.2 SFT corpus and training

  • Reasoning Gym SFT uses first-round teacher revisions that pass performance and execution checks; its single-teacher corpus is concentrated in interpreter loops and direct algorithmic solutions.This supervision may limit the range of strategies learned for successive-round harness improvement.Source: §B.2 · p. 17

B.3 Single-step RL

  • Single-step RL samples from fixed contexts pairing task families, parent harnesses, and execution reports, then applies task and validity rewards to revisions executed with the frozen solver.The task-score term includes partial grader credit; failed edits earn only accrued validity credit.Source: §B.3 · p. 18

B.4 Multistep RL

  • Reasoning Gym multistep RL trains offline in two phases with immediate revision rewards and refreshes revision states between phases.The later-round-state experiment instead uses a fixed mixture of contexts from SFT revision runs.Source: §B.4 · pp. 18–19, Table 8 · p. 19

B.5 Training curves

  • The training-curve appendix reports mean reward, task score, validity-reward baseline, and rollout outcomes for single-step RL.Rollout variance is attributed to batches mixing tasks from different families.Source: Figure 8 · p. 19, §B.5 · p. 19

C Additional Reasoning Gym Results and Analyses

C.1 Single-step results

  • On 27 teacher-covered families, mean held-out scores are 0.600 for Single-step RL and 0.567 for the teacher; across all 26 unseen families, means are 0.528 for Single-step RL and 0.501 for the teacher.The 26-family unseen evaluation includes families excluded from the main figures.Source: §B.1 · p. 16, §C.1 · p. 20, Table 9 · p. 21

C.2 Multistep results

  • After five revisions, all trained proposers finish above Base on expanded held-out sets, with similar canonical scores and Single-step RL highest on format-varied questions.Table 13 reports means over 24 families per question format; family-level development scores show SFT and Single-step RL reverse order on caesar_cipher across formats.Source: Table 13 · p. 23, §C.2 · p. 23

C.3 Training variants

Training variants produce mixed evidence: harder-task training helps single-step performance on format-varied families but underperforms in repeated revision, while composition-focused variants do not consistently yield effective helper use. The section also specifies the QA data and RL training procedures used for these evaluations.

C.3 Training variants

  • 12 to 7 among 40 probe draws: structurally composed harnesses declined at the later composition-prompt checkpoint as mean score increased slightly.The score increase suggests task reward can favor interpreter loops even when the prompt requests helper tools.Source: §C.3 · p. 29, Table 17 · p. 30
  • No harder-task rollout passed the structural-composition screen, so the run received no reward from its composition bonus.Because training pool and validity terms changed together, the run measures their combined effect.Source: §C.3 · p. 29
  • 0.600 versus 0.576 and 0.506 on format-varied families, but harder-task RL finishes below Single-step RL after repeated revision on both development formats.The nineteen-family evaluation includes training families, so it mixes seen and unseen tasks.Source: §C.3 · p. 29, Table 16 · p. 30
  • 0.615 versus 0.660 for Single-step RL, a paired difference of −0.045 ± 0.048 across twelve families; fixed later-round training states show no evidence of improving repeated revision.The checkpoint is evaluated at round five.Source: §C.3 · p. 29

C.4 Harness composition

  • The composition directive raised the composition-prompt model’s score but lowered Single-step RL’s score on the eleven-family format-varied set; five-family probes also scored lower with the directive.A guarded deterministic-helper directive produced direct algorithmic solutions in 158 of 160 draws, but none met structural or functional composition criteria.Source: §C.4 · pp. 29–31, Table 18 · p. 30
  • 25 of 40 composed tool loops at initialization fell to one under the tool-use reward; the combined reward retained 16 of 40 at checkpoint 12, but remained below Single-step RL at round five.Helper outputs reaching the solver remained uncommon, and inspected harnesses showed parsing, tool-communication, and answer-extraction failures around correct helpers.Source: §C.4 · p. 31, Table 19 · p. 31

D.1 Datasets and retrieval

  • HotpotQA supplies in-domain evaluation, while MuSiQue and 2WikiMultihopQA provide out-of-domain evaluation using disjoint development and held-out question sets.The QA training pool contains 2,000 HotpotQA questions; the out-of-domain corpora are built from supplied support and distractor paragraphs.Source: §D.1 · pp. 31–32, Table 21 · p. 32

D.2 RL training

  • For HotpotQA RL, the proposer receives execution reports from six feedback questions and candidates are scored on 64 questions, with exact match plus a validity bonus as reward.Single-step RL uses the seed harness as parent; Multistep RL trains on parents from successive revisions.Source: §D.2 · p. 32

D.3 Multistep evaluation

The QA evaluation tests both repeated adaptation from a shared seed and independent proposals from seed-harness feedback. The protocols distinguish deployable selection using scoring questions from held-out analyses of proposal quality.

D.3 Multistep evaluation

  • Four revision runs each adapt for ten rounds from the same two-hop retrieve-and-summarize seed, using six feedback questions and 64 scoring questions per round.Each run samples eight revisions and advances to the candidate with the highest scoring exact match.Source: §D.3 · pp. 32–33, Table 22 · p. 33
  • QA advances to the highest-scoring candidate even when it falls below its parent, whereas the Reasoning Gym protocol retains a candidate only when its development score improves.Failed or invalid revisions score zero, and QA reports the mean test score of the four current harnesses after each round.Source: §D.3 · p. 33

D.4 Single-step evaluation

  • Independent single-step evaluation scores 80 seed-based proposals on held-out questions and reports unselected mean, fraction above seed, failed-edit rate, Oracle@8, and best-of-80.Oracle@8 and best-of-80 select by held-out scores and are not deployable selection rules.Source: §D.4 · p. 33, Table 23 · p. 34
  • The single-step and successive-revision comparisons differ in feedback distribution, and the best run uses 320 proposals versus 80 for the independent pool.The schematic shows edits adding a third retrieval hop or searching again when a draft answer is absent from retrieved passages.Source: §D.4 · p. 34, Figure 12 · p. 34

E Implementation Details

The implementation presents the proposer with the current harness and execution feedback, then evaluates revisions under distinct feedback and scoring procedures. Composition directives specify desired helper structures, while separate audits assess what proposals actually achieve.

E.1 Structured harness revision

  • The revision interface supplies task configuration, current harness and development score, and execution report, then requests targeted search-and-replace edits with predicted effects.Edits must parse against the current source, and invalid responses may receive validation feedback and be retried.Source: §E.1 · p. 35

E.2 Execution feedback

  • Reasoning Gym reports include scores, instance outcomes, trace excerpts, and revision history; QA reports include traces, answer correctness, and retrieved or missed supporting documents.Independent proposals share a parent report, whereas successive revision obtains feedback from the retained harness.Source: §E.2 · p. 35

E.3 Composition directives

  • Composition directives request reusable computational helpers integrated into the solver’s tool loop, while audits separately measure structural integration and functional contribution.A more restrictive directive requests task-specific helpers rather than a generic code interpreter.Source: §E.3 · p. 35

F Limitations and Future Work

The paper identifies limitations in teacher supervision, proposer agency during revision, and the fixed-solver, shorter-horizon training setting.

Teacher capability and proposer initialization

  • Reasoning Gym supervision comes from one 35B teacher; more diverse harness designs and multistep demonstrations from stronger teachers remain a possible direction.Whether initialization with more harness-design knowledge improves exploration and transfer remains open.Source: F Limitations and Future Work · p. 36

Agentic training of the proposer

  • The proposer uses prescribed revision prompts and externally generated reports, without choosing its own code-inspection or execution steps before submitting edits.Future training could let it inspect code, run targeted tests, and revise from those observations.Source: F Limitations and Future Work · p. 36

Joint training and longer-horizon tasks

  • The solver remains fixed during training, so the experiments do not examine joint harness-and-solver learning or tasks requiring longer sequences of tool use and intermediate decisions.The paper raises transfer to unseen task families and usefulness as the solver changes as open questions.Source: F Limitations and Future Work · p. 36

Related papers

Loading 2609.35738v1…