PaperlayerView exact revision on arXiv

Source-linked AI summary

Harness-Zero: Harness Distillation via Agent-as-Harness

Haoran Ye, Yuxing Lu, Haonan Dong, Zhaochen Su, Guojie Song

arXiv:2609.24974v1cs.AIcs.CLcs.NE

TL;DR

Because optimal harnesses vary and their gains remain tied to deployment, general-purpose agents face a choice between shared and specialized harnesses. Harness-Zero uses a harnessing agent to distill optimized-harness behavior into model parameters, raising macro-average task success from 23.3% to 44.3% under a fixed target harness.

  • Problem

    Harness gains remain tied to deployment; because optimal harnesses vary, general-purpose agents must choose between a shared harness and maintaining specialized ones.

    Source: §1 · pp. 1–2

  • Method

    Harness-Zero uses a harnessing agent guided by an optimized harness to correct student responses in the target harness’s action space, producing demonstrations for fine-tuning.

    Source: HARNESS-ZERO · p. 2

  • Results

    Under the fixed target harness, Harness-Zero raises macro-average task success from 23.3% to 44.3%, above 41.7% with h⋆, and recovers h⋆-specific behaviors at 82.3% average over 28 patterns.

    Source: §5 · p. 8

  • Takeaways & Limitations

    Harness distillation narrows what the deployment harness must provide, but does not eliminate the need for a harness.

    Source: §5 · p. 9

    Harness-Zero depends on a capable harnessing model; weaker models can make review harmful and remove its advantage over code-as-harness.

    Source: §5 · p. 8

Abstract

from arXiv · show

Agent harnesses, the external systems that mediate model-environment interaction, can substantially improve agent performance, but their gains remain tied to the harness at deployment. Because the best harness varies across domains, instances, and models, a general-purpose agent must either settle for a suboptimal shared harness or route among an ever-growing set of specialized ones. We therefore study agent harness distillation: using a domain- or instance-optimized harness as training-time guidance and transferring the behaviors it induces into model weights, so that its gains survive under a single fixed target harness. The challenge is that the two harnesses differ in action space and available information, so guidance from the optimized harness cannot serve directly as supervision for the target one. We introduce Harness-Zero, which enables harness distillation through agent-as-harness. Guided by the optimized harness, a harnessing agent corrects student responses before execution in the target harness's action space, turning harness guidance into training demonstrations. Fine-tuning on the resulting trajectories internalizes harness-induced behavior into the model, so the specialized harness can be removed at deployment. Our experiments spanning knowledge work, tool use, and science domains show that: (1) For frontier LLMs using the same evolved harness, agent-as-harness outperforms code-as-harness. (2) With the specialized harness removed at deployment, Harness-Zero improves the base model's macro-average task success from 23.3% to 44.3%, even exceeding the 41.7% it reaches with that harness still attached. (3) Harness-Zero recovers harness-induced behaviors absent from the base model, with 82.3% average recovery across 28 patterns in the three domains.

1 Introduction

Harness-Zero transfers behaviors induced by optimized harnesses into model parameters, enabling deployment under a fixed target harness; experiments report retained performance gains and recovery of harness-induced behaviors.

1 Introduction

  • Because the best harness varies across domains, instances, and base models, a shared harness sacrifices specialized gains while maintaining many specialized harnesses incurs recurring costs without transferring improvements into the model.The cited work attributes this variation to Liu (2026), Luo et al. (2026b), and Zhang et al. (2026b); recurring costs include context, model calls, tools, and orchestration.Source: §1 · pp. 1–2

HARNESS-ZERO

  • Harness-Zero uses an optimized harness as training guidance to distill its induced behaviors into model parameters for use under a fixed target harness.Agent-as-harness reviews each proposed response, passes sound proposals unchanged or makes the smallest coherent correction in the student's action space, then executes the accepted response through the target harness and fine-tunes on reviewed rollouts.Source: HARNESS-ZERO · p. 2

Agent-as-harness

  • HARNESS-ZERO transfers behaviors induced by an optimized harness into model parameters for deployment under a fixed target harness, while agent-as-harness enables imitation learning across different harness action spaces.Experimental results show that agent-as-harness can outperform code-as-harness, HARNESS-ZERO retains optimized-harness gains after removal, and the distilled model recovers harness-induced behaviors.Source: Agent-as-harness · p. 2

2 Related work

Related work optimizes harnesses, co-evolves them with models, or distills selected behaviors; HARNESS-ZERO instead targets complete harness-behavior transfer into model weights for deployment under a minimal target harness.

Code-as-harness and its optimization

  • Harnesses mediate LLM-environment interaction, while research optimizes prompts, context, workflows, and harnesses and explores editable policies, task-specific generation, and strong-to-weak transfer.HARNESS-ZERO adds agent-as-harness, which can outperform code-as-harness for frontier models and supports distilling optimized harness behavior into model weights.Source: §2 · pp. 2–3

Co-evolution of model and harness

  • Prior work alternates harness and model updates or optimizes them jointly, but harness-dependent gains may not transfer: evaluation-harness changes affected performance more than training method, and cross-harness feedback did not improve held-out minimal-ReAct transfer.HARNESS-ZERO uses an optimized harness to guide a temporary harnessing agent, then distills its behavior into weights without retaining or routing harnesses at deployment.Source: §2 · p. 3

Distillation from privileged guidance

  • EvoHarness-RL and OPHSD show that models can absorb harness-induced behavior, but address individual mechanisms; the former retains an external workspace and the latter lacks an interactive agent loop.HARNESS-ZERO targets distillation of complete tool-using agent harnesses into a model deployed under a minimal target harness.Source: §2 · p. 3

3 HARNESS-ZERO

Harness-Zero adapts an evolved harness into private guidance for a harnessing agent, which corrects student responses under a fixed target harness; SFT then aims to preserve the induced behaviors in the student for deployment under that target harness alone.

3.1 Harness evolution and adaptation

  • Harness-Zero evolves a student-side harness on training tasks, then adapts its components into private reference guidance relative to the fixed target harness.Tools become specifications for student-native equivalents, middleware becomes private review alerts, and skills and memory become diagnostic criteria and intervention guidance.Source: §3.1 · pp. 3–4

3.2 Agent-as-harness

  • Under agent-as-harness, a reviewer uses private guidance to pass or replace each student proposal, while only the accepted response executes through the target harness and enters the student-visible trajectory.A single reviewer session handles the rollout incrementally, conditioning corrections on the current interaction history; its guidance and rejected proposals remain private.Source: §3.2 · pp. 4–5

Code-as-harness vs. agent-as-harness

  • Unlike code-as-harness, which runs the student under the evolved harness, agent-as-harness applies its guidance through minimal, target-valid corrections to proposals under the fixed target harness.The reviewer passes sound proposals unchanged and, when needed, makes the smallest coherent correction; it cannot inspect hidden solutions, verifier feedback, or environment state beyond the student-visible context.Source: Code-as-harness vs. agent-as-harness · p. 5

3.3 Learning from reviewed trajectories

  • SFT trains on accepted responses in reviewed rollouts, including unchanged proposals and harness-guided replacements, while masking reviewer-perspective reasoning from the loss.At deployment, the distilled policy operates under the target harness alone, without the evolved harness, private reference harness, or harnessing agent.Source: §3.3 · p. 5

4 Experiments

Across three domains, Harness-Zero transfers optimized-harness behavior into model weights so it persists under a fixed target harness. Experiments compare agent-as-harness with code-as-harness, evaluate distillation and its supervision signals, and measure behavioral recovery.

4.1 Experimental setup

  • The study reports single-run task success (pass@1, %) for SpreadsheetBench and USPTO, and scenario goal completion (SGC, %) for AppWorld; all three run in Harbor.Distillation uses Qwen3.5-9B with GPT-5.6 Sol as the harnessing agent, then two-epoch LoRA SFT on 487 SpreadsheetBench, 282 AppWorld, and 500 USPTO rollouts.Source: §4.1 · p. 6

4.2 Evaluating agent-as-harness

  • With the evolved harness, agent-as-harness averages 81.1% across six settings, versus 78.1% for meta-harness and 68.6% for mini-SWE-agent.An empty reference harness averages 69.2%; the same model serves as student and harnessing agent, so the gain does not come from a stronger supervising model.Source: §4.2 · p. 6, Table 1 · p. 6

4.3 Evaluating agent harness distillation

  • HARNESS-ZERO raises macro-average performance from 23.3% to 44.3%, a 21.0-point gain, and exceeds the 41.7% achieved by the base model with h⋆.Neither DeepAgents nor Claude Code benefits the base model; generic tools and extended context can be poorly suited to a 9B model and domain-specific tasks.Source: §4.3 · p. 7, Table 2 · p. 7
  • HARNESS-ZERO exceeds h⋆ on SpreadsheetBench and AppWorld, but on USPTO reaches 30.0% versus h⋆’s 38.0%, despite improving over the 12.0% base model.The harness mainly supplies reusable procedures in the first two domains, while USPTO relies on reaction priors, candidate-generation logic, and executable SMILES validation.Source: §4.3 · p. 7

4.4 Comparing alternative distillation signals

  • On USPTO, direct SFT on teacher trajectories or base-student trajectories collected under h⋆ each yields 12% test pass@1, despite 52.0% teacher collection success.Harness-guided review instead starts from the student’s trajectory and translates guidance into corrections compatible with its current state and target action space.Source: §4.4 · p. 7, Table 3 · p. 7
  • On USPTO, review with empty K yields 11%, oracle-answer review yields 15% after SFT despite 98.6% collection success, and K-guided review yields 30% pass@1 from 59.4% collection success.K supplies procedural guidance rather than task answers, supporting reusable behavior such as proposing candidates and validating reactant sets.Source: §4.4 · pp. 7–8, Table 3 · p. 7
  • Executing h⋆ during collection does not ensure transfer: teacher collection success rises from 52.0% to 62.0%, but distilled student performance falls from 12% to 3%.The mismatch prompts unavailable harness-tool calls that exhaust turns; Harness-Zero instead expresses corrections through h.Source: Table 3 · p. 7, §4.4 · p. 8

4.5 Behavioral internalization

  • Under h, HARNESS-ZERO reaches 82.3% average recovery across 28 harness-exclusive behaviors, which the base model scores at 0% by construction.The patterns span memory, skills, tools, and middleware, and each recovery rate is measured on tasks where the behavior appeared under h⋆ but not h.Source: §4.5 · p. 8, Table 4 · p. 8

5 Conclusion and discussion

HARNESS-ZERO turns evolved-harness guidance into training supervision that improves performance under a fixed target harness. The conclusion identifies limitations and outlines ways to make harness distillation more capable and efficient.

Conclusion

  • After SFT, HARNESS-ZERO raises macro-average task success under h alone from 23.3% to 44.3%, exceeding 41.7% with h⋆.Agent-as-harness averages 81.1% versus 78.1% for code-as-harness on frontier models; the student recovers h⋆-specific behaviors at an average rate of 82.3% over 28 patterns.Source: §5 · p. 8

Limitations

  • HARNESS-ZERO depends on a capable harnessing model, and reviewing every proposal increases collection cost; the added calls disappear after distillation.Mean USPTO latency rises to 2.4× during collection. SFT may not internalize deep domain knowledge, and mechanisms such as context management are not fully expressible as student responses; the method narrows but does not eliminate the need for a harness.Source: §5 · pp. 8–9

Future work

  • Future work could improve harnessing predictions, invoke review selectively, enable finer-grained interventions, and use preference learning signals discarded by response-level SFT.The proposed directions include stronger agent world models specialized for harnessing decisions, proxy signals to avoid reviewing sound proposals, token insertion or latent-space steering, and comparisons between rejected and preferred responses at the same state.Source: §5 · p. 9

Conclusion

  • Distilling domain- and task-specific harnesses into shared model parameters could let harness-developed behaviors and knowledge accumulate across agent systems.The authors envision harness development becoming a scalable training signal, with better models building better harnesses whose gains return to model weights.Source: §5 · p. 9

AI Use Statement · (Appendix) · A Skill-guided harness evolution

AI assistants supported grammar and prose polishing and routine coding, while the authors retained responsibility for the method, experimental design, and implementation decisions affecting reported results.

AI Use Statement

  • AI assistants were used to check grammar, polish author-written text, and provide routine coding assistance during implementation.The authors verified all AI-assisted output.Source: AI Use Statement · p. 9
  • The authors made every method, experimental-design, and result-affecting implementation decision, and take full responsibility for the paper.They verified all AI-assisted output.Source: AI Use Statement · p. 9

A.1 Evolution methods … B.1 Reference harness inventory

Harness-Zero evolves a shared student-side harness through failure-focused iterations, then adapts its components into guidance for a harnessing agent. The adaptation preserves component functions while changing how and where they act, including through workspace files, automatic review guidance, and a domain prompt.

A.1 Evolution methods

  • A three-round evolution process analyzes failed-task rollouts with a swarm, consolidates reusable harness changes, and evaluates the same model with the harness but no harnessing-agent intervention.Tasks that pass leave subsequent rounds; evolution targets performance under the student-side harness itself.Source: §A.1 · p. 14
  • Evolution targets failure patterns shared across tasks, generalizes existing components before adding new ones, and requires changes to trace back to trajectory evidence.Task-specific fixes are excluded, and components without evidence of benefit for two consecutive rounds are removed from the manifest.Source: §A.1 · pp. 15–16

A.2 Evolved components

  • SpreadsheetBench and USPTO retain accumulated failure notes in enabled memory, while AppWorld disables memory and injects the same material through bootstrap_instruction middleware.Table 5 lists the components surviving in each domain’s final h⋆; the supplied passage does not provide the full inventory.Source: §A.2 · p. 16

B Harness adaptation

  • Adapting h⋆ to K preserves component functions while changing how and where they act, and can add a domain prompt specifying the harnessing agent’s review policy.Table 6 summarizes the component-level mapping; its specific entries are not supplied here.Source: B Harness adaptation · p. 16

B.1 Reference harness inventory

  • K is mounted read-only in the harnessing agent’s separate workspace, where an index.json guides component discovery and the domain prompt is appended to its system prompt.Review middleware runs automatically on proposals and adds triggered guidance when conditions match; it is not available as files.Source: §B.1 · p. 16
  • Student-side guards can become review-time checks on visible trajectories plus environment action recipes, and one guard can split into multiple checks.USPTO’s answer_guard becomes checks for a missing answer file, bypassed canonicalization, and commitment without candidate enumeration.Source: §B.1 · p. 16

B.2 Worked example: preserving pre-filled cells · B.3 Adapting skills and memory · C Review protocol and harnessing-agent prompt

The appendices explain how harness-specific checks and guidance become student-native corrective actions, including worked examples involving pre-filled cells, skills, and memory. They also describe the review protocol for collecting training trajectories, in which the harnessing agent inspects or repairs updates and follows fixed submission and rationale-recording procedures.

B.2 Worked example: preserving pre-filled cells

  • The pre-filled-cell safeguard checks that example literals remain unchanged, while the reference harness prompts a student-visible comparison before accepting a finish.The evolved student-side harness diffs input and output workbooks; the adapted review middleware instead detects missing comparison evidence in the visible trajectory and requests a verification action.Source: §B.2 · pp. 16, 18

B.3 Adapting skills and memory

  • Skills become direct workflow instructions for the student, while memory turns recurring failures into review conditions and targeted interventions.Examples include replacing overwrites with inspection or restore-and-diff actions, and replacing self-referential verification with assertion-based checks; K supplies the final response through h.Source: §B.3 · p. 20

C.1 Session structure

  • Each harnessing-agent session spans one student trial, retaining its full review history and receiving new visible events, the current proposal, and any triggered guidance.The first update also includes the student system prompt and task instruction; structurally invalid responses are identified in the update.Source: §C.1 · p. 20

C.2 Submission format

  • The harnessing agent submits one PASS or REPLACE decision per update, with a replacement response and other required fields; its intervention budget is set by domain.The budget is 5 for SpreadsheetBench, 3 for AppWorld, and 5 for USPTO; malformed submissions are rejected, and repeated validation failures can fail the trial.Source: §C.2 · pp. 20–21

C.3 System prompt

  • The collection prompt asks the agent to keep trajectories on-policy by using student-visible information and native actions while making targeted corrections and returning control to the student.Training-free evaluation changes the framing and omits on-policy instructions, while the oracle-answer control grants reference-answer access subject to restrictions.Source: §C.3 · pp. 21–22
  • The agent passes only when student-visible observations support every material requirement; otherwise, it chooses another inspection, repair, test, or verification action rather than supplying a teacher-written solution.This keeps the review grounded in student-visible evidence and the steps taken to address gaps.Source: §C.3 · p. 22
  • For each student update, the agent calls submit_review exactly once, sets replacement to null for PASS or provides the complete replacement for REPLACE, and records the components used and a concise reason in private metadata.Source: §C.3 · p. 22

D Trajectory collection and filtering

The collection procedure filters trajectories for task success where applicable and removes private-review information so training data reflects only student-visible text. Reviewer-perspective reasoning in accepted replacement responses is masked, and the resulting datasets contain 487 SpreadsheetBench, 282 AppWorld, and 500 USPTO trajectories.

D Trajectory collection and filtering

  • SpreadsheetBench and AppWorld retain only trials with verifier reward 1.0 and no execution exception, while USPTO keeps all trials and performs better than success-only training.The USPTO ablation uses the same 500 tasks in each condition.Source: D Trajectory collection and filtering · p. 22
  • Trajectories are filtered for private-review traces, including reference-harness paths, recorded candidate-file paths, review-tool names, review metadata keys, and middleware names.Every training trajectory must contain only student-visible text.Source: D Trajectory collection and filtering · p. 22
  • Accepted REPLACE responses with reviewer-perspective reasoning are masked; the resulting sets contain 487 SpreadsheetBench, 282 AppWorld, and 500 USPTO trajectories.Patterns include references to proposals, drafts, student or candidate responses, and review actions.Source: D Trajectory collection and filtering · p. 22

E Implementation details · E.1 Student harness

The student harness gives mini-SWE-agent h only the task and a system prompt for solving it in a Linux sandbox. Its single-command execution interface makes the file system the only state that persists between calls.

E.1 Student harness

  • The mini-SWE-agent h system prompt is the only instruction the student receives beyond the task itself.It frames the student as an agent solving tasks in a Linux sandbox.Source: §E.1 · p. 22
  • The student has one tool, execute, for running a single non-empty bash command per call.The system prompt directs it to issue one execute call at a time and return a short final answer when done.Source: §E.1 · p. 23
  • Each execute call starts a fresh shell, so working-directory and environment changes do not persist, while file-system state does.The command’s combined output and exit code are returned as the observation.Source: §E.1 · p. 23
  • A response without a tool call ends the trial and is treated as the final answer.The prompt asks for a short final answer after completing the task.Source: §E.1 · p. 23
  • The sandbox permits single-level subagent delegation through an agent CLI, with at most four subagents running concurrently.A subagent uses the same base model and execute tool, receives a bounded task, and cannot delegate further.Source: §E.1 · p. 23
  • The evolved h⋆ components in this work do not rely on subagents, and delegation is used sparingly.The authors state that bash plus one-level delegation is expressive enough to map diverse evolved h⋆ components onto h.Source: §E.1 · p. 23

E.2 Models and access

The study evaluates several models through provider APIs and runs harness evolution with Kimi K3, using provider-default decoding and high reasoning effort where applicable.

E.2 Models and access

  • GPT-5.6 Sol, DeepSeek-V4-Pro, and DeepSeek-V4-Flash were accessed through Microsoft Azure; Qwen3.5-9B was evaluated through OpenRouter, and fine-tuned checkpoints were served on NVIDIA A100 GPUs.Qwen3.6-35B-A3B was also evaluated through OpenRouter in the analysis of § F.Source: §E.2 · p. 23
  • All harness-evolution runs used Kimi K3 [Kimi Team, 2026] through Kimi Code [Moonshot AI, 2026].Source: §E.2 · p. 23
  • Models used provider-default decoding parameters, with reasoning enabled and effort set to high when applicable.Source: §E.2 · p. 23

E.3 Training configuration

The study separately fine-tunes Qwen3.5-9B on each domain’s collected trajectories using LoRA supervised fine-tuning and a fixed two-epoch configuration.

E.3 Training configuration

  • LoRA fine-tuning uses rank 32, α=32, batch size 8, a 65,536-token sequence length, and two epochs.The recipe follows Tinker supervised training [Thinking Machines Lab, 2025].Source: §E.3 · p. 23
  • The learning rate uses 5% warmup to a peak of 2 × 10−4, then decays to 10−6; all runs use seed 42.Source: §E.3 · p. 23
  • Each domain is trained separately on its own collected trajectories, and examples exceeding the sequence length are dropped.Source: §E.3 · p. 23

F Additional analysis of agent-as-harness … G.2 Mined patterns

Agent-as-harness gains depend on model capability and incur review overhead, while Harness-Zero’s harness-exclusive behavior analysis defines and catalogs patterns transferred across three domains. The analysis reports 18 SpreadsheetBench, 6 USPTO, and 4 AppWorld patterns.

F Additional analysis of agent-as-harness

  • Agent-as-harness’s relative advantage over meta-harness ((h, K) −h⋆) falls with capability: +1.0% and +4.0% on stronger models, then −1.0% and −12.0% on weaker models.Table 8 extends the USPTO comparison to two weaker models. On the weakest model, the harnessing agent replaces 66.0% of reviewed steps, but its edits are net harmful; review without evolved guidance scores 12.0 versus mini-SWE-agent’s 16.0.Source: Table 8 · p. 23, F Additional analysis of… · p. 23
  • Reviewing every proposal requires at least two model calls per step; on USPTO, agent-as-harness with evolved guidance averages 237.2 s per trial versus 100.1 s for mini-SWE-agent.Each review processes the trajectory prefix and reference harness before the next student step, increasing token use and latency; reviewing only a subset of steps is left to future work.Source: F Additional analysis of… · pp. 23–24

G Harness-exclusive behavior patterns

  • The behavior-pattern analysis uses trajectories from three deployments: mini-SWE-agent (h), meta-harness (h⋆), and HARNESS-ZERO (h).The measurements use the three deployments from Table 2.Source: G Harness-exclusive behavior patterns · p. 24

G.1 Approach

  • For each domain, mining starts from enabled components of h⋆ and reads its memory files, skills, tools, and middleware, then translates rules into semantic binary detectors over trajectories.For example, AppWorld pagination matches use of aw.pages or an explicit page_index loop; a USPTO answer write matches a shell redirect or Python write_text.Source: §G.1 · p. 24
  • A detector’s support tasks show a pattern under h⋆ but not h for the base model; detectors require at least 10 support tasks, and identical detectors are merged.Recovery is the fraction of support tasks where HARNESS-ZERO exhibits the behavior; by construction, the base model exhibits it on 0% under h and 100% under h⋆.Source: §G.1 · p. 24

G.2 Mined patterns

  • Applying binary detectors to enabled student-side harness components and retaining only behaviors with at least 10 support tasks yields 18 patterns on SpreadsheetBench, 6 on USPTO, and 4 on AppWorld.Support tasks are test tasks where the base model shows the pattern under h⋆ but not h; on these tasks, the base model exhibits it on 0% under h and 100% under h⋆.Source: §G.1 · p. 24

Related papers

Loading 2609.24974v1…