PaperlayerView exact revision on arXiv

Source-linked AI summary

Training with Harnesses: On-Policy Harness Self-Distillation for Complex Reasoning

Zhengyang Zhao, Lu Ma, Wentao Zhang

arXiv:2605.08741v1cs.CL

TL;DR

Inference-time harnesses can improve complex reasoning, but their external workflows do not necessarily change the underlying model. OPHSD trains on harness-assisted behavior, achieves strong standalone results across text classification and mathematical reasoning, and suggests that harnesses can serve as temporary training scaffolds.

  • Problem

    Inference-time harnesses improve robustness in practice, but the underlying model may remain brittle without them, leaving open whether a harness’s procedure can be learned by the model itself.

    Source: §1 · p. 1, §3.2 · p. 3

  • Method

    OPHSD trains the model on its own harness-assisted rollouts, using harness-produced contexts as supervisory signals, then removes the harness at inference.

    Source: §1 · p. 2

  • Results

    OPHSD achieves the best standalone results across evaluated text-classification and mathematical-reasoning benchmarks, improving 10.83% over OPSD on HMMT25.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Harnesses can act as temporary training scaffolds whose procedural benefits are transferred back into the model, rather than permanent inference-time fixtures.

    Source: §1 · p. 2

    Empirical validation covers two representative harnesses, not a large-scale sweep across diverse harness designs.

    Source: §5 · p. 9

Abstract

from arXiv · show

Inference-time harnesses substantially improve large language models on complex reasoning tasks. However, the intrinsic capabilities of the underlying model remain unchanged by the addition of these external workflows. To bridge this gap, we introduce \emph{On-Policy Harness Self-Distillation} (OPHSD), which employs the harness-augmented current model as a teacher for self-distillation, thereby introducing extra supervisory signals from the harness beyond training data. OPHSD internalizes task-specific harness capabilities into the student model, yielding robust generalizability and strong standalone performance across diverse reasoning tasks. Evaluated across draft--verify harness for text classification and plan--solve for mathematical reasoning tasks, OPHSD consistently outperforms strong baselines (e.g., +10.83\% over OPSD on HMMT25). Our analysis further indicates that reattaching the harness during inference yields no additional benefits and can even degrade performance, suggesting that complex harnesses need not always be permanent fixtures; instead, they can serve as temporary training scaffolds whose benefits are permanently fed back into the base model. Our code and training data are available at https://github.com/zzy1127/OPHSD-On-Policy-Harness-Self-Distillation.

1 Introduction

OPHSD uses harness-assisted behavior to train a model to perform without external scaffolding. Across text classification and mathematical reasoning, it outperforms strong baselines, while reattaching the harness can provide no further gains or reduce performance.

1 Introduction

  • Inference-time harnesses can improve robustness, but their external procedures leave the underlying model brittle when evidence accumulation, verification, or long-horizon reasoning is needed.The separation also adds latency, token cost, engineering complexity, and failure modes in retrieval, transformation, and control flow.Source: §1 · p. 1
  • SFT imitates static demonstrations, while RL can provide sparse supervision, motivating OPD as a route to dense procedural supervision on student trajectories.The paper asks whether a harness-induced step-by-step procedure can be absorbed into model parameters.Source: §1 · pp. 1–2
  • OPHSD trains the model on its own harness-assisted rollouts so it can reproduce the harness-induced behavior without scaffolding at inference.The method uses reverse KL to match those on-policy rollouts.Source: §1 · p. 2
  • OPHSD achieves the highest average pass@8 across AIME24, AIME25, OlympiadBench, and HMMT25, improving 10.83% over OPSD on HMMT25.It also achieves the best results on LawBench accusation classification and USPTO reaction-type prediction.Source: §1 · p. 2

2 Related Work

Related work frames OPHSD against on-policy distillation and research that designs or optimizes external harnesses. OPHSD instead uses harness-produced supervision to train a model intended for deployment without that runtime.

2.1 On-Policy Distillation

  • OPD trains on student-generated trajectories with teacher supervision, while OPHSD uses harness-generated traces to internalize response patterns for unassisted inference.Reward-based OPD can have high variance and reward-design sensitivity; loss-based OPD offers dense supervision but requires teacher distributions and can be computationally expensive.Source: §2.1 · p. 2

2.2 Harness Engineering

  • A harness is an orchestration layer around model or agent calls, with control flow and intermediate artifacts that can persist across steps or delegated workers.Research on harness engineering also treats the harness itself as an object to represent, modify, and optimize.Source: §2.2 · pp. 2–3
  • Harness-engineering methods optimize external systems that remain in deployment, whereas OPHSD optimizes the model using harness-produced outputs for deployment without the runtime.The distinction is what persists after optimization: the harness or the trained model.Source: §2.2 · p. 3

3 Methodology

OPHSD extends on-policy distillation by replacing static privileged context with a dynamic harness workflow that generates supervisory contexts. The method trains on harness-induced trajectories and applies draft–verify and plan–solve harnesses to classification and mathematical reasoning.

3.1 Preliminaries: On-Policy Distillation and Privileged Context

  • OPD samples student trajectories and uses teacher supervision on those trajectories, while static privileged context gives the teacher informational rather than procedural advantage.The paper argues that internalizing a harness requires generalizing privileged context into a harness-driven workflow.Source: §3.1 · p. 3, §3.2 · p. 3

3.2 OPHSD: On-Policy Harness Self-Distillation

  • OPHSD generalizes privileged input to training-only oracle information and uses a programmatic harness to orchestrate reasoning into a structured terminal context for teacher supervision.The harness can range from a static wrapper to a multi-agent reasoning program, and distilling its trajectories is intended to internalize capabilities beyond static data injection.Source: §3.2 · pp. 3–4
  • The student minimizes reverse KL against a target computed by the base model conditioned on the dynamic harness context.The harness-produced context is treated as a stop-gradient target, keeping the supervisory signal grounded in the base model’s stable prior.Source: §3.2 · pp. 4–5
  • OPHSD can internalize structural reasoning priors such as multi-step decomposition and self-verification, but not genuine real-time external access such as live tool interaction.The student must reproduce harness-induced behavior from the bare input alone.Source: §3.2 · p. 5

3.3 Harness Instantiations

  • For online text classification, draft–verify retrieves similar precedents for a draft, then retrieves same-label confirmers and different-label challengers for verification.The memory bank contains labeled precedents streamed before the input, and the harness evaluation resets it and repopulates it from the test stream.Source: §3.3 · p. 5
  • For mathematical reasoning, plan–solve uses a reference solution to create a strategic sketch, then has a solver derive the answer from that sketch.At harness-baseline evaluation, the reference solution is removed and the model executes both roles from the problem alone.Source: §3.3 · p. 5

4 Experiments

The experiments evaluate OPHSD with draft–verify harnesses for online text classification and plan–solve harnesses for mathematical reasoning. Across these tasks, OPHSD performs strongly without the harness, while analyses examine what behavior it internalizes and where its gains concentrate.

4.1 Experimental Setup

  • Experiments start from Qwen3-8B, using 10k training examples for each text task and 10k DeepMath problems for math benchmarks.Text tasks are evaluated on LawBench and USPTO; math evaluations use AIME24, AIME25, OlympiadBench, and HMMT25.Source: §4 · p. 5, §4.1 · p. 6

4.2 Online Text Classification Results

  • 69.51% / 90.81% are OPHSD’s LawBench F1 and USPTO accuracy, exceeding GRPO by 7.07% / 0.80% and OPSD by 5.26% / 2.80%.The draft–verify harness raises base-model LawBench F1 by 4.93% and USPTO accuracy by 48.95%; after training, reattaching it lowers OPHSD’s scores by 1.10% and 7.19%, respectively.Source: §4.2 · pp. 6–7, Table 2 · p. 6, Figure 2 · p. 6

4.3 Mathematical Reasoning Results

  • OPHSD achieves the highest average pass@8 of 69.50% across four math benchmarks, exceeding OPSD by 2.82%, GRPO by 2.93%, and CRISP by 10.75%.The table reports pass@8 accuracy; testing OPHSD through the plan–solve harness yields 69.13% on the same benchmarks.Source: Table 5 · p. 7, §4.3 · p. 7

4.4 What Does the Student Distilled from the Harness?

  • Without retrieval, OPHSD’s case-citation reasoning rises to at least 75% in the first training phase and at least 90% by training’s end.The judged behavior is spontaneous citation of a past case as evidence; the student learns the reasoning pattern, not memory-bank contents.Source: §4.4 · p. 8
  • OPHSD improves pass rate by 84.62% on problems initially solvable only with the harness and enables standalone solutions to 12.54% of problems initially unsolved in either setting.The analysis groups problems by the base model’s pass rate with and without the harness.Source: §4.4 · p. 8, Figure 4 · p. 8
  • OPHSD maintains stable math-benchmark performance while reducing redundant reasoning; OPSD’s reasoning length collapses after 40 steps, alongside a performance decline.The comparison concerns average output length without the harness; self-distillation methods’ output lengths all fall rapidly early in training.Source: §4.4 · p. 8, Figure 5 · p. 8

4.5 The Advantage Concentrates on the Hardest Problems

  • On hard problems, OPHSD’s gains over the base model are +22.9% on Math, +33.8% on LawBench, and +90.4% on USPTO, above OPSD and GRPO.Problems are divided into Hard, Medium, and Easy groups by base-model performance; the reported gap concentrates on hard groups.Source: §4.5 · pp. 8–9, Figure 6 · p. 9

4.6 Case Study: Internalization and Harness Interference

  • The paper describes the harness as a training scaffold: on LawBench, reattached retrieval can overwrite the student’s bidirectional deliberation, while plan–solve training instills structured planning.The authors report that the external retrieval stream can interfere on LawBench and that the math student learns to flag combinatorial pitfalls.Source: §4.6 · p. 9

5 Conclusion

OPHSD internalizes harness capabilities in model parameters and achieves strong standalone performance across text classification and mathematical reasoning. Validation covers two representative harnesses, not a broad sweep across harness designs.

5 Conclusion

  • OPHSD internalizes harness capabilities in model parameters, making the harness redundant or even counterproductive at inference.The conclusion frames complex harnesses as temporary training scaffolds whose procedural benefits are fed back into the base model.Source: §5 · p. 9
  • The distilled model achieves the highest standalone performance across text classification and mathematical reasoning tasks.The conclusion describes this as empirical evidence that useful aspects of the harness can be internalized.Source: §5 · p. 9
  • Validation covers two representative harnesses rather than a large-scale sweep across diverse harness designs.Source: §5 · p. 9

A Implementation Details

Experiments use Qwen3-8B and implement draft–verify for text classification and plan–solve for mathematics. The section also specifies baseline objectives and training settings.

A.1 Harness

  • Draft–verify retrieves five draft neighbours, confirmers, and challengers; it defaults to a single pass until the memory bank has 10 entries.It uses online labelled examples and resets the bank for harness baseline evaluation to prevent train-test leakage.Source: §A.1 · p. 13
  • Plan–solve uses the reference answer as privileged input during training, then removes it at harness baseline evaluation.The planner temperature is 0.3 and solver temperature is 0.6.Source: §A.1 · p. 13

A.2 Baselines

  • All methods use Qwen3-8B with the verl framework; training uses a learning rate of 1 × 10−6 and batch size 64.Source: §A.2 · p. 13
  • GRPO uses group size 8 and KL coefficient 0; OPSD uses reverse-KL distillation from a fixed initial-policy teacher with a verified-solution prefix.CRISP uses reverse-KL, a static “be concise” prompt, and synchronizes teacher weights from the student every 50 steps.Source: §A.2 · p. 13

A.3 Evaluation

Final results use each method’s highest evaluation score across training. Mathematical-reasoning results average four runs because performance is volatile.

A.3 Evaluation

  • Text classification trains for 300 steps with evaluation every 15; mathematical reasoning trains for 150 steps with evaluation every 10.Source: §A.3 · p. 14
  • Final results report each method’s highest evaluation score across the full training stage.Source: §A.3 · p. 14
  • Mathematical-reasoning results average four runs because of inherent volatility; the paper describes the comparison protocol as fair across methods.Source: §A.3 · p. 14

A.4 LLM-as-Judge Protocol for the Cite-Rate Analysis

The cite-rate analysis uses GPT-4o to label behaviors in model traces. An end-to-end inspection contrasts harness-free and harness-assisted reasoning on a borderline legal-charge question.

A.4 LLM-as-Judge Protocol for the Cite-Rate Analysis

  • GPT-4o labels whether traces cite specific past examples and compare candidate labels; cite-rate aggregates the past-example field over 40 test instances per scenario, method, and step.Source: §A.4 · p. 14

B.1 Text Classification: Where the Harness Narrows the Trained Student’s Deliberation

  • The inspected LawBench-3-3 case compares a trained student without a harness against the same model in draft–verify, with the harness-free mode outperforming the harness mode.Source: §B.1 · p. 14

The Question

  • The case is borderline because trauma-triggered cardiac death makes negligent homicide the legally correct charge rather than the obvious intentional-injury inference.Source: The Question · p. 15

Mode A — Single Forward, No Harness

Mode B — Trained Model with the Harness

With the harness attached, retrieved examples provide one-sided evidence for intentional injury, and the trained model’s reasoning narrows rather than preserving the competing negligent-homicide charge. The verify-stage retrieval does not supply relevant challengers, and the final verdict omits the gold label.

Mode B — Trained Model with the Harness

  • The draft-stage memory-bank neighbours carry intentional-injury labels, while none carries negligent homicide or is semantically related to the fight-pattern facts.The listed top five include unrelated offences such as unlawful trespass, rape, murder, and theft.Source: Mode B — Trained Model with the Harness · pp. 15–16
  • Conditioned on one-sided examples, the model’s draft reasoning collapses two-way weighing into one-way confirmation and omits negligent homicide.The excerpt generalizes from examples in which injurious conduct causing death is classified as intentional injury.Source: Mode B — Trained Model with the Harness · p. 15
  • The verify-stage challenger neighbours come from unrelated semantic clusters, so they do not provide relevant contrast to the draft.The supplied passages identify the challenger-neighbour set but do not report its individual labels.Source: Mode B — Trained Model with the Harness · pp. 15–16
  • Mode B’s verdict was intentional injury (set-F1 = 0); when asked to verify or correct the draft, the model had nothing in front of it pointing back at negligent homicide.Source: Mode B — Trained Model with the Harness · p. 16

Analysis

The analysis contrasts the harness-trained student’s internal reasoning with an example of plan–solve supervision on a math problem. The example traces how explicit planning preserves an ordered-pair distinction that the OPSD student misses.

Analysis

  • The analysis describes Mode A’s internal retrieval as a recall–compare–decide loop that keeps negligent homicide viable alongside intentional injury.By contrast, Mode B’s memory-bank neighbours are dominated by routine intentional-injury cases, and its challenger slot contains unrelated offences; the student then abandons its bidirectional weighing and loses the gold label.Source: Analysis · p. 16

The Question

  • The question asks for the coefficient of x^10 in f(f(x)), where f(x) sums powers of two, and gives 40 as the answer.A stated pitfall is counting (2, 8) and (8, 2) as one unordered pair, which leads to 39 instead.Source: The Question · p. 16

Step 1 (Training): the Harness Generates a Plan

  • During training, the plan-solve harness wraps the student model; in the plan phase it is asked to outline a solution without solving, and for k = 1 find all pairs of powers of two that sum to 10.The solve phase follows the plan and produces 0 + 2 + 10 + 28 = 40. Over eight stochastic harness rollouts at training step 80, four traces are correct.Source: Step 1 (Training): the Harness… · pp. 16–17

Step 2 (Inference): the OPHSD Student Catches the Pitfall

B.2 Mathmatical Reasoning: Plan-Structured Supervision Improves Reasoning Precision

Conversely, the OPSD Student Makes the Error

  • Trained on ground-truth-prefixed traces without explicit enumeration plans, the OPSD student conflates ordered and unordered sequences; among exponent pairs from {1, 2, 4, 8} summing to 10, only (2, 8) occurs, and the total is 39.Source: Conversely, the OPSD Student Makes the… · p. 17

Analysis

The paper contrasts plan–solve supervision with ground-truth imitation as sources of reasoning structure. In the illustrated math example, the former foregrounds a conceptual distinction that the latter misses.

Analysis

  • Both students approach the problem similarly, but OPSD fails to distinguish ordered from unordered sequences at a critical step.The passage characterizes the divergence as structural rigor rather than reasoning capacity.Source: Analysis · p. 17
  • Plan–solve training asks the model to articulate potential pitfalls before calculation, while OPSD’s reference-prefix training lacks that structural grounding.The paper says the OPHSD student internalizes the need to formalize enumeration semantics before calculating.Source: Analysis · p. 17
  • The example presents programmatic harness supervision as embedding rigorous analytical patterns more deeply than standard ground-truth imitation.Source: Analysis · p. 17

C Discussion: Which Harnesses Are OPHSD-Compatible?

The discussion classifies harnesses by whether their benefits can be reproduced from the input and the model’s parametric memory. It also gives a diagnostic for compatibility and flags safety implications of standalone procedural reasoning.

C Discussion: Which Harnesses Are OPHSD-Compatible?

  • OPHSD compatibility depends on whether harness gains can be reproduced from the input and the model’s pre-existing parametric memory.The discussion distinguishes fully distillable procedural priors, partially distillable procedures relying on dynamic content, and non-distillable real-time external interaction.Source: C Discussion: Which Harnesses Are… · p. 18
  • Plan–solve is presented as fully distillable: it restructures processing without adding information unavailable in principle from the input, and OPHSD recovers essentially all harness-induced gains.Source: C Discussion: Which Harnesses Are… · p. 18
  • Draft–verify is partially distillable because its reasoning pattern transfers, but its queried memory-bank content does not.How much harness gain is recovered depends on whether the model’s parametric prior supplies enough domain knowledge; otherwise the procedure operates on weaker evidence.Source: C Discussion: Which Harnesses Are… · p. 18
  • Harnesses using real-time oracles such as live web search, code execution, or stateful environments are classified as non-distillable because their information is not derivable from the input or model weights.Distilling surrounding tool-use procedure is described as plausible but unvalidated and outside the paper’s scope.Source: C Discussion: Which Harnesses Are… · p. 18
  • The proposed compatibility diagnostic removes privileged context at inference and checks whether substantial performance gain over the base model remains.Preserved gain suggests procedural priors; collapse suggests reliance on real-time external grounding.Source: C Discussion: Which Harnesses Are… · p. 18

D Broader Impacts

  • Standalone internalized reasoning can reduce latency, token costs, and engineering complexity, but the paper warns of misuse risks and says high-stakes use still needs ground-truth verification and safeguards such as human review.Source: D Broader Impacts · p. 18

Related papers

Loading 2605.08741v1…