PaperlayerView exact revision on arXiv

Source-linked AI summary

Harness-Aware Distillation for Small Language Model Agents

Moonseok Choi, Taehong Moon, Giung Nam, Juho Lee

arXiv:2610.02858v1cs.AI

TL;DR

When a large agent is distilled, the harness remains in place, but ordinary on-policy distillation does not reliably teach the student to act on its information. HAD adds a contrast between the same teacher’s actions with and without harness information, filters contradictory preferred actions, and outperforms same-harness on-policy distillation baselines across long-horizon tasks.

  • Problem

    On-policy distillation with a shared harness does not reliably transfer the teacher’s ability to act on harness information, even though the harness remains available to the smaller student.

    Source: §1 · pp. 1–2

  • Method

    HAD combines on-policy response distillation with action preferences contrasting the same teacher with and without harness information, filtering preferred actions that contradict harness records.

    Source: §1 · p. 2

  • Results

    HAD achieves the best performance among on-policy distillation methods equipped with the same harness across multiple long-horizon agent tasks.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The analysis suggests HAD learns to recover from stalls while keeping learnable feedback in its weights and reading state information from the harness.

    Source: §1 · p. 2

    Evaluation covered students up to 2B parameters and teachers up to 30B in text-based environments, with a hand-designed harness.

    Source: §5 · p. 9

Abstract

from arXiv · show

Language model agents are deployed with a harness, the software around the model that manages its context, tools, and feedback. When such an agent is distilled into a smaller one, the harness stays in place, so the student mainly needs the teacher-specific abilities that the harness cannot provide, such as acting correctly on harness information. Standard distillation, however, imitates the teacher's full outputs and treats the harness as part of the input. We propose Harness-Aware Distillation (HAD), which focuses distillation on what the teacher adds beyond the harness. HAD complements on-policy distillation with two components: an action preference that contrasts the same teacher's actions with and without the harness information, scored after the student's own reasoning, and a validity check that drops preference pairs whose preferred action contradicts the harness records. We show that the contrast gives the student information that imitating the teacher alone cannot provide, and HAD needs no task rewards, success labels, or future information. Across multiple long-horizon agent benchmarks and models, HAD outperforms on-policy distillation baselines with the same fixed harness. Our analysis shows that HAD enters fewer unproductive loops and recovers from errors more often than the baselines, and suggests that it adaptively keeps learnable feedback in its weights while reading state information from the harness.

1. Introduction

Because the harness remains fixed when a teacher is replaced by a smaller model, the student needs to learn how to act on harness information, not what the harness already provides. HAD adds harness-aware preferences and validity filtering to distillation, and outperforms on-policy distillation baselines using the same harness.

Motivation

  • Adding the harness to on-policy distillation does not reliably transfer the teacher’s ability to use it, and harness utilization improves only inconsistently across methods.Harness utilization is agreement between student actions and harness records.Source: §1 · pp. 1–2
  • On ALFWorld, adding the harness to distillation changes performance from 43.1% to 43.5%, while HAD raises it to 57.4%.Harness utilization changes from 65.7% to 73.1% with the harness added, and to 81.0% with HAD.Source: Figure 2 · p. 2
  • On-policy distillation misses harness-dependent decisions, conflates harness-provided information with teacher judgment, and accepts teacher targets that contradict harness records.For example, a harness may reveal that a target requires an object the agent does not hold.Source: §1 · p. 2

Contributions

  • HAD contrasts the same teacher’s actions with and without harness information, then filters preference pairs whose preferred action contradicts harness records.The preference is scored after the student’s own reasoning; HAD uses no task rewards, success labels, or future information.Source: §1 · p. 2
  • HAD outperforms on-policy distillation baselines with the same harness across multiple long-horizon agent benchmarks and models.The authors attribute its advantage to learning recovery from stalls while using the harness for state information.Source: §1 · p. 2

2. Related Work

Related work covers agent distillation, harness and external-context learning, and preference-based supervision. HAD differs by constructing action preferences from the same teacher’s harness-conditioned and stripped-harness actions at student-visited states.

Distilling language model agents

  • Agent distillation transfers reasoning and tool-use behavior to smaller models, with on-policy methods supervising student-generated sequences using teacher feedback.Prior extensions include step-level supervision, temporal curricula, and selective teacher intervention.Source: §2 · p. 3

Learning with external context and agent harnesses

  • Context distillation and harness self-distillation aim to internalize contextual knowledge or harness-augmented reasoning, while other work studies harness variation and external guidance.This paper instead supervises use of harness information while retaining the harness at deployment.Source: §2 · p. 3

Contrastive supervision and preference-based distillation

  • Prior preference methods construct pairs from contrasting prompts, retrieval-conditioned answers, or teacher and student generations; HAD compares the same teacher with and without harness information.It prefers the harness-conditioned action and excludes pairs that contradict harness records, without task rewards or success labels.Source: §2 · p. 3

3. Method

HAD trains on student-visited histories with the harness enabled, combining response distillation with action preferences formed by contrasting the same teacher with and without harness information. It filters contradictory preferred actions and focuses preference supervision on actions under the student’s own reasoning.

3.1. Harness-Aware Distillation

  • The harness deterministically augments each observation with information computed from environment state and interaction history; stripping removes that information, not its influence on the realized trajectory.The paper makes no assumption about the content of the harness information.Source: §3.1 · p. 3
  • Response distillation trains on student rollouts with the harness enabled, using the teacher’s complete response at each visited history.The response includes reasoning and action, and is distilled with mean token negative log-likelihood.Source: §3.1 · pp. 3–4
  • HAD queries the same teacher on the stripped history and forms action preferences between harness-conditioned and stripped-history actions.Action-token scores are compared under the student’s reasoning rather than the teacher’s reasoning.Source: §3.1 · p. 4
  • The validity checker rejects actions that conflict with harness-induced observations, masking only the preference loss while leaving response distillation unchanged.The final objective combines response distillation with validity-filtered action preferences, with the loss weight adjusted by minibatch gradient-norm balancing.Source: §3.1 · p. 4

3.2. What Harness Awareness Adds to Distillation

  • The preference contrast emphasizes actions more likely with harness information relative to without it, a distinction response distillation alone cannot identify.Harness-conditioned actions are labeled positive and stripped-history actions negative, assuming harness information generally helps the teacher choose better actions.Source: §3.2 · pp. 4–5
  • Filtering removes preferred actions that contradict harness observations, but cannot exclude actions that are valid yet incorrect.If a positive action is rejected, only its preference loss is dropped; the limitation follows from distillation using an incomplete teacher.Source: §3.2 · p. 5

3.3. Why Action-Only Preference under Student Reasoning

  • Action-only preferences target harness-induced action changes, avoiding reasoning differences that can dominate comparisons of full teacher responses.The full-response variant performs worse in the reported ablation.Source: §3.3 · pp. 5–6
  • Scoring teacher actions under student-generated reasoning matches the student’s deployed decision context, while response distillation separately supervises teacher reasoning.The preference term does not condition action supervision on the teacher’s no-harness reasoning.Source: §3.3 · p. 6

4. Experiment

The experiments evaluate HAD against on-policy distillation baselines under a shared harness across long-horizon agent benchmarks. HAD leads the reported comparisons, while analyses examine its error recovery, harness reliance, cross-model performance, and design components.

4.1. Experimental setup

  • The evaluation covers ALFWorld, WebShop, and ScienceWorld with Qwen3 students, plus Gemma-4 on ALFWorld; methods share a task-agnostic harness and training loop.The harness adds state-tracking and behavior-monitoring information to observations without selecting or blocking actions.Source: §4.1 · p. 6

4.2. Main experiment

  • HAD achieves the best performance in every environment; on ALFWorld it reaches 63.4% on unseen and 51.4% on seen tasks, versus 47.0% and 41.4% for the best baseline.The main-results table reports performance with and without the harness; HAD also exceeds its 8B teacher on ALFWorld.Source: §4.2 · p. 7, Table 2 · p. 7

4.3. Analysis

  • HAD avoids stalls in 75.9% of episodes and escapes 59.7% of stalls, versus at most 64.6% stall-free episodes for other models; no baseline clearly improves recovery over the untrained student's 46.8%.Every counted stall is flagged by the harness, suggesting HAD learns to act on harness information to leave a stall.Source: §4.3 · p. 8, Table 3 · p. 8
  • Hiding state tracking lowers HAD's success rate from 63.4% to 37.3% on unseen tasks and from 51.4% to 40.7% on seen tasks; hiding behavior monitoring lowers it to 56.0% and 45.0%.The authors infer that state tracking is more essential at deployment, while HAD may internalize part of behavior-monitoring feedback.Source: §4.3 · p. 8, Table 4 · p. 8

4.4. Cross-model generalization

  • With Gemma-4, HAD reaches 37.3% on unseen and 36.4% on seen ALFWorld tasks, versus 32.1% and 34.3% for the strongest baseline, extending the result across model families.The evaluation uses Gemma-4-E2B distilled from Gemma-4-12B under the same recipe and harness.Source: §4.4 · p. 8, Table 5 · p. 8

4.5. Ablation study

  • Removing the negative, leaving the student to imitate only the harness-chosen action after its own reasoning, lowers SR to 52.4% and harness utilization from 81.0% to 74.4%, close to 73.1% for response distillation alone.Response distillation alone yields 43.5% SR. The contrast with the action chosen without the harness accounts for most harness-utilization gain, while the validity check mainly protects task performance.Source: §4.5 · p. 9
  • With twice the teacher budget, OPD and SOPD each rise from 43.5% to 46.7% and 46.8%, respectively, still below HAD's 57.4%.The baselines receive twice as many student trajectories and updates, favoring them in this comparison.Source: §4.5 · p. 9, Table 7 · p. 9
  • Scoring only the action after the student's reasoning avoids dilution by long reasoning and teacher-style differences; using a teacher reference model does not help, and success peaks at ρ=0.5.The authors say this action-level scoring matches how the student chooses an action at deployment.Source: §4.5 · p. 9

5. Conclusion

HAD distills teacher-specific harness use through action preferences and a validity check, outperforming on-policy distillation methods using the same harness across long-horizon tasks. Evaluation is limited to text environments, smaller students and teachers, and hand-designed harnesses.

5. Conclusion

  • Across multiple long-horizon agent tasks, HAD achieves the best performance among on-policy distillation methods equipped with the same harness.Source: §5 · p. 9
  • HAD contrasts the same teacher’s actions with and without harness information, then drops preferred actions that contradict harness records.Source: §5 · p. 9
  • Evaluation covers students up to 2B parameters, teachers up to 30B parameters, text environments, and hand-designed harnesses.Larger models, tool-calling or software agents, and automatically learned or optimized harnesses remain future work.Source: §5 · p. 9

A.1. Practical Implementation

The appendix specifies practical choices for HAD and analyzes how action-preference learning and validity filtering alter the supervision signal. Its analyses also identify a regularization trade-off and the limits of using harness access as a preference proxy.

A.1. Practical Implementation

  • The checker rejects preferred actions contradicted by harness records at the visited state, without task rewards or future information; rejected pairs leave response distillation intact.It cannot detect actions that are valid but do not advance the task.Source: §A.1 · p. 13
  • The preference gradient is scaled to contribute approximately half as much as the distillation gradient, while β = 0.5 is selected with ρ on held-out validation tasks.Source: §A.1 · p. 13

A.2. Ablation Variants

  • The ablation variants distinguish response distillation alone and no filtering; another comparison changes both the scored span and action-conditioning prefix.The latter retains student-prefix supervision but removes the stripped-history comparison.Source: §A.2 · pp. 13–14

A.3. What Harness-Aware Action Preference Learning Adds

  • The positive response law alone does not generally identify optimal action-preference score differences; the additional teacher query supplies a contrast response imitation lacks.The analysis does not rule out useful harness-dependent behavior learned through response imitation.Source: §A.3 · p. 15

A.4. Regularization Bias and Reference-Relative Preferences

  • With identical positive and negative action laws, isolated reference-free preferences can flatten a nonuniform teacher distribution; a reference-relative variant instead has the reference law as its isolated optimum.Choosing a reference is nontrivial, and the teacher reference ablation performs worse empirically.Source: §A.4 · p. 15

A.5. What Harness-Aware Validity Filtering Adds

  • Without filtering, a rejected action can have a larger positive-to-negative probability ratio than an accepted action, so validity filtering addresses harness access as an imperfect preference proxy.The analysis does not require checking negative candidates during training.Source: §A.5 · p. 16

A.6. Why Action-Only Preference Rather Than Full-Response Preference

The analysis distinguishes action-only comparisons at student reasoning from full-response comparisons that also score teacher reasoning. These objectives can rank actions differently, especially when reasoning contrasts vary across actions.

A.6. Why Action-Only Preference Rather Than Full-Response Preference

  • Full-response preferences score each candidate with its own teacher reasoning, changing both the scored span and action-conditioning context relative to the conditional action comparison.Source: §A.6 · p. 16
  • At a shared prefix, full-response preferences use a reasoning-conditioned action contrast, whereas action-only preferences use the marginal action contrast.A reasoning contrast constant across actions cancels; an action-varying contrast can change the ranking.Source: §A.6 · p. 17
  • With positive responses and action marginals fixed, changing only negative reasoning can reverse full-response action orderings while leaving the conditional HAD objective unchanged.This example concerns isolated preference optima, not the full-response variant combined with response distillation or task utility.Source: §A.6 · pp. 17–18

A.7. Action-Only Preferences at Student Reasoning Prefixes

The appendix formalizes action preferences scored at student reasoning prefixes and relates preference scores to normalized action probabilities. It also qualifies how prefix coverage and the isolated preference objective affect interpretation.

A.7. Action-Only Preferences at Student Reasoning Prefixes

  • At independently sampled student prefixes, teacher reasoning generates candidate actions but is marginalized in their action law, so the same marginal action contrast is taught at each supervised prefix.Source: §A.7 · p. 19
  • Reusing positive teacher reasoning as the scoring prefix changes the target by adding compatibility with that selected reasoning trace.An independently sampled student prefix avoids this extra conditioning.Source: §A.7 · p. 19
  • Student-prefix sampling avoids transferring excess risk between prefix distributions, but does not guarantee low excess risk or improved task return.If a comparison distribution assigns zero mass to a student prefix with positive excess risk, no finite transfer factor suffices.Source: §A.7 · p. 20

A.8. From Scores to Probabilities

  • The isolated preference optimum over normalized probabilities is determined by score differences and action lengths; variable lengths can make probability and score orderings differ.These are isolated preference-loss optima, not optima of the combined HAD objective.Source: §A.8 · p. 21

B.1. Models and tasks

The appendix describes the benchmark settings, shared harness design, and harness-utilization measure used to evaluate the methods. The harness reports state and behavior information, while utilization scores consistency with task-agnostic rules.

B.1. Models and tasks

  • Across ALFWorld, WebShop, and ScienceWorld, the appendix specifies environment settings and measures, with model and evaluation settings summarized in Table 8.WebShop uses a 1,000-product version; ScienceWorld uses version 1.2.3 with teleport simplification and 149 test variations over 30 task types.Source: Table 8 · p. 21, §B.1 · p. 21
  • ALFWorld prompts retain the full alternating-turn history and insert harness lines in the latest turn; the task line uses the trial’s first human annotation.That annotation differs from the templated goal in 10 of 274 evaluation tasks, which are excluded from qualitative examples.Source: §B.1 · p. 21

B.2. Harness

  • The harness shares code and templates across environments, adds only state and behavior reports to observations, and neither supplies task hints nor selects or blocks actions.Information already displayed in an environment remains available in every condition, including evaluation without the harness.Source: §B.2 · p. 22
  • Harness thresholds are fixed before distillation using held-out validation tasks, with firing rates calibrated to avoid interrupting progress in successful episodes.No evaluation task is used for calibration; the training objective of HAD uses no task rewards or success labels.Source: §B.2 · p. 22, Table 10 · p. 22

B.3. Harness utilization

  • Harness utilization averages consistency across five separately scored components, excluding reports whose correct response depends on the task and the budget counter.The unweighted mean prevents components that apply at every step from outweighing those that fire rarely.Source: §B.3 · p. 23
  • The utilization rules check admissible actions, held objects, budget warnings, reported failures, and whether a new state follows a stall report.A stall is counted as consistent if a new state is reached within five steps or the episode ends successfully.Source: §B.3 · pp. 23–24

B.4. Implementation details

All methods use a shared on-policy training loop and common implementation settings, while the OPD-based baselines differ in their supervision. The appendix also documents the prompt format used in the qualitative examples.

B.4. Implementation details

  • In each synchronous training round, the student rolls out 16 tasks with the harness, and collected steps are retained for two policy versions and used once.All methods use full fine-tuning; OPD-family KL uses the teacher’s top 64 tokens, and the learning rate is 10^-5 because 10^-6 did not train students in this setting.Source: §B.4 · p. 24
  • HAD’s validity checker rejects preferred actions that are inadmissible, use an unheld object, or repeat an action reported ineffective at the same state.The teacher samples responses with and without harness information independently at temperature 1.0, so action differences include sampling variation.Source: §B.4 · p. 24
  • The OPD baselines differ in turn weighting, teacher takeover, and step-level supervision, while OPD itself uses token-level reverse KL on student-sampled responses.SAGE-OPD weights turns by teacher-assessed correction need and confidence; Guided-OPD uses a decaying takeover schedule; SOPD trains on a teacher-written step at each student state.Source: §B.4 · p. 24

C. Prompts and Examples

  • The ALFWorld examples show the recorded prompts with only the last three turns, and harness lines appear in the latest user turn before admissible commands.Instructions are identical across ALFWorld cases and are shown in full in Figure 6; the task line is the trial’s first human annotation.Source: C. Prompts and Examples · p. 25, Figure 6 · p. 25

D.1. Analysis: turns

With the harness, HAD takes a similar number of turns to other methods on successful ALFWorld episodes while solving the most tasks. The examples illustrate both successful use of a harness report and a failure to recover from a reported lack of progress.

D.1. Analysis: turns

  • With the harness, HAD takes a similar number of turns to other methods on successful ALFWorld episodes while solving the most tasks.Adding the harness increases turns for most models, including the teacher and untrained student.Source: Table 13 · p. 25, §D.1 · p. 25
  • In the success example, HAD follows the report that the sofa is the only unvisited place and finds the second remote, while baselines make different action errors.OPD attempts to place a remote it is not holding, and SOPD returns to a place already searched.Source: Figure 7 · p. 26
  • In the failure example, HAD repeats inventory after a lack-of-progress report and reaches the step limit; OPD’s reply is not a same-state comparison.OPD is holding a heated mug at that turn, unlike HAD’s state with an unheated mug.Source: Figure 8 · p. 27

D.2. Generalization to other model architecture

On ALFWorld with Gemma models, HAD achieves the highest harness-equipped success rate on both splits and the highest harness utilization. Without the harness on unseen tasks, SOPD exceeds HAD’s success rate.

D.2. Generalization to other model architecture

  • Without the harness on unseen tasks, SOPD reaches 29.9% success rate against HAD’s 23.9%, consistent with HAD relying on the harness retained at deployment.The comparison concerns the no-harness condition on unseen tasks.Source: §D.2 · p. 28, Table 14 · p. 28
  • With the harness, HAD has the highest success rate on both splits and 77.9% harness utilization, above the teacher’s 75.9%.Every OPD baseline ends below the untrained student’s 71.4% harness utilization.Source: Table 14 · p. 28, §D.2 · p. 28
  • With the harness, HAD takes more turns in successful episodes than the other methods while solving the most tasks.Table 14 reports turns as the mean over successful episodes.Source: Table 14 · p. 28, §D.2 · p. 28

D.3. Additional ablations

The additional ALFWorld ablations examine preference-term designs and weights. A separate analysis measures how often harness information changes teacher actions across three benchmarks and where those changes occur.

D.3. Additional ablations

  • On ALFWorld, success is highest at ρ=0.5 among Table 16’s tested preference weights, which range from 0.25 to 2.0.The main experiments use ρ=0.5.Source: §D.3 · p. 28, Table 16 · p. 28
  • Table 15 compares HAD with full-response contrast and teacher-reference variants on ALFWorld.The passage gives no results for these comparisons.Source: §D.3 · p. 28, Table 15 · p. 28

D.4. Where the harness changes the teacher’s action

  • Teacher actions with and without the harness differ at 15.5% of ALFWorld states, 25.6% of WebShop states, and 19.3% of ScienceWorld states, spread throughout episodes.Figure 9 shows the frequency of differences at each episode position.Source: Figure 9 · p. 28, §D.4 · p. 28

Related papers

Loading 2610.02858v1…