PaperlayerView exact revision on arXiv

Source-linked AI summary

Harness Policy Distillation: Aligning Agent Policies with Runtime Semantics

Junze Zhu, Weihao Chen, Zhiyang Zhou, Zhen Wu, Xinyu Dai

arXiv:2610.20261v1cs.LGcs.AI

TL;DR

Product-specific runtime harnesses govern executable actions and termination, but ordinary trajectory supervision leaves the control state behind each action implicit. HPD replays trajectories to create control documents, verifies action targets, and trains shared model views; on TRAJECT-Bench, it achieves higher task success than OPD in the reported comparison.

  • Problem

    Ordinary trajectory supervision conditions reference actions on observed histories but leaves the runtime control state underlying those actions implicit.

    Source: §1 · p. 2

  • Method

    HPD compiles privileged control documents at replayed boundaries, verifies proposed actions against runtime requirements, and jointly trains shared views for control prediction and action selection.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    22.98 task success versus OPD’s 15.61 at nearly equal aggregate F1 on TRAJECT-Bench.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The results support aligning agent policies with runtime semantics as well as task actions.

    Source: §5 · p. 10

    The formulation assumes finite execution and treats fatal contract violations or budget exhaustion as absorbing failure.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Deploying a general-purpose model as an agent means running it inside a product-specific runtime harness that decides which actions are executable, how state advances, and when the loop may terminate. Ordinary trajectory supervision shows which action was taken but leaves the runtime semantics behind that action implicit. We introduce Harness Policy Distillation (HPD), which makes these runtime control semantics an explicit training target alongside actions to align the harness policy. HPD replays reference trajectories through the harness to recover the runtime semantics that are implicit in ordinary trajectory data, and represents them as control documents at each model-call boundary. These documents provide privileged information for a teacher to generate actions, while harness verification ensures that the resulting targets are executable under the runtime contract. We jointly train a shared model on the runtime semantics recovered from control-state privileged information and native history, thereby internalizing these semantics into the model policy. At deployment, the trained model uses the original harness interface without requiring the control document or any additional supplement. On TRAJECT-Bench with the Pi harness, Qwen3-8B trained with HPD reaches 22.98% autonomous task success and 49.23% runtime compliance, against 3.09% and 19.90% for the unadapted model. Harness-verified teacher actions substantially improve the native deployment policy, while joint control-state supervision provides further gains in task completion and runtime compliance. HPD turns executable runtime semantics into dense supervision for learning harness-compatible actions at each decision boundary.

1 INTRODUCTION

Product-specific harnesses govern which actions execute and when agents may stop, but ordinary behavior supervision leaves the runtime control state behind each action implicit. HPD pairs explicit control-state supervision with verified action targets at replayed boundaries, then trains shared views so the model can act through the original harness without a control document.

1 INTRODUCTION

  • The harness governs execution while the model selects actions, so the adaptation target is a policy whose actions remain executable under the product’s runtime contract.The harness also serializes history, exposes approved actions, enforces permissions and workflow stages, inserts tool observations, schedules calls, and defines valid termination.Source: §1 · p. 1
  • Ordinary trajectory, preference, reward, and teacher objectives leave underlying control state to be inferred through their action, preference, reward, or teacher targets.These objectives can learn runtime-sensitive behavior, but their standard forms do not separately specify the control state.Source: §1 · pp. 1–2
  • Runtime-state mismatch arises because action labels do not specify remaining obligations or whether termination is allowed, even on valid reference histories.The resulting errors can affect later decisions because each action changes the next runtime state and model context.Source: §1 · p. 2
  • The harness’s execution history and bookkeeping expose obligations and lifecycle conditions, making boundary control state an explicit prediction target alongside the next action.Tool availability alone does not show whether the corresponding obligation has already been completed.Source: §1 · p. 2
  • HPD compiles privileged control documents at replayed boundaries, verifies proposed actions against runtime requirements, and trains paired control and action targets.Recorded actions advance replay, preserving reference histories for the paired supervision.Source: §1 · p. 2
  • HPD jointly trains a shared model to predict control documents from native history, select actions with documents, and predict actions from native context.At deployment, the model uses the original harness interface without an external control document or additional model call.Source: §1 · p. 2

2 RELATED WORK

Prior work changes action targets, rewards, or the states where supervision is supplied, while harness and agent research highlights the runtime lifecycle. HPD treats runtime facts and reference-derived control labels as supervised targets at replayed decision boundaries.

2 RELATED WORK

  • Trajectory learning, distillation, on-policy distillation, and agentic RL primarily alter action targets, rewards, or visited-state occupancy; HPD adds runtime facts and reference-derived control labels at replayed boundaries.The cited lines of work include behavior cloning, sequence models, teacher-sequence transfer, and supervision or reward on student-generated prefixes.Source: §2 · p. 3
  • Privileged-information and process-supervision methods provide related training-time context or intermediate labels, while HPD pairs replay-derived control documents with harness-verified actions at the same boundary.The passage also distinguishes π-Distill’s privileged teacher/student setup and Training with Harnesses’ on-policy self-teacher.Source: §2 · p. 3
  • Tool-agent and benchmark work treats stateful execution and the deployed interface, event stream, sandbox, and stopping rules as central; HPD makes the runtime lifecycle a supervised object.Evaluation work also separates final task completion from intermediate progress and policy compliance.Source: §2 · p. 3

3 METHODOLOGY

The methodology formulates an agent as a model acting through a harness that serializes runtime states and advances execution. HPD addresses the resulting supervision gap by replaying reference trajectories, compiling control documents, verifying action targets, and training shared native-context and document-conditioned views.

3.1 CLOSED-LOOP EXECUTION UNDER A RUNTIME HARNESS

  • A runtime state includes execution history, tool interfaces, permissions, scheduling, obligations, and remaining budget; replay also tracks recorded tool-observation queues.The formulation assumes finite execution and adds an absorbing failure state for fatal contract violations or budget exhaustion.Source: §3.1 · p. 3
  • The executable process couples a policy’s actions with harness serialization and state transitions, so each action determines the successor context and links boundary learning to closed-loop completion.The harness transition law covers parsing, permission checks, tool execution, observation insertion, scheduling, and stopping; terminal and failure states are absorbing.Source: §3.1 · pp. 3–4

3.2 RUNTIME SEMANTIC ALIGNMENT GAP

  • Policy distillation depends on both the context distribution and the teacher’s conditional action distribution; replayed-reference and free-rollout boundary distributions can differ.On-policy distillation moves supervision toward the expected boundary counts under free rollout.Source: §3.2 · p. 4
  • HPD labels lifecycle stage, remaining obligations, ready calls, and termination conditions alongside a verified action at each replayed reference boundary.Verified records define the replay context distribution; filtering them can change the relative weights of reference histories.Source: §3.2 · p. 4

3.3 HARNESS POLICY DISTILLATION

  • HPD replays recorded actions through the harness, supplies queued observations, and captures the next runtime state before compiling a boundary control document.The document combines observed runtime facts with reference-stage progress and witness-derived labels for the next control, ready frontier, and final permission.Source: §3.3 · pp. 4–5
  • Harness verification checks admissibility, reference-stage consistency, and exact agreement with recorded control decisions, tool names, and normalized arguments before selecting a candidate by reference similarity.The recorded action advances replay, while the selected verified candidate becomes the training target.Source: §3.3 · p. 5
  • Three shared-model views connect native history to control-document reconstruction, document-conditioned action selection, and native-context action prediction.The native-context action view directly supervises the deployment policy, which acts without the control document.Source: §3.3 · p. 5

4 EXPERIMENTS

Experiments evaluate HPD on TRAJECT-Bench/Pi, cross-benchmark transfer, component and schedule ablations, action formation, and adaptation across architectures and harnesses. Full HPD leads the main comparison in task success, while further analyses examine which training views and runtime settings support its performance.

4.1 SETUP

  • The evaluation uses 583 held-out TRAJECT-Bench tasks and reports completion, answer, action-accuracy, reference-path, and runtime-compliance metrics from pi-runtime.Baselines include supervised, preference, reinforcement-learning, and distillation methods.Source: §4.1 · p. 6

4.2 MAIN COMPARISON

  • 22.98 TS: Full HPD leads the TRAJECT-Bench/Pi comparison, exceeding OPD’s 15.61 TS by 7.38 points; adding I/R to HPD-T adds 2.40 TS points.Full HPD also leads in F1, Step-SR, and Exact; the five-seed paired analysis supports both TS differences.Source: Table 1 · p. 6, §4.2 · pp. 6–7
  • SFT raises RC to 55.92 and Step-SR to 47.20 but lowers TS to 2.06 and F1 to 2.40; OPD has the highest RC at 66.21, while HPD has higher TS.Reward-based baselines show that higher answer overlap need not sustain reference-path execution.Source: §4.2 · pp. 6–7

4.3 CROSS-BENCHMARK ROBUSTNESS STUDY

  • On 92 BFCL v4 cases without BFCL-specific training, Full HPD reaches 21.74 TS and 48.12 Step-SR, versus Base’s 13.04 TS and 42.03 Step-SR.HPD-T improves answer overlap and prefix execution over Base but not task completion; HPD’s gains are consistent with runtime-semantic training supporting generalization beyond source-task patterns.Source: §4.3 · p. 7

4.4 HPD COMPONENTS AND TRAINING SCHEDULE

  • Only joint T/I/R raises TS beyond HPD-T; at matched exposure, joint view updates outperform staged schedules on TS and RC.T+R improves Step-SR from 59.25 to 63.13 and RC from 43.74 to 48.03 while TS stays at 20.58; staged T→R drops TS from 20.58 to 7.03 and RC from 43.74 to 18.35.Source: §4.4 · pp. 7–8, Table 2 · p. 7

4.5 TOKEN DISTRIBUTIONS AND STATE-CONDITIONED ACTION ALIGNMENT

  • At identical harness states, HPD assigns 0.9962 mean probability to the reference control entry, versus 0.6242 for Base, 0.5087 for GRPO, and 0.1296 for SFT.HPD’s reference-conditioned entropy profile broadly overlaps Base, while its complete-action state-validity rate is 81.48%, versus 7.40% for SFT despite similar median reference-continuation probabilities.Source: §4.5 · p. 8, Figure 3 · p. 8

4.6 ADAPTATION ACROSS ARCHITECTURES AND HARNESSES

  • Separate adaptation raises TS and F1 across evaluated architectures and harnesses, including 20.07 TS points for Llama3-8B and 9.44–18.35 TS points for ReAct, Compiler, and OpenCode.Each harness adaptation rebuilds replay, control documents, and verification while retaining the T/I/R objective and native-runtime deployment.Source: §4.6 · pp. 8–9, Figure 4 · p. 9

4.7 API MODEL DEPLOYMENT REFERENCES

  • In the shared Pi evaluation, HPD reaches 22.98 TS versus DeepSeek V4 Flash’s 22.30, while GPT-5.4-mini reaches 41.92 F1 versus HPD’s 41.22.The API comparison replaces only the model backend; HPD’s checkpoint is unchanged.Source: Figure 5 · p. 9, §4.7 · p. 9

5 CONCLUSION

HPD targets runtime-state mismatch by training models on both actions and the execution conditions governing them. On TRAJECT-Bench with Pi, it outperforms OPD on task success, and joint control-state training improves completion further.

5 CONCLUSION

  • 22.98 task success for HPD versus OPD’s 15.61 at nearly equal aggregate F1 on TRAJECT-Bench with Pi.HPD-T already benefits from verified action supervision, while joint control-state training further improves completion.Source: §5 · p. 9
  • HPD targets runtime-state mismatch by aligning action supervision with the execution conditions that govern an agent’s policy.The paper argues that this alignment supports training harness-compatible actions at decision boundaries.Source: §5 · pp. 9–10
  • 21.74% task success for Full HPD versus 13.04% for HPD-T on 92 BFCL v4 cases without retraining.The study also reports that separate adaptations improve performance across model backbones and harnesses.Source: §5 · p. 10

A SCOPE AND LIMITATIONS

HPD adapts to a specified harness by replaying reference trajectories, compiling boundary-specific control documents, and training from verified actions and control labels. Its scope depends on harness-specific components, successful reference prefixes, and verifier coverage.

A SCOPE AND LIMITATIONS

  • HPD trains for a specified runtime contract; adapting to another harness requires reconstructing its teacher and training a separate adapter.Reference replay supplies successful reference prefixes as the training distribution.Source: A SCOPE AND LIMITATIONS · p. 13
  • The evaluation verifier checks tool names, required keys, control flow, and termination; argument-level matching uses required keys.Teacher construction requires exact agreement on designated harness-policy and tool-call portions of the reference.Source: A SCOPE AND LIMITATIONS · p. 13
  • Reference replay primarily covers successful recorded prefixes; extending training to states after deployment errors would require verified corrections.Source: A SCOPE AND LIMITATIONS · p. 13

B REPLAY CONSTRUCTION DETAILS

  • The workflow replays trajectories through the harness, compiles a control document at each retained model-call boundary, and trains one student across three views.The student learns native-runtime action prediction, document-conditioned action prediction, and document reconstruction.Source: Figure 6 · p. 13

B.1 ASSUMPTIONS AND HARNESS ADAPTATION

  • Replay requires executable recorded actions, available pre-action prefixes, prefix-derived observations, declared annotation provenance, and verifiable successor transitions.The selected verified candidate supplies the action target; replay advances with recorded actions and consumes tool observations from per-tool queues.Source: §B.1 · pp. 13–14
  • Each harness needs its own serializer, transition law, decision boundaries, control-state adapter, prompt constructors, and verifiers.Pi, Compiler, and OpenCode represent different lifecycle structures through harness-specific adapters.Source: §B.1 · p. 14

B.2 AUTOPROBE BOUNDARY AND FIELD PROVENANCE

  • The probe snapshots Pi at an actual model decision boundary, after prior actions and observations but before the next action is generated.The compiler distinguishes pre-action observations, reference-ledger progress labels, and witness-derived next-decision labels in its privileged control document.Source: §B.2 · p. 14

B.3 TEACHER SAMPLING, VERIFICATION, AND RETRIES

Teacher construction samples actions under privileged control documents, verifies them against the harness and reference, and retains a selected hard target. The experiments then evaluate the resulting views across training configurations and deployment settings.

B.3 TEACHER SAMPLING, VERIFICATION, AND RETRIES

  • Two candidate actions are sampled; the hard gate enforces reference control decisions, tool names, and normalized arguments before similarity ranking selects a verified target.The selected verified action is the shared hard target for T and I, keeping executable tool content reference-aligned.Source: §B.3 · pp. 14–15
  • The first two-candidate attempt accepts 87.90% of raw trajectories, with cumulative coverage reaching 100% by the fifth attempt.When neither candidate passes, construction extends the prefix by one recorded turn and retries.Source: §B.3 · p. 15, Table 3 · p. 15

C.1 DATA CONSTRUCTION AND EVALUATION

  • The partition contains 4,703 training, 584 validation, and 583 held-out test tasks; Pi’s test set has 2,140 gold-history action boundaries.Main end-to-end evaluation uses temperature zero and one rollout per case.Source: §C.1 · p. 15

C.2 TRAINING-STEP CONVENTION

  • Each active view receives 20k batch exposures; joint T/I/R training logs 20k, 40k, and 60k optimizer updates for one, two, and three active views.In the full three-view run, these update counts correspond to approximately 6.7k, 13.3k, and 20k rounds.Source: §C.2 · p. 15

C.3 TRAINING AND COMPARISON METHODS

  • HPD uses harness runtime documents and harness-verified action targets from reference replay and frozen-model candidates, while comparison methods use distinct supervision sources.The principal experiments use Qwen3-8B with LoRA; the comparison includes SFT, preference, reward, and distillation methods.Source: §C.3 · pp. 15–16

C.4 API AND CROSS-BENCHMARK EVALUATION SETTINGS

  • The API comparison keeps the Pi agent, test tasks, settings, and metrics fixed, changing only the model backend; BFCL transfer uses existing checkpoints without retraining.BFCL evaluation retains the same Pi agent and model-side settings while replacing the test tasks.Source: §C.4 · p. 16

C.5 SEED-STUDY RESULTS

Across five supplementary seeds, Full HPD improves over Base and over the T-only ablation. The authors report gains across all four mean metrics and lower TS and F1 standard deviations than T only.

C.5 SEED-STUDY RESULTS

  • 22.14 mean TS for Full HPD, improving by 19.46 points over Base; joint T/I/R training adds 3.94 TS, 1.29 F1, 4.37 Step-SR, and 5.44 RC over HPD-T.The table reports percentages as mean ± standard deviation over five seeds, excluding the main-run seed 42.Source: Table 5 · p. 17, §C.5 · p. 17
  • Full HPD improves all four mean metrics over its T-only ablation; TS and F1 standard deviations are 1.18 and 0.39, versus 1.67 and 0.66 for T only.The authors describe the results as reinforcing the component comparison: verified targets support native-action learning, while joint control-document supervision improves outcomes further.Source: §C.5 · p. 17
  • Adjusted TS p-values are 0.00343 for Full HPD versus HPD-T and 0.00338 versus OPD under the reported paired significance test.The test uses source-cluster differences across five seed runs with Holm correction across TS, F1, Step-SR, and RC.Source: §C.5 · p. 17

C.6 CROSS-BENCHMARK TRANSFER WITHOUT RETRAINING

On 92 BFCL v4 cases without retraining, Full HPD leads task success and Step-SR, while HPD-T leads final-answer F1; the appendix also examines checkpoint progress, loss decomposition, and sampling weights.

C.6 CROSS-BENCHMARK TRANSFER WITHOUT RETRAINING

  • On BFCL v4’s 92-case transfer without retraining, Full HPD scores 21.74 TS and 48.12 Step-SR, while HPD-T leads F1 at 26.12; Full HPD beats OPD by 4.35 TS, 1.79 F1, and 2.25 Step-SR points.Source: §C.6 · p. 17, Figure 7 · p. 18

D.1 CHECKPOINT DYNAMICS

  • Figure 8 tracks loss, task success, and final-answer F1 across a 60k-step full-HPD run; base scores are 3.09% TS and 26.82% F1.The endpoint completes 20k training rounds.Source: Figure 8 · p. 18, Table 6 · p. 19

D.2 LOSS DECOMPOSITION

  • At the 60k-step endpoint, HARNESS-REPLAY, INJECTED, and RUNTIME losses are 0.2415, 0.2283, and 0.0170, respectively.The losses represent different prediction tasks with completion-length normalization, so they are reported separately.Source: §D.2 · p. 19

D.3 SIGNAL-DENSITY SEARCH

  • Injected sampling at 0.2 gives the highest RC (43.57) and Step-SR (56.40), while Runtime at 0.6 gives the highest TS (18.18), Exact (14.24), and F1 (40.05) in the one-seed sweeps.Each axis is varied separately around the shared 1:1:1 default at 20k optimizer steps.Source: §D.3 · p. 19, Figure 9 · p. 19, Table 7 · p. 20

D.4 MODEL-SIZE SCALING STUDY

HPD improves task success and final-answer F1 across seven Qwen3 checkpoints, while the accompanying metric definitions distinguish native-history action ranking, reference-prefix execution, and runtime-constrained task success.

D.4 MODEL-SIZE SCALING STUDY

  • Across seven Qwen3 checkpoints, HPD improves TS and F1 at every size, with TS gains from 4.46 points at 0.6B to 14.41 at 14B; Step-SR and RC also improve at every size.All rows use 583 cases and one seed; 14B and 32B dense checkpoints exceed the 30B-A3B MoE checkpoint on TS.Source: §D.4 · p. 20, Table 8 · p. 20

E METRIC DEFINITIONS

  • Action-Acc ranks the reference next action against a constructed alternative at each gold-history boundary; candidates are scored but not executed.The 2,140 boundaries are evaluated independently, so model-generated rollout histories do not enter this metric.Source: E METRIC DEFINITIONS · pp. 20–21
  • Step-SR credits correct reference steps only until the first divergence, even if a rollout later recovers or reaches an equivalent runtime state..Source: E METRIC DEFINITIONS · p. 21
  • Runtime compliance counts tasks meeting the harness contract, while TS requires obligation completion, final-answer F1 of at least 0.5, exactly one final, native exit, and healthy worker execution.A recovered success can have Exact=0 when it departs from the reference control path.Source: E METRIC DEFINITIONS · p. 21

E.1 TOKEN-DISTRIBUTION DIAGNOSTIC

  • The continuation statistics in the token-distribution diagnostic do not use model-generated prefixes.The supplied text identifies them as panel statistics.Source: §E.1 · p. 21

E.2 INTERPRETING F1 AND TASK COMPLETION

Aggregate final-answer F1 and task success capture distinct outcomes: similar answer overlap can accompany different successful-completion coverage, and paired task-level analyses can separate those patterns.

E.2 INTERPRETING F1 AND TASK COMPLETION

  • Mean F1 decomposes into successful-task coverage and conditional F1 among successful and unsuccessful tasks, so either component can change the aggregate.The tables report aggregate F1 and success rate as separate outcomes.Source: §E.2 · p. 21
  • HPD and OPD have nearly equal aggregate F1 (41.22 versus 41.04), but HPD has higher TS (22.98 versus 15.61) and Exact (18.70 versus 14.07).Aggregate F1 averages answer overlap across tasks, whereas TS counts tasks satisfying execution conditions and the answer threshold.Source: §E.2 · p. 21
  • Task-level comparisons can separate answer overlap on jointly completed tasks, newly completed tasks, lost successes, and tasks completed by neither method.Grouping by execution-only conditions separately distinguishes execution coverage from the answer threshold used in TS.Source: §E.2 · p. 22

F DISTILLATION DERIVATION AND IMPLEMENTATION

The appendix connects replayed document–action records to conditional teacher objectives, then describes how stored targets are sampled and trained across three views of one shared model.

F.1 REPLAY-TEACHER DISTRIBUTIONS

  • Replay records store a point-target document and selected action at each retained boundary; records sharing a serialized context form an empirical mixture when conditioned on that context.The document combines runtime facts with privileged replay labels.Source: §F.1 · p. 22

F.2 CONDITIONAL KL DECOMPOSITION

  • The KL chain rule motivates the R and I losses, while T matches the teacher’s native-runtime action marginal.T is identified as the deployment marginal.Source: §F.2 · p. 22

F.3 STORED TARGETS AND THE IMPLEMENTED LOSS

  • Training stores verifier-approved targets: T and I share an action target, while R predicts the boundary’s control document; uniform retained-record sampling uses completion-length-normalized losses.A deterministic T/I/R cycle updates shared parameters, and deployment uses only the native-harness action view.Source: §F.3 · pp. 22–23

Related papers

Loading 2610.20261v1…