Harness Annealing: Learning to Act with Less External Control

1 INTRODUCTION

• Harness Internalization means assuming specified control responsibilities while maintaining task performance after the associated external support is withdrawn.Evaluation must examine both task performance and control-related behavior after support removal; tools and execution infrastructure remain external.Source: §1 · pp. 2–3
• Existing trajectory-based training typically leaves harness-generated prompts, states, and interventions as conditioning inputs rather than explicit supervision targets.This work instead focuses on transferring specified runtime control responsibilities and evaluating checkpoints with and without corresponding control layers.Source: §1 · p. 2
• For 9B, the H2-targeted checkpoint improves tools-only scores over the H4-trained starting checkpoint by 5.01 points on SWE-QA and 3.09 points on SWE-QA-Pro.The gains vary with model scale and deployment configuration, and further annealing does not uniformly improve performance.Source: §1 · p. 3
• HAT combines explicit control supervision with a curriculum over progressively weaker harness support, supplemented by replay and cue dropout.The curriculum introduces trajectories collected without verification, then without workflow, and finally without state support.Source: §1 · pp. 2–3
• The evaluation covers 9B and 35B models on SWE-QA and SWE-QA-Pro, with the full checkpoint-by-harness matrix revealing that preferred checkpoints vary by scale and deployment objective.Selected annealed checkpoints improve performance under weak support.Source: §1 · p. 3

Adapting the external harness

• Harness-optimization approaches improve external workflows or runtime components, but retain the optimized runtime at deployment.The paper uses a fixed harness-removal ladder to transfer selected responsibilities to the policy.Source: §2 · p. 3

Learning from interaction trajectories

• Trajectory-based training supervises interactions, while this work turns targeted control responsibilities into model outputs and retains observed evidence as input.The Direct–Layered comparison evaluates the construction jointly because segmentation also changes token weighting and optimizer exposure.Source: §2 · p. 3

Distilling guidance and learning harness policies

• Guidance distillation transfers documents, skills, or workflows, whereas this work combines explicit control-decision targets with predetermined layer removals.Every checkpoint is evaluated under every deployment harness to distinguish reduced scaffold use from retained capability after removal.Source: §2 · p. 3

3.1.1 EXECUTION OF AGENTIC TASKS

• Agentic tasks require sequential evidence gathering under partial observability, with sparse intermediate feedback and a need to ground final outcomes in observed evidence.The judge scores final answers without intermediate observations, so it does not directly assess whether answers are evidence-grounded.Source: §3.1.1 · p. 4

3.1.2 FORMALIZING HARNESS OPERATIONS

• The nested harness levels add control layers to tools, with state support at H2, workflow at H3, and verification at H4; higher-level traces include lower-level behaviors.Each level retains preceding support, while tools, execution guards, and token budgets remain external.Source: §3.1.2 · pp. 4–5, Table 1 · p. 4

3.1.3 HARNESS INTERNALIZATION OBJECTIVE

• Internalization is assessed through weak-support performance and score variation across harnesses; low variation alone is insufficient if performance is poor throughout.The method considers H1 performance together with cross-harness variation and the full score matrix.Source: §3.1.3 · p. 5

3.2 HARNESS ANNEALING TRAINING

• HAT addresses the gap between trajectory SFT, where harness decisions remain context, and training that makes selected control decisions explicit targets while progressively introducing weaker-harness demonstrations.The objective is to retain task performance after associated external support is withdrawn.Source: §3.2 · p. 5

3.2.1 LAYER-WISE REASONING SUPERVISION

• Layer-wise supervision makes selected control decisions explicit targets so the policy can act with weaker external support.Records separate observed task and harness cues in OBS from control-oriented judgments in THINK and recorded actions or answers in ACT; loss covers THINK and ACT tokens.Source: §3.2.1 · pp. 5–6

3.3 HARNESS-DEPENDENCY ANNEALING

• The annealing curriculum introduces D3, D2, and D1 sequentially, mixing current-source records with stronger-source replay and applicable cue-dropout variants.An H3-targeted checkpoint continues from H4-trained data and tests transfer under H2, where workflow and verification control are not supplied.Source: §3.3 · p. 6

3.4 DESIGN RATIONALE

• The curriculum targets verification, then workflow, then state control, while replay retains earlier supervision; source records need not match deployment contexts.The final stage introduces tools-only source data, but stronger-support replay remains part of training.Source: §3.4 · p. 7

4 EXPERIMENTS

• The experiments ask whether annealing internalizes control, whether Layered supervision transfers better than trajectory-level SFT, and whether performance improves on out-of-distribution tasks.Training uses SWE-QA-Pro repository-level QA trajectories; evaluation uses its test split and SWE-QA.Source: §4 · p. 7, §4.1 · p. 7

Harness Details

• Training fine-tunes 9B and 35B models on teacher trajectories from four source harnesses, using progressively weaker source-harness mixtures for successive annealing stages.Evaluation uses two modes that are not pooled; the main annealing results use thinking-enabled evaluation.Source: Implementation Details · pp. 7–8

4.2 MAIN RESULTS

• For 9B, H2-targeted improves H1 scores by 5.01 points on SWE-QA and 3.09 points on SWE-QA-Pro; for 35B, H1 scores improve by 0.40 and 1.36 points, respectively.The comparison is against the H4-trained starting checkpoint; Table 3 evaluates every checkpoint under all four deployment harnesses.Source: §4.2 · pp. 8–9, Table 3 · p. 8
• After targeted support is removed, 9B H4-trained under H3, H3-targeted under H2, and H2-targeted under H1 score 77.47, 77.15, and 77.00 on SWE-QA.On SWE-QA-Pro, the corresponding scores are 76.87, 77.96, and 77.73; all red cells exceed Base under the same weak deployment.Source: Table 3 · p. 8, §4.2 · p. 9
• The preferred checkpoint depends on scale and deployment objective: 9B H2-targeted averages 77.17, while 35B H4-trained averages 81.59 across the two benchmarks and four deployment harnesses.At 35B, H2-targeted nevertheless achieves the highest H1 score on both benchmarks.Source: Table 3 · p. 8, §4.2 · p. 9
• Under H1 input cues, stop/continue macro-F1 rises from 64.43 at H4-trained to 87.79 at H1-targeted in a teacher-forced probe.The probe holds reference states fixed and varies input cues, measuring action selection rather than executing runtime harnesses.Source: §4.2 · p. 9

4.3 DIRECT VERSUS LAYERED SUPERVISION

• At H1, Layered supervision outperforms Direct supervision for every source, benchmark, and model scale; for 9B SWE-QA, their score ranges are 53.73–72.48 versus 16.51–32.45.The comparison evaluates two training recipes; adapted checkpoints are independently trained from Base using the indicated source level.Source: Table 4 · p. 9, §4.3 · p. 10

4.4 CROSS-TASK OOD GENERALIZATION

• On 97 banking-knowledge tasks, zero-shot evaluation suggests annealing can improve transfer beyond repository QA, with a peak at an intermediate stage.All checkpoints use the same fixed banking harness; the modest gain over Direct SFT and unmatched training budgets do not isolate control internalization as the source.Source: §4.4 · p. 10, Figure 3 · p. 10

5 CONCLUSION

• Harness-supported experience may help reduce runtime control requirements when a model can reliably assume a control responsibility.The authors frame this as a model–harness feedback loop in which harness development can focus on remaining model limitations.Source: §5 · p. 10
• HAT combines explicit control supervision with demonstrations collected under progressively weaker harnesses.The experiments report improved weak-support performance on SWE-QA and SWE-QA-Pro, but further annealing is not uniformly beneficial.Source: §5 · p. 10
• The proposed co-evolution path is conditional: runtime guidance need not be repeated when the model can reliably take over that control responsibility.The harness can then focus on the model’s remaining limitations.Source: §5 · p. 10

A LIMITATIONS

• Experiments use two Qwen3.5 models, one fixed harness hierarchy, and a single training seed, with repository QA and one banking-domain transfer experiment.Whether the recipe transfers to other model families, task domains, or harness designs remains open.Source: A LIMITATIONS · p. 13
• Later checkpoints receive more training and use different data mixtures, so without equal-budget controls the differences cannot be attributed to annealing order alone.The contributions of replay and cue dropout also cannot be separated.Source: A LIMITATIONS · p. 13
• Direct and Layered supervision are not matched for supervised-token counts or optimizer updates, so their comparison evaluates training recipes rather than trajectory segmentation alone.The runtime hierarchy ranges from H1 tools-only to H4 with tools, state, workflow, and verification.Source: A LIMITATIONS · p. 13, B IMPLEMENTATION DETAILS · p. 13

B.1 TRAINING CONFIGURATION

• The 9B annealing stages continue the preceding LoRA adapter for one epoch each, with seed 42 and 111, 129, and 144 optimizer updates.Each stage uses four GPUs, global batch size 16, bf16, and a 65,536-token cutoff.Source: §B.1 · p. 13
• The 35B stages use six GPUs, global batch size 12, one epoch each, and 151, 156, and 121 optimizer updates for H3-, H2-, and H1-targeted training.Training and data use seed 42, with a 65,536-token cutoff.Source: §B.1 · p. 13
• For 35B, checkpoint selection uses held-out validation loss on 43 training-source question IDs, selecting the H2-targeted adapter at 0.75 epoch.The H4-trained predecessor and H1-targeted adapter are selected at their final updates.Source: §B.1 · p. 13

B.2 TRAINING MIXTURES

• The 9B mixture weights use the original token proxy, whereas the 35B weights use tokenizer-counted supervised tokens.Whole-record sampling makes realized 35B proportions slightly different from target percentages.Source: §B.2 · pp. 13–14
• Table 5 reports current-source and replay percentages for each stage, proceeding from H3-targeted to H1-targeted training after a stage starting from H4-trained.Replay columns denote data sources, not deployment harnesses.Source: Table 5 · p. 14
• For 35B H3 data, cue dropout removes workflow instructions and an added workflow observation; for H2 data, it removes state instructions, the progress ledger, and appended harness state.These variants retain their source index and preserve recorded interaction history; they are not new lower-harness rollouts.Source: §B.2 · p. 14

C TRAINING PATHS AND CONTROL-WITHDRAWAL EVALUATION

• Hℓ-targeted checkpoints continue the curriculum from Layered H4-trained through source level ℓ, whereas Hℓ-trained checkpoints are trained independently from Base on one source level.For example, H3-targeted follows H4 → H3, while H3-trained uses only H3-source training; the index marks the latest source introduced, not the deployment harness.Source: C TRAINING PATHS AND… · p. 14, Table 6 · p. 14
• Table 6 maps cumulative training paths to control-withdrawal evaluations, and every checkpoint is evaluated under all four harnesses.The table states learning objectives; checkpoint names alone do not establish successful internalization.Source: Table 6 · p. 14, C TRAINING PATHS AND… · p. 15
• Standard H3 records retain workflow cues even when H3-targeted is tested under H2, so training inputs and deployment support are specified separately.H1-targeted is a tools-only endpoint, not a step toward tool-free H0 deployment.Source: C TRAINING PATHS AND… · pp. 14–15

D HARNESS DETAILS

• All four deployment harnesses expose Read, Grep, Glob, and read-only Bash; higher levels add state, workflow, and answer-checking support.H2 bundles recent-history retrieval, a progress ledger, and loop detection; H4 checks candidate answers and permits at most two keyword-based repair searches.Source: D HARNESS DETAILS · p. 15, Table 7 · p. 15

E EVALUATION WITH THE LLM JUDGE

• The gpt-5.5 judge scores each final answer using the question, reference answer, and candidate answer, without access to interaction history or tool observations.The judge therefore does not directly assess whether the answer is grounded in trajectory evidence.Source: E EVALUATION WITH THE LLM JUDGE · p. 15
• For each example, the judge assigns five equally weighted integer scores for correctness, completeness, relevance, clarity, and reasoning; their sum is S_i, with S_i ∈ [5, 100].No additional weighting, normalization, or learned calibration is applied, and the primary judge does not emit a binary pass/fail decision.Source: E EVALUATION WITH THE LLM JUDGE · pp. 15–16

E.1 COMPLETE JUDGE PROMPT

• The complete judge prompt directs conservative, strict reference-based scoring, reserving scores of 16–20 for answers that closely match the reference.It instructs the judge to reduce scores for inaccuracies, omissions, or imprecision.Source: §E.1 · p. 16

F DECISIONS AFTER REMOVING HARNESS CUES

• The paired probe tests whether removing input cues changes next-action selection by scoring checkpoints on the same fixed reference states.The selected H4-trained source contains 1,279 records over 464 questions; the probe uses 395 states from 128 questions, and prior model history may retain effects of the original harness.Source: F DECISIONS AFTER REMOVING HARNESS CUES · p. 18, §F.1 · pp. 18–19

F.1 REFERENCE STATES AND PAIRED INPUTS

• The H1–H4 probe conditions are nested input-cue ablations, not reruns of executable harnesses; the question, tool schemas, observations, prior model messages, and target response remain unchanged.H4 keeps original cues; H3 removes verification instructions, H2 also removes workflow instructions, and H1 additionally removes state instructions and harness-generated messages.Source: §F.1 · pp. 18–19

F.2 WHAT THE THREE METRICS MEASURE

• All probe scores use teacher forcing: the model sees reference reasoning and each predicted token is conditioned on the reference token prefix, with no tools executed.The probe tests action selection given that reasoning, not the ability to generate it unaided.Source: §F.2 · p. 19
• The stop/continue macro-F1 metric classifies the first action token as continue when it predicts a tool-call marker and stop otherwise, then averages the two class-specific F1 scores.A non-tool token need not be a valid final answer.Source: §F.2 · p. 19
• Late/final decision accuracy measures agreement with the reference continue/stop decision on a subset of the first metric’s states.There is no independent annotation of evidence sufficiency, so this is not independent corroboration of control transfer.Source: §F.2 · p. 19

F.3 WHERE CONTROL DECISIONS IMPROVE

• Under H1 input cues, stop/continue macro-F1 rises from 64.43 at H4-trained to 87.79 at H1-targeted, while late/final accuracy rises from 62.50 to 83.13 and next-tool accuracy changes from 96.24 to 97.18.Table 9 reports teacher-forced scores on the same reference states; columns vary input cues, not executable harnesses.Source: §F.3 · p. 19, Table 9 · p. 20
• The H4-to-H1 descriptive gap contracts from 31.28 to 6.16 points for stop/continue and from 31.88 to 8.12 points for late/final decisions.This coincides with H4-input stop/continue F1 falling from 95.71 to 93.95 and late/final accuracy falling from 94.38 to 91.25.Source: §F.3 · p. 20

• The task asks how ensure_eol() modifies the output buffer, but this single pair establishes neither an aggregate effect nor a grounded final answer.Source: G WORKFLOW RECOVERY CASE · p. 21, Figure 4 · p. 21
• Under H2, the H3-targeted run changes strategy and returns an answer after 20 tool calls, while the H4-trained run repeats an unresolved search.Source: Figure 4 · p. 21
• Both runs stop below their active step limits without response truncation; the answer cites a samenamed texinfo.py helper without verifying the inherited docutils implementation.Source: Figure 4 · p. 21