HERD: Harness-Evolving Recursive Distillation for Continual Learning

1 Introduction

• Parametric methods risk catastrophic forgetting and loss of plasticity, and raw experience can add noise, duplicates, and conflicts to training.The cited works are McCloskey and Cohen, 1989; Kirkpatrick et al., 2017; Huang et al., 2024; Dohare et al., 2024; and Yang et al., 2026.Source: §1 · p. 1
• Harness-only learning keeps knowledge inspectable and editable but compresses accumulated experience into bounded plain text that loses information.The cited works on in-context learning include Brown et al., 2020; Gutiérrez et al., 2025; and Shi et al., 2025.Source: §1 · p. 1
• HERD evolves its skills to abstract reusable rules, retains high-residual samples in bounded memory, and distills the harness using the previous checkpoint and base model as drift anchors.The previous checkpoint bounds per-stage drift, while the base model bounds cumulative drift; only weights are deployed.Source: §1 · p. 2
• Across 25 stages, HERD’s final mean over seen tasks is 0.706, above every parametric baseline and 27 points above non-parametric methods, while acquisition is 0.713, statistically indistinguishable from Seq-LoRA.Forgetting is 0.010, and general capability falls 6.3 points versus 20.0 to 25.5 for other baselines.Source: §1 · p. 2
• HERD is a hybrid system that evolves a harness of skills and high-residual memory, then recursively internalizes it into weights under drift constraints.The evaluation covers TRACE+17, a 25-stage stream of eight TRACE tasks and 17 Super-NaturalInstructions tasks.Source: §1 · pp. 1–2

2.1 Preliminaries

• The learner retains a prediction model and external state across stages, updates from prior retained state and current-task data, and is evaluated on seen tasks and general-capability probes.Retained state and update compute are bounded.Source: §2.1 · p. 3

2.2 HERD in One Stage

• At each stage, HERD evolves its bounded harness, distills using the skill and verified answer, merges a fresh adapter, refreshes memory, and carries the harness forward.The distillation student sees only the bare prompt.Source: §2.2 · p. 3

2.3 Harness Evolution

• The harness writer uses a bounded task window and frozen-checkpoint errors to draft skills, committing a registry only when it lowers mean residual.Residual memory is refreshed after distillation with the highest-residual whole records; it is not replayed or used as a training target.Source: §2.3 · p. 3

2.4 Drift-Constrained Distillation

• The frozen previous checkpoint teaches from skill-and-answer context, while the student uses the same weights with a fresh adapter and only the bare prompt.Distillation minimizes forward KL over verified-answer positions; scoring the answer’s own prefix avoids prompt-echo paths with vanishing gradients.Source: §2.4 · pp. 3–4
• Checkpoint drift on residual-memory probes limits per-stage change, while base drift on fixed general probes limits cumulative change and protects general capabilities.The two anchors cover complementary risks: small checkpoint steps can accumulate, while the base cannot protect tasks it never knew.Source: §2.4 · p. 4
• Each update measures both drifts, steps on adapter parameters, and uses bounded dual ascent to increase a multiplier only when its budget is exceeded.The previous checkpoint serves as teacher and checkpoint anchor; the immutable base model is the general anchor.Source: §2.4 · pp. 4–5, Figure 2 · p. 4

3.1 Experimental Settings

• TRACE+17 contains 25 tasks, with 500 training, 100 validation, and 100 test examples per task; all methods start from Qwen3.5-4B and share a 32-update-per-stage weight-update budget.The protocol also scores five general-capability categories at stages 4, 8, 16, and 25.Source: §3.1 · p. 5

3.2 Main Results

• After stage 25, HERD matches the strongest parametric method in acquisition, retains more than any weight-updating method, loses the least general capability among them, and deploys without context.Its final average over seen tasks is 0.706, and its acquisition is statistically indistinguishable from Seq-LoRA’s 0.715; Table 1 reports the benchmark comparison.Source: §3.2 · pp. 5–6, Table 1 · p. 6
• HERD takes 10,723 seconds over the stream versus Seq-LoRA’s 10,463 seconds; OPHSD takes 659,000 seconds because it regenerates teacher context through rollouts.ACE also carries 15,410 tokens at every inference.Source: §3.2 · p. 6
• HERD’s mean on tasks 1–8 stays within half a point of its stage-8 level through stage 25, while audited stage-25 drifts are 0.021 and 0.041 against 0.05 budgets.The G-test loss is 6.3 points by stage 25; Figure 3 depicts task scores, G-test macro means, and category changes.Source: §3.2 · p. 6, Figure 3 · p. 7

3.3 Ablation Study

• Without skills, acquisition falls 4.5 points and AA falls 5.0; without both anchors, retention and general capability resemble those of a parametric baseline.OPHSD, with an unchanging harness and unbudgeted drift, has lower acquisition and a larger general-capability loss than HERD.Source: §3.3 · p. 7
• Uniformly filling residual memory keeps acquisition within half a point but doubles forgetting from 0.010 to 0.020, while stage-end DC remains within budget at 0.034.Because checkpoint probes come from memory, uniform selection anchors on records the weights already predict and leaves harder earlier records unguarded.Source: §3.3 · p. 7
• Removing the checkpoint anchor raises forgetting to 0.030 and DC to 0.118; removing the base anchor lets D0 grow from 0.104 at stage 4 to 0.212 at stage 25.Without either anchor, forgetting reaches 0.044, above every parametric baseline, and ∆Cmin reaches −0.560.Source: §3.3 · p. 7
• Under the same adapter, budget, and constraints, replacing distillation with cross-entropy on verified answers doubles general loss to ∆C −0.123 and ∆Cmin −0.320, while both audited drifts remain within budget.At inference, attaching the stage-25 harness reduces forgetting by 0.3 points and changes the other metrics by at most 0.2 points.Source: §3.3 · pp. 7–8

Parametric continual learning

• Parametric approaches isolate updates, replay data, create self-supervised training data, or consolidate memories; HERD uses a harness to select what to internalize and drift budgets to bound updates.The cited approaches include structural methods, replay, self-supervision, and weight-space consolidation.Source: §4 · p. 8

Non-parametric continual learning

• Non-parametric systems adapt external memory, context, or harnesses around a fixed model, so experience passes through bounded context without accumulating in the model itself.HERD retains a harness as its fast layer and internalizes it into weights after each stage.Source: §4 · p. 8

Hybrid continual learning

• Prior hybrid systems consolidate deployment experience, teacher-relabeled harness rollouts, or trajectories into soft prompts; HERD keeps skills and their high-residual examples, then deploys weights alone.Source: §4 · p. 8

Distillation and harness adaptation

• HERD’s teacher uses privileged harness context across stages, explicitly bounds drift, and scores verified answer prefixes rather than student rollouts; its commit gate resembles a simpler regression check.The passage also situates HERD among work on harness adaptation and continual-stream optimizer gains.Source: §4 · p. 8

• The harness iterates on skills and keeps high-residual samples they do not cover in bounded memory, before distillation internalizes the experience into weights.Source: §5 · p. 8
• The previous checkpoint bounds per-stage drift and the base model bounds cumulative drift; HERD deploys only the weights.Source: §5 · p. 8
• Across the 25-stage TRACE+17 stream, HERD matches the strongest parametric baseline in acquisition, forgets least among non-collapsing weight-updating methods, and loses the least general capability.Removing components costs acquisition, retention, or general capability, while attaching the final harness at inference barely changes scores.Source: §5 · p. 8

• Empirical validation covers one base model and one 25-stage stream, rather than a large-scale sweep.Source: Limitations · p. 8
• HERD relies on a verified answer for every training record and assumes known task boundaries, one task per stage, and no revisits.Source: Limitations · pp. 8–9
• The paper points to Appendix E for discussion of these boundaries and corresponding plans.Source: Limitations · p. 9

Harness evolution

• The task window contains whole training records; the frozen checkpoint answers them, and up to four mistakes inform a one-call skill writer.The writer is the base model, and the resulting skill replaces one with the same prefix or identifier.Source: A Implementation Details · p. 14

Harness and routing

• A skill uses the longest common prefix of task prompts for routing, and its instructions are capped at 768 tokens within the 24,576-token registry budget.The registry routes by the longest normalized matching prefix; prompts without a match run bare.Source: A Implementation Details · p. 14

Drift-constrained updates

• Each stage runs 32 updates, measures both drifts before updating, then adjusts multipliers using the same measurement with εC = ε0 = 0.05 and η = 0.5.Both multipliers start at zero; stage-end audited drift is recorded after the final update.Source: A Implementation Details · p. 15

Residual memory and probes

• After distillation, records are ranked by residual under the merged weights and skill; whole records are retained within memory limits and supply next-stage checkpoint probes.The residual refresh deduplicates records and uses identifier order to break ties.Source: A Implementation Details · pp. 14–15

Distillation and stage state

• The previous checkpoint acts as student backbone, teacher, and checkpoint anchor; a fresh adapter is merged into the weights after each stage.A restarted job can replay completed stages from recorded registry, guard, memory, drift, and adapter-hash state.Source: A Implementation Details · p. 15

Benchmark and splits

• TRACE+17 comprises 25 tasks: eight TRACE tasks in canonical order followed by 17 specified Super-NaturalInstructions tasks; prompts are deduplicated, and validation targets and test records are hidden from the model.The disjoint G-test uses 200 test items per category, with a nested 40-item probe and a separate 40-item development panel.Source: B Experiment Details · p. 15

Evaluation

• Scores use each task’s official metric, and general capability is tested on disjoint MMLU, GSM8K, HellaSwag, IFEval, and XCOPA items.Test scores do not select candidates or hyperparameters; evaluation uses greedy decoding with thinking disabled.Source: B Experiment Details · pp. 15–16

Baseline training

• Weight-updating methods receive 32 updates of 32 records per stage, with sequences capped at 16,384 tokens and a common rank-32 LoRA setup unless official code specifies otherwise.Replay, OPR-RU, TreeLoRA, MoRAM, SDFT, and OPHSD use their stated method-specific settings.Source: B Experiment Details · p. 16

Uncertainty

• Learning methods run with seeds 42, 43, and 44; reported means and sample standard deviations accompany 95% item-bootstrap intervals.Intervals use 4,000 draws resampled within tasks and G-test categories, paired between HERD and each baseline.Source: B Experiment Details · p. 16

Acquisition and retention

• HERD’s task score stays within 4 points of its learned-stage score on every task, while several baselines lose 20 points or more on at least one task.Figure 6 breaks out acquisition and change through stage 25 by task; uniform replay’s largest losses are 8 points.Source: C Additional Experimental Results · pp. 16–17, Figure 6 · p. 17

General capability

• HERD’s G-test difference from parametric baselines is 13.7 to 17.7 points, while its difference from the static harness is −1.7 points and statistically indistinguishable from zero.The latter comparison indicates general capability remains statistically indistinguishable from the frozen base under a fixed system prompt.Source: C Additional Experimental Results · p. 17

Drift audit

• HERD’s stage-end drifts stay within budget, with DC at most 0.045 and D0 at most 0.044, although within-stage peaks exceed 0.05 at multiple stages.Without the checkpoint anchor, DC exceeds budget at all 25 stages; without the base anchor, D0 exceeds budget at stage end.Source: C Additional Experimental Results · p. 17, Figure 7 · p. 18

Harness and memory state

• HERD’s harness registry ends at 12,982 tokens, or 53% of its budget, while residual memory is at its token budget from stage 3 onward.At stage 25 it holds 137 whole records representing about 16 tasks; the base multiplier is active at no more than 12% of updates after stage 12.Source: Figure 8 · p. 19, Figure 8a

Training cost

• HERD takes 316 seconds at stage 1 and 284 seconds at stage 25, with a total of 10,723 seconds, within 3% of Seq-LoRA’s total.The table reports stage and total training time as well as inference context; HERD deploys bare weights.Source: C Additional Experimental Results · p. 18, Table 4 · p. 20

Ablations

• At εC = 0.02, acquisition drops 1.8 points and ∆Cmin improves by 4.5; at εC = 0.1, forgetting rises by 1.1 points and audited DC ends at 0.071.The base-anchor removal costs 9.5 points of ∆C; re-scoring deployed weights with the harness changes AA from 0.706 to 0.708 and F from 0.010 to 0.007.Source: C Additional Experimental Results · pp. 18–19, Table 3 · p. 19

Seeds and uncertainty

• The ablations and εC sweep are single runs, whereas Tables 1 and 2 and the principal figures report means over three seeds.Gaps within the approximate reference spread are reported but not interpreted.Source: C Additional Experimental Results · pp. 19–20

HERD against parametric baselines

• At stage 6, HERD routes a NumGLUE-specific skill to the similar prompt prefix, answering with a bare number while retaining a GSM8K derivation.The parametric baselines answer both prompts with a number, while the base, static harness, and HCL produce a sentence and score zero on NumGLUE-cm.Source: D Case Studies · p. 20

Retention between checkpoints

• HERD retains the MeetingBank ordinance title at stage 25 with ROUGE-L 0.73, while Seq-LoRA gives a generic answer with ROUGE-L 0.00.The comparison uses an item learned at stage 3 and requires no replay of earlier answers.Source: D Case Studies · p. 20

HERD’s failing cases

• Probe coverage can fail when a low-residual task leaves memory: SuperNI 192 lost its last probe at stage 17 in all three seeds, and seed 42 fell from 0.83 to 0.52.By stage 25, nine or ten earlier tasks have no probe; the paper identifies a per-task probe floor as a possible repair.Source: D Case Studies · p. 20
• Skill quality varies across seeds: for SuperNI 074, two skills contradict themselves, and their outputs are questions only 21 and 64 times out of 100.The guard passed all three commits, but its mean-residual check does not ensure free generation follows the intended format.Source: D Case Studies · p. 20
• The 32-token anchor probes miss later answer-length failures: HERD’s IFEval pass rate on 17 minimum-word-count items falls to 0.18–0.24 at stage 25.The base scores 0.94 on those items; the paper also reports that a mean drift budget can leave some GSM8K prompts flipped.Source: D Case Studies · p. 20

OPHSD: drift in wording

• OPHSD increasingly reproduces privileged teacher context, with outputs mentioning a plan or reference answer rising from 0.01 at stage 1 to 0.17 at stage 25.On mathematics items the share reaches 0.84, while mathematics ends at 0.24 and instruction following at 0.38.Source: D Case Studies · p. 21

SDFT: collapse by leakage and length

• SDFT’s cutoff outputs rise from 0.01 at stage 1 to 0.94 at stage 12, alongside verbatim reproduction of the teacher instruction rising to 0.95.Its teacher is an EMA of the student, and the loss remains between 0.005 and 0.008 despite the collapse.Source: D Case Studies · p. 21

E Future Works

• Experiments cover one base model and one 25-stage stream; the authors plan tests with longer streams and more task families.The plan is framed in light of HERD’s forgetting of 0.010 and fixed per-stage cost.Source: E Future Works · p. 22
• The long-term goal is to train continually on users’ existing prompts, skills, and memories to internalize organizational procedures, domain knowledge, and codebases.Source: E Future Works · p. 22
• Because user harnesses lack verified answers and task boundaries, proposed directions include alternative accepted targets, residual-based stage boundaries, and retrieval-based skill routing.Candidate targets include user corrections, rubrics, or majority answers from multiple harness rollouts.Source: E Future Works · p. 22
• The recursive loop could extend to self-improvement by using the current checkpoint to write skills and allowing gated harness proposals for training settings and record selection.The proposals would be checked with residual tests and drift audits, with rollback when a gate fails.Source: E Future Works · p. 22