PaperlayerView exact revision on arXiv

Source-linked AI summary

Harness-Model Collaboration for Long-Horizon Continual Learning

Zhen Bi, Zhizhi Peng, Yan Wang, Xueshu Chen, Haofei Zhang, Yaobo Zhang, Zeyu Yang, Qing Shen, Jungang Lou

arXiv:2610.34724v1cs.AI

TL;DR

Long-horizon agents accumulate reusable procedural knowledge, but model adaptation and external memory alone trade persistent learning against rapid reuse. HaMo combines strategy-level memory with experience-conditioned continual adaptation and retrieval-coverage-based trajectory selection. Across within-domain and cross-domain continual learning, it achieves a better balance between adaptation and retention than harness-only or model-only approaches, while cross-domain forgetting remains only partially mitigated.

  • Problem

    Long-horizon agents lack a systematic way to accumulate reusable procedural knowledge across tasks, while model adaptation and external memory separately trade persistent learning against rapid reuse.

    Source: §1 · p. 1

  • Method

    HaMo stores trajectories as strategy-level experience cards, selects complementary training trajectories by retrieval coverage, and continually trains the model to execute retrieved experience in context.

    Source: §1 · pp. 1–2

  • Results

    Across within-domain and cross-domain continual learning, HaMo achieves a better balance between adaptation and retention than harness-only or model-only approaches.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The results highlight the value of coordinating external experience accumulation with internal model adaptation for long-horizon agents.

    Source: §6 · p. 9

    HaMo partially mitigates cross-domain forgetting but does not prevent it completely, and retention depends on the domain sequence.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Long-horizon agents continuously encounter reusable procedural knowledge, yet existing systems typically rely either on model adaptation or external memory, trading persistent learning for rapid reuse. This limitation is especially important for Web agents that repeatedly execute related multi-step workflows. We propose HaMo, a Harness-Model collaborative framework that combines fast non-parametric experience accumulation with slow parametric adaptation. HaMo distills trajectories into strategy-level experience cards for immediate retrieval, while an experience-conditioned continual learning objective trains the model to interpret, ground, and execute retrieved strategies across changing contexts. A retrieval-coverage-based selection mechanism further identifies compact and complementary trajectories for parameter updates. Experiments show that HaMo outperforms harness-only and model-only adaptation under within-domain and cross-domain shifts, improving both retention and adaptation. These results highlight the importance of jointly preserving procedural experience and internalizing the competence to transfer and execute it across tasks.

1 Introduction

Long-horizon agents need to reuse procedural knowledge across related tasks, but model-only and harness-only approaches each leave a different gap. HaMo combines persistent strategy memory with experience-conditioned model adaptation, and its evaluations report stronger adaptation and retention than either approach alone.

1 Introduction

  • Model-only adaptation can preserve past interactions as behavioral competence but is costly and may interfere with acquired capabilities under sequential shifts; harness memory enables immediate reuse without retraining.Memory alone does not ensure the model can interpret and execute retrieved experience.Source: §1 · p. 1
  • HaMo combines persistent strategy-level experience cards for immediate reuse with model adaptation that learns to interpret and execute retrieved strategies in changing contexts.Cards contain task context, initial conditions, and a strategy-level solution outline; the model is conditioned on retrieved experience during learning and inference.Source: §1 · pp. 1–2
  • HaMo selects compact, complementary trajectories by retrieval coverage so parameter updates focus on broadly reusable strategies without training on every stored trajectory.The selection favors experiences that support many tasks while covering distinct strategy contexts.Source: §1 · p. 2
  • Across ScienceWorld, ALFWorld, and WebShop, HaMo achieves stronger continual adaptation and better retention than harness-only or model-only learning under within-domain and cross-domain shifts.Harness-only methods reuse experience rapidly but reach a performance ceiling, while model-only adaptation can suffer substantial forgetting under distribution shifts.Source: Figure 1 · p. 1, §1 · p. 2

2 Related Work

Prior work addresses long-horizon interaction and experience reuse, while continual-learning methods separately develop external memory or parameter adaptation. HaMo coordinates the two by assigning immediate reuse to strategy memory and selected experience plus replay to parameter updates.

2.1 LLMs for Long-Horizon Tasks

  • Research on long-horizon tasks spans reasoning, planning, interaction, and experience reuse, including trajectory compression, rollout aggregation, global memory, and reusable workflows.The paper focuses instead on retaining and consolidating reusable experience across continual long-horizon task streams.Source: §2.1 · p. 2

2.2 Continual Learning for LLM Agents

  • External-memory methods retain experience without frequent weight updates, while parameter-based continual learning uses techniques such as replay and parameter protection to mitigate forgetting.These lines of work do not directly decide which experience should remain externally retrievable and which should drive parameter updates.Source: §2.2 · p. 2
  • HaMo coordinates rapidly updated strategy memory for immediate reuse with selected trajectories and replay for less frequent parameter adaptation.This differs from prior work combining evolving experience with continual world-model learning, which the paper names as related context.Source: §2.2 · p. 2

3 Method

HaMo couples persistent harness memory with continual model adaptation through a shared experience-conditioned interface. It distills trajectories into cards, selects complementary training examples by retrieval coverage, and trains a model to execute retrieved strategies in context.

3 Method

  • HaMo uses persistent external memory to make strategy-level experience available immediately, while continual parameter adaptation trains the model to interpret and execute that experience in context.Training and inference use the same initial dialogue format: the request, initial observation, and injected matched experience.Source: §3 · p. 2

3.1 Harness-Level Memory

  • Each expert trajectory is distilled into a card combining its task description, initial observable conditions, and extracted solution outline; new cards are added to retained memory.Cards remain mapped one-to-one to their source trajectories, allowing selected cards to identify training trajectories.Source: §3.1 · pp. 2–3
  • Retrieval coverage estimates which query cards a candidate card can support, using leave-one-out nearest-neighbor retrieval among candidate cards.The method retrieves three nearest neighbors for each query and reverses those relations to estimate candidate reuse scope.Source: §3.1 · p. 3
  • The selection greedily adds cards with the largest marginal coverage, favoring broadly reusable experiences that add strategy contexts not already covered.A coverage-saturation curve and its knee determine the stage-specific training budget; unselected cards remain in memory and available for retrieval.Source: §3.1 · pp. 3–4

3.2 Model-Level Adaptation

  • Model adaptation combines newly selected anchor trajectories with historical replay, conditioning expert-action training on the harness experience matched to each trajectory.Replay preserves exposure to earlier task behaviors and experience-conditioned strategy use as parameters evolve.Source: §3.2 · p. 4
  • Experience-conditioned training teaches the model to interpret strategy guidance and instantiate it across changing objects, observations, states, and interaction histories rather than memorize each stored strategy.At inference, the harness supplies retrieved guidance and the model adapts it to the evolving execution trajectory.Source: §3.2 · pp. 4–5
  • A single LoRA adapter is updated across stages while backbone parameters remain frozen, keeping the trainable parameter budget fixed.The optional post-training memory-release step is disabled in reported experiments, so all experience cards remain available across stages.Source: §3.2 · p. 4

4 Experimental Setting

The experiments cover three text-based interactive benchmarks and test continual learning within ScienceWorld and across sequential domain shifts. Comparisons include harness and model adaptation baselines, using environment-specific scores and stage-wise evaluation.

4.1 Datasets and Continual Learning Protocol

  • ScienceWorld, ALFWorld, and WebShop cover scientific experimentation, household tasks, and instruction-conditioned online shopping in text environments.Their action spaces and task semantics differ.Source: §4.1 · p. 5
  • Within-domain learning adds four sample-disjoint ScienceWorld subsets, while cross-domain learning sequentially trains on ScienceWorld, ALFWorld, and WebShop and evaluates encountered domains after each stage.The main cross-domain order is S → A → W; ablations also consider A → W → S and W → S → A.Source: §4.1 · p. 5

4.2 Baselines

  • Baselines include a fixed-parameter Base+Harness, AgentTuning, ETO, and component-ablation variants that isolate harness-only and model-only adaptation.HaMo uses a single continually updated LoRA adapter with a frozen backbone; Model only excludes external harness and historical replay.Source: §4.2 · p. 5

4.3 Metrics

  • Scores use each environment’s native metric on a 0–100 scale, with higher values better and episodes capped at 50 interaction turns.ScienceWorld scores task progress, ALFWorld task completion, and WebShop product-match reward; Figure 3 reports stage-wise and average ScienceWorld scores, while Figure 4 shows cross-domain scores by training stage.Source: §4.3 · p. 5, Figure 3 · p. 6, Figure 4 · p. 6

5 Results

HaMo improves continual-learning performance across within-domain and cross-domain evaluations, while the ablations show that retention, acquisition, and aggregate performance vary with adaptation components, training order, and model scale. A WebShop case study illustrates constraint checking against the current task.

5.1 Main Results

  • HaMo’s ScienceWorld score rises from 38.13 to 60.49 with Qwen3-8B and from 38.61 to 61.15 with Qwen3-14B across four stages.Four-stage averages are 52.22 and 53.09, exceeding Base+Harness by 13.88 and 12.44 points, respectively.Source: §5.1.1 · p. 6
  • After the final S →A →W stage, HaMo scores 52.43 on ScienceWorld, 66.79 on ALFWorld, and 59.64 on WebShop, outperforming every listed baseline on each domain.These scores exceed the strongest non-HaMo baseline by 13.02, 19.71, and 5.16 points, respectively.Source: §5.1.2 · p. 6

5.2 Ablation Results

  • On Qwen3-8B, Model only’s ScienceWorld score falls from 53.98 after the first stage to 24.57 finally, while HaMo finishes at 52.43.HaMo exceeds Model only by 27.86 points on final ScienceWorld, although Model only scores higher on final ALFWorld (69.34 versus 66.79).Source: Figure 5 · p. 7, §5.2.1 · p. 7
  • Across tested orders, final-stage totals are 178.86 for S →A →W, 188.41 for A →W →S, and 181.94 for W →S →A.A →W →S has the highest total, while W →S →A has the highest final ALFWorld and WebShop scores (74.09 and 60.43).Source: Figure 6 · p. 7, §5.2.2 · pp. 7–8
  • First-trained-domain retention depends on order: ScienceWorld returns to 52.43 in S →A →W, while it falls from 55.19 to 47.42 in W →S →A.In A →W →S, ALFWorld rises from 66.79 to 73.72; in W →S →A, WebShop rises from 54.99 to 60.43.Source: Figure 7 · p. 7, §5.2.2 · p. 8

5.2.4 Case Study: Experience-Guided Constraint Ver-

  • In the WebShop case, HaMo checks available options and the current $60 budget, scoring 100 versus Base’s 25.The candidate title did not establish the requested caramel flavor; the case does not isolate the contributions of retrieved guidance and parameter adaptation.Source: Table 1 · p. 8, §5.2.4 · pp. 8–9

5.2 Ablation Results

  • HaMo remains effective with Qwen3-1.7B, 4B, and 8B, and the largest backbone achieves the strongest overall results.Larger models generally show stronger acquisition of newly introduced tasks, especially after adapting to later domains.Source: Figure 8 · p. 8, §5.2.3 · p. 8

6 Conclusion

HaMo pairs persistent strategy-level memory with experience-conditioned parameter adaptation. Across three benchmarks, it offers a better balance between adaptation and retention than harness-only or model-only approaches.

6 Conclusion

  • Across ScienceWorld, ALFWorld, and WebShop, HaMo balances adaptation and retention better than harness-only or model-only approaches, with stronger robustness under within-domain and cross-domain learning.The paper presents coordinating external experience accumulation with internal model adaptation as a direction for long-horizon agents that learn from interaction while preserving capabilities.Source: §6 · p. 9
  • HaMo combines persistent strategy-level memory with experience-conditioned parameter adaptation for long-horizon continual learning.New procedural knowledge is available for immediate reuse while the model gradually improves at interpreting and executing retrieved strategies.Source: §6 · p. 9
  • The experiments cover ScienceWorld, ALFWorld, and WebShop, and report stronger robustness under both within-domain and cross-domain continual learning.Source: §6 · p. 9

Limitations

HaMo mitigates but does not eliminate forgetting during cross-domain training; retention varies with the domain sequence.

Limitations

  • HaMo offsets some forgetting from cross-domain training but does not fully prevent catastrophic forgetting.Source: Limitations · p. 9
  • Subsequent adaptation may still reduce performance on domains learned earlier.Source: Limitations · p. 9
  • Retention depends on the domain sequence, so the results support partial mitigation rather than complete immunity to forgetting.Source: Limitations · p. 9

Future Work

Future work proposes more adaptive use of retrieved strategies and broader evaluation of HaMo’s components, costs, and transfer scope.

Future Work

  • A proposed extension would learn when to retrieve a strategy, reject an unsuitable card, or refresh guidance during execution.The suggested decisions could use retrieval confidence, task progress, and explicit checks of strategy preconditions to respond to misleading experience.Source: Future Work · p. 9
  • Controlled ablations, repeated runs, and longer task streams are proposed to clarify component contributions, variability, and persistent forgetting.The suggested ablations cover replay, anchor selection, experience conditioning, and decoding temperature.Source: Future Work · p. 9
  • End-to-end cost measurements and tests on multimodal agents and changing web environments are proposed to examine memory-maintenance tradeoffs and transfer beyond text-based benchmarks.Source: Future Work · p. 9

A Continual Adaptation and Inference Procedures

The procedures retain experience cards while selecting a compact training collection by coverage, then adapt an adapter using matched experience and replay. At inference, retrieved experience guides actions alongside the evolving interaction history.

A Continual Adaptation and Inference Procedures

  • Coverage is computed within the incoming card collection; selecting anchors changes the training collection, while all experience cards remain in memory.The implementation uses lazy priority-queue updates for the same marginal-coverage selection, and the historical collection supplies replay rather than an additional trainable model.Source: A Continual Adaptation and Inference… · p. 11
  • For adaptation, HaMo retrieves or matches experience for replay trajectories, pairs expert actions with context and experience, and trains the same LoRA adapter while keeping the backbone frozen.Training uses the experience-conditioned behavioral-cloning objective; the replay collection is empty at the first stage.Source: A Continual Adaptation and Inference… · p. 11
  • At inference, the task and initial observation retrieve experience, which conditions actions over an interaction loop that records observations and stops at termination or the 50-step limit.Subsequent actions also depend on the evolving interaction history; no expert solution or future observation is needed to construct the inference query.Source: A Continual Adaptation and Inference… · p. 11

B Implementation and Experimental Settings

The experiments carry a frozen-backbone LoRA adapter across adaptation stages and use stochastic inference. Evaluation covers sample-disjoint within-domain stages and cross-domain task orders.

B Implementation and Experimental Settings

  • Training updates only a rank-16, α=32 LoRA adapter for three epochs at 2 × 10^-4; the backbone stays frozen and the adapter carries across stages.Table 2 is labeled as summarizing implementation and evaluation settings.Source: B Implementation and Experimental… · p. 11, Table 2 · p. 11
  • Inference uses temperature 0.7 to retain action diversity, but the paper does not attribute gains to temperature without an ablation.Sampling is intended to allow alternative actions when the current state differs from a retrieved example; it is a decoding choice, not a training regularizer or guarantee against overfitting.Source: B Implementation and Experimental… · pp. 11–12
  • Episodes end at environment termination or 50 turns; within-domain evaluation tests after four sample-disjoint ScienceWorld training stages, while cross-domain evaluation measures encountered domains.Native task scores are reported on a 0–100 scale, with partial credit retained for ScienceWorld and WebShop.Source: B Implementation and Experimental… · p. 12

Related papers

Loading 2610.34724v1…