PaperlayerView exact revision on arXiv

Source-linked AI summary

Recurrent Harness for Agentic Continual Learning

Yuxin Xiao, Zhiyun Lu, Jianyu Wang, Yangtian Zhang, Yang Song, Qifan Wang, Bailin Wang

arXiv:2610.12668v1cs.LGcs.AIcs.CL

TL;DR

Agentic continual learning must adapt harnesses across streams and distribution shifts, but existing approaches typically evolve harnesses only forward or rely on replay or validation data. ReHarness revises retrieved reflections using downstream trajectory feedback while grounding new content in source trajectories. Across three benchmarks and three LLM backbones, it consistently outperforms strong baselines and improves stability and plasticity.

  • Problem

    Existing harnesses typically evolve only through forward information flow or require additional replay or validation data, limiting strictly online harness optimization.

    Source: §1 · p. 1, §3.2 · p. 4

  • Method

    ReHarness combines forward read and write with backward revision, using each instance’s trajectory to revise retrieved reflections while grounding new content in source trajectories.

    Source: §1 · p. 1

  • Results

    Across three benchmarks and three LLM backbones, ReHarness consistently outperforms strong baselines, achieving 0.240 overall normalized gain and ranking first or tied in eight of nine benchmark–backbone combinations.

    Source: §1 · p. 2, §4.2 · p. 6

  • Takeaways & Limitations

    ReHarness produces source-grounded revisions, corrects overgeneralized or outdated reflections under distribution shifts, and improves stability and plasticity.

    Source: §1 · p. 2, §6 · p. 10

    Under the strictly streaming setting, instances cannot be revisited for counterfactual outcomes, validation sets are unavailable, and frontier API agents do not expose parameters for gradient updates.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

Language model agents increasingly operate over long interaction streams, requiring online adaptation to distribution shifts through continual learning. However, existing agent harnesses typically either evolve their states only through forward information flow or rely on additional replay or validation data for harness optimization. To address this, we propose ReHarness, a recurrent harness framework for agentic continual learning that complements forward read and write with backward revision under a strictly online streaming protocol. We ask what information could have been captured differently in the harness state to better inform the current instance in the stream. Specifically, after each instance, we leverage the resulting trajectory as feedback to revise the reflections retrieved for that instance, thereby propagating feedback backward through the textual harness state. Across three continual learning benchmarks and three language model backbones, ReHarness consistently outperforms strong raw-context baselines, memory- and reflection-based methods, and prompt optimization approaches. Further analysis shows that ReHarness produces source-grounded and meaningful revisions, corrects overgeneralized or outdated experience under distribution shifts, and improves both stability and plasticity during continual learning.

1 Introduction

Agentic continual learning needs harnesses that adapt across long streams and distribution shifts, but existing approaches mainly move experience forward or depend on extra data for optimization. ReHarness adds backward revision of retrieved reflections using downstream trajectory feedback, and the paper reports consistent improvements across benchmarks and backbones.

Motivation

  • Existing harnesses typically write and retrieve experience through forward information flow, leaving memories and reflections rarely revised according to later effectiveness.Other approaches optimize harness states with additional training or validation data unavailable under a strictly online streaming protocol.Source: §1 · p. 1

Approach

  • ReHarness adds backward revision after each instance, using its trajectory to revisit retrieved reflections while grounding new content in their source trajectories.The revised harness state is carried forward without instance replay or gradient-based parameter updates.Source: §1 · pp. 1–2

Contributions

  • Across three benchmarks and three LLM backbones, ReHarness consistently outperforms raw-context, memory- and reflection-based, and prompt-optimization baselines.The paper also reports statistically significant gains and consistent improvements across 16 evaluated tasks.Source: §1 · p. 2
  • ReHarness produces source-grounded revisions, corrects overgeneralized or outdated reflections under distribution shifts, and improves stability and plasticity.The paper reports that it can outperform the strongest full-history baseline without a large revision set while using roughly one-third as many tokens.Source: §1 · p. 2

2 Related Work

Prior work studies continual learning for agents through persistent runtime states, memory, prompts, and reflections. ReHarness addresses a gap in harness evolution: existing approaches often depend on replay or validation data, or only propagate information forward.

Continual Learning

  • Continual learning studies adaptation to evolving streams under distribution shifts while retaining acquired capabilities, with agentic systems accumulating learning signals in memory, prompts, or persistent runtime states.Benchmarks assess long-stream performance, including plasticity to shifts and stability of acquired capabilities.Source: §2 · p. 2

Agent Harness

  • Harness research spans storing, retrieving, and optimizing memories, alongside methods that abstract experience into reflections and curate them for future use.Some memory methods optimize offline, while others accumulate past demonstrations through online in-context learning.Source: §2 · p. 2
  • Existing approaches often rely on validation sets or trajectory replay, or support only forward information flow rather than revising experience based on downstream effectiveness.The paper positions backward revision as closing this recurrent loop in an online setting.Source: §2 · p. 3, Figure 1 · p. 3

3 Framework

The framework moves from forward-only harnesses to recurrent adaptation by revising retrieved reflections with downstream feedback. It presents the streaming constraints, revision mechanism, and ReHarness’s read–revise–write cycle.

3.1 From Stateful to Recurrent Agent Harnesses

  • Forward-only harnesses write reflections from past trajectories and retrieve them for later decisions, but an overgeneralized reflection can persist and mislead future instances after a distribution shift.The sorting example shows that a reflection for descending order can be retrieved again for an ascending-order query even after a corrective reflection is written.Source: §3.1 · pp. 3–4

3.2 Textual Backward Feedback through Revision

  • Strict streaming limits backward optimization because instances are seen once, validation sets are unavailable, and frontier API agents expose no parameters for gradient updates.The framework therefore uses natural-language feedback in the textual harness state.Source: §3.2 · p. 4
  • A revision agent revisits retrieved reflections using their source trajectories and current-instance evidence, correcting exaggerated hypotheses or adding missed evidence while keeping new content source-grounded.In the sorting example, revision adds the condition “larger first only if descending”; the revised reflection then informs a new forward write and the next harness state.Source: §3.2 · p. 4, §3.3 · p. 4

3.3 ReHarness: Closing the Recurrent Loop

  • ReHarness retrieves top-k reflections, revises each using its source trajectory and current effectiveness feedback, then writes a new reflection from the current trajectory.The framework’s Table 1 caption reports that ReHarness outperforms all baselines on mean normalized gain across benchmark–backbone combinations.Source: §3.3 · p. 4, Table 1 · p. 5

4 Experiment

Across three benchmarks, ReHarness achieves the strongest overall normalized gain and remains competitive across varied tasks. Its compact reflections also reduce total token use relative to raw-context baselines, while revisions remain largely source-grounded.

4.1 Experiment Setup

  • Comparisons include raw-context, memory, reflection, and online prompt-optimization baselines, evaluated with Claude Opus 4.8, GPT-5.5, and Gemini 3.7 Flash.Retrieval uses the top five entries ranked by Qwen3-Embedding-8B.Source: §4.1 · p. 5
  • Experiments use three continual-learning benchmarks under a strictly online stream protocol, with harness state reset between streams and normalized gain measured against a stateless backbone.Each experiment uses three independently sampled stream rollouts with shared initialization and random seed across methods.Source: §4.1 · pp. 5–6

4.2 ReHarness Consistently Improves Agentic Continual Learning

  • ReHarness achieves the highest overall normalized gain of 0.240, ranks first or tied in eight of nine benchmark–backbone combinations, and significantly outperforms every baseline.Against Top-K Raw, padj = 0.039; against Full Raw, padj = 0.043.Source: §4.2 · p. 6
  • On CL-Bench, revised reflections remain over 95% grounded for all three backbones, with grounding decreasing by at most 0.3% per revision.The table reports grounded, leakage, and unsupported claim rates by backbone and their average change per revision.Source: Table 2 · p. 7
  • Performance varies by task: ReHarness gains are marginal on exploitable poker but consistent across WritingBench tasks, and it remains competitive across MemoryArena domains.The authors attribute lower effectiveness on poker to noisy hand-level rewards that provide a less diagnostic signal for backward revision.Source: Figure 2 · p. 6, §4.2 · p. 7
  • Across one Claude rollout on CL-Bench, ReHarness uses 41.57M total tokens versus 52.66M for Top-K Raw and 142.12M for Full Raw.ReHarness's compact reflections lower repeated decision-time context costs despite additional reflection and revision calls.Source: §4.2 · p. 7

5 Analysis

ReHarness revisions remain largely source-grounded and improve future-instance performance over a no-revision harness, while counterfactual replay indicates that revisions can be more effective when available on the same instance. Examples and analyses show how revisions respond to overgeneralization and distribution shifts, and how their extent varies with stability, retrieval size, and use in subsequent reflection generation.

5.1 Is ReHarness Revision Source-Grounded?

  • Over 95% of revised claims remain grounded in their source trajectories, with leakage at 0.8–2.1% and unsupported claims at 0.5–2.6%.Grounding decreases by at most 0.3% per revision.Source: §5.1 · p. 7

5.2 Is ReHarness Capable of Revision?

  • Counterfactual replay substantially improves over ReHarness across tasks, while online ReHarness beats the no-revision baseline across tasks, especially in database exploration, sales prediction, and blind spectrum monitoring.Replay applies the revised state to the same instance and violates the online streaming protocol; ReHarness applies it only to the next instance.Source: §5.2 · p. 8

5.3 What Does ReHarness Revise during Continual Learning?

  • Revisions correct speculative transmitter guidance in blind spectrum monitoring and update an outdated price-column strategy after a database schema shift.The revised database strategy enumerates price-like columns, allowing the agent to find the answer in two queries.Source: §5.3 · p. 8, Figure 3 · p. 9

5.4 How Does ReHarness Revise during Continual Learning?

  • ReHarness achieves the highest average stability gain and highest plasticity gain for every backbone, while several forward-only methods have negative average stability gains.Stability measures gain on the first post-shift instance; plasticity measures gain on subsequent instances before the next shift.Source: §5.4 · p. 9, Table 4 · p. 9

5.5 What Makes ReHarness Effective?

  • On CL-Bench with Claude, normalized gain rises sharply from k = 1 to 5, stays relatively stable for k ∈{5, 10, 20}, and declines for larger k.The analysis fixes k = 5 in the main experiments; larger revision sets may add cost and reflection drift.Source: §5.5 · p. 10, Figure 4 · p. 10
  • Removing revised reflections from demonstrations for the memory agent lowers normalized gain on CL-Bench with Claude from 0.217 to 0.191.The authors interpret this as evidence that revised experience helps guide subsequent reflection generation.Source: §5.5 · p. 10

6 Discussion

The discussion presents ReHarness as a recurrent harness for continual learning and reports consistent benchmark gains, while noting weaker performance on exploitable poker and added storage overhead.

6 Discussion

  • ReHarness complements forward read and write with backward revision of past reflections, grounding new content in source trajectories.Source: §6 · p. 10
  • Across three continual learning benchmarks and three LLM backbones, ReHarness consistently outperforms strong baselines with statistically significant gains across challenging tasks.Source: §6 · p. 10
  • ReHarness underperforms on exploitable poker with noisy downstream rewards and stores raw trajectories for future revision, adding storage overhead.The paper notes it remains more token-efficient than the strongest full-history raw-context baseline.Source: §6 · p. 10

A Appendix

The appendix points readers to task-level normalized-gain tables, agent prompts, and full qualitative revision examples.

A Appendix

Loading 2610.12668v1…